SiteDistrict Bots and Crawlers - SiteDistrict

SiteDistrict Bots and Crawlers

Our bots and crawlers, and the requests they make

At SITE DISTRICT, we have a number of automated scripts and processes that make requests to sites hosted on SiteDistrict, as well as to sites hosted elsewhere.

This page documents those bots, the User-Agent strings sent, their purpose, and other relevant details.

Bots

Site Checker / Monitor

  • TypeSite Checker / Uptime Monitor
  • User-AgentSiteDistrict-Site-Checker/1.0 (+https://sitedistrict.com/sitedistrict-bots)
  • PurposeMontitor sites for unexpected downtime or very slow responses
  • Details

    The SITEDISTRICT uptime monitor and response time checker is an internal system used to monitor both sites hosted on SiteDistrict, as well as sites that have been added to our dashboard, that have not yet been migrated to our platform. The site monitor makes requests at regular intervals, to check both the HTTP response code and the response time for the site located at a given domain. Sites that cannot be reached due to a DNS, TCP, or TLS issue, do not return an HTTP 200, or take longer than 45 seconds to respond, are considered "down". To reduce false positives, a site is not considered down until two consecutive checks fail. Once the first check has failed, another requests is sent about 30 seconds later to confirm the downtime.

WP Cron

  • TypeWordPress Cron Trigger
  • User-AgentSiteDistrict-WP-Cron/1.0 (+https://sitedistrict.com/sitedistrict-bots)
  • PurposeRun scheduled WordPress cron jobs at the time they are scheduled
  • Details

    The SITEDISTRICT WP Cron system is a distributed system running across all servers where WordPress sites are hosted. It constantly checks the MySQL database of sites that are up & running, and schedules requests based on the schedule found inside the cron option of the wp_options database table. The system then makes requests to the /wp-cron.php script on the respective sites, bypassing any DNS lookups. In addition to pulling the legacy WP cron schedule from the wp_options table, it also runs jobs scheduled by Action Scheduler.

Site Migrator

  • TypeSite Migrator / Importer
  • User-AgentSiteDistrict-Site-Migrator/1.0 (+https://sitedistrict.com/sitedistrict-bots)
  • PurposeImport WordPress sites from other WordPress hosts
  • Details

    The SITEDISTRICT site migrator is a system that allows SiteDistrict users to migrate a site to our hosting platform, without leaving the SiteDistrict dashboard. The system logs into the remote WordPress site, installs our migration plugin, and then makes requests to export details about the site, as well as the database and files. Details on using this tool can be found on our Importing a WordPress Site page.

Speed Tester

  • TypeServer & Page Response Time Tester
  • User-AgentSiteDistrict-Speed-Tester/1.0 (+https://sitedistrict.com/sitedistrict-bots)
  • PurposeMeasure the cached / uncached server response times of WordPress sites, hosted both on SITEDISTRICT, as well as other hosts
  • Details

    Our platform and dashboard include a Speed Test tool that can be used to crawl a site to measure the server response times for pages on the site. This tool is one of the best methods available for comparing raw WordPress hosting performance for nearly identical copies of sites, one hosted on SITEDISTRICT, and the other at another hosting provider. This tool is run simultaneously on both the imported copy of a site, and the original / live copy of the site, as part of our migration process, which is documented on our Testing an Imported Site page.

Host Check

  • TypeHosting Checker / Identifier
  • User-AgentSiteDistrict-Host-Check/1.0 (+https://sitedistrict.com/sitedistrict-bots)
  • PurposeIdentify the WordPress hosts for a domain or site
  • Details

    Our system checks domains associated with sites hosted on SiteDistrict, as well as sites belonging to clients, but hosted elsewhere, which have not yet been migrated. When sites are added to our platform, HTTP requests are made to identify the hosting provider. In addition, we regularly monitor domains that have already been added, to detect if DNS has been pointed at or away from our platform. This part of the system relies on both DNS lookups, and also HTTP requests, when we cannot determine the host directly from DNS, such as for domains using Cloudflare with their proxy enabled.

SSL Validator

  • TypeTLS certificate Alt-name Validator
  • User-AgentSiteDistrict-SSL-Validator/1.0 (+https://sitedistrict.com/sitedistrict-bots)
  • PurposeCheck whether the www subdomain for a domain points at SITEDISTRICT
  • Details

    The SITEDISTRICT platform automatically issues SSL / TLS certificates for domains that are attached to sites, or that have been added as Domain Redirects. This bot checks to see if a www DNS record exists and points at SiteDistrict, before trying to include it in the list of Subject Alternative Name (SAN) for the certificate.

 

Amazing WordPress Hosting

 

Built WithAround The
Copyright © 2016 - 2024 SiteDistrict, All Rights Reserved