ChangeVaneBot
Who we are
- ChangeVane checks public websites for the people responsible for them: what changed, what broke, and whether fixes held. Every request our crawler makes identifies itself as
ChangeVaneBot/0.1 (+https://changevane.com/bot). - It reads HTML. It doesn't run JavaScript, submit forms or log in, and it only ever sends GET and HEAD requests.
robots.txt comes first
- Before requesting anything else from a website, we read its robots.txt and follow it. Rules addressed to
ChangeVaneBot take precedence over the rules for every bot (*). - If a website's robots.txt can't be read — it answers with a server error (5xx), with 429 Too Many Requests, or not at all — we don't crawl that website. A robots.txt that doesn't exist (404) means there are no rules.
- We honour a
Crawl-delay of up to 30 seconds between requests. If you ask for longer, we wait 30 seconds. - We follow up to 10 redirects. We read the robots.txt of the website we are about to request, then follow redirects from there without checking robots.txt again at each step.
What we request from the website being checked
- The sitemaps your robots.txt lists — or, if it lists none,
/sitemap.xml, /sitemap_index.xml and /wp-sitemap.xml. - Pages, with GET. We read HTML only, up to 5 MB per page; a response that isn't HTML is not downloaded.
- At most 500 pages per check (a free check stops sooner), following links at most 10 deep. Each request gets 15 seconds, and a check stops after 10 minutes.
- At most 5 requests at a time, each starting at least 0.2 seconds after the last — or your Crawl-delay apart, if it is longer.
- Links and images on those pages: one
HEAD request each, which returns headers only. If a server refuses HEAD, we send one GET and read only its headers. Each address is checked once per check — at most 500 on the website and 200 elsewhere.
Other websites it links to
- Before checking a link to another website, we read that website's robots.txt — once per check — and request nothing it disallows. If it can't be read, we request nothing from that website. We send one request at a time to any website other than the one being checked.
Where it connects
- Public addresses only, on ports 80 and 443 — never private networks, localhost or cloud metadata addresses. We check this again at every redirect.
How often
- When someone asks for a free check of a website — at most once an hour per website, whoever asks — and once a week for each person who has subscribed to weekly reports about it.
How to stop it
- Add this to your robots.txt. We read robots.txt at the start of every check, so it applies from the next one:
User-agent: ChangeVaneBot
Disallow: /
Contact
- Questions, or something our bot got wrong:
bot@changevane.com. We'll answer.