What our crawler does, how often it visits, and how to stop it.
A check is one request to a product's homepage, on a schedule. We write down what came back every time, not only when something changed.
If you found AppWatchBot in your logs, this is what it was doing, how often it comes back, and how to make it stop.
We look up the domain, ask for the homepage, follow any redirects, and note what came back and where it ended up. Nothing else on the page is read and nothing from it is stored.
The homepage, not the bare domain. Plenty of products live at a subdomain while the bare domain has nothing on it, so reading that as a dead product would be wrong about a lot of working ones.
It depends on how much the product has been changing. One that moves gets checked more often than one that has looked identical for months, and anything that has started failing is checked sooner, so if it comes back we see that within a day.
It identifies itself as AppWatchBot and it respects robots.txt. Where a product has an official API we use that instead of fetching pages.
If it is causing you a problem, tell us and we will stop fetching your site. You do not have to give a reason and we will not ask for one.
Blocking us is not the same as being down. A 401, 403, 429 or 451 tells us something about our own request and nothing about your product, so it changes nothing either way.
That matters more than it sounds. Bot protection is most common on the products doing best, so if we treated a block as a failure, the best defended products would be exactly the ones we wrongly marked dead.
Whether anyone is still working on it. A product abandoned two years ago answers exactly like one shipping weekly, so we never claim otherwise on the strength of a check.