AppWatchBot

Every request carries one of these, and nothing we run ever presents itself as a browser:

• AppWatchBot/0.1 (+https://appwatch.ai/bot)
• AppWatchBot/0.1 (+https://appwatch.ai/bot; liveness check only, no content stored)
• AppWatchBot/0.1 (+https://appwatch.ai/bot; freshness check only)
• AppWatchBot/0.1 (+https://appwatch.ai/bot; archive.org lookups and archived copies only, never the live site)
• AppWatchBot/0.1 (+https://appwatch.ai/bot; one homepage screenshot, rendered in a browser)

Add this to your robots.txt and we stop, on every path, within a day:

User-agent: AppWatchBot
Disallow: /

We read robots.txt before we read anything else and we honour Crawl-delay. That includes the check that your site is up: while it is blocked, your listing keeps the state we last saw and says when we last checked. We keep a copy of your file for up to a day, so a change takes effect within one. If you would rather your app was not listed at all, use our contact form: we remove it, and removal also stops the crawling.

Your homepage, following its redirects. Beyond that, a short list and nothing else:

• Your pricing page. If your homepage does not link one, we try /pricing and /plans.
• Your icon, from the icon or apple-touch-icon link your homepage names, or /favicon.ico.
• sitemap.xml and a changelog or blog feed when you publish one, because they tell us whether anything changed for a fraction of the cost of fetching the site again.
• A screenshot of the top of your homepage. It is one page load in a browser, so it fetches your page's own scripts, styles and images as any visitor would, and a second load a moment later if the first one fails. Video and other streams are refused.
• When a homepage is an empty page whose only content is a script sending the visitor to another address on the same site, as parked domains are, that one address.
• When somebody runs our free page analyzer on your site, that page and up to three of its stylesheets, once.

Other sites we read are the places apps are listed or written about: launch directories and their sitemaps, publishers' feeds and the articles they link, link shorteners (to see where a link lands), and app store listings and sitemaps. The same rules apply to them.

The check that your site is up reads the response headers and never the page: one request, one more per redirect, and a TLS handshake to read your certificate. The point of the sitemap and the feed is to avoid downloading your pages, not to find more of them: we do not crawl your site or follow your internal links.

We start at most one request a second to your site, counting every subdomain as one site, and wait longer if your robots.txt sets a Crawl-delay, up to a minute. Two things are the exceptions, both listed above: the screenshot, whose browser loads your page's resources together as any visitor's does, and the page analyzer, which a person starts and which reads its few files in turn.

In practice it is far less. The check that your site is up runs about once a week for a live app, once a day while an app is in trouble, and once a month for one that has stopped responding. Reading your homepage for the facts below, and your sitemap and feed for changes, happen at most once a day each. Your pricing page is read at most once a month, and your icon about twice a year, or once a month while we have found none.

If your server answers 429 or a 500, 502, 503 or 504, or does not answer, we space our requests out further, from five seconds and doubling up to a minute, and we honour Retry-After up to a minute. Each healthy answer halves that again.

For your app: the domain, a name and tagline, when we first saw it, whether it still responds, the technologies your page loads, how you charge (free, freemium, paid or on request, and whether there is a free or an enterprise plan), the languages your site says it offers, your icon, and the screenshot. Each fact is stored with the evidence behind it.

We do not store your pages. The fetch happens, the few facts above are extracted, and the document is discarded rather than archived. The screenshot is a picture of the top of the page, not a copy of it. We take no contact details from your pages, and nothing about anybody's personal life: business profiles only, permanently. The full account is in our privacy notice.

AppWatch, a registry of the app market. We keep a record of which apps exist, whether they are still running, and what they charge, so that founders can see what is arriving in their market. The crawling exists to keep that record honest: without it we would be publishing claims about your product that nobody had checked.

If we have something wrong about your app, or you want it corrected or removed, use our contact form. We answer.

Assistants that fetch a page because somebody asked a question get everything we publish. OAI-SearchBot, Claude-SearchBot, PerplexityBot, Applebot and the rest are named in our robots.txt rather than merely permitted, so a rule we add later cannot block them by accident.

Crawlers that collect training corpora get everything except the app records under /apps/. Those hold the check history behind more than 17,000 products, which only exists because we kept checking, and it is the one thing here that could not be rebuilt. This page, the help centre, the market pages and the rest of the site stay open to them.

We do not block Google-Extended. It controls grounding in Gemini as well as training, and blocking it would take us out of answers somebody asked for.