AppWatchBot

Every request carries one of these, and nothing we run ever presents itself as a browser:

• AppWatchBot/0.1 (+https://appwatch.ai/bot)
• AppWatchBot/0.1 (+https://appwatch.ai/bot; liveness check only, no content stored)
• AppWatchBot/0.1 (+https://appwatch.ai/bot; freshness check only)

Add this to your robots.txt and we stop, on every path, on the next request:

User-agent: AppWatchBot
Disallow: /

We read robots.txt before we read anything else and we honour Crawl-delay. If you would rather your app was not listed at all, use our contact form: we remove it, and removal also stops the crawling.

Your homepage, and occasionally a page you link from it such as pricing. We also read sitemap.xml and a changelog or blog feed when you publish one, because they tell us whether anything changed for a fraction of the cost of fetching the site again.

Most visits are a single request that reads the response header and stops. The point of the sitemap and the feed is to avoid downloading your pages, not to find more of them: we do not crawl your site, follow your internal links, or look for anything you have not linked from your homepage.

At most one request a second, and in practice far less: a live app is visited about once a week, an app in trouble once a day while that lasts, and an app that has stopped responding once a month. We do not run parallel requests against the same site.

If your server slows down or returns errors, we slow down automatically and stay slower until it recovers.

For your app: the domain, a name and tagline, when we first saw it, whether it still responds, the technologies your page loads, whether you have a pricing page, and the languages your site says it offers. Each of those is stored with the evidence behind it.

We do not store your pages. The fetch happens, the few facts above are extracted, and the document is discarded rather than archived. We take no contact details from your pages, and nothing about anybody's personal life: business profiles only, permanently. The full account is in our privacy notice.

AppWatch, a registry of the app market. We keep a record of which apps exist, whether they are still running, and what they charge, so that founders can see what is arriving in their market. The crawling exists to keep that record honest: without it we would be publishing claims about your product that nobody had checked.

If we have something wrong about your app, or you want it corrected or removed, use our contact form. We answer.

Assistants that fetch a page because somebody asked a question get everything we publish. OAI-SearchBot, Claude-SearchBot, PerplexityBot, Applebot and the rest are named in our robots.txt rather than merely permitted, so a rule we add later cannot block them by accident.

Crawlers that collect training corpora get everything except the app records under /apps/. Those hold the check history behind more than 17,000 products, which only exists because we kept checking, and it is the one thing here that could not be rebuilt. This page, the help centre, the market pages and the rest of the site stay open to them.

We do not block Google-Extended. It controls grounding in Gemini as well as training, and blocking it would take us out of answers somebody asked for.