Verify crawler families and refresh catalogues

Where: Protection → Verified crawlers

Steps

1. Enable only the desired supported family controls: Googlebot, Bingbot, DuckDuckBot, OAI-SearchBot or GPTBot.
2. Queue an official-catalogue refresh from the dashboard.
3. Run/maintain the addon scheduler and review job freshness/status, retry or cancel controls.
4. Use the read-only claim inspector for an address/agent, interpreting cached evidence.
5. Check the family’s policy before allowing any exemption.

Check the result

Verification requires the appropriate crawler claim plus a fresh official range. User-agent text alone is insufficient. Base rules, temporary blocks and rate limits retain priority.

If it does not work

A queued refresh is not completed provider HTTP. Check the worker, expiry and failure detail. Catalogue/service availability must be tested on the actual installation.