Which AI crawlers can access your site
CrawlBit checks crawlability, schema, content and live citations in one scan, then writes the fixes. Free to start.
Scan my site free →Check in one click whether GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot and Google-Extended are allowed to read your site. Many sites block AI bots by accident in robots.txt, on Shopify as anywhere else, and quietly disappear from AI answers. Paste your URL to check.
CrawlBit checks crawlability, schema, content and live citations in one scan, then writes the fixes. Free to start.
Scan my site free →AI search visits have grown fast, and a visitor sent by an AI engine tends to be far more valuable than an average search visitor. But an AI engine can only cite you if its crawler is allowed in. If your robots.txt blocks GPTBot, ClaudeBot, PerplexityBot or Google-Extended, you are invisible to that engine, often without realizing it.
If a crawler is blocked and you want the visibility, remove its Disallow rule, or add an explicit allow. For example:
User-agent: GPTBot Allow: / User-agent: ClaudeBot Allow: / User-agent: PerplexityBot Allow: /
Blocking is a valid choice if you do not want AI training on your content. Just make it a deliberate one, not an accident.