Public crawler help

About ListurboBot

You may have noticed ListurboBot in your WAF logs. This page explains who we are, what we do, and how to allow us through your firewall.

1Who we are

Listurbo is a Generative Engine Optimization (GEO) SaaS platform. We help brands get cited, recommended, and mentioned in AI engines like ChatGPT, Gemini, Claude, Perplexity, and Grok.

We do not build a training corpus. We do not redistribute fetched content to third parties. ListurboBot only retrieves content when one of our customers explicitly asks us to evaluate a site they own or are auditing.

2When ListurboBot fetches your site

ListurboBot fetches a page only when a Listurbo user actively runs one of these tools against your URL:

  • Sitemap discovery — reads /robots.txt and /sitemap.xml to enumerate the site’s public pages.
  • AI crawler rule check — reads your robots.txt to see which AI crawlers it allows and which it blocks. The crawlers we look for are GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, Googlebot, Google-Extended, Bingbot, PerplexityBot, Perplexity-User, xAI-Bot.
  • Schema markup audit — parses HTML for JSON-LD (Q&A, step-by-step guides, articles and company details) so we can suggest improvements.
  • Content freshness scan — compares page last-modified + content hash against the previous evaluation snapshot to detect stale pages.

Fetch volume is bounded by the customer’s plan quota. We respect Crawl-Delay if your robots.txt declares one, and we never run aggressive concurrent crawls.

3Our user-agent string

Every request ListurboBot makes carries this exact User-Agent header. Allowlist by this string and you’ll catch all of our traffic and nothing else.

ListurboBot/1.0 (+https://listurbo.com/help/bot)
ListurboBot

Identifies us — bot name we self-declare. Use this token alone for substring matching.

/1.0

Crawler version. Industry-standard format. Will increment as we evolve our fetch behavior.

(+https://listurbo.com/help/bot)

Link to this exact page. The + prefix is the standard convention so log parsers can extract the URL.

4Allow us in robots.txt (one line)

Add the snippet below to your site’s /robots.txt file. ListurboBot reads robots.txt on every fetch session and respects this directive.

User-agent: ListurboBot
Allow: /

Tip: If you already have a global User-agent: * with Allow: /, ListurboBot is already permitted at the robots.txt layer — but your WAF may still block us. See section 5.

5WAF allowlist steps (3 platforms)

If your robots.txt already allows ListurboBot but we’re still blocked, your WAF (Web Application Firewall) is rejecting us at the network layer. Pick your WAF platform and follow the steps below.

1

Sign in to Cloudflare dashboard at dash.cloudflare.com and select the target domain.

2

In the left sidebar, navigate to Security → WAF → Custom rules.

3

Click Create rule. Set Field = User Agent, Operator = contains, Value = ListurboBot.

4

Set Action = Skip, then under “WAF components to skip” select All managed rules and All custom rules.

5

Name the rule “Allow ListurboBot”, click Deploy, and confirm the rule shows status = Active.

Verify (within 24 hours)

Open Security → Events. Filter by User-Agent contains “ListurboBot”. Recent requests should show Action = Skip instead of Block / Challenge.

Expected outcome

Listurbo evaluations succeed for your domain. Your URLs return 200 OK to ListurboBot. Tool results pages no longer show the “fetch blocked” warning.

1

Log in to your WordPress admin dashboard as an administrator.

2

In the left menu, open Wordfence → Tools → Diagnostics.

3

Scroll to the Custom Patterns panel. Click Add new pattern.

4

Set Pattern type = User-Agent, Pattern value = ListurboBot, Action = Allowlist (always allow). Save.

5

Open Wordfence → Tools → Live Traffic and confirm “Custom patterns” shows the new entry as Active.

Verify (within 24 hours)

Open Wordfence → Tools → Live Traffic. Filter by User-Agent containing “ListurboBot”. Recent visits should be marked Allowed, not Blocked or Throttled.

Expected outcome

Listurbo evaluations succeed against the WordPress site. ListurboBot bypasses brute-force protection, country blocking, and rate limits.

1

SSH into your nginx server. Open the relevant server block config (typically /etc/nginx/sites-available/yoursite.conf).

2

Inside the server block (or a relevant location / block), add the SecRule below.

3

Run sudo nginx -t to confirm the configuration is syntactically valid.

4

Reload nginx with sudo systemctl reload nginx. No downtime.

SecRule REQUEST_HEADERS:User-Agent "@contains ListurboBot" \
    "id:1001,phase:1,pass,nolog,allow,ctl:ruleEngine=Off"

Verify (within 24 hours)

Run tail -f /var/log/nginx/access.log and watch the lines that contain ListurboBot. Recent ListurboBot requests should return 200, not 403.

Expected outcome

ModSecurity skips its rule engine for ListurboBot requests. Your site continues to be protected against all other malicious User-Agents.

6Why this matters for your GEO

Blocking ListurboBot also means you are blocking GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, Googlebot, Google-Extended, Bingbot, PerplexityBot, Perplexity-User, xAI-Bot in most cases — the same WAF rules that target our user-agent string typically reject other AI crawlers as well. When AI engines cannot read your site’s content, your brand will not be cited, recommended, or mentioned in their answers. Opening your site to all legitimate AI crawlers is the core action of GEO optimization.

7Contact us

If you have questions about ListurboBot’s behavior, want to report unusual traffic, or need help adjusting your WAF, reach out via the channels below.