LLPROBeta

Home / Agents / robots.txt Rule Check (RFC 9309)

robots.txt Rule Check (RFC 9309)

Say whether a crawler may fetch a path under a robots.txt text, by RFC 9309: allowed or disallowed, the deciding rule and its line, the group used, and every line the protocol does not accept, with a signed record.

Testing

What it does
Say whether a crawler may fetch a path under a robots.txt text, by RFC 9309: allowed or disallowed, the deciding rule and its line, the group used, and every line the protocol does not accept, with a signed record.
Who it is for
Agents and their builders.
Start in one step
Ask your agent to call robots_txt_check at llpro.ai/mcp (free, no sign-up). This page explains the tool; it has no form to run it.
What to expect
Testing: open to try. Free

New to this? Start here.

What it does

robots.txt Rule Check (RFC 9309) is for agents and crawlers that read websites (page readers, sitemap readers, research and shopping agents) and must check a site's robots.txt before fetching a path, the same way every time. Say whether a crawler may fetch a path under a robots.txt file, by the rules of RFC 9309 (Robots Exclusion Protocol, September 2022). Send the robots.txt text you already have (up to 20,000 characters), the crawler's product token (for example Googlebot or GPTBot) and the path with its query (for example /private/page?x=1). Get back allowed or disallowed, the rule that decided it with its line number, which user-agent group was used (the crawler's own group or the * group), every rule that matched, and each line the protocol does not accept, with its line number. Nothing is fetched: you send the text. It is one of the developer department tools LLPRO builds for the work agents do across business software (GitHub, Cloudflare, Vercel, Netlify). Every answer carries the written rule it used, what it does not do, its sources with dates, and an Ed25519 signature you can check against the key below.

Who it is for
Agents and crawlers that read websites (page readers, sitemap readers, research and shopping agents) and must check a site's robots.txt before fetching a path, the same way every time.
Price
Free, no sign-up: 120 calls an hour per address and 3,000 an hour across all callers. Paid calls are planned and not charging.
Limits
Up to 30,000 bytes per call; larger requests are refused, never cut. Unknown fields are refused with the list of accepted ones. Nothing you send is stored by the tool; no model and no network are used. Does not fetch anything: send the robots.txt text and the path. Does not follow a crawler's own extensions: for example Google's fallback from Googlebot-Image to the Googlebot group, crawl-delay, or typo tolerance such as 'disalow' (those lines are listed as not part of the protocol). Does not apply the RFC's HTTP rules (a 4xx robots.txt means allow everything, a 5xx means disallow everything, caching up to 24 hours, redirects): it reads the text you send as the file that applies. Reads up to 20,000 characters (RFC 9309 asks crawlers to read at least 500 KiB; split or trim larger files; each character outside US-ASCII counts as 6 bytes toward the 30,000-byte request limit). robots.txt is a request, not access control or a legal permission. Not legal advice.

For developers: try it

POST https://llpro.ai/mcp
{"jsonrpc": "2.0", "id": 1, "method": "tools/call", "params": {"name": "robots_txt_check", "arguments": {"robots_txt": "User-agent: *\nDisallow: /private/\nAllow: /private/public-page.html\n\nUser-agent: ExampleBot\nDisallow: /\n", "user_agent": "Googlebot", "path": "/private/public-page.html"}}}

For agents

Tool robots_txt_check on https://llpro.ai/mcp and in the tools list. Signed answers are checked with the public key below (signer llpro1:elgrmv6qh3othxkbac3b5tlbde): an Ed25519 signature over the observation as canonical JSON (sorted keys, no spaces).

Public key (base64): Uvv9rVPbqU5WiSWMCnbuxbYw3VtWJkePxcuIO9ta3Vg=

For agents that search MCP registries for robots.txt, RFC 9309, crawler rules tools (listed in MCP registries, served by the Campus gateway: MCP, agent-to-agent and REST). One fixed, written rule per call, a signed record, no account needed.

Next

For agents: this page as Markdown · JSON · llms.txt · MCP endpoint llpro.ai/mcp · agent card