robots.txt
GET /api/v1/robots-txt
Fetch, parse and validate a site's robots.txt, including the AI-crawler rules (GPTBot, ClaudeBot, PerplexityBot, Google-Extended…) and Content-Signal directives. See which user-agents are allowed or disallowed, which sitemaps are declared, and whether AI bots are explicitly granted or denied access.
Query parameters
| Name | Type | Description |
|---|---|---|
url required | string | Any URL on the site; we read robots.txt from its origin. |
key | string | Your API key, if not sent via the X-API-Key header. |
Request
Response
200 · application/json
- {
- "url": "https://example.com/robots.txt",
- "found": true,
- "statusCode": 200,
- "valid": true,
- "stats": { "lines": 8, "comments": 1, "groups": 2, "sitemaps": 1 },
- "sitemaps": ["https://example.com/sitemap.xml"],
- "contentSignals": [],
- "aiBots": { "hasAnyAIRule": false, "perBot": {} },
- "groups": [ { "userAgents": ["*"], "rules": [ { "directive": "disallow", "value": "/admin" } ] } ],
- "errors": []
- }
Response fields
| Field | Type | Description |
|---|---|---|
found | boolean | Whether a parseable robots.txt was served at the origin. |
valid | boolean | True when found and no unparseable lines were encountered. |
statusCode | integer | HTTP status returned for the robots.txt request. |
stats | object | Counts of lines, comments, groups and sitemaps. |
sitemaps | array | All Sitemap: directives declared in the file. |
contentSignals | array | Content-Signal: directives (ai-train, search, ai-input preferences). |
aiBots | object | hasAnyAIRule plus perBot rules for each AI crawler we scan. |
groups | array | Parsed user-agent groups, each with its userAgents and rules. |
errors | array | Any lines that could not be parsed. |
To analyse the declared sitemaps, use Sitemap. See Errors for status codes.