What Are Robots Directives?
Robots directives are instructions embedded in your HTML that tell search engine crawlers how to handle a page. The most common method is the meta robots tag: <meta name="robots" content="index, follow">. These directives control whether a page should be indexed (appear in search results) and whether search engines should follow the links on the page.
Why Robots Directives Matter for SEO
A misconfigured robots directive can completely remove your page from search results. A single noindex tag will tell Google not to include the page in its index, regardless of how well-optimised the content is. Similarly, a nofollow directive prevents search engines from following links on the page, which can break your internal linking structure and prevent link equity from flowing to other pages.
These directives are powerful tools when used intentionally, such as keeping admin pages or thin content out of search results. But accidental use is one of the most common and damaging SEO mistakes.
Common Robots Directives
| Directive | Effect |
|---|---|
index | Allows the page to appear in search results. This is the default behaviour. |
noindex | Tells search engines not to include the page in search results. |
follow | Allows search engines to follow and crawl links on the page. This is the default. |
nofollow | Tells search engines not to follow any links on the page. |
noarchive | Prevents search engines from showing a cached copy of the page. |
nosnippet | Prevents search engines from displaying a text snippet or video preview in results. |
max-snippet:[n] | Limits the text snippet in search results to a specified number of characters. |
Robots Directives Best Practices
- Do not noindex important pages - Double-check that your key landing pages, product pages, and blog posts are not accidentally set to noindex.
- Use noindex intentionally - Apply noindex to pages that should not appear in search results: thank-you pages, internal search results, staging environments.
- Check after site migrations - Noindex tags are commonly left behind after moving from a staging site to production.
- Prefer robots.txt for blocking crawling - Use robots.txt to prevent crawling of entire sections, and meta robots for page-level control.
- Avoid conflicting signals - Do not include a page in your sitemap while also setting it to noindex. Send consistent signals.
How RankNibbler Checks Your Robots Directives
RankNibbler scans the meta robots tag on your page and reports its content. If a noindex or nofollow directive is detected, it is prominently flagged in the audit results as a warning, since these directives can significantly affect your search visibility. Pages with no meta robots tag are reported as using the default index/follow behaviour.
Check your robots directives now. Visit the RankNibbler homepage and enter a URL for a free audit.