Robots.txt
A file that tells crawlers what they may and may not crawl.
What it means
robots.txt is a text file at the root of your site that gives directives to crawlers about which paths they may crawl and which they should avoid, and points them to your XML sitemap.
How it affects your SEO
Used well, it keeps crawlers off staging, search results, and parameter URLs to preserve crawl budget. Misused, it can accidentally block your whole site from Google — a common cause of sudden de-indexation.
Robots.txt — frequently asked questions
Can robots.txt block a page from ranking?
robots.txt blocks crawling, not indexing. A blocked page can still appear in results from links pointing to it. To remove it from the index, use a noindex directive instead.
What happens if I delete robots.txt?
Google assumes crawling is fully allowed. A missing or misconfigured file can waste crawl budget or accidentally expose staging and parameter URLs to crawlers.
Related SEO terms
Tells Google which version of a page is the “real” one when duplicates exist.
Google’s three speed-and-stability metrics: LCP, INP, and CLS.
Structured data that labels content (reviews, FAQ, products) for search engines.
How many pages Google will crawl on your site per day.
Is this hurting your rankings?
Run a free 60-second audit and find out where robots.txt — and dozens of other factors — stand on your site.
Get my free SEO score