What Is a Robots.txt File?
A robots.txt file is a set of directions that inform search engines like google which pages to crawl and which pages to keep away from, guiding crawler entry however not essentially retaining pages out of Google’s index.
A robots.txt file seems like this:
Robots.txt recordsdata could appear difficult. Nonetheless, the syntax (pc language) is simple.
Earlier than explaining the main points of robots.txt, we’ll make clear how robots.txt differs from different phrases that sound related.
Robots.txt recordsdata, meta robots tags, and x-robots tags information search engines like google in dealing with website content material however differ of their stage of management, whether or not they’re situated, and what they management.
Take into account these specifics:
- Robots.txt: This file is situated within the web site’s root listing and supplies site-wide directions to go looking engine crawlers on which areas of the location they need to and shouldn’t crawl
- Meta robots tags: These tags are snippets of code within the part of particular person webpages and supply page-specific directions to search engines like google on whether or not to index (embody in search outcomes) and observe (crawl the hyperlinks on) every web page
- X-robot tags: These code snippets are used primarily for non-HTML recordsdata, resembling PDFs and pictures and are carried out within the file’s HTTP header
Additional studying: Meta Robots Tag & X-Robots-Tag Defined
Why Is Robots.txt Essential for web optimization?
A robots.txt file is essential for web optimization as a result of it helps handle internet crawler actions to stop them from overloading your web site and crawling pages not supposed for public entry.
Beneath are a number of causes to make use of a robots.txt file:
1. Optimize Crawl Finances
Blocking pointless pages with robots.txt permits Google’s internet crawler to spend extra crawl price range (what number of pages Google will crawl in your website inside a sure time-frame) on pages that matter.
Crawl price range can fluctuate based mostly in your website’s dimension, well being, and variety of backlinks.
In case your website has extra pages than its crawl price range, essential pages might fail to get listed.
Unindexed pages received’t rank, which suggests you’ve wasted time creating pages that customers by no means see in search outcomes.
2. Block Duplicate and Non-Public Pages
Not all pages are supposed for inclusion within the search engine outcomes pages (SERPs), and a robots.txt file allows you to block these private pages from crawlers.
Take into account staging websites, inside search outcomes pages, duplicate pages, or login pages. Some content material administration techniques deal with these inside pages mechanically.
WordPress, for instance, disallows the login web page “/wp-admin/” for all crawlers.
3. Conceal Assets
Robots.txt allows you to exclude assets like PDFs, movies, and pictures from crawling if you wish to maintain them non-public or have Google deal with extra essential content material.
How Does a Robots.txt File Work?
A robots.txt file tells search engine bots which URLs to crawl and (extra importantly) which URLs to keep away from crawling.
When search engine bots crawl webpages, they uncover and observe hyperlinks. This course of leads them from one website to a different throughout varied pages.
If a bot finds a robots.txt file, it reads that file earlier than crawling any pages.
The syntax is simple. You assign guidelines by figuring out the user-agent (the search engine bot) and specifying directives (the foundations).
You need to use an asterisk (*) to assign directives to all user-agents directly.
For instance, the next instruction permits all bots besides DuckDuckGo to crawl your website:
Semrush bots crawl the online to collect insights for our web site optimization instruments, resembling Web site Audit, Backlink Audit, and On Web page web optimization Checker.
Semrush bots respect the foundations in your robots.txt file, that means they received’t crawl your website should you block Semrush bots from crawling.
Nonetheless, blocking Semrush bots limits the usage of sure Semrush instruments.
For instance, should you block the SiteAuditBot from crawling your website, you may’t audit your website with the Web site Audit instrument. This instrument helps analyze and repair technical points in your website.
If you happen to block the SemrushBot-SI from crawling your website, you may’t use the On Web page web optimization Checker instrument successfully.
Because of this, you lose the chance to generate optimization concepts that might enhance your webpages’ rankings.
Discover a Robots.txt File
Your robots.txt file is hosted in your server, similar to different recordsdata in your web site.
You may view any web site’s robots.txt file by typing the location’s homepage URL into your browser and including “/robots.txt” on the finish.
For instance: “https://semrush.com/robots.txt.”
Examples of Robots.txt Information
Listed below are some real-world robots.txt examples from well-liked web sites.
YouTube
YouTube’s robots.txt file tells crawlers to not entry consumer feedback, video feeds, login/signup pages, and age verification pages.
The principles in YouTube’s robots.txt file discourages indexing user-specific or dynamic content material that doesn’t assist search outcomes and should increase privateness issues.
G2
G2’s robots.txt file tells crawlers to not entry sections with user-generated content material, like survey responses, feedback, and contributor profiles.
The principles in G2’s robots.txt recordsdata assist defend consumer privateness by proscribing entry to probably delicate private data. The principles additionally stop makes an attempt to control search outcomes.
Nike
Nike’s robots.txt file makes use of the disallow directive to dam crawlers from accessing user-generated directories, like “/checkout/” and “*/member/inbox.”
The principles in Nike’s robots.txt file stop delicate consumer information from showing in search outcomes and scale back alternatives to control web optimization rankings.
Search Engine Land
Search Engine Land’s robots.txt file makes use of the disallow tag to discourage indexing of “/tag/” listing pages, which regularly have low web optimization worth and might trigger duplicate content material points.
The principles in Search Engine Land’s robots.txt file encourage search engines like google to deal with higher-quality content material and optimize the location’s crawl price range—one thing particularly essential for big web sites like Search Engine Land.
Forbes
Forbes’s robots.txt file instructs Google to not crawl the “/check/” listing, which probably accommodates testing or staging environments.
The principles in Forbes’s robots.txt file stop unfinished or delicate content material from being listed, assuming it’s not linked to from elsewhere.
Explaining Robots.txt Syntax
A robots.txt file consists of a number of directive blocks, with every block specifying a user-agent (a search engine bot) and offering “permit” or “disallow” directions.
A easy block might seem like this:
Consumer-agent: Googlebot
Disallow: /not-for-google
Consumer-agent: DuckDuckBot
Disallow: /not-for-duckduckgo
Sitemap: https://www.yourwebsite.com/sitemap.xml
The Consumer-Agent Directive
The primary line of every directive block specifies the user-agent, which identifies the crawler.
For instance, use these traces to stop Googlebot from crawling your WordPress admin web page:
Consumer-agent: Googlebot
Disallow: /wp-admin/
When a number of directives exist, a bot might select probably the most particular one.
Think about you have got three units of directives: one for *, one for Googlebot, and one for Googlebot-Picture.
If the Googlebot-Information consumer agent crawls your website, it should observe the Googlebot directives.
Nonetheless, the Googlebot-Picture consumer agent will observe the extra particular Googlebot-Picture directives.
The Disallow Robots.txt Directive
The disallow directive lists components of the location a crawler shouldn’t entry.
An empty disallow line means no restrictions exist.
For instance, the under rule permits all crawlers entry to your complete website:
Consumer-agent: *
Permit: /
To dam all crawlers out of your complete website, use the under block:
Consumer-agent: *
Disallow: /
The Permit Directive
The permit directive permits search engines like google to crawl a subdirectory or particular web page, even in an in any other case disallowed listing.
For instance, use the under rule to stop Googlebot from accessing all weblog posts besides one:
Consumer-agent: Googlebot
Disallow: /weblog
Permit: /weblog/example-post
The Sitemap Directive
The sitemap directive tells search engines like google—particularly Bing, Yandex, and Google—the place to search out your XML sitemap (a file that lists all of the pages you need search engines like google to index).
The picture under reveals what the sitemap directive seems like:
Together with a sitemap directive in your robots.txt file is a fast solution to share your sitemap.
Nonetheless, you must also submit your XML sitemap on to search engines like google by way of their webmaster instruments to hurry up crawling.
The Crawl-Delay Directive
The crawl-delay directive tells crawlers what number of seconds to attend between requests, which helps keep away from server overload.
Google not helps the crawl-delay directive. To set crawl charges for Googlebot, use Google Search Console.
Bing and Yandex do assist the crawl-delay directive.
For instance, use the under rule to set a 10-second delay after every crawl motion:
Consumer-agent: *
Crawl-delay: 10
Additional studying: 15 Crawlability Issues & Repair Them
The Noindex Directive
A robots.txt file tells search engines like google what to crawl and what to not crawl however can’t reliably maintain a URL out of search outcomes—even should you use a noindex directive.
If you happen to use noindex in robots.txt, the web page can nonetheless seem in search outcomes with out seen content material.
Google by no means formally supported the noindex directive in robots.txt and confirmed so in September 2019.
To reliably exclude a web page from search outcomes, use a meta robots noindex tag as an alternative.
Create a Robots.txt File
Use a robots.txt generator instrument to shortly create a robots.txt file.
Observe these steps to create a robotx.txt file from scratch:
1. Create a File and Identify It Robots.txt
Open a .txt doc in a textual content editor or internet browser.
Identify the doc “robots.txt.”
Now you can begin typing directives.
2. Add Directives to the Robots.txt File
A robots.txt file accommodates a number of teams of directives, and every group contains a number of traces of directions.
Every group begins with a user-agent and specifies:
- Who the group applies to (the user-agent)
- Which directories (pages) or recordsdata the agent ought to entry
- Which directories (pages) or recordsdata the agent shouldn’t entry
- A sitemap (optionally available) to inform search engines like google which pages and recordsdata you deem essential
Crawlers ignore traces that don’t match the above directives.
Think about you don’t need Google to crawl your “/shoppers/” listing as a result of it’s for inside use solely.
The primary group in your file would seem like this block:
Consumer-agent: Googlebot
Disallow: /shoppers/
You may add extra directions for Google after that, just like the one under:
Consumer-agent: Googlebot
Disallow: /shoppers/
Disallow: /not-for-google
Then press enter twice to begin a brand new group of directives.
Now think about you need to stop entry to “/archive/” and “/assist/” directories for all search engines like google.
A block stopping entry to these :
Consumer-agent: Googlebot
Disallow: /shoppers/
Disallow: /not-for-google
Consumer-agent: *
Disallow: /archive/
Disallow: /assist/
When you’re completed, add your sitemap:
Consumer-agent: Googlebot
Disallow: /shoppers/
Disallow: /not-for-google
Consumer-agent: *
Disallow: /archive/
Disallow: /assist/
Sitemap: https://www.yourwebsite.com/sitemap.xml
Save the file as “robots.txt.”
3. Add the Robots.txt File
After saving your robots.txt file, add the file to your website so search engines like google can discover it.
The method of importing your robots.txt file depends upon your internet hosting atmosphere.
Search on-line or contact your internet hosting supplier for particulars.
For instance, search “add robots.txt file to [your hosting provider]” for platform-specific directions.
Beneath are some hyperlinks explaining learn how to add robots.txt recordsdata to well-liked platforms:
After importing, affirm that the file is accessible and that Google can learn it.
4. Check Your Robots.txt File
First, confirm that anybody can view your robots.txt file by opening a personal browser window and getting into your sitemap URL.
For instance, “https://semrush.com/robots.txt.”
If you happen to see your robots.txt content material, check the markup.
Google supplies two testing choices:
- The robots.txt report in Search Console
- Google’s open-source robots.txt library (superior)
Use the robots.txt report in Search Console in case you are not a sophisticated consumer.
Open the robots.txt report.
If you happen to haven’t linked your website to Search Console, add a property and confirm website possession first.
If you have already got verified properties, choose one from the drop-down after opening the robots.txt report.
The instrument experiences syntax warnings and errors.
Edit errors or warnings instantly on the web page and retest as you go.
Modifications made throughout the robots.txt report aren’t saved to your website’s stay robots.txt file, so copy and paste corrected code into your precise robots.txt file.
Semrush’s Web site Audit instrument can even verify for robots.txt points.
Arrange a undertaking and run an audit.
When the instrument is prepared, navigate to the “Points” tab and seek for “robots.txt.”
Click on “Robots.txt file has format errors” if it seems.
View the listing of invalid traces.
Click on “Why and learn how to repair it” for particular directions.
Examine your robots.txt file commonly. Even small errors can have an effect on your website’s indexability.
Robots.txt Greatest Practices
Use a New Line for Every Directive
Place every directive by itself line to make sure search engines like google can learn them and observe the directions.
Incorrect instance:
Consumer-agent: * Disallow: /admin/
Disallow: /listing/
Appropriate instance:
Consumer-agent: *
Disallow: /admin/
Disallow: /listing/
Use Every Consumer-Agent Solely As soon as
Record every user-agent as soon as to maintain the file organized and scale back the danger of human error.
Complicated instance:
Consumer-agent: Googlebot
Disallow: /example-page
Consumer-agent: Googlebot
Disallow: /example-page-2
Clear instance:
Consumer-agent: Googlebot
Disallow: /example-page
Disallow: /example-page-2
Writing all directives beneath the identical user-agent is cleaner and helps you keep organized.
Use Wildcards to Make clear Instructions
Use wildcards (*) to use directives broadly.
To stop search engines like google from accessing URLs with parameters, you could possibly technically listing them out one after the other.
Nonetheless, you may simplify your instructions with a wildcard.
Inefficient instance:
Consumer-agent: *
Disallow: /footwear/vans?
Disallow: /footwear/nike?
Disallow: /footwear/adidas?
Environment friendly instance:
Consumer-agent: *
Disallow: /footwear/*?
The above instance blocks all search engine bots from crawling all URLs beneath the “/footwear/” subfolder with a query mark.
Use ‘$’ to Point out the Finish of a URL
Use “$” to point the tip of a URL.
To dam search engines like google from crawling all of a sure file kind, utilizing “$” helps you keep away from itemizing all of the recordsdata individually.
Inefficient:
Consumer-agent: *
Disallow: /photo-a.jpg
Disallow: /photo-b.jpg
Disallow: /photo-c.jpg
Environment friendly:
Consumer-agent: *
Disallow: /*.jpg$
Use “$” fastidiously as a result of errors can result in unintended unblocking.
Add feedback by beginning a line with “#”—crawlers ignore something that begins with a hash.
For instance:
Consumer-agent: *
#Touchdown Pages
Disallow: /touchdown/
Disallow: /lp/
#Information
Disallow: /recordsdata/
Disallow: /private-files/
#Web sites
Permit: /web site/*
Disallow: /web site/search/*
Builders typically add humorous feedback utilizing hashes since most customers by no means see the file.
For instance, YouTube’s robots.txt file reads: “Created within the distant future (the 12 months 2000) after the robotic rebellion of the mid 90’s which worn out all people.”
And Nike’s robots.txt reads “simply crawl it” (a nod to its “simply do it” tagline) and options the model’s emblem.
Use Separate Robots.txt Information for Completely different Subdomains
Robots.txt recordsdata solely management crawling on the subdomain the place they reside, which suggests you might want a number of recordsdata.
In case your website is “area.com” and your weblog is “weblog.area.com,” create a robots.txt file for each the area’s root listing and the weblog’s root listing.
5 Robots.txt Errors to Keep away from
When creating your robots.txt file, be careful for the next widespread errors:
1. Not Together with Robots.txt within the Root Listing
Your robots.txt file have to be situated in your website’s root listing to make sure search engine crawlers can discover it simply.
For instance, in case your web site’s homepage is “www.instance.com,” place the file at “www.instance.com/robots.txt.”
If you happen to put it in a subdirectory, like “www.instance.com/contact/robots.txt,” search engines like google might not discover it and will assume you haven’t set any crawling directions.
2. Utilizing Noindex Directions in Robots.txt
Don’t use noindex directions in robots.txt—Google doesn’t assist the noindex rule within the robots.txt file.
As a substitute, use meta robots tags (e.g., ) on particular person pages to manage indexing.
3. Blocking JavaScript and CSS
Keep away from blocking entry to JavaScript and CSS recordsdata by way of robots.txt until crucial (e.g., proscribing entry to delicate information).
Blocking crawling of JavaScript and CSS recordsdata makes it tough for search engines like google to grasp your website’s construction and content material, which might hurt your rankings.
Additional studying: JavaScript web optimization: Optimize JS for Search Engines
4. Not Blocking Entry to Your Unfinished Web site or Pages
Block search engines like google from crawling unfinished variations of your website to maintain it from being discovered earlier than you’re prepared (additionally use a meta robots noindex tag for every unfinished web page).
Engines like google crawling and indexing an in-development web page might result in a poor consumer expertise and potential duplicate content material points.
Utilizing robots.txt to maintain unfinished content material non-public till you’re able to launch.
5. Utilizing Absolute URLs
Use relative URLs in your robots.txt file to make it simpler to handle and keep.
Absolute URLs are pointless and might trigger errors in case your area modifications.
❌ Instance with absolute URLs (not really helpful):
Consumer-agent: *
Disallow: https://www.instance.com/private-directory/
Disallow: https://www.instance.com/temp/
Permit: https://www.instance.com/important-directory/
✅ Instance with relative URLs (really helpful):
Consumer-agent: *
Disallow: /private-directory/
Disallow: /temp/
Permit: /important-directory/
Preserve Your Robots.txt File Error-Free
Now that you just perceive how robots.txt recordsdata work, you need to guarantee yours is optimized. Even small errors can have an effect on how your website is crawled, listed, and displayed in search outcomes.
Semrush’s Web site Audit instrument makes analyzing your robots.txt file for errors simple and supplies actionable suggestions to repair any points.
