Identifying the Problem
If you see the “Blocked by robots.txt” error in Google Search Console, it means your robots.txt file is stopping Googlebot from accessing some of your URLs. This can impact your site’s search visibility.
What is robots.txt?
- Location: Found at the root of your website.
- Function: Guides search engine crawlers like Googlebot on which pages they can and cannot access.
- Commands: Uses instructions like “User-agent,” “Disallow,” and “Allow” to manage access.
Deciding Which URLs to Block
Certain URLs shouldn’t be indexed to save Google’s crawl budget and avoid unnecessary indexing. Examples include:
- Dynamic URLs: URLs with parameters like search terms, filters, or product variations.
- User-Specific URLs: Pages related to user accounts, shopping carts, or checkout processes.
When to Unblock URLs
While blocking some URLs is fine, you might accidentally block important pages, affecting your site’s visibility in search results.
Steps to Fix the Issue
Find the Blocked URLs
- Use Google Search Console’s “Index Coverage” report to identify pages with the “Blocked by robots.txt” error.
- Check if these URLs should be accessible to Googlebot.
Edit Your robots.txt File
- Access the file using an FTP client or your website’s content management system (CMS).
- Make a backup copy before making any changes.
Review the robots.txt File
- Look for directives blocking Googlebot from the affected URLs.
- Common issues include broad “Disallow” directives or incorrect syntax.
Modify the File
- Remove or adjust the “Disallow” directives to allow Googlebot access.
- Example: Change Disallow: /products/ to Disallow: /products/sale/ to enable access to other product pages.
- Ensure correct syntax and avoid overusing wildcard characters.
Test Your Changes
- Use a robots.txt testing tool to check the effectiveness of your changes.
- Verify that Googlebot can now access the previously blocked URLs.
Update and Submit the File
- Save the updated robots.txt file to your website’s root directory.
- Allow time for Googlebot to re-crawl your site and update its index.
Additional Tips
- Be Precise: Use specific directives to manage access to particular pages or directories.
- Avoid Overblocking: Blocking too many pages can harm your website’s visibility.
- Use “Allow” Directives: Explicitly allow access to certain pages when necessary.
- Submit a Sitemap: Use Google Search Console to submit a sitemap to prioritize important pages.
- Monitor Regularly: Keep an eye on your robots.txt file and Google Search Console to ensure everything functions correctly.
Example robots.txt File
This example allows Googlebot to access all pages except those in the “/admin/” and “/checkout/” directories. It explicitly permits access to images.
Final Thoughts
Before making changes to your robots.txt file, review your website’s structure and goals carefully. Incorrect modifications can have unintended consequences. Following these steps will help you resolve the “Blocked by robots.txt” issue and improve your site’s visibility in search results.
