Robots.txt Blocking Indexing for My Site - Support

Resolved Jessie Lambert
5 months, 3 weeks ago

Rank Math free

I’ve been troubleshooting this for weeks and am hoping you can help.

The Robots.txt file is blocking site indexing. However, it is just this:

User-agent: *
Disallow:

I’ve spoken to my host and we manually added a Robots.txt to see if that will help. But they suggested I contact you as this is the only site I’m having this issue for, and it is also the only site I’m using Rankmath Pro on.

To solve this, I’ve already tried renaming the sitemap, removed old sitemap and recreated it, requested reindexing, requested recrawl of the robots.txt file (although what’s shown in the report is correct — it should not be blocking anything). I’ve purged site and server cache. I’m not using a CDN.

Please let me know if you have any experience with this. All the troubleshooting I’ve done is not getting me anywhere.

Viewing 6 replies - 1 through 6 (of 6 total)

RankBot
5 months, 3 weeks ago
Hello,

We apologize for the issue you are facing with your robots.txt file blocking site indexing. Based on the information you’ve provided, it seems that your robots.txt file is set up correctly with the following directive:
```
User-agent: *
Disallow:
```
This configuration should allow all user agents to crawl your site without restrictions. However, if Google is still reporting that your site is blocked, there are a few steps you can take to further troubleshoot this issue:
1. Check the Robots Testing Tool: Use the Robots Testing Tool to verify that Googlebot is not being blocked from crawling your site. Enter your URL and select "Googlebot" as the User Agent, then click TEST. This will show you if there are any rules in your robots.txt that might be affecting Googlebot’s access.
2. Clear Cache Again: Since you’ve already purged your site and server cache, ensure that there are no caching plugins or server-side caching mechanisms that might be serving an outdated version of your robots.txt file.
3. Check for Other Directives: Ensure there are no other directives in your Rank Math settings or any other SEO plugins that might be affecting indexing. Sometimes, settings in the SEO plugin can override the robots.txt file.
4. Inspect Google Search Console: Look for any specific errors or warnings in Google Search Console related to indexing or crawling. This can provide more insight into what might be causing the issue.
5. Manual Review: If possible, manually review the contents of your robots.txt file directly by visiting yourdomain.com/robots.txt to ensure it reflects the changes made.
If after these steps the issue persists, please reply to this ticket with your site URL so we can assist you further. A human agent from our support team will take over and provide more tailored assistance.

Thank you for your patience, and we hope to resolve this issue for you soon!

Sources:
Common robots.txt Issues & How to Fix Them » Rank Math
How to Fix Indexing Issues in Google Search Console » Rank Math
Jessie Lambert
5 months, 3 weeks ago

Rank Math free

I did everything listed above.

1 – everything is showing valid/”Allowed by Disallow”
2 – Cleared site, server, and CDN caches
3 – I’m only using Rankmath for SEO and am not aware of any other plugins that are creating Robots.txt files. I’m using Advanced Custom Fields Pro, CookieYes, Custom Post Types UI, Elementor, Elementor Pro, Happy Elementor Addons, Imagify, Meta Pixel for WordPress, Ranmath, Rankmath PRO, Replafce Image, Seraphinite Accelerator, Site Kit by Google, Speed Optimizer (Siteground), and UpdraftPlus. I also have Upsell Plubin and Advanced Custom Fields, but they are deactivated.
4 – I’m getting the same error in your screenshot — “Blocked by Robots.txt”
5 – I’ve manually reviewed, and nothing seems wrong:
User-agent: *
Disallow:

NOTE: I did update the Robots.txt file Sunday, March 9 at 1pm from “…Disallow: / ” to “Disallow:” — so this may be an issue with needing to wait more, but it has been over 24 hours.

Jessie Lambert
5 months, 3 weeks ago

Rank Math free

My URL is waycraftmedia.com

Reinelle
5 months, 3 weeks ago

Hello,

We always recommend using our default and recommended rules for your robots.txt:
https://rankmath.com/kb/how-to-edit-robots-txt-with-rank-math/#default-rules

According to Google, the disallow: rule is an invalid rule and hence matches nothing:
https://support.google.com/webmasters/thread/59279351/using-robots-txt-disallow-without-parameters?hl=en

However, you may try to update your robots.txt rules in Rank Math > General Settings > Edit robots.txt.

If it’s still not updating, then one of your plugins (or plugins under /mu-plugins) or theme is generating the robots.txt on your site.

Looking forward to helping you.

Jessie Lambert
5 months, 3 weeks ago

Rank Math free

That was the issue! I don’t recall changing the Robots.txt file after installing RankMath, but my host (Siteground) shared what it should be (before it has “Disallow: / “). When I updated it to the following, I cleared all caches (site, server, CDN), initiated a recrawl of robots.txt in Search Console, deleted the existing sitemap, and added the sitemap back. Now I’m indexed!

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://yoursite.com/sitemap_index.xml

Note: My host did recommend creating a robots.txt in my root folder vs using the one in RankMath, so I also did that. Just sharing in case this helps someone else! But please let me know if there are any issues with doing this and I will revert back.

Thank you so much for the help!

Princess
5 months, 3 weeks ago

Hello,

We are super happy that this resolved your issue. If you have any other questions in the future, know that we are here to help you.

If you don’t mind me asking, could you please leave us a review (if you haven’t already) on
https://wordpress.org/support/plugin/seo-by-rank-math/reviews/#new-post

about your overall experience with Rank Math? We appreciate your time and patience.

If you do have another question in the future, please feel free to create a new forum topic, and it will be our pleasure to assist you again.

Thank you.

RankBot
2 months, 2 weeks ago

Hello,

Since we did not hear back from you for 15 days, we are assuming that you found the solution. We are closing this support ticket.

If you still need assistance or any other help, please feel free to open a new support ticket, and we will be more than happy to assist.

Thank you.

Viewing 6 replies - 1 through 6 (of 6 total)

The ticket ‘Robots.txt Blocking Indexing for My Site’ is closed to new replies.