Blank Robots.txt error? - Support

Resolved Blogging Lecture
2 years, 7 months ago

Rank Math free

Hi team, We mentioned a site below that has blank robots.txt and there was a blank sitemap that we fixed, there are many search console core web vitals error that is now requested to be fixed.

I wanted to know, if these things like core web vitals errors, blank sitemaps, and blank robots.txt can be the reason for the sudden ranking drop in google, we loosed several keywords, that were ranking on the first page, but now it is on 4-5 page or even disappeared.

We are using Cloudways and didn’t upload any custom robots.txt file but Rankmath plugin is showing “Contents are locked because a robots.txt file is present in the root folder”.

Please help, how to solve this using WordPress. And can this and other issues told above can be the reason of google ranking drop?

Thanks for taking the time.

Viewing 4 replies - 1 through 4 (of 4 total)

Rakibuzzaman Siam
2 years, 7 months ago
Hello,

Thank you for contacting us and sorry for any inconvenience that might have been caused due to that.

The issues you’re facing on Google Search Console are not related to your robots.txt or your sitemap. You can try to URLs below to resolve those issues:

1. Excluded by NoIndex Tag:
https://rankmath.com/kb/excluded-by-noindex-tag/

2. Alternate Page with Proper Canonical Tag:
https://rankmath.com/kb/alternate-page-with-proper-canonical-tag/

3. Not found (404):
https://rankmath.com/kb/find-the-souce-of-404-errors/

4. Page With Redirect:
https://rankmath.com/kb/page-with-redirect/

5. Duplicate Without User-Selected Canonical:
https://rankmath.com/kb/duplicate-without-user-selected-canonical/

Regarding the robots.txt, I can see the robots.txt file is being generated by Rank Math properly at /robots=1. Please check if there is a robots.txt file in the root directory of your website. If there is a file, delete it and clear your website cache including any server-level cache, and check again.

But if there’s no physical robots.txt file present in your server’s root directory, then this might be happening because the redirection isn’t taking the place. In order to fix this issue, you can try adding some rules to your site, similar to the following rewrite rule, and check if the robots.txt is working properly:

# For Nginx server:
```
# START Nginx Rewrites for Rank Math Robots.txt File
rewrite ^/robots.txt$ /?robots=1 last;
rewrite ^/([^/]+?)-robots([0-9]+)?.txt$ /?robots=$1&robots_n=$2 last;
# END Nginx Rewrites for Rank Math Robots.txt File
```
Here’s an article you can check as well: https://ubiq.co/tech-blog/rewrite-url-parameters-nginx/

# For Apache server:
```
# START of Rank Math Robots.txt Rewrite Rules
RewriteEngine On
RewriteBase /
RewriteRule ^robots.txt$ /index.php?robots=1 [L]
# END of Rank Math Robots.txt Rewrite Rules
```
Here’s how you can edit .htaccess file using Rank Math:
https://rankmath.com/kb/edit-htaccess/

Please do take a backup before editing your .htaccess file.

You can also ask your web host to do that for you.

Let us know how it goes. Looking forward to helping you.

Thank you.
Blogging Lecture
2 years, 7 months ago

Rank Math free

It got solved but we are facing a problem these days. We had all the articles in the top 10 and now most of them have gone away.

Recently we changed the canonical URL of those posts and it solved the issue we got our ranking back. We thought that it was happening because of duplicate pages but now again the posts are losing their ranking. We are not sure what is causing this issue.

One thing we saw right now is that the post we added canonical URLs manually has been removed from the sitemap. Would you please tell me how to get those URLs back on the sitemap page (https://modicarefamily.com/post-sitemap.xml)? There are almost 400 posts but there are only a few in the sitemap.

And also why the ranking is going up and down. It is not a Google update for sure, it is something that is blocking Google bots and creating some technical issues. We are waiting for Rankmath’s support reply.

Blogging Lecture
2 years, 7 months ago

Rank Math free

The traffic dropped from 150K+ per day to 10K per day. It happened a few days back, we did some changes to website design and some search console error fixes, and the traffic got recovered but now again traffic has reduced. Please tell me why this is happening, is it because of some issues in the sitemap, robots.txt, or anything else?

Miguel
2 years, 7 months ago

Hello,

Please note that we don’t add the pages with custom canonical to the sitemap because usually the canonical is used to consolidate the duplicate content and as such the custom canonical is not added to the sitemap because we don’t have a way to check if the URL is getting duplicated in the sitemap or not.

Also, pages with a different canonical rather than their own URL will not get indexed and as such it’s not necessary to include them in the sitemap.

This could also explain why the rankings are going down.

You are essentially telling Google to index other pages when visiting the ones with custom canonical and reducing the overall exposure of the website.

Another possibility for the drop in traffic is the recent changes to Google algorithms which can cause this.

If you want to verify that, we would suggest the following tutorial: https://rankmath.com/blog/google-algorithm-update-recovery/

Don’t hesitate to get in touch if you have any other questions.

RankBot
2 years, 6 months ago

Hello,

Since we did not hear back from you for 15 days, we are assuming that you found the solution. We are closing this support ticket.

If you still need assistance or any other help, please feel free to open a new support ticket, and we will be more than happy to assist.

Thank you.

Viewing 4 replies - 1 through 4 (of 4 total)

The ticket ‘Blank Robots.txt error?’ is closed to new replies.