Hello,
Thanks for contacting us and sorry for any inconvenience that might have been caused due to that.
I checked your site and everything seems fine there. The 403 error means Googlebot is being blocked from accessing your sitemap. Please check with your host if there is any server-side firewall that is blocking the web crawlers.
Let us know how that goes. Looking forward to your update.
Hello,
I had contacted my server before contacting you and they told me it’s not a server related problem. they say google bots are providing access without any problem.
Hello,
Please check your robots.txt file and make sure it’s the default one to let Google crawl your sitemap correctly. Currently, that file doesn’t even have the appropriate entry to the sitemap in there.
Here’s how you can edit that file with Rank Math: https://rankmath.com/kb/how-to-edit-robots-txt-with-rank-math/
Also, make sure that you don’t have any rules in Cloudflare or any other firewall blocking the crawling from Google to your website.
Hope this helps solve your issue.
Don’t hesitate to get in touch if you require any further assistance.
Hello,
When I entered the google search console robots.txt tool, I encountered the error that your robots.txt file could not be found.
When google bots wanted to scan my robots.txt file on 18.11.2021 Could not find robots.txt file.
What should I do to fix the problem?
Hello,
Navigate to your robots.txt file in Rank Math which is located under WordPress Dashboard > Rank Math > General Settings > Edit robots.txt as shown below:
Remove the already entered robots.txt rules from this file then paste the default Rank Math’s robots.txt rules:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: yoursite.com/sitemap_index.xml
Note: Replace yoursite.com with your actual domain name.
Click the Save Changes button after applying these directives.
We hope that helps. Please don’t hesitate to contact us again if you need further assistance.
Hello,
My robots.txt settings are already as you suggested.
Hello,
Rank Math dynamically generates robots.txt. When I access your robots.txt https://www.colaker.net/robots.txt it displays a different set of rules than the one set within the Rank Math’s robots.txt.
This suggests that there might be a file name robots.txt already present on your server. Kindly access your server through FTP and remove this file.
Looking forward to your update. Thank you.
I don’t have robots.txt file installed on my server. If there was a robots.txt file on my server, I could not edit the robots.txt file via the rankmath plugin, it would be locked.
I checked again for you and I couldn’t find any robots.txt file on my server.
Hello,
There’s currently something blocking the access by Googlebot to your website. Even the robots.txt file on your website is returning a 403 error which indicates that is being blocked.
Check every setting on your server, firewall, and Cloudflare in case you use that platform as well.
Unfortunately, there’s nothing else we can do from our side since we don’t block access to Googlebot for obvious reasons so this must be happening from other sources.
Don’t hesitate to get in touch if you have any other questions.
Hello,
Since we did not hear back from you for 15 days, we are assuming that you found the solution. We are closing this support ticket.
If you still need assistance or any other help, please feel free to open a new support ticket, and we will be more than happy to assist.
Thank you.
