Website not indexed by Googlebot – problem with http header or sitemap or robots

#268892
  • Resolved Monica
    Rank Math free

    Hi guys!
    I have a problem with my website and I would like to get help.
    Two weeks ago my web hosting changed its servers and I have had several problems since then. What I can’t solve is about indexing the website through Google Seach Console.
    Before the server change, I hadn’t done any seo to my website, but it was anyway on the first page in the searches because its previous version, the old website, which was not in wordpress but in html, written manually line by line, had a 93/100 seo score. Since the situation was good, I had postponed, with the new wordpress site, the seo work but the change of the server has forced me to face the problem. A friend recommended Rank Math and I think it’s a great tool. Currently the mobile version of my website is on the first page, in the very first place in the Google searches, but the desktop version is totally absent. It is present on Google but not indexed because I reported the url but it does not appear anywhere.
    The sitemap, both the one created with Rank Math and the one I’ve created manually, have been rejected by Google Search Console for a general HTTP error: 403 error. Some tests reported an error in the robots.txt file or in the robots meta tag, maybe there is a command conflict or access/permission problem. I am attaching the screenshots of some header checkers tests I made and the settings I have in Rank Math. I’ve also tried to do a Googlebot simulation following this article ( https://gentofsearch.com/blog/chrome-googlebot-simulator/ ), and while other robots including Googlebot Mobile and Googlebot Desktop upload the website without any problem, Googlebot gives a 404 error on all resources and doesn’t show the webpage. Apparently Googlebot can’t access to the folders wp_content and wp_includes In some forum it is said to delete the line Disallow: / wp_content / and Disallow: / wp_includes / from the robots.txt file and replace them with Noindex, but it doesn’t work. Just to leave nothing to chance, I have to tell you that I’ve also these plugins: Easy WP SMTP, Elementor, Element Pack Lite – Addon for Elementor, Elementor PRO, Site Kit by Google and WP-Optimize – Clean, Compress, Cache. Maybe any of them could give problems? Anyone have any idea what the real problem is and how I can fix it?
    It is really frustrating because the internet is my only sales channel as my clients are international. Traffic has decreased from 14K to 1,4 and the number of inquires is almost zero, while they were 2-3 per day until a month ago. I really need help with this, I feel like I haven’t enough knowledge for solving the problem on my own.
    Thanks a lot to anyone who can provide any clarification or advice.

Viewing 3 replies - 1 through 3 (of 3 total)
  • Hello,

    Thanks for contacting us. Sorry for the unexpected delay and any inconvenience that might have been caused due to that.

    I can check that your site is already indexed on Google. If you check for site:yourdomain.com/ you will be able to see the results for your domain.

    I checked your sitemap and it is returning a 404 error. Please follow the steps given below to resolve the issue:

    1. Flush the Sitemap cache by following this video screencast:
    https://i.rankmath.com/pipRDp

    2. Exclude the Sitemap files of the Rank Math plugin in your caching plugin. The cache could be via a plugin or from the server. For plugins or Cloudflare, please follow this article:
    https://rankmath.com/kb/exclude-sitemaps-from-caching/

    After that check your sitemap with this tool: https://www.xml-sitemaps.com/validate-xml-sitemap.html

    If it validates fine then remove all previously submitted sitemaps from your GSC account and resubmit your main sitemap(sitemap_index.xml). Wait for Google to crawl your sitemap again and the issue should be resolved.

    Also, you are not using the robots.txt file created by Rank Math. It seems like there is a physical robots.txt file is present in your site’s root directory. Please remove that file so Rank Math’s robots.txt can take over and allow Google to recognize and crawl your sitemap.

    And all other settings inside Rank Math seem fine. Please follow the above-mentioned steps and wait for Google to crawl your site again.

    Let us know how that goes. Looking forward to helping you.

    Monica
    Rank Math free

    Hi! Thank you so much for your advice. I’ve followed the procedure you’ve indicated step by step but Google Search Console still cannot read the sitemap (error 403). Is there anything else I can do?

    Prabhat
    Rank Math agency

    Hello,

    I accessed the robots.txt URL on your website and it returns a 404 error.

    Here’s a screenshot: https://i.rankmath.com/Z7QKrL

    It seems like there might be permission issues on the server due to which, the robots.txt file is not generated and the sitemap is also not accessible by Googlebot. https://rankmath.com/kb/cant-edit-robots-txt/#num-3-permission-issue-on-the-web-server

    Please get in touch with your web host regarding this as they would be in a better position to assist you.

    Please also mention to your web host that upon accessing the sitemap as Googlebot, it is returning a 403 (forbidden) error.

    Once fixed, you can clear your website’s cache and wait for Google to crawl the website again to see if the issue gets fixed.

    Please let us know how that goes.

    Thank you.

    Hello,

    Since we did not hear back from you for 15 days, we are assuming that you found the solution. We are closing this support ticket.

    If you still need assistance or any other help, please feel free to open a new support ticket, and we will be more than happy to assist.

    Thank you.

Viewing 3 replies - 1 through 3 (of 3 total)

The ticket ‘Website not indexed by Googlebot – problem with http header or sitemap or robots’ is closed to new replies.