Hello,
Thanks for contacting us and sorry for the inconvenience caused.
If the search engines have already discovered the URLs then they will still show the error for them even if they are returning a 404 error. You can ignore them as the search engines will eventually stop crawling them.
If you are completely sure that the images are no longer present on your site, you should get in touch with the support of Optimole as they will be in a better position to help you figure out why their plugin is still discovering those images.
Hope that helps. Let us know if you need any other assistance.
I contacted Optimole.
For urls that will be deleted from search engines, I tell you bluntly. It is not true! I got penalty after penalty for undeletable urls. I had the same problem with the previous site as well. A year and a half wasn’t enough to delete even a tenth of the URLs I no longer had.
Hello,
Thank you for your reply. We’re sorry to hear that you’re having issues with removing URLs from search engines. We understand your frustration and we want to help you resolve this problem.
First of all, we want to assure you that Rank Math does not create any URLs that you don’t have on your site. Rank Math only helps you optimize your existing URLs for SEO and generate sitemaps for them.
Secondly, we want to explain that removing URLs from search engines is not something that Rank Math or any other plugin can do for you. It is something that the search engine does automatically or where it does not, you have to do it yourself by following the guidelines and tools provided by the search engines themselves.
From our experience, we know that when Google re-crawls your site, as long as the deleted pages are not in your sitemap, Google does not include them in its index – unless Google is discovering those pages from other sources, like maybe you linked to them from other pages or other website links to them.
That said, if you still want to remove these pages from search results – you can use the Google Removal tools in Google Search Console > Indexing > Removals. You can learn more about removing URLs fro Google here: https://ahrefs.com/blog/remove-urls-from-google/
We hope this helps you understand how to remove URLs from search engines and improve your SEO performance. If you have any other questions or issues, please let us know. We are always happy to help.
Have a great day!
I’ve followed Ahrefs’ guidance in the past, but with little success. Google keeps the old cache and doesn’t re-crawl the URLs. What if I use a very permissive robots.txt file instead? What should I write to my robots?
Hi there,
Using your robots.txt file is not a good idea if you want to remove old URLs from Google. Crawling is not the same thing as indexing. Your robots.txt file only tells Google not to crawl your pages, it does not prevent Google from indexing them and definitely does not remove a page that’s already indexed. If Google finds links to your pages from other sources, it may still show them in the search results.
As we mentioned earlier, the best way to remove old URLs from Google is to use the Removals tool in Google Search Console.
To move forward with this, can you please confirm that you have used the Removal tool, if yes, can you please try again – this time use loom.com to screen record the steps you took. Send us the video alongside some examples of the pages in question so we can take a look.
We really look forward to hearing back from you. Thank you.
I may have encountered the problem that has been plaguing me for more than 2 years and on 2 sites; There is a block for robots.txt files. They told me to contact Cloudflare. Whether it’s a firewall issue or something else, you should include it in your site’s SEO analysis, because it’s a serious problem to fix.
Hello,
We’re not sure we understand your message completely. We checked your robots.txt file and it seems to be working fine for us. You can see the screenshot here: https://i.rankmath.com/i/dqmiqh
Also, please note that the content analysis test only checks the content of your page as it is seen by search engines. It does not take into account your robots.txt file or any other settings. The content analysis test is meant to help you improve your SEO and readability.
That said, we don’t think that your robots.txt file is the reason why your old pages are still showing up on Google. To fix that issue, you need to follow the instructions that we shared with you earlier.
Best
Hello,
I have updated the sensitive data as requested. Can you please check further?
Thank you.
Hello,
I’ve checked the robots.txt file, and it is not blocking Ahrefs. Please refer to the screenshot where the Google bot is properly seeing your robots.txt.
From the error message, it looks like the Firewall is blocking the Ahrefsbot.
To fix this, you may get in touch with your web host and ensure that the ModSecurity is disabled or not blocking the Ahrefsbots.
You can refer to this article for more information:
https://help.ahrefs.com/en/articles/1402668-troubleshooting-common-issues-with-site-audit-access
Hope that helps.
Thank you.
Hello,
Since we did not hear back from you for 15 days, we are assuming that you found the solution. We are closing this support ticket.
If you still need assistance or any other help, please feel free to open a new support ticket, and we will be more than happy to assist.
Thank you.
