Hello,
Thank you for contacting Rank Math and bringing your concern to our attention. I’m sorry for any inconvenience this issue may have caused you.
Please note that we only report the 404s. We do not have information on how or why they are generated. You can get further insight into them by enabling the advanced monitor at WordPress Dashboard > Rank Math > General Settings > 404 Monitor:
https://rankmath.com/kb/monitor-404-errors/#advanced-404-monitor
That will give you more information like referrer and user-agent. You can use that information to track down the source or find out if it is just bots generating those errors.
You can also fix these errors by redirecting them to a related page on your website:
https://rankmath.com/kb/fix-404-errors/
Or, you can add them to the exclusion list at WordPress Dashboard > Rank Math > General Settings > 404 Monitor > Exclude Paths
On the other hand, I’ve checked your /robots.txt, and it seems that you’re not using the recommended robots.txt of Rank Math.
Please update it as well as you’re allowing all bots to access your website files/directory.
I hope that helps.
Thank you.
Hello,
Ok.
A question out of scope maybe, is there any feature by RM for deleted pages/posts being crawled “referentially”, or clearing caches can help those being crawled externally on referential?
Thank you for your suggestion about robot.txt, furthermore, is deleting the robot text file completely would result in the same as RM recommended settings?
Thank you
Hello,
We don’t have a feature to check and analyze your website to look for broken links. Our 404 module only reports when those 404 errors are triggered by accessing a page that no longer exists.
Regarding the robots.txt file, if you delete the physical file from the root of your website our plugin will create a dynamic one with the default options.
Don’t hesitate to get in touch if you have any other questions.
Hello,
I deleted the robot.txt from the root and added the default values manually. However Google Lighthouse report pointed out this as a problem and the sitemap url defined as “invalid”
Here below the link to the SS.
https://ibb.co/nmmTvxz
What can you say about that?
Thanks,
Hello,
I checked your robots.txt configuration and it seems that you haven’t added any protocol to your sitemap URL as you can see in my screenshot here:
Please ensure to add your sitemap URL like so: http://supplementtrends.com/sitemap_index.xml
Let us know how that goes. Looking forward to helping you.
Hello,
I was not aware of it just copied as it is somewhere 🙂
Thank you for your support
Hello,
We are super happy that we have addressed your concerns. If you have any other questions in the future, know that we are here to help you.
If you don’t mind us asking, could you please leave us a review (if you haven’t already) on https://wordpress.org/support/plugin/seo-by-rank-math/reviews/#new-post about your overall experience with Rank Math? We appreciate your time and patience.
If you have another question in the future, please feel free to create a new forum topic, and it will be our pleasure to assist you again.
Thank you.
