Robot.txt issue, indexing issue,

Hey there Rank Math staff! First off I am so impressed with the support that you offer people with a free plug-in! This is absolutely amazing. To be honest I’d pay for this kind of support!

I’m having an issue and I’m confused. Let me give you an example concerning a certain page (in sensitive data).

The example page in question shows on google search console that it is indexed, and it even says that the page is mobile-friendly. Which is great. But then if I go to live test it states the URL is not on google and says “URL will be indexed only if certain conditions are met.” and if I go to mobile-friendly test- it states “Page cannot be reached”

This is where it gets more confusing for me. Because when I type in google search bar: site:(mywebsite.com/example page in question) It indeed shows up.

This caused me then to go to robot.txt tester and see what is happening. I see in the tester that the latest robot was April 21. When I test it states “An error has occured while trying to parse the robots.txt file”

I’m wondering if this is why my pages are not being ranked high on google. They used to be and then I switched over to Rank Math. We did a lot of SEO work and got our site pretty great (as far as your plug-in shared) but we noticed that we started dropping in ranking, and our homepage is ranked far higher than specific pages, which is weird (Even though they have great SEO, backlinks, internal links, etc etc.)

I’ve tried to talk to some google support people, and they have mentioned that it could be because I’m using a CDN to deliver images? Another suggested firewalls, or blocking bots??

I’m really confused. I’ve thought about deleting the Robot.txt file from file manager, but then I’m curious how Google sees the virtual robot.txt file?? Will that solve my problems?

THANK YOU SOOOOO MUCH for your help. Seriously would be happy to pay for this kind of support.

• This topic was modified 4 years, 8 months ago by Richard.