pages on the website blocked by robots.txt

Hello,

We apologize for the issue you are facing with your website’s pages being blocked by robots.txt. This can often lead to errors like "Unable to open Desktop URL" and "Unable to open Mobile URL."

To resolve this issue, you should check your robots.txt file for any directives that might be blocking Googlebot from crawling your pages. Here are the steps you can follow:

1. Access your robots.txt file: You can find it at `https://rankmath.com/kb/blocked-due-to-other-4xx-issue/.
2. Review the file: Look for any Disallow directives that might be preventing access to the pages in question.
3. Modify or remove directives: If you find any rules that are blocking important pages, consider modifying or removing them to allow Googlebot access.

For more detailed guidance on editing your robots.txt file with Rank Math, you can refer to this guide.

Additionally, you can use the Robots Testing Tool to check if specific URLs are blocked. Enter the URL in the tool and select ‘Googlebot’ as the User Agent to see if it’s disallowed.

If you continue to experience issues after checking your robots.txt file, please reply to this ticket with the specific URLs that are being blocked, and a human agent from our support team will assist you further. Thank you for bringing this concern to our attention!

Sources:
Common robots.txt Issues & How to Fix Them » Rank Math
How to Fix Indexing Issues in Google Search Console » Rank Math