Blocked due to unauthorized request (401)

Hello,

We apologize for the issues you are facing with your website’s indexing and the errors in Google Search Console. Thank you for raising these concerns.

Blocked Due to Unauthorized Request (401)

The 401 error typically indicates that access to a resource is restricted, often due to issues with your robots.txt file or permissions set on your server. To resolve this:

1. Check Your Robots.txt File: Ensure that your robots.txt file is not blocking Googlebot from accessing your site. You can edit your robots.txt file by going to Rank Math SEO → General Settings → Edit .robots.txt. If you need to regenerate it, make sure it includes the following rules:

   User-agent: *
   Disallow: /wp-admin/
   Allow: /wp-admin/admin-ajax.php
   Sitemap: yoursite.com/sitemap_index.xml

   

2. Check File Permissions: Ensure that the file permissions on your server are correctly set, especially if you are using a Linux server.

3. Regenerate Your .htaccess File: A corrupt .htaccess file can also lead to 401 errors. You can regenerate it by going to Rank Math SEO → General Settings → Edit .htaccess. If this section is unavailable, ensure you’ve switched to the Advanced Mode in Rank Math.
   

Warning – Rule Ignored by Googlebot (Crawl-Delay)

The warning regarding Crawl-Delay indicates that Googlebot does not recognize this directive in your robots.txt file. Googlebot does not support the Crawl-Delay directive, so you can safely remove this line from your robots.txt file to avoid confusion.

Sitemap Errors

Regarding the sitemap errors, it seems there are invalid URLs being generated. The URLs you provided (e.g., `https://rankmath.com/wp-content/uploads/2022/10/edit-htacess-file.jpg) appear to be malformed and should not be present in your sitemap. To fix this:

1. Check for Plugin Conflicts: Sometimes, plugins can interfere with sitemap generation. You can use the Health Check plugin to identify any conflicts.
2. Review Your Posts and Pages: Ensure that all posts and pages have valid URLs and that there are no custom canonicals that might be affecting the sitemap.
3. Regenerate Your Sitemap: You can regenerate your sitemap by going to Rank Math SEO → Sitemap Settings and ensuring everything is set up correctly.

If you continue to experience issues after trying these steps, please reply to this ticket, and a human agent from the Rank Math Support team will assist you further.

Thank you for your patience, and we hope to resolve these issues quickly so your website can go live!

Sources:
How to Fix the ‘Blocked Due to Other 4xx Issue’ Error » Rank Math
Rank Math Sitemap Issues and Their Fixes » Rank Math
How To Fix Blocked Due to Access Forbidden (403) Error in Google Search Console » Rank Math

Hello,

We checked your website and we don’t see any blockage happening when we load the site using Googlebot.

Can you please share a screenshot of the report you received about the 401 and the affected URLs?

As for the sitemap issues, the Blurb feature in Divi affects the generation of sitemap which is a common problem with our users here. Here are some example tickets with the same issue:

https://support.rankmath.com/ticket/invalid-urls-in-my-sitemap/
https://support.rankmath.com/ticket/sitemap-in-google-search-console/
https://support.rankmath.com/ticket/sitemap-can-be-read-but-has-errors-invalid-url-3/

This type of issue can’t be resolved by an update from our plugin. The Blurb feature is coming from Divi so you need to contact them instead.

Last but not least, as for the robots.txt issue, please share more details about it so we can assist you further.

Looking forward to helping you.