X-Robots-Tag Issues

Hello,

We apologize for the issue you are facing with the X-Robots-Tag settings and the sitemap registration in Google Search Console. It sounds like there may be a configuration issue that needs to be addressed.

Here are a few steps you can take to troubleshoot and resolve this problem:

1. Check Your Server Configuration: Ensure that the X-Robots-Tag is correctly set in your server configuration files (like .htaccess for Apache or nginx.conf for Nginx). You can use the following examples:

   • For Apache:

     Header set X-Robots-Tag "index, follow"

   • For Nginx:

     add_header X-Robots-Tag "index, follow";

2. Verify Plugin Settings: Double-check the settings in your HTTP header plugin to ensure that it is correctly configured to set the X-Robots-Tag to index, follow. Sometimes, plugins may have conflicting settings or may not apply changes as expected.

3. Clear Caches: If you are using any caching plugins or server-side caching, clear all caches to ensure that the latest headers are being sent.

4. Test with cURL: You can test the response headers directly using a tool like cURL. Run the following command in your terminal (replace yourwebsite.com with your actual domain):

   curl -I https://support.rankmath.com

   Look for the X-Robots-Tag in the response headers to see if it reflects the correct settings.

5. Check for Conflicting Plugins: Sometimes, other SEO or security plugins may interfere with header settings. Temporarily disable other plugins to see if that resolves the issue.

6. Review Google Search Console: After making changes, give Google some time to re-crawl your site. You can also request a re-crawl of your sitemap in Google Search Console.

If you continue to experience issues after trying these steps, please reply to this ticket with your site URL so we can assist you further. A human agent from our support team will take over and provide more specific guidance.

Thank you for raising this concern, and we hope to help you resolve it soon!

Sources:
Can I NoIndex PDF Files Using an SEO Plugin? » Rank Math
Can I NoIndex PDF Files Using an SEO Plugin? » Rank Math

Hello,

Thank you for contacting us. We apologize for any inconvenience this issue may have caused.

Can you please confirm if you’re inspecting your sitemap URL in your Google Search Console and getting this error? If so, then it is normal and you don’t have to worry. You’ll get the error if you inspect your sitemaps because, by default, sitemaps are set to noindex as they are meant to be crawled and not to be displayed on search results pages.

You should only submit your sitemap to the sitemap section of your GSC account and shouldn’t use the URL inspection tool for the sitemap URLs.
https://rankmath.com/kb/submit-sitemap-to-google/

Hope that helps and please don’t hesitate to let us know if you have any other questions.

Thank you.

Hello,

The error you see from Google indicates that Google already knows about these URLs, but they haven’t indexed them yet.

Here is what Google says about this status:

Discovered – currently not indexed: The page was found by Google, but not crawled yet. Typically, Google tried to crawl the URL but the site was overloaded; therefore Google had to reschedule the crawl. This is why the last crawl date is empty on the report.

The discovery could be due to internal linking, hence there’s no referring sitemap as Google didn’t use that for the discovery of the said URL. Please note that sitemaps are just a backup crawling mechanism.

You need to wait for Google to crawl and index those URLs. Please refer more to this guide as well: https://rankmath.com/kb/discovered-currently-not-indexed/

Hope that helps.