Critical: X-Robots-Tag: noindex Header on sitemap_index.xml Persists Despite Exh

#1065116
  • Resolved 이상배

    1 month, 2 weeks ago

    Rank Math free

    Critical: X-Robots-Tag: noindex Header on sitemap_index.xml Persists Despite Exhaustive Troubleshooting (Domain: noblelee.kr)

    Dear Rank Math Support Team,

    I am encountering a critical issue with my website’s (noblelee.kr) sitemap, https://noblelee.kr/sitemap_index.xml. Google Search Console continuously reports that this sitemap is being restricted from indexing due to an ‘X-Robots-Tag: noindex’ HTTP header. This is severely impacting my site’s ability to be properly indexed by Google.

    I have performed extensive and systematic troubleshooting, and all evidence points to Rank Math SEO as the source of this ‘noindex’ header. I kindly request your immediate assistance in resolving this.

    Here’s a detailed summary of the steps I have taken and the findings:

    **1. Initial Problem Confirmation (Google Search Console):**
    * My sitemap (https://noblelee.kr/sitemap_index.xml) consistently shows “URL is not on Google” and “Indexing allowed? No: ‘X-Robots-Tag’ http header detected noindex” in Google Search Console’s Live Test. The latest test was on 2025. 7. 10. 오후 1:42:54.

    **2. Hosting Provider Confirmation (FastComet):**
    * I contacted my hosting provider, FastComet. They confirmed that they are NOT generating these headers from the server side. Their tests showed that the ‘x-robots-tag: noindex’ header only appears when my WordPress plugins are enabled, clearly indicating the issue originates from the website (plugins), not the server.

    **3. Comprehensive WordPress-Level Troubleshooting:**

    **a. .htaccess File Verification:**
    * I have carefully reviewed my .htaccess file in the public_html directory. There are no directives such as ‘Header set X-Robots-Tag “noindex”‘ or similar rules that would directly add this HTTP header.

    **b. WordPress Search Engine Visibility:**
    * I have verified that the “Search Engine Visibility” option in WordPress Settings -> Reading is UNCHECKED, ensuring that I am not discouraging search engines from indexing the site.

    **c. Plugin Conflict Test (Excluding Rank Math SEO):**
    * Following a systematic approach, I deactivated ALL other WordPress plugins (except Rank Math SEO) to isolate the issue.
    * After deactivating all other plugins and clearing all caches, a Google Search Console Live Test for https://noblelee.kr/sitemap_index.xml *still showed the ‘X-Robots-Tag: noindex’ header.* This strongly indicates that Rank Math SEO itself, or a conflict involving Rank Math, is the source.

    **d. Rank Math SEO Plugin Settings Re-verification:**
    * **Global Meta Settings:**
    * Under Rank Math SEO -> Titles & Meta -> Global Meta, I confirmed that “Noindexed Category and Tag Archives” is set to ‘Off’ (회색).
    * In the ‘Robot Meta’ section, “Noindex” and “No Archive” are unchecked.
    * **Individual Content Type Settings:**
    * I have meticulously checked ‘Robot Meta’ settings for all post types (Posts, Pages, Media, Categories, Tags) under Titles & Meta. None of them have “Noindex” or “No Archive” explicitly checked.
    * **Sitemap Settings:**
    * Under Rank Math SEO -> Sitemap Settings, the sitemap URL (https://noblelee.kr/sitemap_index.xml) is clearly displayed, indicating the sitemap feature is active. There is no explicit “Enable/Disable Sitemap” toggle, which seems to be by design.

    **e. Thorough Cache Clearance:**
    * After every setting change and plugin activation/deactivation, I have consistently cleared ALL caches:
    * SpeedyCache: Used “CLEAR ALL CACHE AND THE SELECTIONS”.
    * Rank Math Cache: Used “분석 캐시 삭제” (Clear Analysis Cache) and “캐시 지우기” (Clear Cache) in Database Tools.
    * Browser Cache: Performed hard refreshes (Ctrl+F5 / Cmd+Shift+R).

    **4. Persistent Issue:**
    Despite all these exhaustive steps, the ‘X-Robots-Tag: noindex’ header continues to be present on my sitemap_index.xml, as confirmed by Google Search Console’s Live Test. This suggests a deeper, less obvious configuration or a potential bug within the Rank Math plugin itself, or a complex interaction with the active theme (Kadence).

    **Request for Assistance:**
    I am at a loss as to why this header persists. Could you please investigate this issue from your end, checking for any known bugs, specific settings I might have overlooked within Rank Math’s less obvious configurations, or any potential conflicts with WordPress core/Kadence theme that Rank Math might be causing?

    My ultimate goal is to have my sitemap properly indexed by Google.

    Thank you for your time and expertise.

    Sincerely,
    Sangbai Lee
    n*******@n********
    noblelee.kr

Viewing 3 replies - 1 through 3 (of 3 total)
  • RankBot

    1 month, 3 weeks ago

    Hello,

    We apologize for the issue you are facing with the ‘X-Robots-Tag: noindex’ header on your sitemap. It sounds like you’ve done a thorough job troubleshooting this problem, and we appreciate you providing such detailed information.

    To address the ‘X-Robots-Tag: noindex’ issue, here are a few steps you can take:

    1. Check Rank Math Settings:

      • Go to Rank Math > General Settings > Sitemap Settings and ensure that there are no settings that might be causing the sitemap to be marked as noindex.

    2. Review Custom Code:

      • If you have added any custom code snippets or modifications in your theme’s functions.php file or any other plugin, please review them to ensure they are not adding the noindex header.

    3. Plugin Conflicts:

      • Sometimes, other plugins can conflict with Rank Math. You can use the Health Check plugin to troubleshoot potential conflicts by disabling all other plugins except Rank Math and checking if the issue persists.

    4. Server Configuration:

      • Although your hosting provider has confirmed that they are not adding this header, it might be worth double-checking any server-level configurations or security settings that could be affecting the headers.

    5. Contact Support for Advanced Help:

      • If the issue continues after these checks, we recommend replying to this ticket so a human agent from our support team can assist you further. They can provide more in-depth troubleshooting and help resolve this critical issue.

    Thank you for raising this concern, and we hope to help you get this resolved quickly!

    Sources:
    Robots Meta Tag vs X-Robots: Understanding the Difference » Rank Math

    이상배

    1 month, 2 weeks ago

    Rank Math free

    Hello,

    I have updated the sensitive data as requested. Can you please check further?

    Thank you.

    Jeremy

    1 month, 2 weeks ago

    Hello,

    Thank you for contacting Rank Math support, and sorry for any inconvenience that might have been caused due to that.

    This is not an issue as XML Sitemaps are designed to be crawled by search engines, not indexed. The X-Robots-Tag: noindex directive ensures that the sitemap itself does not appear in search results, but this does not prevent Google from crawling its contents and indexing the pages listed within it.

    There’s no need to worry, as this is the expected behavior for sitemaps.

    Please avoid submitting your XML sitemaps to Google Search Console’s URL Inspection tool as that tool is meant for HTML pages.

    You can properly submit your sitemap to your Google Search Console account by following this guide: https://rankmath.com/kb/submit-sitemap-to-google/

    However, if you still wish to remove the noindex tag from your sitemap for any specific reason, you can use the filter in your site:

    add_filter( 'rank_math/sitemap/http_headers', function( $headers ) {
	if ( '/sitemap_index.xml' !== $_SERVER['REQUEST_URI'] ) {
		return $headers;
	}
	unset( $headers['X-Robots-Tag'] );
	return $headers;
} );

    Here’s how you can add filter/hook to your WordPress site:
    https://rankmath.com/kb/wordpress-hooks-actions-filters/

    Hope that helps and please do not hesitate to let us know if you need our assistance with anything else.

    RankBot

    2 weeks, 5 days ago

    Hello,

    Since we did not hear back from you for 15 days, we are assuming that you found the solution. We are closing this support ticket.

    If you still need assistance or any other help, please feel free to open a new support ticket, and we will be more than happy to assist.

    Thank you.

Viewing 3 replies - 1 through 3 (of 3 total)

The ticket ‘Critical: X-Robots-Tag: noindex Header on sitemap_index.xml Persists Despite Exh’ is closed to new replies.