difference between keeping in sitemap and un-indexing custom posts

#350222
  • Resolved Graeme Harris

    3 years, 8 months ago

    Rank Math free

    This is a couple staged question.

    I have a large number of custom location type posts I use on a locations directory page.

    After noticing that Search Console had discovered these posts and their archive pages but had not indexed them; and still after some time had not indexed them, I thought it may make sense to non-index the custom location posts because it is really only their archive pages which I am trying to rank and maybe having hundreds of these location custom posts was cluttering my sitemap.

    I notice under Rank Math>Sitemap Settings> I can set the check box to include in sitemap for both the custom post type, and their taxonomy archives.

    And then under Rank Math>Titles & Meta>Post Types>Locations Custom Post Type, I can set the Robots Meta to NoIndex; which I have now done.

    So now when I go and look at my sitemap, I had 3 XMLs there because I have 580 custom location posts and a sitemap limit of 200 set, now these 3 XMLs are all empty, but still sitting there in the sitemap.

    Could you please comment on the difference between including a post or page in the sitemap and having that post or page set to NoIndex?

    Is this the same thing?

    I expected to keep the custom posts in the sitemap so Google could see the structure; while setting them not to be crawled with the NoIndex because each individual post isn’t intended to rank in Google for anything, rather only the archive page is intended to rank.

    Could any of the other Titles & Meta setting pages be interfering with the indexing of the archive pages for my custom post taxonomy’s even when I have those individual taxonomy archive pages set to Index ? For example the settings for “Post Formats”?

    If anyone has any experience or best practice for large numbers of custom posts used to create directory websites and whether to index vs no index these pages I would be very glad to hear any opinion.

    Thank you so very much!

    G

Viewing 5 replies - 1 through 5 (of 5 total)
  • Anas

    3 years, 8 months ago

    Rank Math business

    Hello,

    Thank you for contacting Rank Math.

    By default Rank Math only adds the post/pages in the sitemap which are set to index.

    If you want to add noindex pages in the sitemap, you need to add this filter in the theme’s functions.php file:

    /**
* Filter to add <code>noindex</code> URLs in the Sitemap
*
* @param bool $value Whether to include noindex terms in Sitemap.
* @param string $type Object Type.
*
* @return boolean
*/
add_filter( 'rank_math/sitemap/include_noindex', function( $value, $type ) {
	return true;
}, 10, 2 );

    Here is a guide you can follow to add the filter:
    https://rankmath.com/kb/wordpress-hooks-actions-filters/

    Then, exclude the sitemaps from caching by following this guide:
    https://rankmath.com/kb/exclude-sitemaps-from-caching/

    No, the Post format or any other Titles & Meta settings will not interfere with the indexing of other post types.

    Crawling and indexing of the website depend on the authority and crawl budget of the website.

    It is completely upto Google if they want to index or not index your website. So, if you haven’t willingly or accidentally made your pages noindex and Google still isn’t indexing your pages – it means that either Google doesn’t want to index or hasn’t crawled your pages yet because it hasn’t allocated that much crawl budget to your website.

    Please check this website for more details.

    Please make sure that you have submitted the sitemap in the Search Console:
    https://rankmath.com/kb/submit-sitemap-to-google/

    Hope that helps and please do not hesitate to let us know if you need our assistance with anything else.

    Graeme Harris

    3 years, 8 months ago

    Rank Math free

    Thank you this helps a lot, so now I can include them in the sitemap but no index if I wish to.

    Are you able to point me toward any content or have any knowledge of other RankMath users who have location directory websites and whether or not they sitemap and index every location’s post page?

    I’m struggling to find any examples or opinions on this topic.

    In some ways it seems to me because I have a small website and probably not much crawl budget as you put it from Google, that I should noindex the hundreds of location post pages.

    On the other hand, all these pages are relevant to their taxonomy archive explorer.

    And although I have a small website like 30 pages, and now adding hundreds of location posts, is that really a big deal to index when you consider other websites have posts and pages numbering in tens of thousands sometimes…

    I know you can’t answer this question directly for me, but any experience or any recommended reading to help me try and make my decision how to proceed would be really appreciated.

    Thanks so much for the awesome support as always

    G

    Jaideep Asrani

    3 years, 8 months ago

    Hello,

    If you don’t want to index the post types then there is no point in including them in the sitemap. Google will be able to see your site structure when it crawls your site. If you keep the posts in the sitemap it will just exhaust your crawl budget as if even they are set to noindex, Google will still have to crawl them to see that.

    And it is completely up to you how you want to display your site in the SERPs. If your archives are targeting a location and your target users search for the location-based keywords then you can keep your archives indexed and posts noindexed.

    Hope this helps. Let us know if you need any other assistance.

    Thanks.

    Graeme Harris

    3 years, 8 months ago

    Rank Math free

    Hey Jaideep,

    yes this does help, so thank you very much.

    Been a lot of learning implementing this directory on my website and like always I really appreciate the support.

    Thanks again!

    Jeremy

    3 years, 8 months ago

    Hello,

    We are super happy that we could address your concern. If you have any other questions in the future, know that we are here to help you.

    If you don’t mind me asking, could you please leave us a review (if you haven’t already) on https://wordpress.org/support/plugin/seo-by-rank-math/reviews/#new-post about your overall experience with Rank Math? We appreciate your time and patience.

    If you do have another question in the future, please feel free to create a new forum topic, and it will be our pleasure to assist you again.

    Thank you.

Viewing 5 replies - 1 through 5 (of 5 total)

You must be logged in to reply to this ticket.