Hello,
Thank you for contacting Rank Math support and sorry for the inconvenience caused to you.
Please check the page source of the URLs in question. Check if there is a “noindex” robots tag in there.
Then, determine if it is a post, a page, a category, or a tag archive.
Once you determine what type of URLs are set to index, check their settings at WordPress Dashboard > Rank Math > Titles & Meta.
https://i.rankmath.com/ApCKiM
Check each tab and ensure the settings are what you want them to be.
Do note that sometimes it is good to have certain links set as noindex.
This is true, especially for the “Misc. Pages”, “Authors”, and “Tags”. Ideally, you don’t want them to be indexable because that might create duplicate content issues and dilute down your website’s authority.
If the pages set as noindex are amongst the pages I mentioned above; you can ignore them or make them indexable using the provided options.
If you see noindex only on individual pages or posts, you can check the Advanced Tab of that post/page and select the “Index” option:
https://i.rankmath.com/MhGm7v
I hope that helps, and please do not hesitate to let us know if you need our assistance with anything else.
I understand you but I am doing this so that adsense bot does not crawl pages like photos documents cart checkout and some others like them. Ideally they should be noindex so I have set them to no index so why are they showing errors in search console or if I set my pages to noindex does google adsense bot still crawl them ?
Hello,
Thank you for the follow-up, and apologies for the delay.
If in case you don’t want Google to crawl any of your documents (.pdf as an example), cart, checkout, etc., then you can just add a Disallow statement in your robots.txt like this:
Disallow: /*.pdf
Disallow: /cart
Disallow: /checkout
Please refer to this tutorial guide for more details on editing the robots.txt file with Rank Math: https://rankmath.com/kb/how-to-edit-robots-txt-with-rank-math/
You can follow the pattern for disallowing those links you don’t want to be crawled by Google.
I hope that helps. Thank you, and please don’t hesitate to contact us anytime if you need further assistance with anything else.
Because somebody is saying about * / user agent or bla bla so Just provide me the complete code to disallow above links and allow all other links
Hello,
Thank you for getting back to us.
If you wish to ensure these pages are not indexed by search engines you need to add this code to your robots.txt file:
User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Please check your robots.txt file within the root folder.
Hope this helps. Let us know if you have any other questions.
Hello,
We are super happy that this resolved your issue. If you have any other questions, know that we are here to help you.
If you don’t mind me asking, could you please leave us a review (if you haven’t already) on https://wordpress.org/support/plugin/seo-by-rank-math/reviews/#new-post about your overall experience with Rank Math? We appreciate your time and patience.
If you do have another question in the future, please feel free to create a new forum topic, and it will be our pleasure to assist you again.
Thank you.
