Hello,
Thank you for contacting Rank Math and bringing your concern to our attention.
If you have already fixed the disallow issue in your robots.txt, you may need to resubmit your site for indexing.
Also When I originally indexed the site, I included all the articles, pages, and categories. Is that correct to include them all?
Since you have originally set them as index, I would recommend that they remain indexed. If you set them to noindex, you might experience a sudden drop of traffic on your site if they are ranking well on Google search results pages.
I hope that helps.
Thank you, and please don’t hesitate to contact us anytime if you need further assistance with anything else.
Thank Reinelle, appreciate that.
Hello,
You’re welcome.
We are glad that we have addressed your concern. Please don’t hesitate to contact us anytime to assist you further with anything else.
Looking forward to helping you.
Thank you.
Thanks for that, however I’m getting the following error on GSC:
Indexed, though blocked by robots.txt
I had some issues yesterday with Ezoic and my DNS settings, my site went down and displayed an older cached version. Could that have caused the above error, or is it something else?
When I inspect URLs in GSC it says they are indexed and OK.
Thanks
Matt
Hello,
That might be the reason, but Could you please check if the URLs are already indexed? Or some of them left behind?
Since you have also mentioned “GSC says they are indexed and OK.” I would suggest you to monitor the GSC for any possible errors.
Hope that helps and please do not hesitate to let us know if you need our assistance with anything else.
Thanks for the reply.
They seem to be indexed on GSC.
I’ve just analyzed on Rankmath and it’s saying: Public Blog
Attention: Search Engines can’t see your website. You must go to your Reading Settings and uncheck the box for Search Engine Visibility.
However, this check box is NOT ticked.
Could this be the issue? Any ideas why it’s saying that?
Cheers
In the No Index meta section, it says: Your page contains noindex meta or header.
Remove the noindex robots meta tag. Only ever use it on pages you want to keep out of the search engines.
But in GSC:
Crawl
Last crawl
5 Jul 2021, 21:42:32
Crawled as
Googlebot smartphone
Crawl allowed?
Yes
Page fetch
Successful
Indexing allowed?
Yes
Hello,
Could you please check the affected URL/s using our External SEO Analyzer and see if the issue is already fixed there?
If so, please head over to Rank Math > Status & Tools > Database Tools > and click the Clear SEO Analysis and Remove Transients buttons to flush the SEO Analysis cache
Please note that some options are only available when you enable the Advanced Mode in Rank Math. If it’s not showing, you must have chosen to use the Easy Mode. Please follow the steps in this link to switch to Advanced Mode:
https://rankmath.com/kb/advanced-mode/
Once done, clear your website’s cache and Start again the sitewide analysis in Rank Math > SEO Analysis.
If the issue still persists, please also share the affected URL to check further.
I hope that helps.
Thank you, and looking forward to your update.
Hi guys,
I’m still getting “Indexed, though blocked by robots.txt” on 4 posts, the others seem ok now.
When I inspect the URL I get the following:
Last crawl
4 Jul 2021, 16:44:29
Crawled as
Googlebot smartphone
Crawl allowed?
warning
No: blocked by robots.txt
Page fetch
warning
Failed: Blocked by robots.txt
Indexing allowed?
Yes
Indexing
User-declared canonical
None
Google-selected canonical
Inspected URL
In the Rankmath Analyzer it’s showing green ticks and says:
Noindex Meta ?Does your content contain a noindex robots meta tag?
Your page contains no noindex meta or header.
WWW Canonicalization ?Does your site appear on more than one URL?
Both www and non-www versions of your URL are redirected to the same site.
OpenGraph Meta ?Does your site use OpenGraph meta tags?
Opengraph meta tags have been found.
Robots.txt ?Does your site have a valid robots.txt file
Your site has a robots.txt file which includes one or more Disallow: directives. Make sure that you only block parts you don’t want to be indexed.
Schema Meta Data ?Does your content include relevant Schema.org metadata?
Schema.org data has been found on your page.
Are they waiting to be crawled by Google, or is it something else?
Thanks
Matt
Hello,
As the posts were last crawled by Google on July 4, Google has not seen the changes for these URLs.
You can wait for Google to re-crawl these URLs.
You can ignore the Indexed, though blocked by robots.txt warning, as the posts are already indexed.
Also, make sure your robots.txt file looks similar to the below-mentioned one.
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://yourdomain.com/sitemap_index.xml
We really look forward to helping you.
Hello,
We are super happy that we could address your concern. If you have any other questions in the future, know that we are here to help you.
If you don’t mind us asking, could you please leave us a review (if you haven’t already) on https://wordpress.org/support/plugin/seo-by-rank-math/reviews/#new-post about your overall experience with Rank Math? We appreciate your time and patience.
If you have another question in the future, please feel free to create a new forum topic, and it will be our pleasure to assist you again.
Thank you.
