Hello,
We apologize for the issue you are facing with the URLs showing as "Crawled – currently not indexed" in Google Search Console. This can often happen with URLs that have parameters, such as those generated by WooCommerce for product attributes.
To address this issue, here are some recommendations:
1. Use Noindex Meta Tag
You can add a noindex meta tag to these URLs to prevent them from being indexed by search engines. This way, the content will still be accessible to users, but search engines will not consider it for indexing. You can do this by configuring the settings in Rank Math:
- Go to the Rank Math dashboard.
- Navigate to Titles & Meta settings.
- Under the Products tab, you can set the noindex option for specific product attribute URLs.
2. Canonical Tags
If you have a main product page that you want to be indexed, ensure that the canonical tag on the attribute pages points to the main product page. This tells search engines which version of the page is the preferred one to index.
3. Exclude Parameters in Google Search Console
You can also exclude specific URL parameters in Google Search Console:
- Go to your Google Search Console account.
- Navigate to Legacy tools and reports > URL Parameters.
- Here, you can specify parameters like
attribute_size and attribute_color to tell Google how to handle them.
4. Improve Internal Linking
Ensure that your important pages are well-linked internally. This helps search engines understand which pages are more important and should be indexed.
5. Content Quality Check
Make sure that the pages with these attributes provide valuable content. If they are thin on content, consider consolidating them or improving their quality.
For more detailed guidance, you can refer to our article on fixing indexing issues: How to Fix "Crawled – Currently Not Indexed" Error in Google Search Console.
Thank you for raising this concern! If you have further questions or need assistance with specific settings in Rank Math, please reply to this ticket, and a human agent will assist you further.
Hello,
In this case, you may use this code to set those filtered URLs’ to noindex and remove the self-canonical so search engines won’t crawl them:
add_filter( 'rank_math/frontend/robots', function( $robots ) {
$url = home_url( $_SERVER['REQUEST_URI'] );
if( (strpos($url,'?attribute_') !== false) ) {
$robots["index"] = 'noindex';
$robots["follow"] = 'nofollow';
}
return $robots;
});
And here’s a guide to adding the code to the website:
https://rankmath.com/kb/wordpress-hooks-actions-filters/
Once done, give Google some time to recrawl your site.
Looking forward to helping you.
Thanks for your help.
If I follow this logic and I want to apply the same thing to my URLs with /feed, would this script be the right one?
add_filter( ‘rank_math/frontend/robots’, function( $robots ) {
$url = home_url( $_SERVER[‘REQUEST_URI’] );
if ( strpos( $url, ‘/feed’ ) !== false ) {
$robots[‘index’] = ‘noindex’;
$robots[‘follow’] = ‘nofollow’;
}
return $robots;
});
Hello,
Yes, the filter you have shared will set the feed URLs to noindex.
Please do not hesitate to let us know if you need our assistance with anything else.
Hello,
Since we did not hear back from you for 15 days, we are assuming that you found the solution. We are closing this support ticket.
If you still need assistance or any other help, please feel free to open a new support ticket, and we will be more than happy to assist.
Thank you.
