Hello,
We apologize for the issue you are facing with the addition of numerous replacement pages with proper canonical tags on your site. It sounds like you are dealing with a significant number of filter URLs that may be affecting your site’s performance.
To address this, you can indeed use your robots.txt file to block these filter URLs from being indexed by search engines. The exact phrase you should add to your robots.txt file would depend on the specific structure of your URLs. However, a common approach is to disallow any URL that contains the ?filter parameter. You can add the following line to your robots.txt file:
User-agent: *
Disallow: /*?filter=
This directive tells search engines not to index any URL that includes ?filter= in it.
Make sure to check your robots.txt file by going to Rank Math SEO → General Settings → robots.txt in your WordPress dashboard to add this rule.
Additionally, if you want to ensure that these filter pages do not affect your site’s performance, consider implementing 301 redirects for any URLs you do not want indexed or using canonical tags appropriately to point to the main versions of your pages.
If you need further assistance or if this does not resolve your issue, please reply to this ticket, and a human agent from the Rank Math Support team will assist you further.
Thank you for raising this concern!
Sources:
Common robots.txt Issues & How to Fix Them » Rank Math
Hi,
Thanks for the answer.
I tried to write a rule as you mentioned in the robots.txt file.
After some time, pages did start to be blocked by robots.txt, but still a lot of pages with the filter? prefix continue to appear.
This is a site with less than 1000 properly indexed pages, and currently there are 46,000 pages added with filter? And all the time more and more are added.
In the map settings in rankmath Include archives for terms of Taxonomies in the XML sitemap are now turned off.
I would appreciate guidance.
Thank you very much!
Hello,
If you blocked them through robots.txt, those URLs will end up in this error:
https://rankmath.com/kb/fix-submitted-url-blocked-by-robots-txt-error/
In this case, you may use and customize this code to set those filtered URLs’ to noindex and remove the self-canonical:
add_filter( 'rank_math/frontend/robots', function( $robots ) {
$url = home_url( $_SERVER['REQUEST_URI'] );
if( (strpos($url,'?filter') !== false) ) {
$robots["index"] = 'noindex';
$robots["follow"] = 'nofollow';
}
return $robots;
});
And here’s a guide to adding the code to the website:
https://rankmath.com/kb/wordpress-hooks-actions-filters/
Let us know how this goes.
Hello,
Since we did not hear back from you for 15 days, we are assuming that you found the solution. We are closing this support ticket.
If you still need assistance or any other help, please feel free to open a new support ticket, and we will be more than happy to assist.
Thank you.