Hello,
Thanks for contacting us, and sorry for any inconvenience that might have been caused due to that.
Those URLs are generated dynamically when someone clicks on the Add to Cart button. You can add a similar rule like below to your robots.txt file to prevent Google from crawling those URLs:
Disallow: */?add-to-cart*
If it seems to be confusing on what exact rule you should add to your robots.txt file, please share a sample URL with us here, so that we can generate a rule for you.
Here’s how you can edit robots.txt file using Rank Math:
https://rankmath.com/kb/how-to-edit-robots-txt-with-rank-math/#edit-robotstxt
Looking forward to helping you.
Thank you.
Hello,
I have updated the sensitive data as requested. Can you please check further?
Thank you.
Hello,
Please refer to this robots.txt configuration instead.
Disallow: */?add-to-cart*
Disallow: */?attachment_id*
Disallow: */cart/*
After adding these rules, you should give Google some time to recrawl your website and the error from GSC should go away.
Let us know how that goes.
Hello,
We are super happy that this resolved your issue. If you have any other questions in the future, know that we are here to help you.
If you don’t mind me asking, could you please leave us a review (if you haven’t already) on https://wordpress.org/support/plugin/seo-by-rank-math/reviews/#new-post about your overall experience with Rank Math? We appreciate your time and patience.
If you do have another question in the future, please feel free to create a new forum topic, and it will be our pleasure to assist you again.
Thank you.
Hi, why is rankmath not set by default with this additional solution?
I guess, no one wants the crawler to crawl those pages by default.
Regards
Hello @nikolajworoschilow,
We don’t add those rules by default, since those are not applicable to every user. As the rules can be different for individual users, we prefer to add standard robots.txt rule that is applicable for almost every website.
Hope that clarifies your doubt.
Thank you.
Yes, after further research I’ve understood.
As a user of rank math, while still in the learning process, it would be great to have some kind of hint on the robot.txt settings page with a typical example to copy and paste for a quick fix.
I discovered this issue on my website by today, after 1 month of letting google crawl my site.
That’s not efficient to me. 😛
I’ve used the following code to deal with the crawler.
Hope this helps others to get a quick fix.
User-agent: *
Disallow: */cart/*
Disallow: */checkout/*
Disallow: */my-account/*
Disallow: */search/*
Disallow: *?add-to-cart=*
Disallow: *?remove_item=*
Disallow: *?s=*
Disallow: *?redirect_to=*
Disallow: *?attachment_id*
Disallow: */wp-admin/*
Disallow: */wp-login/*
Allow: /wp-admin/admin-ajax.php
Sitemap: https://***YOUR-SITE***.**COM**/sitemap_index.xml
Regards
Hello @nikolajworoschilow,
You can find a few examples and rules here:
https://rankmath.com/kb/how-to-edit-robots-txt-with-rank-math/#robotstxt-rules
For the rule you’ve added, yes, they should work fine for your site.
Please do not hesitate to let us know if you need our assistance with anything else.
Thank you.
Hello,
Since we did not hear back from you for 15 days, we are assuming that you found the solution. We are closing this support ticket.
If you still need assistance or any other help, please feel free to open a new support ticket, and we will be more than happy to assist.
Thank you.
