Hello,
We apologize for the issue you are facing with the add-to-cart URLs being indexed despite your Disallow directive in the robots.txt file. This can happen if Google has already indexed those URLs before the directive was added or if there are other factors at play.
Here are some steps you can take to resolve this issue:
-
Verify the robots.txt File: Ensure that your robots.txt file is correctly configured and accessible. You can check it by visiting `https://rankmath.com/kb/cant-edit-robots-txt/.
-
Check for Other Directives: Make sure there are no conflicting directives in your robots.txt file that might be allowing Google to index those URLs.
-
Use the URL Removal Tool: In Google Search Console, you can use the URL removal tool to temporarily hide those URLs from search results. This won’t remove them permanently but can help in the short term.
-
Add Noindex Meta Tag: If you want to ensure that these pages are not indexed, consider adding a noindex meta tag to those pages. This will instruct search engines not to index them, even if they are crawled.
-
Monitor Changes: After making these changes, monitor your Google Search Console for updates on the indexing status of those URLs.
For more detailed guidance on fixing indexing issues, you can refer to our comprehensive guide.
If you continue to experience issues, please reply to this ticket with your site URL so that a human agent can assist you further. Thank you for raising this concern!
Sources:
How to Fix Indexing Issues in Google Search Console » Rank Math
How to Fix the "Indexed, though blocked by robots.txt" Error » Rank Math
Hello,
Thank you for contacting us. We apologize for any inconvenience this issue may have caused.
In addition to the disallow directives, you can add the following filter on the website to set the URLs with the string ?add-to-cart to noindex:
add_filter( 'rank_math/frontend/robots', function( $robots ) {
// Get the current page URL
$current_url = home_url( $_SERVER['REQUEST_URI'] );
// Define the string to check for in the URL
$string_to_check = '?add-to-cart';
// Check if the current URL contains the specific string
if ( strpos( $current_url, $string_to_check ) !== false ) {
// Change the robots tag to noindex
$robots['index'] = 'noindex';
}
// Return the modified robots array
return $robots;
});
Here’s how you can add a filter/hook to your WordPress site: https://rankmath.com/kb/wordpress-hooks-actions-filters/
Once done, allow Google some time to recrawl your site to reflect the changes.
Let us know how it goes. Looking forward to helping you.
Thank you.
Ok I have add the code, I will let you know when I see some change in SCG
Hello,
Sure, do let us know how that goes.
In the meantime, don’t hesitate to get in touch with us if you have any other questions.
Hello,
Since we did not hear back from you for 15 days, we are assuming that you found the solution. We are closing this support ticket.
If you still need assistance or any other help, please feel free to open a new support ticket, and we will be more than happy to assist.
Thank you.
