High Resource Usage Due to Bots Interaction with Product Filters

Hello,

We apologize for the issue you are facing with high resource usage due to bot interactions with your product filters. Thank you for providing detailed information about the steps you’ve already taken to mitigate the problem.

To optimize your sitemap and help prevent crawler bots from triggering your product filters, consider the following approaches:

1. Review Your Sitemap Structure: Ensure that your sitemap only includes essential URLs that you want search engines to crawl. If your product filters generate numerous URLs, it might be beneficial to exclude them from the sitemap.

2. Use the robots.txt File: You can further refine your robots.txt file to block specific parameters or paths that lead to product filters. For example:

   User-agent: *
   Disallow: /path-to-your-product-filters/

   This will prevent all bots from accessing those specific paths.

3. Implement Canonical Tags: If you have multiple URLs for the same product due to filters, ensure that you are using canonical tags to point to the main product page. This helps search engines understand which version of the page should be indexed.

4. Limit Crawling Frequency: You can set a crawl delay in your robots.txt file to limit how often bots can access your site. For example:

   User-agent: *
   Crawl-delay: 10

   Note that this may not apply to all bots, as Googlebot manages its own crawl rate.

5. Monitor Bot Traffic: Use tools like Google Search Console to monitor how often bots are crawling your site and which URLs they are accessing. This can help you identify any problematic areas that need further optimization.

6. Consider Using a CDN: If you haven’t already, implementing a Content Delivery Network (CDN) can help manage traffic spikes and reduce server load during high bot activity.

For more detailed guidance on managing bot traffic and optimizing your sitemap, you can refer to our articles on bot traffic management and sitemap issues.

If you continue to experience issues or need further assistance, please reply to this ticket, and a human agent from our support team will take over and provide additional help.

Thank you for reaching out, and we hope this helps resolve your issue!

Sources:
Bot Traffic: How It Affects Your Website and What to Do About It » Rank Math
Common robots.txt Issues & How to Fix Them » Rank Math

Hello,

Thank you so much for getting in touch.

Rank Math’s sitemap does not include dynamic URLs like product filter pages, so the issue likely stems from internal links rather than the sitemap itself. Bots probably discovered these URLs through crawlable links on your site. To prevent indexing, we recommend setting those links to nofollow.

Additionally, your robots.txt blocks the entire site, not just filter URLs so please review this to ensure it aligns with your goals. Here’s a small modification to the robots.txt directives we can suggest:

User-agent: Yandex
Disallow: /renk/turuncu/?filter_renk=*

User-agent: YandexBot
Disallow: /renk/turuncu/?filter_renk=*

User-agent: YandexBot/3.0
Disallow: /renk/turuncu/?filter_renk=*

User-agent: bingbot/2.0
Disallow: /renk/turuncu/?filter_renk=*

Lastly, you can also use Bing’s URL Inspection Tool to identify how bots are discovering these pages.

Don’t hesitate to get in touch with us if you have any other questions.

Hello,

The filtered URL is already set to noindex in the page’s source code, which should inform bots they shouldn’t be crawled.

However, it seems that those bots are still crawling them; that’s why you may need to add the disallow rules for that bot to stop crawling them.

Looking forward to helping you.

Hello,

We are super happy that we could address your concern. If you have any other questions in the future, know that we are here to help you.

If you don’t mind me asking, could you please leave us a review (if you haven’t already) on https://wordpress.org/support/plugin/seo-by-rank-math/reviews/#new-post about your overall experience with Rank Math? We appreciate your time and patience.

If you do have another question in the future, please feel free to create a new forum topic, and it will be our pleasure to assist you again.

Thank you.