High Resource Usage Due to Bots Interaction with Product Filters

#974178
  • Hüseyin Kocatepe

    16 hours, 34 minutes ago

    Rank Math free

    Request for Sitemap Optimization

    Dear RankMath Support Team,

    I hope this message finds you well. I am reaching out regarding a significant resource usage issue on our website caused by bot interactions with our product filters. Over the course of a single day, we experienced a massive 14GB of cache buildup and CPU usage that exceeded 100%, leading to the temporary shutdown of our website.

    After investigating the issue, we identified that both YandexBot and BingBot were causing this spike due to their interactions with the product filters. As a temporary solution, we added the following blocks to our **robots.txt** and **.htaccess** files to prevent further issues:

    **robots.txt**:

    `
    User-agent: Yandex
    Disallow: /

    User-agent: YandexBot
    Disallow: /

    User-agent: YandexBot/3.0
    Disallow: /

    User-agent: bingbot/2.0
    Disallow: /

    Sitemap: https://longoni.com.tr/sitemap_index.xml
    `

    **.htaccess**:

    `
    RewriteEngine On
    RewriteCond %{HTTP_USER_AGENT} YandexBot/3.0 [NC]
    RewriteRule .* – [F,L]

    # Block YandexBot user-agent
    RewriteEngine On
    RewriteCond %{HTTP_USER_AGENT} YandexBot [NC]
    RewriteRule .* – [F,L]
    `

    While these changes have helped mitigate the issue, we believe the root cause is related to our sitemap. Given that you are the developers of RankMath, we would appreciate your guidance on optimizing our sitemap to prevent crawler bots from triggering the product filters, thus resolving the issue in a more permanent way.

    Could you kindly advise us on the best approach to optimize our sitemap in this context?

    Thank you in advance for your help. Looking forward to your response.

    Best regards,
    Hüseyin Kocatepe
    longoni.com.tr
    h****************@g******** or h*****************@g********

Viewing 2 replies - 1 through 2 (of 2 total)
  • RankBot

    16 hours, 34 minutes ago

    Hello,

    We apologize for the issue you are facing with high resource usage due to bot interactions with your product filters. Thank you for providing detailed information about the steps you’ve already taken to mitigate the problem.

    To optimize your sitemap and help prevent crawler bots from triggering your product filters, consider the following approaches:

    1. Review Your Sitemap Structure: Ensure that your sitemap only includes essential URLs that you want search engines to crawl. If your product filters generate numerous URLs, it might be beneficial to exclude them from the sitemap.

    2. Use the robots.txt File: You can further refine your robots.txt file to block specific parameters or paths that lead to product filters. For example:

      User-agent: *
Disallow: /path-to-your-product-filters/

      This will prevent all bots from accessing those specific paths.

    3. Implement Canonical Tags: If you have multiple URLs for the same product due to filters, ensure that you are using canonical tags to point to the main product page. This helps search engines understand which version of the page should be indexed.

    4. Limit Crawling Frequency: You can set a crawl delay in your robots.txt file to limit how often bots can access your site. For example:

      User-agent: *
Crawl-delay: 10

      Note that this may not apply to all bots, as Googlebot manages its own crawl rate.

    5. Monitor Bot Traffic: Use tools like Google Search Console to monitor how often bots are crawling your site and which URLs they are accessing. This can help you identify any problematic areas that need further optimization.

    6. Consider Using a CDN: If you haven’t already, implementing a Content Delivery Network (CDN) can help manage traffic spikes and reduce server load during high bot activity.

    For more detailed guidance on managing bot traffic and optimizing your sitemap, you can refer to our articles on bot traffic management and sitemap issues.

    If you continue to experience issues or need further assistance, please reply to this ticket, and a human agent from our support team will take over and provide additional help.

    Thank you for reaching out, and we hope this helps resolve your issue!

    Sources:
    Bot Traffic: How It Affects Your Website and What to Do About It » Rank Math
    Common robots.txt Issues & How to Fix Them » Rank Math

    This is an automated reply from Rank Math Support Bot. Your ticket has been assigned to a human agent and we will get back to you shortly.

    Hüseyin Kocatepe

    16 hours, 1 minute ago

    Rank Math free

    AI provided good answers, but when I look at our sitemap, there are no links related to product filters. I need guidance on how to properly configure the sitemap in the plugin settings.

Viewing 2 replies - 1 through 2 (of 2 total)

You must be logged in to reply to this ticket.