website is heavily crawled by web bots - Support

Resolved Zimo
3 years, 1 month ago

Rank Math free

Hi have received this info from hosting service due to high cpu usage. is there anything that can be done in Rankmath?

174065 “Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)”
58331 “Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)”
5559 “Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/)”
4825 “Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)”
To resolve this issue I would highly recommend checking and optimizing the SEO of your website.

Viewing 7 replies - 1 through 7 (of 7 total)

Jaideep Asrani
3 years, 1 month ago
Hello,

Thanks for contacting us and sorry for any inconvenience that might have been caused due to that.

The Yandex bot is crawling your site to index it on the Yandex search engine. So you shouldn’t block the Yandex bot if you want to index your site in Yandex. For the other bot, you can add the rule given below in your robots.txt file from WP Dashboard > Rank Math > General Settings > Edit Robots.txt to block them:
```
User-agent: SemrushBot
Disallow: /
User-agent: AhrefsBot
Disallow: / 
User-agent: BLEXBot
Disallow: /
```
Hope this helps. Let us know if you need any other assistance.
Zimo
3 years, 1 month ago

Rank Math free

after adding the suggested code. These lines started showing up on backend and some product pages on front end:
see sensitive data

Zimo
3 years, 1 month ago

Rank Math free

Hello,

I have updated the sensitive data as requested. Can you please check further?

Thank you.

Zimo
3 years, 1 month ago

Rank Math free

Hi, this is what the settings look like now (see link). and there is no robot.txt file in wp file manager. is there any basic rule setting i am missing?

Prabhat
3 years, 1 month ago

Rank Math agency

Hello,

We’re really sorry for the unfortunate delay. For some reason, the ticket was stuck in drafts.

Assuming you are using the latest version of all the plugins and the themes (please update if not yet), it would seem like a conflict with one of the plugins or the theme you are using.

To determine that, please follow this troubleshooting guide:
https://rankmath.com/kb/check-plugin-conflicts/

Here is a video you can follow as well: https://www.youtube.com/watch?v=FcIWRhcUP2c

Only the site admin would know and your users won’t be affected by this troubleshooting.

If the issue persists, please edit the first post on this ticket and include your WordPress & FTP logins in the designated Sensitive Data section.

Please do take a complete backup of your website before sharing the information with us.

It is completely secure and only our support staff has access to that section. If you want, you can use the below plugin to generate a temporary login URL to your website and share that with us instead:

https://wordpress.org/plugins/temporary-login-without-password/

You can use the above plugin in conjunction with the WP Security Audit Log to monitor what changes our staff might make on your website (if any):

https://wordpress.org/plugins/wp-security-audit-log/

Please do take a complete backup of your website before sharing the information with us.

We really look forward to helping you.

Zimo
3 years ago

Rank Math free

Hello,

I have updated the sensitive data as requested. Can you please check further?

Thank you.

Jaideep Asrani
3 years ago

Hello,

You have only shared the FTP details. Can you please share the login details of your WordPress admin dashboard?

We really look forward to helping you.

Thanks.

RankBot
2 years, 11 months ago

Hello,

Since we did not hear back from you for 15 days, we are assuming that you found the solution. We are closing this support ticket.

If you still need assistance or any other help, please feel free to open a new support ticket, and we will be more than happy to assist.

Thank you.

Viewing 7 replies - 1 through 7 (of 7 total)

The ticket ‘website is heavily crawled by web bots’ is closed to new replies.