Hello,
Thank you for contacting Rank Math.
Can you please share the URL with us so we can check?
We look forward to helping you.
https://www.lifeca.com/
My robots.txt is following；
User-agent: *
Disallow: /wp-admin/
Disallow: /ueditor/net/upload/image/
Disallow: /rss/
Disallow:/CaseDetail.aspx?id=*
Disallow:/123_/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://www.lifeca.com/sitemap_index.xml
The log of spiders: (example)
2022-09-15 14:24:17 404 /CaseDetail.aspx?id=461 101.67.29.180
2022-09-15 14:05:45 404 /CaseDetail.aspx?id=294 39.173.107.41
Regards
Xia Lin
Hello,
I checked your website in a robots.txt validator tool and the affected page shouldn’t be crawled anymore as you can see in my screenshot below:
Can you please try clearing your website cache including any server-level caching services and observe if the bots still keep on crawling that specific URL?
Looking forward to helping you.
I have set the robots.txt as:
User-Agent: *
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-admin/
Sitemap: https://www.lifeca.com/sitemap_index.xml
But when I visited https://www.lifeca.com/robots.txt, it still was
User-agent: *
Disallow: /wp-admin/
Disallow: /ueditor/net/upload/image/
Disallow: /rss/
Disallow: /CaseDetail.aspx?id=*
Disallow: /YiMinNewsDetail.aspx?id=*
Disallow: /yiminnewsdetail.aspx?id=*
Disallow: /123_/
Disallow: /Image_*
Allow: /wp-admin/admin-ajax.php
User-agent: YisouSpider
Disallow: /
User-agent: DotBot
Disallow: /
Sitemap: https://www.lifeca.com/sitemap_index.xml
Why is there no real-time update?
Thanks
ok. I went back the Version of 1.0.96. It works now.
Hello,
This usually happens if the robots.txt file is cached.
Can you please clear the cache from your CDN and your website cache and check again?
Let us know how it goes.
We are here to assist you.
Hello,
We are super happy that your issue is resolved.
If you have any questions in the future, please feel free to create a new forum topic, and it will be our pleasure to assist you again.
Thank you.
