Hello,
We apologize for the issue you are facing with your robots.txt configuration. Thank you for providing the details and the URL of your site.
The rules in your current robots.txt file are set to disallow certain parameters and paths, which may be specific to your site’s structure and SEO strategy. Here’s a breakdown of what each rule does:
User-agent: * – This applies to all web crawlers.Disallow: /*?orderby= – This prevents crawlers from accessing URLs with the orderby parameter, which is often used for sorting.Disallow: /*/page/ – This blocks access to paginated pages, which can help avoid duplicate content issues.Disallow: /*/trackback/ – This prevents crawlers from accessing trackback URLs, which are not typically useful for SEO.Disallow: /product/���-���/ – This seems to be blocking a specific product URL, possibly due to encoding issues or specific content.Allow: / – This allows access to all other pages on your site.- `Sitemap: https://rankmath.com/wp-content/uploads/2021/09/Enter-URL-in-robots.txt-Tester.jpg – This specifies the location of your sitemap, which is good practice.
If your current settings are working well for your site and you are not experiencing any issues with indexing or visibility in search engines, you may choose to keep them as they are. However, if you want to align with the recommendations from the tutorial, you can consider adjusting your rules accordingly.
For a more tailored approach, we recommend reviewing the tutorial again and considering how those suggestions fit with your site’s goals. If you’re unsure or want to ensure everything is set up correctly, you might also consider our professional setup service at Rank Math Services, which requires a PRO subscription. This service can help ensure your plugin setup adheres to the latest SEO standards.
If you have further questions or need assistance with specific changes, please reply to this ticket, and a human agent will assist you further.
Thank you for reaching out!
Sources:
How to Edit Your Robots.txt With Rank Math SEO » Rank Math
This is an automated reply from Rank Math Support Bot. Your ticket has been assigned to a human agent and we will get back to you shortly.
Hi,
I apologize, but I’m not much of a tech person.
As with your bot reply – how can I tell if the current settings are doing good for my shop or not?
I want to make sure that AI can read and understand my shop as well as traditional, old-school crawlers.
Please advise:
1.
How can I check whether the current setting is preferred for my shop:
User-agent: *
Disallow: /*?orderby=
Disallow: /*/page/
Disallow: /*/trackback/
Disallow: /product/���-���/
Allow: /
Sitemap: https://the-charmer.com/sitemap.xml
2.
Should I implement your tutorial’s suggestion?
If I do, how can I confirm that it is more suitable for my shop?
Thank you,
I appreciate a human reply here
Hello,
Thank you so much for getting in touch.
The robots.txt settings on your site are tailored to your store’s structure and may differ from the example in our tutorial. Unless you are experiencing indexing issues, we recommend keeping the current configuration, as the directives you added help control unnecessary crawling for archives and filters.
However, since you are using Rank Math, we suggest updating the sitemap line to:
Sitemap: https://the-charmer.com/sitemap_index.xml
Here’s how you edit your robots.txt in Rank Math plugin: https://rankmath.com/kb/how-to-edit-robots-txt-with-rank-math/
Hope that helps.
Thank you for your response.
…by the time you responded, I already changed the robots.txt to your tutorial recommendation, tested it, and so far, can’t see any issues.
Thanks for the site map tip, I fixed that.
Is there anything else you recommend I check or set in order to shape my shop even more for search engine / Ai crawlers?
Hello,
We recommend setting up your llms.txt for AI bots to understand what’s on your site and give more accurate answers.
Here’s a link for more information:
https://rankmath.com/kb/llms-txt/
Looking forward to helping you.
