editing robots.txt

Hello,

We apologize for the issue you are facing with editing your robots.txt file. It’s great that you have already located your current robots.txt file and are aware of the entries you want to add.

When using the Rank Math SEO tool to edit your robots.txt, you should include all the entries you want in the final version. This means you will need to combine your existing entries with the new ones you wish to add. Here’s how your final robots.txt should look:

User-agent: *
Disallow: /wp-content/uploads/wc-logs/
Disallow: /wp-content/uploads/woocommerce_transient_files/
Disallow: /wp-content/uploads/woocommerce_uploads/
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

User-agent: Googlebot
Disallow: /woo-variation-swatches/v1/
User-agent: AdsBot-Google
Disallow: /woo-variation-swatches/v1/

Sitemap: https://www.searchenginejournal.com/google-blocking-external-resources/357974/#close

Make sure to copy all the existing rules and add the new user-agent rules at the end. This way, you ensure that all directives are included and that there are no conflicts.

It’s understandable to be nervous about making changes to your robots.txt file, as mistakes can lead to unintended consequences for your site’s indexing. If you have any doubts or need further assistance, please feel free to reply to this ticket, and a human agent from the Rank Math Support team will assist you further.

Thank you for raising this concern!

Sources:
I Can’t Edit My Website’s Robots.txt » Rank Math

Hello,

Thank you for your query and we are so sorry about the trouble this must have caused.

Rank Math generates the robots.txt file virtually, which is why you don’t see it in your cPanel.

However, if you can’t edit the robots.txt in our plugin, either a physical robots.txt file is present on your website’s root directory or your hosting isn’t giving you permission to edit the file. You can check this with them and ensure that the path is editable and the physical robots.txt file is removed from your website’s root directory.

You can go ahead and use the following robots.txt directives instead:

User-agent: *
Disallow: /wp-content/uploads/wc-logs/
Disallow: /wp-content/uploads/woocommerce_transient_files/
Disallow: /wp-content/uploads/woocommerce_uploads/
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

User-agent: Googlebot
Disallow: /woo-variation-swatches/v1/
User-agent: AdsBot-Google
Disallow: /woo-variation-swatches/v1/

Sitemap: https://www.watersportswarehouse.co.za/sitemap_index.xml

Hope that helps.

Hello,

The 404 error on your sitemap is not related to the robots.txt file.

Right now, we can see your sitemap is working properly without any issues:

You can update your robots.txt file again from WP Dashboard > Rank Math > General Settings > Edit Robots.txt, and if you face the 404 again, please follow the steps below:

1. Flush the Sitemap cache by following this video screencast:
https://i.rankmath.com/pipRDp

2. Exclude the Sitemap files of the Rank Math plugin in your caching plugin. The cache could be via a plugin or from the server. For plugins or Cloudflare, please follow this article:
https://rankmath.com/kb/exclude-sitemaps-from-caching/

Let us know how it goes. Looking forward to helping you.

Thank you.