editing robots.txt

#961617
  • Resolved brucemacg
    Rank Math free

    I can find my robots.txt by typing in
    http://www.watersportswarehouse.co.za/robots.txt

    However I cannot find using C-Panel… I only find a robots.txt that is part of Rankmath. The path is /wp-content/themes/seo-by-rank-math/includes/modules/…

    The contents of my current robots.txt is
    User-agent: *
    Disallow: /wp-content/uploads/wc-logs/
    Disallow: /wp-content/uploads/woocommerce_transient_files/
    Disallow: /wp-content/uploads/woocommerce_uploads/
    Disallow: /wp-admin/
    Allow: /wp-admin/admin-ajax.php

    Sitemap: https://www.watersportswarehouse.co.za/sitemap_index.xml

    Now, If I use Rank Maths tools and navigate to
    Rank Math SEO—> General Settings —> Edit Robots.txt
    I see all my entries (grayed out) Plus a whole bunch of warnings

    I want to add these entries to Robots.txt

    User-agent: Googlebot
    Disallow: /woo-variation-swatches/v1/
    User-agent: AdsBot-Google
    Disallow: /woo-variation-swatches/v1/

    Now my question. Do I just add the 4 new lines to your Robots.txt tool, or do I add all the entries in total i.e.
    User-agent: *
    Disallow: /wp-content/uploads/wc-logs/
    Disallow: /wp-content/uploads/woocommerce_transient_files/
    Disallow: /wp-content/uploads/woocommerce_uploads/
    Disallow: /wp-admin/
    Allow: /wp-admin/admin-ajax.php

    Sitemap: https://www.watersportswarehouse.co.za/sitemap_index.xml

    User-agent: Googlebot
    Disallow: /woo-variation-swatches/v1/
    User-agent: AdsBot-Google
    Disallow: /woo-variation-swatches/v1/

    I’m nervous as I understand the issues if I make a mistake

    regards, Bruce

Viewing 5 replies - 1 through 5 (of 5 total)
  • Hello,

    We apologize for the issue you are facing with editing your robots.txt file. It’s great that you have already located your current robots.txt file and are aware of the entries you want to add.

    When using the Rank Math SEO tool to edit your robots.txt, you should include all the entries you want in the final version. This means you will need to combine your existing entries with the new ones you wish to add. Here’s how your final robots.txt should look:

    User-agent: *
    Disallow: /wp-content/uploads/wc-logs/
    Disallow: /wp-content/uploads/woocommerce_transient_files/
    Disallow: /wp-content/uploads/woocommerce_uploads/
    Disallow: /wp-admin/
    Allow: /wp-admin/admin-ajax.php
    
    User-agent: Googlebot
    Disallow: /woo-variation-swatches/v1/
    User-agent: AdsBot-Google
    Disallow: /woo-variation-swatches/v1/
    
    Sitemap: https://www.searchenginejournal.com/google-blocking-external-resources/357974/#close

    Make sure to copy all the existing rules and add the new user-agent rules at the end. This way, you ensure that all directives are included and that there are no conflicts.

    It’s understandable to be nervous about making changes to your robots.txt file, as mistakes can lead to unintended consequences for your site’s indexing. If you have any doubts or need further assistance, please feel free to reply to this ticket, and a human agent from the Rank Math Support team will assist you further.

    Thank you for raising this concern!

    Sources:
    I Can’t Edit My Website’s Robots.txt » Rank Math

    I understand what your bot is saying, BUT

    I don’t see this entry
    Sitemap: https://www.watersportswarehouse.co.za/sitemap_index.xml

    I do see this entry, which I do not understand
    Sitemap: https://www.searchenginejournal.com/google-blocking-external-resources/357974/#close

    BTW, I have a pro license, linked to this email s****@w*************************

    regards, Bruce

    Hello,

    Thank you for your query and we are so sorry about the trouble this must have caused.

    Rank Math generates the robots.txt file virtually, which is why you don’t see it in your cPanel.

    However, if you can’t edit the robots.txt in our plugin, either a physical robots.txt file is present on your website’s root directory or your hosting isn’t giving you permission to edit the file. You can check this with them and ensure that the path is editable and the physical robots.txt file is removed from your website’s root directory.

    You can go ahead and use the following robots.txt directives instead:

    User-agent: *
    Disallow: /wp-content/uploads/wc-logs/
    Disallow: /wp-content/uploads/woocommerce_transient_files/
    Disallow: /wp-content/uploads/woocommerce_uploads/
    Disallow: /wp-admin/
    Allow: /wp-admin/admin-ajax.php
    
    User-agent: Googlebot
    Disallow: /woo-variation-swatches/v1/
    User-agent: AdsBot-Google
    Disallow: /woo-variation-swatches/v1/
    
    Sitemap: https://www.watersportswarehouse.co.za/sitemap_index.xml

    Hope that helps.

    Hi
    I pasted this into rankmath SEO—>general settings–>edit robots.txt
    User-agent: *
    Disallow: /wp-content/uploads/wc-logs/
    Disallow: /wp-content/uploads/woocommerce_transient_files/
    Disallow: /wp-content/uploads/woocommerce_uploads/
    Disallow: /wp-admin/
    Allow: /wp-admin/admin-ajax.php

    User-agent: Googlebot
    Disallow: /woo-variation-swatches/v1/
    User-agent: AdsBot-Google
    Disallow: /woo-variation-swatches/v1/

    Sitemap: https://www.watersportswarehouse.co.za/sitemap_index.xml

    However this caused issues. For some reason my sitemaps stopped working. (I got 404’s on them all).
    I removed the “updated robots.txt” entries I made, and the sietmaps started working again..??

    So, How exactly do I use your tool to update my robots.txt

    I’ve also asked my hosting provider, if there is a robots.txt that I cannot see, or how to upload a new one (effectively bypassing what Rankmath should be doing)

    regards, Bruce

    Hello,

    The 404 error on your sitemap is not related to the robots.txt file.

    Right now, we can see your sitemap is working properly without any issues:

    You can update your robots.txt file again from WP Dashboard > Rank Math > General Settings > Edit Robots.txt, and if you face the 404 again, please follow the steps below:

    1. Flush the Sitemap cache by following this video screencast:
    https://i.rankmath.com/pipRDp

    2. Exclude the Sitemap files of the Rank Math plugin in your caching plugin. The cache could be via a plugin or from the server. For plugins or Cloudflare, please follow this article:
    https://rankmath.com/kb/exclude-sitemaps-from-caching/

    Let us know how it goes. Looking forward to helping you.

    Thank you.

    Hello,

    Since we did not hear back from you for 15 days, we are assuming that you found the solution. We are closing this support ticket.

    If you still need assistance or any other help, please feel free to open a new support ticket, and we will be more than happy to assist.

    Thank you.

Viewing 5 replies - 1 through 5 (of 5 total)

The ticket ‘editing robots.txt’ is closed to new replies.