Robots.txt editing with dissalow CDN

#959914
Viewing 3 replies - 1 through 3 (of 3 total)
  • RankBot

    3 weeks, 3 days ago

    Hello,

    We apologize for the issue you are facing with editing your robots.txt file. To address the problem with your CDN and ensure that Google can crawl your site properly, you can add a specific rule to your robots.txt file. Here’s how you can do it:

    Steps to Edit Your Robots.txt File:

    1. Access Your WordPress Dashboard: Log in to your WordPress admin area.
    2. Navigate to Rank Math Settings: Go to Rank Math > General Settings > Edit Robots.txt.
    3. Add the Disallow Rule: In the editor, you can add the following line: 
      Disallow: /cdn-cgi/

      This rule will prevent crawlers from accessing the CDN-related paths.

    4. Save Changes: After adding the rule, make sure to save your changes.
    5. Test Your Robots.txt File: You can use the Robots Testing Tool to check if the changes are effective. Enter your website URL and select the appropriate user-agent (you can use "All (robots.txt)"). Click on TEST to see if Google can access your site properly.

    Important Notes:

    • Since you mentioned using QUIC.cloud CDN, the above rule should help in allowing Google to crawl your site without issues related to CDN paths.
    • If you encounter any difficulties while editing the robots.txt file or if you have further questions, please feel free to reply to this ticket, and a human agent from our support team will assist you further.

    Thank you for raising this concern, and we hope this helps you resolve the issue quickly!

    Sources:
    Common robots.txt Issues & How to Fix Them » Rank Math

    This is an automated reply from Rank Math Support Bot. Your ticket has been assigned to a human agent and we will get back to you shortly.

    World

    3 weeks, 1 day ago

    Rank Math free

    Thanks for your answer, it’s pretty clear except one thing. Please can you tell me exactly which should be the entire robots.txt including the CDN disallow rule? I don’t know on which line to put the code. THanks a lot!

    Jeremy

    3 weeks, 1 day ago

    Hello,

    Thank you so much for getting in touch.

    We can see that your robots.txt is loading just fine: https://agoodframe.com/robots.txt

    As mentioned in the guide, you can choose to ignore the error or add the mentioned disallow rule. Here’s how you edit your robots.txt in Rank Math plugin: https://rankmath.com/kb/how-to-edit-robots-txt-with-rank-math/

    Don’t hesitate to get in touch with us if you have any other questions.

Viewing 3 replies - 1 through 3 (of 3 total)

You must be logged in to reply to this ticket.