how to add code in robot.txt

#95196
  • Resolved Stylus Design

    3 years, 6 months ago

    Rank Math free

    Hello,

    I need to add the following code in robot.txt

    Disallow: /wp-admin/
    Disallow: /*?*wordfence_lh

    If I click the field and add the code in the RankMath Edit robots.txt, do I need to add the following default code before the above?
    User-agent: *
    Disallow: /wp-admin/
    Allow: /wp-admin/admin-ajax.php

    Sitemap: https://example.com/sitemap_index.xml

    After this, do I also need to add the whole code in Google search console’s robots.txt Tester? If I click See live robots.txt, I only see this,
    Disallow: /wp-admin/
    Disallow: /*?*wordfence_lh

    It is confusing and I’m not clear at all, please advise.

Viewing 3 replies - 1 through 3 (of 3 total)
  • Michael Davis

    3 years, 6 months ago

    Rank Math pro

    Hello,

    Thank you for getting in touch with us.

    If I click the field and add the code in the RankMath Edit robots.txt, do I need to add the following default code before the above?

    Yes you do. The robots.txt is generated by Rank math dynamically if there is no static robots.txt file in your root folder. To add custom rules, you will need to read the default Rank math defined rules then add your custom definitions. You can do this in Rank math > General Settings > Robots.txt

    You can check whats defined in your robots.txt by going to https://example.com/robots.txt or testing your site URL with this tool: https://technicalseo.com/tools/robots-txt/

    Hope that helps and please do not hesitate to let us know if you need our assistance with anything else.

    Stylus Design

    3 years, 6 months ago

    Rank Math free

    Thank you for your clarification, Michael. That helps.

    Michael Davis

    3 years, 6 months ago

    Rank Math pro

    Hello,

    We are super happy that this was helpful.

    If you don’t mind me asking, could you please leave us a review (if you haven’t already) on https://wordpress.org/support/plugin/seo-by-rank-math/reviews/#new-post about your overall experience with Rank Math? We appreciate your time and patience.

    If you do have another question in the future, please feel free to create a new forum topic, and it will be our pleasure to assist you again.

    Thank you.

Viewing 3 replies - 1 through 3 (of 3 total)

The ticket ‘how to add code in robot.txt’ is closed to new replies.