Robots.txt Optimization

  • Hi

    many many thanks to you guys and your wonderful plugins
    I want to ask you if it’s possible, do me a favor
    I don’t know how to optimize Robots.txt and beg you to write it for me.
    I find this one from an SEO Expert (maybe!) useful
    https://mimos.ir/robots.txt

    I’ll wait for any further response
    thanks a lot

Viewing 5 replies - 1 through 5 (of 5 total)
  • Hello,

    Thank you for contacting Rank Math today.

    Rank Math will generate a virtual robots.txt file for your site and thus it is not recommended to create a static file since Rank Math updates this file dynamically depending on the robots setting that you configure on the Rank Math > Titles & Meta settings.

    Looking forward to helping you. Thank you.

    ​​​​​​​

    Hi Michael
    thanks for your response
    I know it and it’s wonderful
    but how about this?

    Disallow: /cgi-bin/
    disallow: /wp-includes/
    disallow: /amp/
    disallow: /wp-admin/
    disallow: /feed/
    disallow: /page/
    disallow: *?*
    disallow: */?checkout=*
    disallow: */?s=*
    disallow: */search?q=*
    disallow: */?*
    disallow: *utm_*
    disallow: */?lmxnuvmyq=*
    disallow: */?DA=*
    disallow: */?ypexbygeq=*
    disallow: */?SD=*
    disallow: */?paged=*
    disallow: */?DD=*
    Allow: *.js
    allow: *.css
    allow: /wp-admin/admin-ajax.php

    Can I add them? what is this addresses?

    thanks a lot

    Hi Milad,

    Thanks for the reply.

    Before adding these directives on your robots file it is best to be sure that it does what you want. In the example above, the robots file would block indexing for all links that contain /page and all /amp pages

    If you are sure that this is the desired behavior then you can head over to Rank Math > General Settings > Edit robots.txt file and add the directives.

    I hope this info helps. Thank you.

    Hi Michael
    thanks for your response
    which of the line above is popular?
    I want more optimization for rank math
    I search a lot about it. but I couldn’t find anything.

    thanks a lot

    Hello,

    Thank you for contacting Rank Math today.

    When you Disallow links it implies they wont be accessed or indexed by search engines. I would recommend you leave the robots.txt file as it is and only add Disallow/Allow if you are sure about it.

    I hope this helps you. Thank you.

    ​​​​​​

Viewing 5 replies - 1 through 5 (of 5 total)

You must be logged in to reply to this ticket.