robots.txt not working

#855
  • Resolved Marc
    Rank Math free

    I want to change the content of my virtual robots.txt. I have used the tool inside Rank Math located in “General Settings” -> “Edit robots.txt”. But the file is not changing and I have the default content.

    I have no robots.txt file on the root of the www folder of my site. I have checked that.

    I used to do this functionality with an other plugin (virtual robots.txt) but I want to migrate that functionality to Rank Math. I have deactivated and unistalled the plugin in order to not mismatch with Rank Math.

    Please, help me.

    • This topic was modified 5 years, 9 months ago by Marc.
Viewing 15 replies - 1 through 15 (of 18 total)
  • Hello,

    Thank you for contacting Rank Math today.

    Please ensure that there are no server restrictions on that file by contacting your hosting provider first and asking them to check that.

    Also please deactivate this plugin: /pc-robotstxt/

    If that doesn’t help, please share your WP admin access in sensitive data field along with your FTP.

    Looking forward to help you.

    Marc
    Rank Math free

    The older plugin works fine so, there is no reason my provider is blocking that functionality.

    I have look at the plugins folder of my wordpress. There is no “/pc-robotstxt/” folder. That’s make sense because I have already unistalled it.

    I have changed sensitive data form with access to my wordpress folder with FTP and wordpress access.

    Hello,

    Can you please add the sensitive data again as it seems like it is not stored?

    Looking forward to helping you. Thank you.

    P.S. Please make sure the access of the FTP is of the root folder as that’s where the Robots.txt files are placed.

    Marc
    Rank Math free

    It is not possible to save sensitive data. I edit sensitive data form, I put content. I submit. No error. But on the conversation the sensitive data is not showing.

    When you resolve the issue on your web reply me to put the data.

    If you want, I can send it by email.

    Marc
    Rank Math free

    Finally I could edit sensitive data. I have logged out and in to my user and I have filled the optional field of the reason of the edit in the form.

    Please help me.

    Hello,

    I don’t see robots.txt file on your server, to me it looks like your hosting provider is restricting the access to that file and its not allowing to change it.

    Can you please contact your hosting provider and confirm if there are no restrictions set?

    Looking forward to helping you. Thank you.

    Marc
    Rank Math free

    I have open a ticket on my hosting. They told me there is not a server restriction to robots.txt file. So, if the plugin generates that file, is a plugin problem.

    As you can see with the plugin “virtual robots.txt” you have activated on my site it is working fine. I don’t know if you user wordpress virtual robots or creates a robots.txt on the server. But my server provider says there is no limitation on server side.

    Hello,

    While trying to connect to the FTP, we are getting the following error message:
    ftp error, 200

    Can you please confirm the details again?

    Looking forward to helping you. Thank you.

    Marc
    Rank Math free

    I have written down ftp server I give up last time.

    I have tested ftp connection detail with filezilla and all is fine. I suppose the error was the ftp server.

    Hello,

    Upon checking further, we have identified the issue.

    We will be releasing an update in this week which shall fix the issue.

    We seek your patience and understanding in this matter.

    Thank you.

    Hello rankmath, I have same problem with Marc.

    I already changed my robots.txt in rankmath plugin but it doesn’t effect the actual robots.txt on my website : https://sahabataqiqah.co.id/robots.txt

    It makes my sites can’t be indexed by Google.

    Really appreciate your help!

    Regards,
    Romadhon Hapsa
    Sahabat Aqiqah Web Master

    Attachment:

    Screenshot 1 (Robots.txt setting in rank math plugin)
    https://snag.gy/ebnQAN.jpg

    Screeshot 2 (Robots.txt setting in actual url)
    https://snag.gy/vCjU5o.jpg

    Screenshot 3 (Site can’t be indexed blocked by robots.txt)
    https://snag.gy/4eUpAf.jpg

    Todd
    Rank Math free

    Hello,

    Thank you for getting in touch.

    This is happening because you have made your entire website noindex.

    You can fix that from WordPress Dashboard > Settings > Reading.

    After you do that, this issue will be fixed.

    Hope that helps and please do not hesitate to let us know if you need our assistance with anything else.

    Thank you for your reply Toddy,

    I’ve done what you say which is changing the settings in reading.

    The result is:
    – Actual Robots.txt setting changed, which is same with in the plugin
    https://snag.gy/6lIEcP.jpg

    -BUT, I still can’t get my website indexed on Google, they says it still blocked by robots.txt
    https://snag.gy/ePl2n1.jpg

    What should I do then?

    Todd
    Rank Math free

    Hello,

    Please share a screenshot of your robots.txt settings in Rank Math.

    Hi Todd,

    Here is the screenshot: https://snag.gy/beEMO8.jpg

    Thank you

Viewing 15 replies - 1 through 15 (of 18 total)

The ticket ‘robots.txt not working’ is closed to new replies.