Hello,
Thank you for contacting the support, and sorry for any inconvenience that might have been caused due to that.
I visited your website and I was able to replicate the issue. The robots.txt gives a 404 (Not Found) error on your website.
Can you please try to clear your website cache and see if this fixes the robots.txt file?
If the issue persists, we might need to take a closer look at the settings. Please edit the first post on this ticket and include your WordPress & FTP logins in the designated Sensitive Data section.
Please do take a complete backup of your website before sharing the information with us.
It is completely secure and only our support staff has access to that section. If you want, you can use the below plugin to generate a temporary login URL to your website and share that with us instead:
https://wordpress.org/plugins/temporary-login-without-password/
You can use the above plugin in conjunction with the WP Security Audit Log to monitor what changes our staff might make on your website (if any):
https://wordpress.org/plugins/wp-security-audit-log/
We really look forward to helping you.
Hello,
I have updated the sensitive data as requested. Can you please check further?
Thank you.
Thanks for the swift response Jeremy.
I have purged my NitroCache but i think this issue is not related.
I have provided a temporary admin access per your recommendation.
FTP is a little more tricker as it requires certificate to access and it will be hard for me to have to go any change everything afterwards.
If you are familiar with driving linux shell I will activate the plugin called wpterm and you can run commands to check the system.
e.g ls -l /opt/bitnami/wordpress/
https://store.mrlb.com.au/wp-admin/tools.php?page=wpterm
If I could understand how robots.txt is supposed to be generated I could have a look at the file and owner permissions and see if maybe it is a permission issue.
Hello,
Thank you so much for providing temporary login access.
I tried conducting some troubleshooting on the backend and the issue persists.
Since this is a 404 issue, this could also be related to your NGINX configuration. Please check this one with your hosting provider if there’s a missing rule in your NGINX configuration file and while you’re at it, you can also check if this issue is about file permission.
Let us know how that goes. Looking forward to helping you.
Hi Jeremy,
Thanks for getting back to me.
I am the hosting provider so I can fix any permission or NGINX issue. However I really need to understand how your plugin and robots.txt work together.
I read in RankMaths that robotos.txt is “virtual” but there were no more details than that. The word virtual is very vague. To me it implies there will be no actual robots.txt on the filesystem but rather some virtual one that is perhaps generated by the plugin when the file is requested.
Could you please just simply advise if RankMaths plugin tries to create a robots.txt on the file system? And is it trying to create it in the WordPress root folder?
Cheers, Matt
Hi Jeremy,
I have created a robots.txt in wordpress root folder and RankMaths can see it however it’s locked (since I created it)
I can browse to it via https://store.mrlb.com.au/robots.txt so i don’t believe there would be a NGINX issue.
If I can just understand what rank maths needs to create a virtual robots.txt then i think i can resolve this.
THanks, Matt
Hello,
Allow me to further check this one with our developers. We should get back to you shortly.
We appreciate your time and patience on this one.
Thanks Jeremy. I’ll stand by.
Hello,
Thank you so much! We will keep this ticket open for you.
In the meantime, please do not hesitate to let us know if you need our assistance with anything else.
Hello,
We tried to make some changes but it seems that the website is being aggressively cached and our changes don’t take any effect immediately.
Could you please make sure that you disable the caching temporarily while we debug the issue and also please make sure that your server has at least the following variables:
PHP Memory Limit: 512M
Max input time: 150
PHP time limit: 120
PHP max input variables: 1000
Looking forward to hearing back from you.
Hi Miguel
Thanks for looking into this.
I made the PHP suggestions you requested and also turned off caching.
Cheers, Matt
Hello,
I tried adding a physical robots.txt file in your root folder via your website’s backend and it seems the robots.txt still throwing 404.
We may also need to ask for your FTP so we can further troubleshoot the issue.
Kindly include your FTP login in the Sensitive Data section.
Looking forward to helping you.
Hi Jeremy
You must have deleted the robots.txt because I couldn’t find it. I tested this previous and after correct chown/chmod it becomes visible without a 404.
I will have a look and see if there is a way i can provide SFTP access.
Cheers, Matt
Hello,
Sure, please let us know if you are able to provide an FTP so we can further troubleshoot this deeper.
Looking forward to helping you.
Hi Jeremy,
I had to specifically setup an FTP server over night. Whilst this is done, and the user I have created is different to that of the instance wordpress runs under. The user account can still list and create files. I have tested it this morning.
I will update the sensitive area now.
Thanks, Matt
