Hello,
Thank you for contacting Rank Math, and sorry for any inconvenience caused.
Can you please share the affected URL with us so we can check?
You can add the URL in the sensitive data section.
We look forward to helping you.
Hello,
I have updated the sensitive data as requested. Can you please check further?
Thank you.
Hello,
There are some Disallow rules in your robots.txt file that are blocking the crawlers from crawling some pages. If you want them to crawl the whole site, please navigate to WP Dashboard > Rank Math > General Settings > Edit Robots.txt and remove the rules for the blocked pages:
Let us know how that goes. Looking forward to helping you.
Thank you, Jaideep
Do I remove the whole line “Disallow: /*?”
What does it mean and how do you think this came into my robots.txt
Flywheel, which is my host, told me all is well. But they have been getting bad lately.
So I really appreciate your knowledgable answer.
I will switch to a different host in the coming weeks (this is not the only issue with them)
Hello,
If you haven’t added it manually then the rules must be the default ones of the plugin that you are using to create the robots.txt file. If you remove the Disallow rule for the *? all the URLs with query parameters will be available for Google to crawl. It is up to you if you want to index those pages or not.
Hope this helps. Let us know if you need any other assistance.
Thanks, Jaideep.
I only use Rankmath and I have never paid any attention to robots.txt files until now.
Could it be that this has been done by the hosting company?
I remember, years back, when there was another mess with some code on my website that their server software put there and that also harmed my sites.
Hello,
Yes, it could be from your hosting provider. You can edit the robots.txt file from WP Dashboard > Rank Math > General Settings > Edit Robots.txt. If the file is not editable then it means that there is a physical file in your site’s root directory. You can edit the file directly from there or remove it so our plugin can generate its default robots.txt file.
Hope this helps. Let us know if you need any other assistance.
Thanks.
I don’t see those robots.txt files in rankmath (general setttings – robots.txt)
I created a screenshot but there is no upload button here.
So I have to go into the root directory and find a folder called (robots.txt)?
Hello,
It seems like you are using the Easy Mode of our plugin. Some options are only available in the Advanced Mode. Please follow this guide to switch to the Advanced Mode: https://rankmath.com/kb/switch-from-easy-to-advanced-mode/
You will be able to access the Edit Robots.txt option after that. And if it is in the root directory, it will be a TXT file, not a folder.
Hope this helps. Let us know if you need any other assistance.
Thanks, Jaideep
I turned on Advanced mode.
But the robots.txt window is still empty.
BUT I CAN write in it.
Would this overwrite the robots.txt file in the root folder?
If yes, what should I put in?
thanks,
Robert
Hello,
Rank Math would not be able to override the robots.txt in your website’s root directory. The robots.txt file we generate is virtual, and you may need to delete or at least rename the static file in your website files if you want to use Rank Math’s robots.txt.
You can also get in touch with your web host for assistance locating the static file on your site.
For the recommended rules to use in robots.txt, please refer to this link:
https://rankmath.com/kb/how-to-edit-robots-txt-with-rank-math/#default-rules
For your screenshot question earlier, you can use a free service tool like https://imgbb.com/ and share the generated link here so we can check.
I hope that helps.
Thank you.
Hi Reinelle,
What is the robots.txt section then good for? I can’t even see my sites robots.txt files?
Not complaining, I’m really puzzled about the function of this.
Hello,
Rank Math generates the robots.txt like other websites, but we generate it on the fly (virtually) so if you have a physical robots.txt file in your website’s root directory, Rank Math’s virtually robots.txt will be disabled.
If you are still having issues editing the robots.txt, we might need to take a closer look at the settings. Please edit the first post on this ticket and include your WordPress & FTP logins in the designated Sensitive Data section.
Please do take a complete backup of your website before sharing the information with us.
It is completely secure and only our support staff has access to that section. If you want, you can use the below plugin to generate a temporary login URL to your website and share that with us instead:
https://wordpress.org/plugins/temporary-login-without-password/
You can use the above plugin in conjunction with the WP Security Audit Log to monitor what changes our staff might make on your website (if any):
https://wordpress.org/plugins/wp-security-audit-log/
We really look forward to helping you.
Hello,
Since we did not hear back from you for 15 days, we are assuming that you found the solution. We are closing this support ticket.
If you still need assistance or any other help, please feel free to open a new support ticket, and we will be more than happy to assist.
Thank you.
