Hello,
Thank you for contacting Rank Math and bringing your concern to our attention.
In such case, you may add a HTTP header as 503 Service Temporarily Unavailable with a Retry-after combination to tell Google that you’re busy.
Returning a 503 in combination with a Retry-After header tells Google how many minutes to wait before coming back. This does not mean Google will crawl again in exactly X minutes, but it’ll ensure Google doesn’t come back around to take a look anytime before then.
To do that, you may edit the file index.php at your root directory and add the below lines after <?php
header("HTTP/1.1 503 Service Temporarily Unavailable");
header("Status: 503 Service Temporarily Unavailable");
header("Retry-After: 3600");
The value 3600 indicates the delay time in seconds. That means that the above sample will tell GoogleBot to return after an hour. You may also set specific date and time for this one like below:
Retry-After: Sun, 22 Jan 2023 08:00:00 GMT
Please be careful with what you add here, as adding a faulty date might result in unexpected results.
Once done, you may set a 503 Service Temporarily Unavailable page on front-end of your site. For fancy notices, you can use a plugin like WP Maintenance Mode.
Here’s what Google says,
https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt#http-status-codes
A 503 (service unavailable) error results in fairly frequent retrying. If the robots.txt is unreachable for more than 30 days, Google will use the last cached copy of the robots.txt. If unavailable, Google assumes that there are no crawl restrictions.
If you need to temporarily suspend crawling, we recommend serving a 503 HTTP status code for every URL on the site.
Be sure that you’ve set 5xx error on front-end:
If we are able to determine that a site is incorrectly configured to return 5xx instead of a 404 status code for missing pages, we treat the 5xx error from that site as a 404. For example, if the error message on a page that returns a 5xx status code is “Page not found”, we would interpret the status code as 404 (not found).
Hope that helps, and please do not hesitate to let us know if you need our assistance with anything else.
Thank you.
