Hello,
We apologize for the issue you are facing with your website being de-indexed from Google. It can be quite concerning when such a situation arises.
From the robots.txt file you shared, it appears that you have several Disallow directives. While these directives are meant to control which parts of your site Google can crawl, it’s important to ensure that none of them inadvertently block access to your main content.
To troubleshoot this issue:
- Check for Blocking Directives: Ensure that there are no
Disallow rules that might be preventing Googlebot from crawling your main pages. The User-agent: * directive applies to all bots, including Googlebot, so any Disallow rule under it could potentially block access.
- Use the Robots Testing Tool: You can use the Robots Testing Tool to check if Google is blocked from crawling specific URLs on your site. This tool will help you identify if any of your rules are causing issues.
- Review Your Sitemap: Make sure that your sitemap is correctly configured and submitted in Google Search Console. This helps Google discover your pages more efficiently.
- Check Google Search Console: Look for any specific messages or errors related to indexing issues in your Google Search Console account. This can provide insights into why your site was de-indexed.
- Consider Recent Changes: If you made any recent changes to your site or its configuration, consider whether those changes could have affected indexing.
If you find any Disallow directives that are blocking important pages, you will need to modify your robots.txt file accordingly. You can refer to our guide on editing robots.txt with Rank Math for assistance.
If you’re still having trouble after checking these points, please reply to this ticket, and a human agent from the Rank Math Support team will assist you further.
Thank you for raising this concern, and we hope to help you resolve it as soon as possible!
Sources:
Common robots.txt Issues & How to Fix Them » Rank Math
How to Fix Indexing Issues in Google Search Console » Rank Math
Discovered – Currently Not Indexed in Google Search Console » Rank Math
Why Did My Website Disappear From Google Search? » Rank Math
This is an automated reply from Rank Math Support Bot. Your ticket has been assigned to a human agent and we will get back to you shortly.
Hello,
Thank you for reaching out to Rank Math support.
We can confirm that your sitemap and robots.txt files are functioning properly without any issues. Additionally, we have verified that your website has not been de-indexed on Google, as shown in the screenshot below:
If you are concerned about your website not appearing for specific keywords, please note that your rankings are influenced by how well you optimize for those keywords. You can read this guide to learn more about optimizing your posts for SEO: https://rankmath.com/kb/score-100-in-tests/.
We hope this clarifies the issue. Please let us know if you have any other questions or concerns.
Please concern your response.
I understand your concern.
As I have already told you, my website has 260 URLs, and Google has already indexed all of them. I have 26 pages on Google of my website. But as you see today Google shows only up to 5 pages. The rest of the URLs are completely De-Indexed from Google.
I already checked on Google Search Console i.e. URL Inspection – My website all the URLs are De-Indexed. Most of the URLs are showing under [ Crawled – currently not indexed ] issues at GSC.
My website is affiliate marketing and the festive season is going. I regularly post the content too. Not that my website is New. 8 months old my website.
So my main point is why this will happen now and suddenly. Google De-Indexed my whole website.
I am sharing some screenshots of my GSC and Google Pages.
https://imgur.com/a/LnIMKmq
https://imgur.com/a/KqjVN20
This is a critical problem for me. I again request you please look into this matter and do the needful. ASAP.
Hello,
The “Crawled – currently not indexed” means Google has crawled your page but has not indexed it yet. Finding a certain number of URLs with this status is entirely normal because Google does not index all the URLs we submit.
This isn’t an error, but there are some reasons for excluding the pages from its index.
Here’s a link for more information:
https://rankmath.com/kb/crawled-currently-not-indexed/
Please note that Google considers various factors while indexing. Your frequency of posting and your site’s domain authority are just two of the many factors Google considers when indexing new web pages.
Google assigns a crawl budget to your website depending on these factors (especially these two), and that has a direct effect on how soon (i.e. how efficiently) your content can be indexed.
Looking forward to helping you.
I understand your concern.
But my main point is Google already Indexed all the pages. I regularly Post the content too. Then why (suddenly) Google De-Indexed all my pages? My website doesn’t have the domain authority right now.
So please can you give my suggestion on how my website URLs will be Indexed fastly? and also checked there were no issues on the Robot.txt file that I have shared image with you. Please check and give suggestions.
https://imgur.com/a/h0xayWi
I am glad for your response.
Hello,
When you see most of the pages under the “Crawled – Currently not indexed” that means that the pages lack unique content and Google has already discovered the pages and crawled them to try and index them, but found the content was lacking so it doesn’t want to index them.
This has nothing to do with the technical aspects of the website, which is what our plugin can control.
Don’t hesitate to get in touch if you have any other questions.
I have done everything that you have concern. But still my website is continuously De-indexeig from Google. And urls are increasing in GCS as crawled currently not indexed error.
As you see in the shared screenshots out of these 2 URLs, one is original and the second is a duplicate URL i.e [ ?coupon_type=sale ] that I want to De-Index or not show on Google Search. Also, give a command on the robots.txt file as I have shared with you. If you say to re-direct the duplicate URLs to the original URLs then where i can do, becoz these duplicate URLs are not shown in anywhere.
https://imgur.com/a/ckOEYIX
https://imgur.com/a/6QfgUOn
Google continuously De-Indexing my pages whether are originally URLs or duplicate URLs.
This was a faurated time for my website. Please suggest what to do? So that my original URLs are able to be indexed and shown on Google Search.
Hello,
Those filtered URLs are found on your content with these buttons:
In this case, you can add this code to your site to noindex those URLs with filters to stop search engines from crawling them:
add_filter( 'rank_math/frontend/robots', function( $robots ) {
$url = home_url( $_SERVER['REQUEST_URI'] );
if (strpos($url,'?coupon_type=') !== false || strpos($url,'wp_action=')) {
$robots['index'] = 'noindex';
$robots['follow'] = 'nofollow';
return $robots;
}
return $robots;
});
If you’re not sure how to add this code, you can follow this guide:
https://rankmath.com/kb/wordpress-hooks-actions-filters/
Once done, remove these disallow rules you added to your robots.txt for those filtered URLs for search engines to know they are already noindexed:
Disallow: /?coupon_type=all&wpu_action=register
Disallow: /?wpu_action=lost-pass
Disallow: /?coupon_type=code
Disallow: /?coupon_type=sale
Disallow: /?wpu_action=register
Disallow: /?coupon_type=code&wpu_action=lost-pass
Disallow: /?coupon_type=sale&wpu_action=lost-pass
Disallow: /?coupon_type=all
Disallow: /?coupon_type=code&wpu_action=register
Disallow: /?coupon_type=all&wpu_action=register
Disallow: /?coupon_type=all&wpu_action=lost-pass
Disallow: /?coupon_type=sale&wpu_action=register
Disallow: /?coupon_type=all&wpu_action=register
Disallow: /?coupon_type=sale&wpu_action=lost-pass
Looking forward to helping you.
As suggested,
I have added the shared code on the Snippets Plugin and removed all Disallow Rules on Robots.txt. I have shared a screenshot of Code Snippets and RankMath Robots.txt. Please have a look.
https://imgur.com/a/Ck1GGZr
https://imgur.com/a/0I8r5kL
Is there anything else I want to do? Please let me know.
Can you please tell me How and When my De-Indexed pages will be Indexed and shown on Google Search?
Thank you.
Hello,
You added the code as an HTML Snippet, please modify the snippet and add it as a PHP snippet.
Cheers!
Thanks for your response.
Can you please tell me how to do it?
Please have look into this matter and do the needful.
Thankyou.
Hello,
Please change the Code Type from HTML Snippet to PHP Snippet. You can do that by selecting the “HTML Snippets” dropdown and changing it to PHP Snippets. Here’s a short video:
We hope this helps clarify the issue. Please let us know if you have any other questions or concerns.
As per your suggestion,
I have changed the Code Type from HTML Snippet to PHP Snippet. Also, shared a screenshot please have a look.
https://imgur.com/a/mr5F0KE
Is there anything else I want to do? Please let me know.
Again, I request you to please tell me How and When my De-Indexed pages will be Indexed and shown on Google Search. How much time does it take to Index my pages?
ThankYou!
