Pages with Feed and swcfpc=1 are not being indexed by google.

Hello,

I understand your concern about certain pages with "feed" and "swcfpc=1" parameters not being indexed by Google. If these pages shouldn’t be crawled and indexed, there are a few steps you can take to address this issue:

1. Robots.txt File: You can block these specific pages from being crawled by adding directives to your website’s robots.txt file. This file tells search engine crawlers which pages or files they should not crawl.

2. Noindex Meta Tag: You can add a noindex meta tag to the pages you don’t want to be indexed. This tag tells search engines not to include those pages in their index.

3. Canonical Tags: If these pages are duplicates or variations of other pages on your site, you can use canonical tags to specify the preferred version of the content that should be indexed.

4. Content Audit: Perform a content audit to identify low-quality or duplicate content on your website. Remove or improve these pages to enhance the overall quality of your site.

If you need further assistance in implementing these solutions or have any other questions, please let us know by assigning this ticket to a human agent using the available buttons. Thank you for reaching out, and we’re here to help you resolve this issue.

Closing spiel.

Sources:
Why Google Hasn’t Indexed Your Post, Page, and/or Website? » Rank Math
How to Fix "Crawled – Currently Not Indexed" Error in Google Search Console » Rank Math

Hello,

Thanks for contacting us, and sorry for any inconvenience that might have been caused due to that.

Feed URLs are not created for humans but for RSS feed crawlers and readers. They are only basic code versions of your actual content pages.

Those are not meant to be indexed and hopefully, you won’t want these pages to be indexed as Google doesn’t like them and would most likely not show them to users anyway.

In this case, please add this rule to your robots.txt to disallow them from being crawled:

User-Agent: *
Disallow: */feed/
Disallow: *swcfpc=1*

Here’s how you can edit the robots.txt file using Rank Math:
https://rankmath.com/kb/add-sitemaps-to-robots-txt/#num-2-2-navigate-to-edit-robots-txt

This should save your crawl budget and after the next few crawls, the warnings should be removed from GSC.

Hope that helps, and please do not hesitate to let us know if you need our assistance with anything else.

Thanks.

Hello,

Your robots.txt is already working fine.

This can happen when you use a content delivery network (CDN) like Cloudflare since Rank Math does not create a physical robots.txt file on your site.

Instead, the file is created and displayed when a user visits your robots.txt URL.

So, you can safely ignore this error if your robots.txt file is already available.

Here’s a link for more information:
https://rankmath.com/kb/fix-common-robots-txt-issues/#robots-txt-is-not-valid-in-pagespeed-insights

Looking forward to helping you.

Hello,

Apologies for the issue but the robots.txt is still in the wrong format.

Please use this rules instead:

User-Agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: */feed/
Disallow: *swcfpc=1*
Disallow: /cdn-cgi/

Sitemap: https://confidentpersonality.com/sitemap_index.xml

As for the issue with the swcfpc=1 pages getting indexed by Google. Let’s try forcing it to set to noindex in their robots meta.

Please use this filter code for your website:

add_filter('rank_math/frontend/robots', function ($robots) {
	$url = home_url($_SERVER['REQUEST_URI']);
	if (strpos($url, 'swcfpc=')) {
		$robots["index"] = 'noindex';
		$robots["follow"] = 'nofollow';
	}
	return $robots;
});

You may refer to this guide on how to add filters to your website: https://rankmath.com/kb/wordpress-hooks-actions-filters/

Let us know how that goes. Looking forward to helping you.

Hello,

We tested your site and can see that the robots.txt is now set up correctly and blocking those URLs. After Googlebot has crawled your site for a while it will no longer report it as crawled but not indexed.

The report of being blocked by robots.txt in the URL inspection tool indicates that it is now setup correctly.

Hope that helps, and please do not hesitate to let us know if you need our assistance with anything else.