Using "robots" meta tag to get website indexed - Support

Resolved Matt Eastwood
3 years, 9 months ago

Rank Math free

Hello, Rank Math team,

A while ago, I opened ticket #317540 asking for suggestions to get new websites indexed by Google. Even months after release, Google wouldn’t index e.g. https://www.mrm-messer.de, even though the sitemaps are fine and, as you can see in the aforementioned ticket, I took many steps to try and get it done.

It’s a problem I had with two of my own projects, and that we had with several projects at the office in the small advertising agency I work at.

Last week, one of my colleagues found a solution. He added this exact meta tag to the <head> of affected websites, e.g., https://derknoll.de:

<meta name=”robots” content=”index, follow” />

This prompted Google to index https://derknoll.de/ within hours, after several months of no indexing. The site was built in WordPress using Oxygen page builder and Slim SEO.

My colleague tried this on a handful of other websites over the weekend and they all had the same positive results.

On my own project, https://www.mrm-messer.de, launched in early December, still not indexed and built using WP, Beaver Builder and Rank Math, I see this meta tag in the <head>:

<meta name=”robots” content=”follow, index, max-snippet:-1, max-video-preview:-1, max-image-preview:large”/>

How would I go about editing this meta tag so it matches the one that gets the website indexed?

Thanks!
Matt.

P.S.: My colleague found the same long <meta> tag and split it in two, for derknoll.de:

<meta name=’robots’ content=’max-image-preview:large, max-snippet:-1, max-video-preview:-1′ />
<meta name=”robots” content=”index, follow” />

Viewing 9 replies - 1 through 9 (of 9 total)

Jeremy
3 years, 9 months ago

Hello,

Thank you for contacting the support, and sorry for any inconvenience that might have been caused due to that.

I checked your website and it is already set to index in the robots meta tag.

The robot’s meta tag used by your colleague is just the same configuration as in your website although how the attribute values are presented is in a different order still, the configuration is still the same.

Could you please check your Google Search Console for coverage issues so we can further analyze the situation?

Looking forward to helping you on this one.

Matt Eastwood
3 years, 9 months ago

Rank Math free

Hello Jeremy,

Thanks for your quick reply and suggestions. 🙂 I realize I should have provided more context. mrm-messer.de is just one example. We’ve been facing the same issue with the following websites since mid-December 2021, when we discovered that schmuckmemory.de hadn’t been indexed since its launch in July, 2021:

https://www.mrm-messer.de/
https://www.schmuckmemory.de/
https://derknoll.de/
https://hotel-koenigshof.eu

In the 3 months we’ve investigated this issue, we’ve kept a close eye on GSC, submitted and frequently re-submitted the sitemaps both as …/sitemap-index.html and …/?sitemap=1, put backlinks on other already-indexed websites, posted in SEO forums, spoke to a marketing agency specializing in SEO, followed the advice in tickets #317540 and #320425, as well as the help pages from the Rank Math knowledgebase suggested in those tickets.

Incidentally, the marketing agency we contacted told us that getting websites indexed has been a rampant problem among webmasters since late 2020, when Google had problems detecting canonical contents and GSC was shut down over several months. Since then, indexing has been very slow and intransparent. Apparently, people suspect Google has decreased the importance it places on GSC as a signal for indexing.

For our websites, GSC sometimes has trouble fetching the sitemaps (404), or it fetches them and discovers 0 URLs. However, most of the time, GSC properly reads the sitemaps and simply won’t index the sites, even when we explicitly request indexing of specific URLs. The URL inspection tool says: “This page is not in the index, but not because of an error. See the details below to learn why it wasn’t indexed.” Here’s a GSC screenshot for schmuckmemory.de from today: https://paste.pics/GAJ7J

After all this, what I’m saying is that last Friday, my colleague discovered the solution to our problem. He placed this specific meta tag in the <head> of affected websites:

<meta name=”robots” content=”index, follow” />

He put the other arguments in a separate meta tag.

Afterwards, they were indexed within mere hours. We’ve had a 100% success rate with this, and you can imagine we’re baffled as to why this works. We just know it does. We’re aware that splitting the long meta tag in two doesn’t, on the face of it, really change things, since it’s still the same information. But the evidence is incontrovertible.

For comparison, the meta tag that Rank Math places (e.g. in mrm-messer.de) is this:

<meta name=”robots” content=”follow, index, max-snippet:-1, max-video-preview:-1, max-image-preview:large”/>

We did notice that it says “follow, index” and the one that works says “index, follow”. Google themselves are using the “index, follow” order, cf. here: https://developers.google.com/search/docs/advanced/robots/robots_meta_tag

Plus, as I mentioned, there are the other arguments that stand after “follow, index” that we split into a separate meta tag.

Hence, my question is: How can we achieve this effect on our websites that run Rank Math? We’d love to at least give this a shot, since we have reason to believe that the exact syntax mentioned above will, in fact, get our websites indexed by Google.

I hope there’s a way to do this. Looking forward to your response.

Thanks, and best regards,
Matt.
Jaideep Asrani
3 years, 9 months ago
Hello,

The robots meta that Rank Math adds is according to Google’s guidelines and it shouldn’t have any effect on indexing whether you have follow, index or index, follow in there. All Google looks for is the index in the robots meta to confirm that you want them to index the pages.

And even if it had any effect, Google still needs to crawl the URL first in order to see the tags but the screenshot that you have shared shows that Google hasn’t even crawled your URL yet.

Having said that, if you want to use your suggested setup then please navigate to WP Dashboard > Rank Math > Titles & Meta > Global Meta and uncheck all the options in the Advanced Robots Meta:

After that add the filter given below in your theme’s functions.php file:
```
add_filter( 'rank_math/frontend/robots', function( $robots ) {
	$robots = array('index' => 'index', 'follow' => 'follow');
	return $robots;
});
```
Hope this helps. Let us know if you need any other assistance.

Thanks.
Matt Eastwood
3 years, 9 months ago

Rank Math free

Hey Jaideep,

Thanks so much for your insights! This helps me understand the matter better.

I’m gonna give this a shot nonetheless. For schmuckmemory.de, I disabled the settings you mentioned:

https://paste.pics/GAYFE

I then updated WP and all plugins, then cleared all cache I could find. Yet, the full meta tag is still in the source:

<meta name=”robots” content=”index, follow, max-snippet:-1, max-video-preview:-1, max-image-preview:large”/>

(the order swapping worked, though!)

Is there anything else I need to do to remove the max-snippet, max-video-preview and max-image-preview arguments for this test?

Thanks 🙂
Matt.

Kamlesh
3 years, 9 months ago

Rank Math free

Hello,

Could you please head over to your WordPress Dashboard > Rank Math > Titles & Meta and check if the Robots Meta is disabled for Homepage?

If it is enabled, please uncheck all the options in the Advanced Robots Meta from there as well.

Let us know how it goes.

Looking forward to helping you.

Thank you.

Matt Eastwood
3 years, 9 months ago

Rank Math free

On “Homepage”, I get this message:

Static page is set as the front page (WP Dashboard > Settings > Reading). To add SEO title, description, and meta for the homepage, please click here: Edit Page: Schmuckmemory

I went ahead and un-set the checkboxes on the front page and that did the trick.

I found the same options on “Post Formats”, “Authors” and “Misc Pages” too, so I disabled them there as well.

Poking GSC again. I’ll be sure to let you know what happens!

Jeremy
3 years, 9 months ago

Hello,

On “Homepage”, I get this message:
Static page is set as the front page (WP Dashboard > Settings > Reading). To add SEO title, description, and meta for the homepage, please click here: Edit Page: Schmuckmemory

If your homepage is set to a static page which you can configure under WordPress Dashboard > Settings > Reading > “Your homepage displays”, you can edit its meta directly to that page editor since this is treated as an actual page under WordPress and not a dynamic page.

Poking GSC again. I’ll be sure to let you know what happens!

Let us know how that goes. We will keep this ticket open for you.

If you have further questions, please let us know here and we are more than happy to answer.

Looking forward to helping you.

Matt Eastwood
3 years, 9 months ago

Rank Math free

Hey guys, this is to let you know that I tried this on schmuckmemory.de and it is now being indexed by Google. That said, I also found that the connection to GSC hadn’t been established from within Rank Math using my Rank Math account, and the GSC implementation in General Settings > Webmaster Tools wasn’t active. Either of the two might have done the trick.

This thread has been truly helpful, thanks for all your suggestions and insights. I will certainly keep all this in mind for future projects.

Reinelle
3 years, 9 months ago

Hello,

Thank you for the update.

We are glad it worked for you.

If you have any other concerns, please don’t hesitate to contact us anytime to assist you further.

Looking forward to helping you.

Thank you.

Viewing 9 replies - 1 through 9 (of 9 total)

The ticket ‘Using “robots” meta tag to get website indexed’ is closed to new replies.