Crawler issues - Support

Resolved Stuart
3 years, 9 months ago

Rank Math free

Hi,

Ive been reading about crawler / canonical tags issues in your support forums but Im still having issues.

Google seems happy

Bing says – Index Issues – Large number of pages pointing to the same canonical URL. Canonical tags can be used to help de-dupe URLs for pages with the same or very similar content. Bing has detected that your site is using what appears to be the same canonical URL for a large number of pages that are not the same. This may be indicative of a problem with your HTML template, content publishing system, or site code. You should review the source of your pages and check if the URLs used inside the <link rel=”canonical” href=”<url>” /> are not accidentally pointing to the same location for all (or too many) pages.

Facebook – OMG – CRASHES MY SERVER

I have an excel spreadsheet from Content King who scanned my website and I can see whats going on, I just cant fix it.
I’ve emailed this to s******@r*********** with this ticket in the title – look at line 248 onwards.

The issue is with my job categories, Ive recently added a manual redirect but the hit count is in excess of 25000 hits.

I’m so stuck, my developer is stuck and so is a digital marketer. Please help.

Ive spoken to the theme developer, they say they are ready to help if needed, Ive also spoken to the plugins people at wp job manager and they said to install another plugin for crawlers, like a black hole, but that wont work as facebook is whitelisted.

what am I doing wrong?

ideas welcomed. Thank you.

Viewing 7 replies - 1 through 7 (of 7 total)

Stuart
3 years, 9 months ago

Rank Math free

One thing Ive just noticed as well. Job categories are blocked in the rank math dashboard, yet when i check them manually some are showing as blocked and others show as indexed when checkeing them in the advanced tab.

I have screenshots I can email if required.

Is that weird why some show as blocked and other show to index?

Miguel
3 years, 9 months ago

Hello,

Thank you for contacting us and sorry for any inconvenience that might have been caused due to that.

Upon inspecting your website there a couple of things we noticed that we would like to expose here in order to better help narrow down the issue:

1. Some jobs categories have different robots meta tags like you mentioned. The global settings for the jobs categories are set to noindex so the only reason these settings are not applied to all of them is because you either edited them before or after making the global changes.

2. You are blocking with robots.txt all the pages with filters getting applied to them. However it seems that they are still able to being indexed and all of them have the canonical of the main jobs page.

3. One last thing: you added the redirection before or after you started noticing the issue?

Looking forward to helping you.

Stuart
3 years, 9 months ago

Rank Math free

Hi,

Did you have any thoughts please?

I have pro on the website now, cant wait to get stuck in.

Stuart
3 years, 9 months ago

Rank Math free

Hi Miguel,

Thanks for help so far.

Replies to points above….

1.) I have now been through every single category option in jobs, locations, job types, resumes, disability and equality, everything , it took me ages and checked twice and marked them as the 5 blue ticks you mentioned above for no index, no follow etc.

2.) My web chap is blocking facebook and maybe bing as the server would be crashing constantly otherwise from high error log messages filling the memory.

3.) I added the redirection after issues were noticed.

Newthings now.

4.) I let Bing rescan the website, from the colsole it seems to be these strings, ?filter

https://somersetjobs.co.uk/jobs/?filter_location=portishead,wells&query_type_location=or&filter_job_listing_category=clothing-materials&query_type_job_listing_category=or
3
https://somersetjobs.co.uk/jobs/?filter_location=portishead%2Cwells&query_type_location=or&posted_before=1-hour
3
https://somersetjobs.co.uk/jobs/?filter_location=portishead,wells&query_type_location=or&filter_job_listing_salary=50k-plus&query_type_job_listing_salary=or
3
https://somersetjobs.co.uk/jobs/?query_type_location=or&filter_job_listing_salary=25000-34999&query_type_job_listing_salary=or&filter_location=somerton,wells
3
https://somersetjobs.co.uk/jobs/?query_type_location=or&filter_job_listing_salary=25000-34999&query_type_job_listing_salary=or&filter_location=ilminster,wells
3
https://somersetjobs.co.uk/jobs/?filter_location=wells,weston-super-mare&query_type_location=or&filter_job_listing_category=healthcare-jobs&query_type_job_listing_category=or
3
https://somersetjobs.co.uk/jobs/?filter_location=wells,weston-super-mare&query_type_location=or&filter_job_listing_category=fmcg-jobs&query_type_job_listing_category=or
3
https://somersetjobs.co.uk/jobs/?filter_location=wells,weston-super-mare&query_type_location=or&filter_job_listing_category=management-jobs&query_type_job_listing_category=or
3
https://somersetjobs.co.uk/jobs/?filter_location=wells%2Cweston-super-mare&query_type_location=or&posted_before=30-days
3
https://somersetjobs.co.uk/jobs/?query_type_location=or&filter_location=ilminster,wells,weston-super-mare
3
https://somersetjobs.co.uk/jobs/?filter_job_listing_category=school&query_type_job_listing_category=or&filter_location=wells&query_type_location=or&filter_job_listing_type=part-time-jobs&query_type_job_listing_type=or
3
https://somersetjobs.co.uk/jobs/?query_type_location=or&filter_location=jobs-in-cheddar,portishead,wells
3
https://somersetjobs.co.uk/jobs/?query_type_location=or&filter_location=bridgwater,portishead,wells
3
https://somersetjobs.co.uk/jobs/?filter_location=portishead,wells&query_type_location=or&filter_job_listing_type=temporary&query_type_job_listing_type=or
3
https://somersetjobs.co.uk/jobs/?query_type_location=or&filter_location=wells

5.) I recently found this article as it seems to be the filters but i might be wrong?? It mentions

“non of the SEO specific features will work here, like noindex, nofollow etc you have to create regex rule and give 410 to search engines”

Does this make any ideas come to mind?

https://magento.stackexchange.com/questions/95537/can-you-block-google-from-crawling-filters/97427

Reinelle
3 years, 9 months ago

Hello,

Thank you for the update, and sorry for the delay.

I would just like to summarize things up. Bing is saying about too many pages with the same canonical (those pages with filter), so in order to fix that – you want those pages to be non-indexable (noindex) so Bing doesn’t see them. Could you please confirm if this is correct?

If so, I’ve checked your CSV file, and as you mentioned, the URLs with filters are set to noindex. However, upon checking the URLs you have followed up (from Bing’s rescan), it seems that they are not included in your CSV and their meta tags are still set to index (page’s source code). We wanted to check their robots meta individually, but the credentials you have provided seem to be not working. Could you please check them as well?

I hope that helps. Thank you, and looking forward to your update.

Stuart
3 years, 9 months ago

Rank Math free

Hi Reinelle,

Thank you for reply, apologies for my late one.

Yes, that’s right, those filter pages need to be no index.
Login details do work, I just tested them.

If it still shows the login pge after password just use the black admin bar at top of screen.

The URLs provided were just a small sample of the log from bing.

Facebook get stuck in an infinate filter loop, it’s a nightmare, facebook is blocked currently/.
Michael Davis
3 years, 9 months ago

Rank Math pro
Hello,

To fix the issue with index/noindex, you can consult with the job plugin you are using to see if there is a way to get all pages with filters then use th following hook to set all filtered pages to noindex:
```
/**
 * Allows filtering of the robots meta data.
 *
 * @param array $robots The meta robots directives.
 */
add_filter( 'rank_math/frontend/robots', function( $robots ) {
     global $post;
     if(filtered() == true){
        $robots['index'] = "noindex";
	return $robots;
	}
	return $robots;
});
```
P.S: I also noticed you have a lot of plugins that need update including Rank math free and Pro. We recommend to always keep your plugins up to the latest updates

Hope that helps and please do not hesitate to let us know if you need our assistance with anything else.
RankBot
3 years, 8 months ago

Hello,

Since we did not hear back from you for 15 days, we are assuming that you found the solution. We are closing this support ticket.

If you still need assistance or any other help, please feel free to open a new support ticket, and we will be more than happy to assist.

Thank you.

Viewing 7 replies - 1 through 7 (of 7 total)

The ticket ‘Crawler issues’ is closed to new replies.