Question about Google Crawl

#1107835
  • Waqas Bhatti

    3 weeks, 2 days ago

    Rank Math free

    Hmm, just wondering, can it be done without using Google Cloud –> as this page (https://rankmath.com/blog/google-indexing-api/) mentions it, but notice once the free trial ends, I’ll have to pay up. So, I’m wondering if it can be done without it. As I’m watching the Google Search console for page index, it looks like the last update was on 20th September, and only 220 out of 1849 have been done… which makes me wonder how long it will take for Google to complete everything? Also, want to check I have set it up correctly?

    Also, I have further questions to ask:
    1) Wondering when Google will re-update the page that has already been indexed, with the contents having been updated, so it doesn’t show the old version.
    2) How long does Google see the correct URL link, not a wrong link, so that it first discovers the URL link? As I know, a new sitemap was submitted on 20th September.
    3) When I create new products/pages, do I need to resubmit the new sitemap to Google Search Console, or does it already see it and schedule it when approve?

    So far, I’m happy with the Rank Maths free version a lot, with all the information it gives me to get the site on the SEO side, and I’m going through the webpage to bring it in line with the correct version.

Viewing 6 replies - 1 through 6 (of 6 total)
  • RankBot

    3 weeks, 2 days ago

    Hello,

    We apologize for the issues you’re facing regarding Google crawling and indexing. Thank you for raising your concerns.

    To address your questions:

    1. Using Google Cloud: The Google Indexing API is indeed a feature that can be integrated with Google Cloud, but it is not the only way to ensure your pages are indexed. You can still rely on standard methods such as submitting your sitemap through Google Search Console and ensuring your content is high-quality and relevant to encourage Google to crawl your site more frequently.

    2. Indexing Timeline: Google does not have a fixed schedule for re-indexing pages. It can take anywhere from a few days to several weeks for Google to re-crawl and update indexed content, depending on various factors like your site’s authority, the frequency of updates, and the overall crawl budget assigned to your site. If you have submitted a new sitemap, it should help Google discover new or updated content more efficiently.

    3. Correct URL Discovery: When you submit a new sitemap, Google will typically start discovering URLs from that sitemap relatively quickly, but again, the exact timing can vary. It’s important to ensure that there are no issues preventing Google from accessing your URLs, such as incorrect robots.txt settings or noindex tags.

    If you would like us to check your Rank Math settings or provide guidance on setup, we recommend reviewing our tutorials first. For comprehensive assistance, you might consider our professional setup service available at Rank Math Services, which requires a PRO subscription.

    If you have further questions or need specific help with your site, please reply to this ticket with your site URL so we can assist you better.

    Thank you for using Rank Math! We’re glad to hear you’re finding value in the free version. Please let us know if there’s anything else we can help with!

    Sources:
    Why Google Hasn’t Indexed Your Post, Page, and/or Website? » Rank Math
    Why Google Hasn’t Indexed Your Post, Page, and/or Website? » Rank Math

    This is an automated reply from Rank Math Support Bot. Your ticket has been assigned to a human agent and we will get back to you shortly.

    Waqas Bhatti

    3 weeks, 2 days ago

    Rank Math free

    Hello,

    I have updated the sensitive data as requested. Can you please check further?

    Thank you.

    Waqas Bhatti

    3 weeks, 2 days ago

    Rank Math free

    Hello,

    I have updated the sensitive data as requested. Can you please check further?

    Thank you.

    Reinelle

    3 weeks, 1 day ago

    Hello,

    In this case, please follow these steps to flush your sitemap cache:

    1. Flush the Sitemap cache by following this video screencast:
    https://i.rankmath.com/pipRDp

    2. Exclude the Sitemap files of the Rank Math plugin in your caching plugin. The cache could be via a plugin or from the server. For plugins or Cloudflare, please follow this article:
    https://rankmath.com/kb/exclude-sitemaps-from-caching/

    3. Disable the transient cache for the sitemap by applying this filter. This will make sure that your XML sitemap always reflects the latest changes on your site:

    /** 
 * Filter if XML sitemap transient cache is enabled.
 *
 * @param boolean $unsigned Enable cache or not, defaults to true
 */
add_filter( 'rank_math/sitemap/enable_caching', '__return_false');

    If you’re not sure how to add this code, you can follow this guide:
    https://rankmath.com/kb/wordpress-hooks-actions-filters/

    Lastly, head over to your website files /wp-content/uploads/rankmath/, and delete any XML files from that path.

    Regarding your HTML Sitemap, its purpose is for your site visitors/users.

    Aside from that, crawlers can understand and organize your website faster and more easily with an HTML sitemap. When the crawlers know how to organize your site, it can increase your search engine visibility.

    Here’s a link for more information:
    https://rankmath.com/kb/html-sitemap/

    Looking forward to helping you.

    Waqas Bhatti

    2 weeks, 4 days ago

    Rank Math free

    Hi Reinelle, okay, just gone through the information given, some I have done and some I don’t understand, which made me more confused about how.

    So 1 and 2 are done.
    3, from the link (https://rankmath.com/kb/wordpress-hooks-actions-filters/) is giving me random stuff that I don’t understand, where I need to apply this code “/**
    * Filter if XML sitemap transient cache is enabled.
    *
    * @param boolean $unsigned Enable cache or not, defaults to true
    */
    add_filter( ‘rank_math/sitemap/enable_caching’, ‘__return_false’);” As it seems too complex for me to understand, I need something simple for me to follow it. As I’m not that top-end expert, but more like a newbie and starter technical person, I just couldn’t follow it, as I had no idea where and how to do it 😩🤷‍♂️.
    4 that say “/wp-content/uploads/rankmath/”, what I get is page not found “This page doesn’t seem to exist.
    It looks like the link pointing here was faulty. Maybe try searching?” when I do it as https://a7sports.co.uk/wp-content/uploads/rankmath/. So I’m lost completely with it.

    For the HTML Sitemap, okay, I will enable it to be seen on the website, as I was a bit unsure if it was supposed to be in the public domain. Also, with this HTML Sitemap, I got a further question to ask: can HTML Sitemap display the layout better, or is there no version for it, just as it is one long random stuff it shows. As I notice how competitor show their HTML Sitemap layout, compared to what Rank Maths does on my website, I’m working on it. Let me know if you need the competitor’s sitemap link so you can see the difference between how they do it and how Rank Maths does it on mine.

    Also, to clarify further, as I see the question I asked, which has not been answered, about:
    1) This page https://rankmath.com/blog/google-indexing-api/ requires the Instant Indexing plugin and Google Cloud account, as I followed the information from the link given. But I see a problem, once the free trial ends, I know it gonna stop, since there’s no payment for it. I’m wondering if this is the only way or if there’s a different way that doesn’t require Instant Indexing and Google Cloud, and still gets updated every time I make any changes to the pages, or move the link to a new version. So Google show the correct version to the customer on the search engine.
    2) So, regarding Google, pages that have already been indexed and now show on the search engine. As I notice currently, the pages I have done, which show an updated version of the contents, but Google is still showing the old content that was captured and shown via the search engine. So, when will it be updated to reflect the change made to it? Do you need an image on this that I’m referring to? So you understand what I am seeing at the moment.
    3) I notice from the 404 monitor from Rank Maths that it has shown me the link to the old version, which Google had pick up first time before I used Rank Maths (so whats surprised to see they was existed, and I was like that don’t exist now, so why it still there?), which I already updated to the new and correct link version. So, it makes me wonder when Google will update the link to say this old link is no longer in use and see this link that customers see when viewing the website. I’m trying to reduce unnecessary redirection links for no reason. As I’m wondering, is this link to the sitemap, which shows all the current links set from the website? The reason why I gave “20th September 2025” reference is that this is what Google Search Console is telling me.

    Additional question about the sitemap that I didn’t ask before in my previous message, I need to understand under the Rank Math SEO plugin for the General section, which focuses on the sitemap. “Links Per Sitemap” says “Max number of links on each sitemap page” I know Rank Maths knowledge base says recommend 200 links. As far as I know, the website I got had approximately 2000 links (all of the products, categories, brands, and pages). So, could I do it as 2000 Links per Sitemap? So it shows everything rather than just the first 200 links only. Due to this, I read about “Max number of links on each sitemap page”. Or keep it as 200 (but done 201 from what you stated to me to do for “Flush the Sitemap cache by following this video screencast”), and it means it will add every page that I have created (like 2000 pages done so far), and it will be shown in the sitemap by every 200 link per section/sitemap folder.

    Apologies for asking these queries, as I’m just making sure it is going in the right direction and working; otherwise, I’m questioning myself if this has even been done correctly and getting the website to be seen clearly to Google and pages is ranking well over other competitor I see for same product I’m selling to the customer to buy from us instead of those competitor.

    Wait for your reply on this matter.

    Jeremy

    2 weeks, 4 days ago

    Hello,

    If step 1 and 2 is followed, try checking the sitemap if the updated content is shown there. If they are, then no need to follow step 3. If you still want us to disable the transient caching for your sitemap, you may share your website’s WP and FTP login here with us.

    For the HTML sitemap, at the moment, the layout you see is shown as is. There’s no way to customize it in our settings. The look of the HTML sitemap depends entirely on your theme’s styling.

    As for your questions:

    1. You don’t need to rely solely on the Instant Indexing plugin and Google Cloud. By default, Google will eventually pick up changes through its normal crawling process, especially if your site is properly optimized and submitted in Search Console. The Indexing API is just an optional way to speed things up, but it’s not required for Google to update your pages.

    2. Google decides when to refresh the cached version of your pages, and unfortunately, we can’t influence the exact timing. In general, websites with higher authority and consistent publishing schedules tend to see updates reflected faster in the search results.

    3. Google may still discover and show old URLs if they are linked somewhere. We recommend checking the affected URL in Google Search Console’s URL Inspection tool as it will show you the referring page where Google found it. From there, you can update or remove the source link to help reduce unnecessary redirections.

    4. You can increase the number of links per sitemap if you wish. It’s fine to set it to 2000, as long as the sitemap loads quickly, since Google may discourage crawling very large or slow-loading sitemaps. Keeping it at 200 will simply split your 2000 links into multiple sitemap pages, which is also perfectly valid.

    Don’t hesitate to get in touch with us if you have any other questions.

Viewing 6 replies - 1 through 6 (of 6 total)

You must be logged in to reply to this ticket.