Which custom fields cause post to be included in sitemap?

#43760
  • Resolved Cheryl

    4 years ago

    Rank Math free

    Hello, I’m still searching for the issue behind my sitemap issues from the last ticket.
    Now I’ve been looking at the specific fields for the post type concerned. The issue affects only the post type “Listings” under the MyListing theme.

    I’m puzzled as to why a majority of posts are not even included in the sitemap despite being set to “index”.

    Could you tell me please, which custom fields exactly cause a post to be included or excluded in a sitemap? And what must they contain?

    I’ve noticed some fields where the content seems to differ in my posts, could any of these be triggering a problem/exclusion?:
    rank_math_permalink
    rank_math_primary_job_listing_category
    rank_math_snippet_jobposting_unpublish
    any others?

    Thanks in advance,
    Cheryl

    ____________________________________
    FIXED! Rank Math Sitemap only shows the posts which are marked as index and whose canonical URL matches with the post URL. Also check for http vs. https in the canonical! > That did the trick!
    ____________________________________

    • This topic was modified 3 years, 12 months ago by Cheryl.
Viewing 15 replies - 1 through 15 (of 18 total)
1 2
  • Michael Davis

    4 years ago

    Rank Math pro

    Hello,

    Thank you for contacting Rank Math today.

    The custom field that dictates whether the post will be indexed is the “rank_math_robots” custom field. This field will save a serialized array if the post is set to index as shown below:
    a:1:{i:0;s:5:"index";}

    Looking forward to helping you. Thank you.

    ​​​​​​

    Cheryl

    4 years ago

    Rank Math free

    Hi Michael,

    thanks a lot!

    And you know, I thought so… but strangely, I have hundreds of posts where “index” is activated, but they are not showing up in the sitemaps. Do you have any idea if any other settings or fields could be causing this?

    Should all “indexed” posts be included in the sitemap? Or does RankMath evaluate the pages and only include those that fit some pattern?

    In blog posts, for example, I have 175 published posts, set as “index”, but only 128 are in https://www.ecoco.bio/post-sitemap.xml.
    I can’t see what’s different between those that are included and those that are not. E.g:
    https://www.ecoco.bio/vmail-fuer-dich-podcast-100/ is included
    https://www.ecoco.bio/vmail-fuer-dich-podcast-101/ is not included.

    Post type “Pages” https://www.ecoco.bio/page-sitemap.xml:
    Missing e.g.:
    homepage,
    https://www.ecoco.bio/wiki/
    https://www.ecoco.bio/faq/
    https://www.ecoco.bio/neues-angebote/
    https://www.ecoco.bio/suche/
    https://www.ecoco.bio/vmail-fuer-dich-podcast/
    Some of these are really important pages for the website.

    Post type “Listings”
    https://www.ecoco.bio/job_listing-sitemap1.xml and several more numbered further…
    There are more than 1500 published listings, but only a fraction shows up in the sitemaps.

    Could it be some conflict with something on the pages? E.g. a particular Elementor element? But then again, the Listings don’t use Elementor. As mentioned, I have tried deactivating all plugins and switching the theme but couldn’t resolve the issue.

    I’d be very grateful for any insights or ideas you could share. I’m trying to get to the bottom of this but need to understand better how the pages end up in the sitemap in the first place or why they might be excluded.

    Thanks in advance!
    Cheryl

    Michael Davis

    4 years ago

    Rank Math pro

    Hello,

    Thank you for contacting Rank Math today.

    Rank Math does not really use any special criteria to add links to the sitemap. The plugin will only check the index attribute.

    I suspect that your caching plugin is aggressively caching your pages and causing the links not be updated. To fix this, I can suggest that you purge both your site and server cache and then perform a fresh install of the plugin to clear the local cache.

    PS: No data loss will be experienced in this process.

    Looking forward to helping you. Thank you.

    ​​​​​​

    Cheryl

    4 years ago

    Rank Math free

    Hi Michael,

    thanks for the tip!
    I’ve tried purging the cache and re-starting the systems, just re-installed RankMath, but not seen any success. There are still pages missing in the sitemaps and 404s on the sitemap URLs.

    Perhaps I should try excluding the sitemap from caching in the first place?
    Is there an option in RankMath to exclude the sitemap from caching?

    If not: I’m using Breeze on Cloudways, with Varnish and Redis.
    Breeze has an option to exclude URLs from caching. Would this work?:
    /(.*)sitemap(.*).xml

    Would you say I also need to exclude the sitemap from CDN? That would be the Cloudways CDN. There is an “exclude content” field, would this work?:
    .php, /(.*)sitemap(.*).xml

    Thanks,
    Cheryl

    Todd

    4 years ago

    Rank Math free

    Hi Cheryl,

    Yes, please add this in the exclusion list:
    /(.*)sitemap(.*).xml

    That should prevent sitemap files from being cached.

    Cheryl

    4 years ago

    Rank Math free

    Hi Todd,

    thanks for your help!
    I excluded sitemaps from Breeze, CDN, Varnish and EWWW. No success.

    Since you tried to Troubleshoot my site with Health Check the other day, I moved to a different host, Cloudways, but still have the same problem there. At least the Troubleshoot works now – but when I deactivate all Plugins except RankMath and switch to default theme, the issue is still there. So it must be something in WordPress, right?
    Do you perhaps have any more ideas where I could be looking?

    I’d be really grateful for any tip you might have that could point me in the right direction.

    Thanks in advance,
    Cheryl

    Cheryl

    4 years ago

    Rank Math free

    Hi again,

    I am now wondering if the issue could be related to permissions? Could you tell me where the sitemap xml files are stored please, so I can check if the permission settings might be causing a 404?

    Thanks,
    Cheryl

    Cheryl

    4 years ago

    Rank Math free

    Hi again,

    I found this message in Debug Log after loading sitemap pages:

    [19-Apr-2020 15:16:18 UTC] PHP Notice: the_widget was called incorrectly. Widgets need to be registered using register_widget(), before they can be displayed. Please see Debugging in WordPress for more information. (This message was added in version 4.9.0.) in xxx/wp-includes/functions.php on line 5167

    Do you have any idea what this could be and how to fix it?

    Thanks in advance,
    Cheryl

    Michael Davis

    3 years, 12 months ago

    Rank Math pro

    Hello,

    Thank you for contacting Rank Math and sorry for any inconvenience that might have been caused due to that.

    In regards to the last error, this is likely related to a theme functionality that is trying to register widgets.
    The sitemap file is created dynamically and thus you won’t find it on the file system.
    ​​​​​​​
    We might need to take a closer look at the settings. Please edit the first post on this ticket and include your WP logins in the designated Sensitive Data section.
    Sensitive Data Section

    It is completely secure and only our support staff has access to that section. If you want, you can use the below plugin to generate a temporary login URL to your website and share that with us instead:

    https://wordpress.org/plugins/temporary-login-without-password/

    You can use the above plugin in conjunction with the WP Security Audit Log to monitor what changes our staff might make on your website (if any):

    https://wordpress.org/plugins/wp-security-audit-log/

    We really look forward to helping you.

    Cheryl

    3 years, 12 months ago

    Rank Math free

    Hello,

    I have updated the sensitive data as requested. Can you please check further?

    Thank you.

    Cheryl

    3 years, 12 months ago

    Rank Math free

    Hi Michael,

    I have updated the sensitive date with the login.

    Thanks a lot for looking into this!
    Let me know if you need anything further.

    Best regards,
    Cheryl

    Michael Davis

    3 years, 12 months ago

    Rank Math pro

    Hi Cheryl,

    Thanks for the reply.

    We are checking this further and we will be in touch as soon as we are done with the investigations.

    Your patience is highly appreciated. Thank you.

    Todd

    3 years, 12 months ago

    Rank Math free

    Hi,

    I am sorry but when we just enable the MyLisiting Child theme + Rank Math during troubleshooting mode – we can’t access the backend.

    There is some issue with you theme and we will need your FTP details to look further into this.

    Thank you.

    Cheryl

    3 years, 12 months ago

    Rank Math free

    Hello,

    I have updated the sensitive data as requested. Can you please check further?

    Thank you.

    Cheryl

    3 years, 12 months ago

    Rank Math free

    Hi Todd!

    Sorry about that! Last time I tried troubleshouting it seemed to work fine.
    I’d be really interested to hear if you spot anything strange. I have added SFTP details, if you want to take a look.

    Thank you,
    Cheryl

Viewing 15 replies - 1 through 15 (of 18 total)
1 2

The ticket ‘Which custom fields cause post to be included in sitemap?’ is closed to new replies.