Optimizing Sitemap Performance a sitemap with around 20,000 pages

#988830
  • Abhishek

    1 week, 5 days ago

    Rank Math free

    Question: Optimizing Sitemap Performance for a Large WordPress Site on AWS EC2 with Bitnami
    I am hosting a WordPress site on an AWS EC2 instance using a Bitnami stack. The database is hosted on Amazon RDS, and Redis is used as a caching layer along with the WP-Optimizer plugin. The site currently has a sitemap with around 20,000 pages, which is expected to grow to 100,000 pages.

    The issue is that the sitemap and wp-admin pages are loading very slowly or eventually showing a 500 Internal Server Error. However, all other pages on the site are loading smoothly without issues. The server has enough memory and storage available, and the following configurations are in place:

    Caching: Redis and WP-Optimizer are enabled.
    Plugins: Rank Math Pro is used for generating the sitemap.
    Logs: Apache error logs show warnings like:
    Constant WP_MEMORY_LIMIT already defined
    Undefined array keys “find” and “replace” in the Rank Math Pro plugin files.
    Timeout errors (AH01075).
    Server: EC2 instance with sufficient CPU, RAM, and disk space.
    PHP-FPM: Default timeout settings.
    Here’s what I’ve tried so far:

    Optimized the wp-config.php file to remove duplicate constants like WP_MEMORY_LIMIT.
    Updated all plugins, including Rank Math Pro, to the latest versions.
    Increased PHP-FPM timeout in the configuration file.
    Ensured the server resources are sufficient.
    Despite these efforts, the sitemap still takes too long to load, and wp-admin occasionally shows internal server errors.

    Questions:

    How can I optimize the sitemap generation and loading process for such a large number of pages (20,000 to 100,000)?
    Are there specific configurations or adjustments needed for Rank Math Pro to handle large sitemaps efficiently?
    Is there any additional tuning required for Redis, Apache, or PHP-FPM to address these issues?
    Would splitting the sitemap into smaller chunks or switching to a different sitemap generator plugin help resolve this problem?
    Are there any known issues with large sitemaps on Bitnami stacks, and how can they be mitigated?
    Any guidance on resolving these performance issues would be greatly appreciated. Thank you!

Viewing 4 replies - 1 through 4 (of 4 total)
  • RankBot

    1 week, 5 days ago

    Hello,

    We apologize for the issues you are facing with your sitemap and wp-admin pages. Given the complexity of your setup with a large number of pages, here are some suggestions to optimize the sitemap generation and loading process:

    1. Optimize Sitemap Generation

    • Chunking the Sitemap: Consider splitting your sitemap into smaller chunks. Rank Math allows you to set the number of links per sitemap. You can adjust this in the Rank Math SEO → Sitemap Settings section. This can help reduce the load time for each individual sitemap file.

    2. Rank Math Pro Configurations

    • Disable Caching for Sitemaps: If you have caching enabled for your sitemaps, it might cause issues with dynamic generation. You can disable caching for sitemaps by adding the following filter in your theme’s rank-math.php file: 
      add_filter( 'rank_math/sitemap/enable_caching', '__return_false' );
    • Check for Plugin Conflicts: Ensure that there are no conflicts with other plugins that might be affecting the sitemap generation. You can use the Health Check plugin to troubleshoot this.

    3. Server Configuration Adjustments

    • Increase PHP-FPM Timeout: Since you are experiencing timeout errors, consider increasing the PHP-FPM timeout settings further. You can adjust the request_terminate_timeout and request_slowlog_timeout settings in your PHP-FPM configuration.
    • Apache Configuration: Ensure that your Apache configuration is optimized for handling large requests. You may want to increase the Timeout directive in your Apache configuration file.

    4. Redis and Caching Layer Tuning

    • Redis Configuration: Make sure that Redis is properly configured to handle large datasets. You might want to check the Redis memory limit and eviction policy to ensure it can handle the cache effectively.

    5. Monitoring and Logs

    • Check Logs for Errors: Continue monitoring your Apache error logs for any additional warnings or errors that could provide insight into what might be causing the slow performance or internal server errors.

    If these adjustments do not resolve the issues, please reply to this ticket with your site URL so we can assist you further. A human agent from our support team will take over and provide more tailored assistance based on your specific setup.

    Thank you for raising this concern, and we hope to help you resolve it soon!

    Sources:
    Rank Math Sitemap Issues and Their Fixes » Rank Math

    This is an automated reply from Rank Math Support Bot. Your ticket has been assigned to a human agent and we will get back to you shortly.

    Jeremy

    1 week, 1 day ago

    Hello,

    Thank you so much for getting in touch.

    For large sitemaps, we recommend reducing the number of links per sitemap in Rank Math → Sitemap Settings → General Settings → Links per Sitemap to improve performance. This helps avoid timeouts and slow loading.

    Additionally, consider increasing PHP timeout values and server resources (CPU/RAM) to handle sitemap generation more efficiently. Since you’re using Redis, ensure it’s properly configured and not overloaded.

    If issues persist, you can check for any conflicts with WP-Optimizer caching and exclude the sitemap from caching to prevent delays.

    Don’t hesitate to get in touch with us if you have any other questions.

    Abhishek

    1 day, 14 hours ago

    Rank Math free

    I still have the issue. A white page is showing when reloading the page. Why can’t you solve a single issue?

    Jeremy

    17 hours, 45 minutes ago

    Hello,

    We checked your sitemap page and it is showing a Database Error:

    Rank Math support

    Can you please check this with your hosting provider first?

    Looking forward to helping you.

Viewing 4 replies - 1 through 4 (of 4 total)

You must be logged in to reply to this ticket.