-
Hey there!
I’ve recently had a mail from Google Search Console saying that some pages had not been indexed and to review my site.
When I was to the Robots testing tool https://www.google.com/webmasters/tools/robots-testing-tool I had the following message
You have a robots.txt file that we are currently unable to fetch. In such cases we stop crawling your site until we get hold of a robots.txt, or fall back to the last known good robots.txt file. Learn more.
The curious thing is that I do note recall being touching anything recently about that.
When I try to reach https://alexdepablos.com/robots.txt it is redirected to https://alexdepablos.com, but I do not have any redirection as far as I know in the .htaccess
# SGS XMLRPC Disable Service
<Files xmlrpc.php>
order deny,allow
deny from all
</Files>
# SGS XMLRPC Disable Service END# BEGIN W3TC Browser Cache
<IfModule mod_mime.c>
AddType text/css .css
AddType text/x-component .htc
AddType application/x-javascript .js
AddType application/javascript .js2
AddType text/javascript .js3
AddType text/x-js .js4
AddType video/asf .asf .asx .wax .wmv .wmx
AddType video/avi .avi
AddType image/avif .avif
AddType image/avif-sequence .avifs
AddType image/bmp .bmp
AddType application/java .class
AddType video/divx .divx
AddType application/msword .doc .docx
AddType application/vnd.ms-fontobject .eot
AddType application/x-msdownload .exe
AddType image/gif .gif
AddType application/x-gzip .gz .gzip
AddType image/x-icon .ico
AddType image/jpeg .jpg .jpeg .jpe
AddType image/webp .webp
AddType application/json .json
AddType application/vnd.ms-access .mdb
AddType audio/midi .mid .midi
AddType video/quicktime .mov .qt
AddType audio/mpeg .mp3 .m4a
AddType video/mp4 .mp4 .m4v
AddType video/mpeg .mpeg .mpg .mpe
AddType video/webm .webm
AddType application/vnd.ms-project .mpp
AddType application/x-font-otf .otf
AddType application/vnd.ms-opentype ._otf
AddType application/vnd.oasis.opendocument.database .odb
AddType application/vnd.oasis.opendocument.chart .odc
AddType application/vnd.oasis.opendocument.formula .odf
AddType application/vnd.oasis.opendocument.graphics .odg
AddType application/vnd.oasis.opendocument.presentation .odp
AddType application/vnd.oasis.opendocument.spreadsheet .ods
AddType application/vnd.oasis.opendocument.text .odt
AddType audio/ogg .ogg
AddType video/ogg .ogv
AddType application/pdf .pdf
AddType image/png .png
AddType application/vnd.ms-powerpoint .pot .pps .ppt .pptx
AddType audio/x-realaudio .ra .ram
AddType image/svg+xml .svg .svgz
AddType application/x-shockwave-flash .swf
AddType application/x-tar .tar
AddType image/tiff .tif .tiff
AddType application/x-font-ttf .ttf .ttc
AddType application/vnd.ms-opentype ._ttf
AddType audio/wav .wav
AddType audio/wma .wma
AddType application/vnd.ms-write .wri
AddType application/font-woff .woff
AddType application/font-woff2 .woff2
AddType application/vnd.ms-excel .xla .xls .xlsx .xlt .xlw
AddType application/zip .zip
</IfModule>
<IfModule mod_expires.c>
ExpiresActive On
ExpiresByType text/css A31536000
ExpiresByType text/x-component A31536000
ExpiresByType application/x-javascript A31536000
ExpiresByType application/javascript A31536000
ExpiresByType text/javascript A31536000
ExpiresByType text/x-js A31536000
ExpiresByType video/asf A31536000
ExpiresByType video/avi A31536000
ExpiresByType image/avif A31536000
ExpiresByType image/avif-sequence A31536000
ExpiresByType image/bmp A31536000
ExpiresByType application/java A31536000
ExpiresByType video/divx A31536000
ExpiresByType application/msword A31536000
ExpiresByType application/vnd.ms-fontobject A31536000
ExpiresByType application/x-msdownload A31536000
ExpiresByType image/gif A31536000
ExpiresByType application/x-gzip A31536000
ExpiresByType image/x-icon A31536000
ExpiresByType image/jpeg A31536000
ExpiresByType image/webp A31536000
ExpiresByType application/json A31536000
ExpiresByType application/vnd.ms-access A31536000
ExpiresByType audio/midi A31536000
ExpiresByType video/quicktime A31536000
ExpiresByType audio/mpeg A31536000
ExpiresByType video/mp4 A31536000
ExpiresByType video/mpeg A31536000
ExpiresByType video/webm A31536000
ExpiresByType application/vnd.ms-project A31536000
ExpiresByType application/x-font-otf A31536000
ExpiresByType application/vnd.ms-opentype A31536000
ExpiresByType application/vnd.oasis.opendocument.database A31536000
ExpiresByType application/vnd.oasis.opendocument.chart A31536000
ExpiresByType application/vnd.oasis.opendocument.formula A31536000
ExpiresByType application/vnd.oasis.opendocument.graphics A31536000
ExpiresByType application/vnd.oasis.opendocument.presentation A31536000
ExpiresByType application/vnd.oasis.opendocument.spreadsheet A31536000
ExpiresByType application/vnd.oasis.opendocument.text A31536000
ExpiresByType audio/ogg A31536000
ExpiresByType video/ogg A31536000
ExpiresByType application/pdf A31536000
ExpiresByType image/png A31536000
ExpiresByType application/vnd.ms-powerpoint A31536000
ExpiresByType audio/x-realaudio A31536000
ExpiresByType image/svg+xml A31536000
ExpiresByType application/x-shockwave-flash A31536000
ExpiresByType application/x-tar A31536000
ExpiresByType image/tiff A31536000
ExpiresByType application/x-font-ttf A31536000
ExpiresByType application/vnd.ms-opentype A31536000
ExpiresByType audio/wav A31536000
ExpiresByType audio/wma A31536000
ExpiresByType application/vnd.ms-write A31536000
ExpiresByType application/font-woff A31536000
ExpiresByType application/font-woff2 A31536000
ExpiresByType application/vnd.ms-excel A31536000
ExpiresByType application/zip A31536000
</IfModule>
<IfModule mod_deflate.c>
AddOutputFilterByType DEFLATE text/css text/x-component application/x-javascript application/javascript text/javascript text/x-js text/html text/richtext text/plain text/xsd text/xsl text/xml image/bmp application/java application/msword application/vnd.ms-fontobject application/x-msdownload image/x-icon application/json application/vnd.ms-access video/webm application/vnd.ms-project application/x-font-otf application/vnd.ms-opentype application/vnd.oasis.opendocument.database application/vnd.oasis.opendocument.chart application/vnd.oasis.opendocument.formula application/vnd.oasis.opendocument.graphics application/vnd.oasis.opendocument.presentation application/vnd.oasis.opendocument.spreadsheet application/vnd.oasis.opendocument.text audio/ogg application/pdf application/vnd.ms-powerpoint image/svg+xml application/x-shockwave-flash image/tiff application/x-font-ttf application/vnd.ms-opentype audio/wav application/vnd.ms-write application/font-woff application/font-woff2 application/vnd.ms-excel
<IfModule mod_mime.c>
# DEFLATE by extension
AddOutputFilter DEFLATE js css htm html xml
</IfModule>
</IfModule>
<FilesMatch "\.(css|htc|less|js|js2|js3|js4|CSS|HTC|LESS|JS|JS2|JS3|JS4)$">
FileETag MTime Size
<IfModule mod_headers.c>
Header set Pragma "public"
Header append Cache-Control "public"
Header unset Set-Cookie
</IfModule>
</FilesMatch>
<FilesMatch "\.(html|htm|rtf|rtx|txt|xsd|xsl|xml|HTML|HTM|RTF|RTX|TXT|XSD|XSL|XML)$">
FileETag MTime Size
<IfModule mod_headers.c>
Header set Pragma "public"
Header set Cache-Control "max-age=3600, public"
</IfModule>
</FilesMatch>
<FilesMatch "\.(asf|asx|wax|wmv|wmx|avi|avif|avifs|bmp|class|divx|doc|docx|eot|exe|gif|gz|gzip|ico|jpg|jpeg|jpe|webp|json|mdb|mid|midi|mov|qt|mp3|m4a|mp4|m4v|mpeg|mpg|mpe|webm|mpp|otf|_otf|odb|odc|odf|odg|odp|ods|odt|ogg|ogv|pdf|png|pot|pps|ppt|pptx|ra|ram|svg|svgz|swf|tar|tif|tiff|ttf|ttc|_ttf|wav|wma|wri|woff|woff2|xla|xls|xlsx|xlt|xlw|zip|ASF|ASX|WAX|WMV|WMX|AVI|AVIF|AVIFS|BMP|CLASS|DIVX|DOC|DOCX|EOT|EXE|GIF|GZ|GZIP|ICO|JPG|JPEG|JPE|WEBP|JSON|MDB|MID|MIDI|MOV|QT|MP3|M4A|MP4|M4V|MPEG|MPG|MPE|WEBM|MPP|OTF|_OTF|ODB|ODC|ODF|ODG|ODP|ODS|ODT|OGG|OGV|PDF|PNG|POT|PPS|PPT|PPTX|RA|RAM|SVG|SVGZ|SWF|TAR|TIF|TIFF|TTF|TTC|_TTF|WAV|WMA|WRI|WOFF|WOFF2|XLA|XLS|XLSX|XLT|XLW|ZIP)$">
FileETag MTime Size
<IfModule mod_headers.c>
Header set Pragma "public"
Header append Cache-Control "public"
Header unset Set-Cookie
</IfModule>
</FilesMatch>
<IfModule mod_headers.c>
Header set Referrer-Policy "no-referrer-when-downgrade"
</IfModule>
# END W3TC Browser Cache
# BEGIN W3TC Page Cache core
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteCond %{HTTPS} =on
RewriteRule .* - [E=W3TC_SSL:_ssl]
RewriteCond %{SERVER_PORT} =443
RewriteRule .* - [E=W3TC_SSL:_ssl]
RewriteCond %{HTTP:X-Forwarded-Proto} =https [NC]
RewriteRule .* - [E=W3TC_SSL:_ssl]
RewriteCond %{HTTP:Accept-Encoding} gzip
RewriteRule .* - [E=W3TC_ENC:_gzip]
RewriteCond %{HTTP_COOKIE} w3tc_preview [NC]
RewriteRule .* - [E=W3TC_PREVIEW:_preview]
RewriteCond %{REQUEST_URI} \/$
RewriteRule .* - [E=W3TC_SLASH:_slash]
RewriteCond %{REQUEST_METHOD} !=POST
RewriteCond %{QUERY_STRING} =""
RewriteCond %{HTTP_COOKIE} !(comment_author|wp\-postpass|w3tc_logged_out|wordpress_logged_in|wptouch_switch_toggle) [NC]
RewriteCond "%{DOCUMENT_ROOT}/wp-content/cache/page_enhanced/%{HTTP_HOST}/%{REQUEST_URI}/_index%{ENV:W3TC_SLASH}%{ENV:W3TC_SSL}%{ENV:W3TC_PREVIEW}.html%{ENV:W3TC_ENC}" -f
RewriteRule .* "/wp-content/cache/page_enhanced/%{HTTP_HOST}/%{REQUEST_URI}/_index%{ENV:W3TC_SLASH}%{ENV:W3TC_SSL}%{ENV:W3TC_PREVIEW}.html%{ENV:W3TC_ENC}" [L]
</IfModule>
# END W3TC Page Cache core
# BEGIN WordPress
# The directives (lines) between "BEGIN WordPress" and "END WordPress" are
# dynamically generated, and should only be modified via WordPress filters.
# Any changes to the directives between these markers will be overwritten.
RewriteEngine On
RewriteRule .* - [E=HTTP_AUTHORIZATION:%{HTTP:Authorization}]
RewriteBase /
RewriteRule ^index\.php$ - [L]# add a trailing slash to /wp-admin
RewriteRule ^([_0-9a-zA-Z-]+/)?wp-admin$ $1wp-admin/ [R=301,L]RewriteCond %{REQUEST_FILENAME} -f [OR]
RewriteCond %{REQUEST_FILENAME} -d
RewriteRule ^ - [L]
RewriteRule ^([_0-9a-zA-Z-]+/)?(wp-(content|admin|includes).*) $2 [L]
RewriteRule ^([_0-9a-zA-Z-]+/)?(.*\.php)$ $2 [L]
RewriteRule . index.php [L]# END WordPress
# SGO Unset Vary
Header unset Vary
# SGO Unset Vary END# Alex's changes
Redirect /twitter https://twitter.com/adpablos
Redirect /linkedin https://www.linkedin.com/in/alejandrodepablos/
Redirect /instagram https://www.instagram.com/adpabloslopez/
RedirectMatch 301 /en/how-to-notify-with-a-whatsapp-a-calendar-event/ /en/technology/whatsapp-notification-from-google-calendar/
RedirectMatch 301 /category/tecnologia/ /tecnologia/
RedirectMatch 301 /category/ingenieria-de-software/ /ingenieria-de-software/# END Alex's changes
So… I do not know what has happened.
I want and look into my FTP and I did not see any robots.txt in the public_html folder… Do not know if that is normal or not. I have the Edit robots.txt menu in Rank Math with the default settings.
Could you help me with that?
Thanks so much!
-
Hello,
Thank you for contacting Rank Math, and sorry for the delay and inconvenience caused.
It looks like the robots.txt file is cached:
Can you please clear the cache from your CDN and your website cache and check again?
Let us know how it goes.
We are here to assist you.
The ticket ‘Robots.txt’ is closed to new replies.