ray/doc/source/robots.txt

# Algolia-Crawler-Verif: C37A60AFB138256B
User-agent: *
Allow: /
Allow: /*/latest/
Allow: /en/latest/   # Fallback for bots that don't understand wildcards
Allow: /*/master/
Allow: /en/master/   # Fallback for bots that don't understand wildcards
Sitemap: https://docs.ray.io/en/latest/sitemap.xml
[docs] add algolia crawler verification (#28094) 2022-08-25 10:36:26 +02:00			`# Algolia-Crawler-Verif: C37A60AFB138256B`
[docs] add robots txt (#24726) as per discussion on slack, this should avoid having old versions of the docs indexed by search engines, see readthedocs/readthedocs.org#2430 for reference. Signed-off-by: Max Pumperla <max.pumperla@googlemail.com> 2022-05-18 11:52:22 +02:00			`User-agent: *`
[docs] add algolia crawler verification (#28094) 2022-08-25 10:36:26 +02:00			`Allow: /`
[docs] add robots txt (#24726) as per discussion on slack, this should avoid having old versions of the docs indexed by search engines, see readthedocs/readthedocs.org#2430 for reference. Signed-off-by: Max Pumperla <max.pumperla@googlemail.com> 2022-05-18 11:52:22 +02:00			`Allow: /*/latest/`
			`Allow: /en/latest/ # Fallback for bots that don't understand wildcards`
			`Allow: /*/master/`
			`Allow: /en/master/ # Fallback for bots that don't understand wildcards`
[docs] add algolia crawler verification (#28094) 2022-08-25 10:36:26 +02:00			`Sitemap: https://docs.ray.io/en/latest/sitemap.xml`