ray/doc/source/robots.txt

User-agent: *
Allow: /*/latest/
Allow: /en/latest/   # Fallback for bots that don't understand wildcards
Allow: /*/master/
Allow: /en/master/   # Fallback for bots that don't understand wildcards
Disallow: /
[docs] add robots txt (#24726) as per discussion on slack, this should avoid having old versions of the docs indexed by search engines, see readthedocs/readthedocs.org#2430 for reference. Signed-off-by: Max Pumperla <max.pumperla@googlemail.com> 2022-05-18 11:52:22 +02:00			`User-agent: *`
			`Allow: /*/latest/`
			`Allow: /en/latest/ # Fallback for bots that don't understand wildcards`
			`Allow: /*/master/`
			`Allow: /en/master/ # Fallback for bots that don't understand wildcards`
			`Disallow: /`