The documentation improvement project has been discussing several aspects of search engine optimizations recently to help users find the the docs they need more efficiently (e.g. not being directed by a search engine to old versions of the docs). We've come up with a multi-part approach which we'd like to form consensus on and implement soon: 1) using canonical URLs This is metadata that we'll add to every page's header with Sphinx's help which hints to search engines about where to find the latest / canonical version of page. For example: The canonical URL for https://docs.ceph.com/docs/luminous/something would be https://docs.ceph.com/something OR https://docs.ceph.com/latest/something. See next item. 2) URL changes To support the canonical URLs we're proposing a change the URL structure such that accessing https://docs.ceph.com will redirect to https://docs.ceph.com/latest which would currently display `mimic` and soon `nautilus` while retaining also the explicit versioned URLs like `/mimic/`. We've been looking at the build infrastructure and have come up with an approach to automate these links using the new machine-readable release information. 3) robots.txt We'll explicitly instruct search engines to not index (a) wip branches and (b) releases below a certain threshold (e.g. < jewel).
