ray/doc/source/ray-contribute/docs.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "8c7dab40",
   "metadata": {},
   "source": [
    "(docs-contribute)=\n",
    "\n",
    "# Contributing to the Ray Documentation\n",
    "\n",
    "There are many ways to contribute to the Ray documentation, and we're always looking for new contributors.\n",
    "Even if you just want to fix a typo or expand on a section, please feel free to do so!\n",
    "\n",
    "This document walks you through everything you need to do to get started.\n",
    "\n",
    "## Building the Ray documentation\n",
    "\n",
    "If you want to contribute to the Ray documentation, you'll need a way to build it.\n",
    "You don't have to build Ray itself, which is a bit more involved.\n",
    "\n",
    "You just have to [clone the Ray repository](https://docs.ray.io/en/master/ray-contribute/development.html#clone-the-repository), then [prepare the Python environment](https://docs.ray.io/en/master/ray-contribute/development.html#prepare-the-python-environment).\n",
    "\n",
    "Next, change into the `ray/doc` directory:\n",
    "\n",
    "```shell\n",
    "cd ray/doc\n",
    "```\n",
    "\n",
    "**Note**: If you are on an Apple Silicon (M1) read the instructions below for installing the dependencies.\n",
    "\n",
    "Make sure you activate the Python environment you are using (e.g. venv, conda, etc.) and then to install the documentation dependencies, run the following command:\n",
    "\n",
    "```shell\n",
    "pip install -r requirements-doc.txt\n",
    "```\n",
    "\n",
    "Additionally, it's best if you install the dependencies for our linters with\n",
    "\n",
    "```shell\n",
    "pip install -r ../python/requirements_linters.txt\n",
    "```\n",
    "\n",
    "so that you can make sure your changes comply with our style guide.\n",
    "Building the documentation is done by running the following command:\n",
    "\n",
    "```shell\n",
    "make develop\n",
    "```\n",
    "\n",
    "which will build the documentation into the `_build` directory.\n",
    "After the build finishes, you can simply open the `_build/html/index.html` file in your browser.\n",
    "It's considered good practice to check the output of your build to make sure everything is working as expected.\n",
    "\n",
    "Before committing any changes, make sure you run the \n",
    "[linter](https://docs.ray.io/en/latest/ray-contribute/getting-involved.html#lint-and-formatting)\n",
    "with `../scripts/format.sh` from the `doc` folder,\n",
    "to make sure your changes are formatted correctly.\n",
    "\n",
    "For reproducing CI build failures locally, you might want to use `make html`, which\n",
    "is the same as `make develop` but treats warnings as errors.\n",
    "\n",
    "## Building Docs for Apple Silicon (M1)\n",
    "\n",
    "If you are using an Apple Silicon (M1) some of the dependencies required for building the docs don't have binary packages available by default (not available in PyPI).\n",
    "\n",
    "The simplest way to install those dependencies is with `conda` (https://docs.conda.io/en/latest/miniconda.html) first, that way `pip` won't try to install them by building them from scratch.\n",
    "\n",
    "To do that, make sure you create and/or activate the conda environment, and then install the dependencies with:\n",
    "\n",
    "\n",
    "```shell\n",
    "conda install -c conda-forge xgboost lightgbm\n",
    "```\n",
    "\n",
    "After that you can install the other dependencies as described above.\n",
    "\n",
    "## The basics of our build system\n",
    "\n",
    "The Ray documentation is built using the [`sphinx`](https://www.sphinx-doc.org/) build system.\n",
    "We're using the [Sphinx Book Theme](https://github.com/executablebooks/sphinx-book-theme) from the\n",
    "[executable books project](https://github.com/executablebooks).\n",
    "\n",
    "That means that you can write Ray documentation in either Sphinx's native \n",
    "[reStructuredText (rST)](https://www.sphinx-doc.org/en/master/usage/restructuredtext/index.html) or in\n",
    "[Markedly Structured Text (MyST)](https://myst-parser.readthedocs.io/en/latest/).\n",
    "The two formats can be converted to each other, so the choice is up to you.\n",
    "Having said that, it's important to know that MyST is\n",
    "[common markdown compliant](https://myst-parser.readthedocs.io/en/latest/syntax/reference.html#commonmark-block-tokens).\n",
    "If you intend to add a new document, we recommend starting from an `.md` file.\n",
    "\n",
    "The Ray documentation also fully supports executable formats like [Jupyter Notebooks](https://jupyter.org/).\n",
    "Many of our examples are notebooks with [MyST markdown cells](https://myst-nb.readthedocs.io/en/latest/index.html).\n",
    "In fact, this very document you're reading _is_ a notebook.\n",
    "You can check this for yourself by either downloading the `.ipynb` file,\n",
    "or directly launching this notebook into either Binder or Google Colab in the top navigation bar.\n",
    "\n",
    "## What to contribute?\n",
    "\n",
    "If you take Ray Tune as an example, you can see that our documentation is made up of several types of documentation,\n",
    "all of which you can contribute to:\n",
    "\n",
    "- [a project landing page](https://docs.ray.io/en/latest/tune/index.html),\n",
    "- [a getting started guide](https://docs.ray.io/en/latest/tune/getting-started.html),\n",
    "- [a key concepts page](https://docs.ray.io/en/latest/tune/key-concepts.html),\n",
    "- [user guides for key features](https://docs.ray.io/en/latest/tune/tutorials/overview.html),\n",
    "- [practical examples](https://docs.ray.io/en/latest/tune/examples/index.html),\n",
    "- [a detailed FAQ](https://docs.ray.io/en/latest/tune/faq.html),\n",
    "- [and API references](https://docs.ray.io/en/latest/tune/api_docs/overview.html).\n",
    "\n",
    "This structure is reflected in the\n",
    "[Ray documentation source code](https://github.com/ray-project/ray/tree/master/doc/source/tune) as well, so you\n",
    "should have no problem finding what you're looking for.\n",
    "All other Ray projects share a similar structure, but depending on the project there might be minor differences.\n",
    "\n",
    "Each type of documentation listed above has its own purpose, but at the end our documentation\n",
    "comes down to _two types_ of documents:\n",
    "\n",
    "- Markup documents, written in MyST or rST. If you don't have a lot of (executable) code to contribute or\n",
    "  use more complex features such as\n",
    "  [tabbed content blocks](https://docs.ray.io/en/latest/ray-core/walkthrough.html#starting-ray), this is the right\n",
    "  choice. Most of the documents in Ray Tune are written in this way, for instance the\n",
    "  [key concepts](https://github.com/ray-project/ray/blob/master/doc/source/tune/key-concepts.rst) or\n",
    "  [API documentation](https://github.com/ray-project/ray/blob/master/doc/source/tune/api_docs/overview.rst).\n",
    "- Notebooks, written in `.ipynb` format. All Tune examples are written as notebooks. These notebooks render in\n",
    "  the browser like `.md` or `.rst` files, but have the added benefit of adding launch buttons to the top of the\n",
    "  document, so that users can run the code themselves in either Binder or Google Colab. A good first example to look\n",
    "  at is [this Tune example](https://github.com/ray-project/ray/blob/master/doc/source/tune/examples/tune-serve-integration-mnist.ipynb).\n",
    "\n",
    "## Fixing typos and improving explanations\n",
    "\n",
    "If you spot a typo in any document, or think that an explanation is not clear enough, please consider\n",
    "opening a pull request.\n",
    "In this scenario, just run the linter as described above and submit your pull request.\n",
    "\n",
    "## Adding API references\n",
    "\n",
    "We use [Sphinx's autodoc extension](https://www.sphinx-doc.org/en/master/usage/extensions/autodoc.html) to generate\n",
    "our API documentation from our source code.\n",
    "In case we're missing a reference to a function or class, please consider adding it to the respective document in question.\n",
    "\n",
    "For example, here's how you can add a function or class reference using `autofunction` and `autoclass`:\n",
    "\n",
    "```markdown\n",
    ".. autofunction:: ray.tune.integration.docker.DockerSyncer\n",
    "\n",
    ".. autoclass:: ray.tune.integration.keras.TuneReportCallback\n",
    "```\n",
    "\n",
    "The above snippet was taken from the\n",
    "[Tune API documentation](https://github.com/ray-project/ray/blob/master/doc/source/tune/api_docs/integration.rst),\n",
    "which you can look at for reference.\n",
    "\n",
    "If you want to change the content of the API documentation, you will have to edit the respective function or class\n",
    "signatures directly in the source code.\n",
    "For example, in the above `autofunction` call, to change the API reference for `ray.tune.integration.docker.DockerSyncer`,\n",
    "you would have to [change the following source file](https://github.com/ray-project/ray/blob/7f1bacc7dc9caf6d0ec042e39499bbf1d9a7d065/python/ray/tune/integration/docker.py#L15-L38).\n",
    "\n",
    "## Adding code to an `.rST` or `.md` file\n",
    "\n",
    "Modifying text in an existing documentation file is easy, but you need to be careful when it comes to adding code.\n",
    "The reason is that we want to ensure every code snippet on our documentation is tested.\n",
    "This requires us to have a process for including and testing code snippets in documents.\n",
    "\n",
    "In an `.rST` or `.md` file, you can add code snippets using `literalinclude` from the Sphinx system.\n",
    "For instance, here's an example from the Tune's \"Key Concepts\" documentation: \n",
    "\n",
    "```markdown\n",
    ".. literalinclude:: doc_code/key_concepts.py\n",
    "    :language: python\n",
    "    :start-after: __function_api_start__\n",
    "    :end-before: __function_api_end__\n",
    "```\n",
    "\n",
    "Note that in the whole file there's not a single literal code block, code _has to be_ imported using the `literalinclude` directive.\n",
    "The code that gets added to the document by `literalinclude`, including `start-after` and `end-before` tags,\n",
    "reads as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ba88d95f",
   "metadata": {
    "vscode": {
     "languageId": "python"
    }
   },
   "outputs": [],
   "source": [
    "# __function_api_start__\n",
    "from ray.air import session\n",
    "\n",
    "\n",
    "def objective(x, a, b):  # Define an objective function.\n",
    "    return a * (x ** 0.5) + b\n",
    "\n",
    "\n",
    "def trainable(config):  # Pass a \"config\" dictionary into your trainable.\n",
    "\n",
    "    for x in range(20):  # \"Train\" for 20 iterations and compute intermediate scores.\n",
    "        score = objective(x, config[\"a\"], config[\"b\"])\n",
    "\n",
    "        session.report({\"score\": score})  # Send the score to Tune.\n",
    "\n",
    "\n",
    "# __function_api_end__"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6971eeb0",
   "metadata": {},
   "source": [
    "This code is imported by `literalinclude` from a file called `doc_code/key_concepts.py`.\n",
    "Every Python file in the `doc_code` directory will automatically get tested by our CI system,\n",
    "but make sure to run scripts that you change (or new scripts) locally first.\n",
    "You do not need to run the testing framework locally.\n",
    "\n",
    "In rare situations, when you're adding _obvious_ pseudo-code to demonstrate a concept, it is ok to add it\n",
    "literally into your `.rST` or `.md` file, e.g. using a `.. code-cell:: python` directive.\n",
    "But if your code is supposed to run, it needs to be tested.\n",
    "\n",
    "## Creating a new document from scratch\n",
    "\n",
    "Sometimes you might want to add a completely new document to the Ray documentation, like adding a new\n",
    "user guide or a new example.\n",
    "\n",
    "For this to work, you need to make sure to add the new document explicitly to the \n",
    "[`_toc.yml` file](https://github.com/ray-project/ray/blob/master/doc/source/_toc.yml) that determines\n",
    "the structure of the Ray documentation.\n",
    "\n",
    "Depending on the type of document you're adding, you might also have to make changes to an existing overview\n",
    "page that curates the list of documents in question.\n",
    "For instance, for Ray Tune each user guide is added to the\n",
    "[user guide overview page](https://docs.ray.io/en/latest/tune/tutorials/overview.html) as a panel, and the same\n",
    "goes for [all Tune examples](https://docs.ray.io/en/latest/tune/examples/index.html).\n",
    "Always check the structure of the Ray sub-project whose documentation you're working on to see how to integrate\n",
    "it within the existing structure.\n",
    "In some cases you may be required to choose an image for the panel. Images are located in\n",
    "`doc/source/images`. \n",
    "\n",
    "## Creating a notebook example\n",
    "\n",
    "To add a new executable example to the Ray documentation, you can start from our\n",
    "[MyST notebook template](https://github.com/ray-project/ray/tree/master/doc/source/_templates/template.md) or\n",
    "[Jupyter notebook template](https://github.com/ray-project/ray/tree/master/doc/source/_templates/template.ipynb).\n",
    "You could also simply download the document you're reading right now (click on the respective download button at the\n",
    "top of this page to get the `.ipynb` file) and start modifying it.\n",
    "All the example notebooks in Ray Tune get automatically tested by our CI system, provided you place them in the\n",
    "[`examples` folder](https://github.com/ray-project/ray/tree/master/doc/source/tune/examples).\n",
    "If you have questions about how to test your notebook when contributing to other Ray sub-projects, please make\n",
    "sure to ask a question in [the Ray community Slack](https://forms.gle/9TSdDYUgxYs8SA9e8) or directly on GitHub,\n",
    "when opening your pull request.\n",
    "\n",
    "To work off of an existing example, you could also have a look at the\n",
    "[Ray Tune Hyperopt example (`.ipynb`)](https://github.com/ray-project/ray/blob/master/doc/source/tune/examples/hyperopt_example.ipynb)\n",
    "or the [Ray Serve guide for RLlib (`.md`)](https://github.com/ray-project/ray/blob/master/doc/source/serve/tutorials/rllib.md).\n",
    "We recommend that you start with an `.md` file and convert your file to an `.ipynb` notebook at the end of the process.\n",
    "We'll walk you through this process below.\n",
    "\n",
    "What makes these notebooks different from other documents is that they combine code and text in one document,\n",
    "and can be launched in the browser.\n",
    "We also make sure they are tested by our CI system, before we add them to our documentation.\n",
    "To make this work, notebooks need to define a _kernel specification_ to tell a notebook server how to interpret\n",
    "and run the code.\n",
    "For instance, here's the kernel specification of a Python notebook:\n",
    "\n",
    "```markdown\n",
    "---\n",
    "jupytext:\n",
    "    text_representation:\n",
    "        extension: .md\n",
    "        format_name: myst\n",
    "kernelspec:\n",
    "    display_name: Python 3\n",
    "    language: python\n",
    "    name: python3\n",
    "---\n",
    "```\n",
    "\n",
    "If you write a notebook in `.md` format, you need this YAML front matter at the top of the file.\n",
    "To add code to your notebook, you can use the `code-cell` directive.\n",
    "Here's an example:\n",
    "\n",
    "````markdown\n",
    "```{code-cell} python3\n",
    ":tags: [hide-cell]\n",
    "\n",
    "import ray\n",
    "import ray.rllib.agents.ppo as ppo\n",
    "from ray import serve\n",
    "\n",
    "def train_ppo_model():\n",
    "    trainer = ppo.PPOTrainer(\n",
    "        config={\"framework\": \"torch\", \"num_workers\": 0},\n",
    "        env=\"CartPole-v0\",\n",
    "    )\n",
    "    # Train for one iteration\n",
    "    trainer.train()\n",
    "    trainer.save(\"/tmp/rllib_checkpoint\")\n",
    "    return \"/tmp/rllib_checkpoint/checkpoint_000001/checkpoint-1\"\n",
    "\n",
    "\n",
    "checkpoint_path = train_ppo_model()\n",
    "```\n",
    "````\n",
    "\n",
    "Putting this markdown block into your document will render as follows in the browser:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "78cac353",
   "metadata": {
    "tags": [
     "hide-cell"
    ],
    "vscode": {
     "languageId": "python"
    }
   },
   "outputs": [],
   "source": [
    "import ray\n",
    "import ray.rllib.agents.ppo as ppo\n",
    "from ray import serve\n",
    "\n",
    "def train_ppo_model():\n",
    "    trainer = ppo.PPOTrainer(\n",
    "        config={\"framework\": \"torch\", \"num_workers\": 0},\n",
    "        env=\"CartPole-v0\",\n",
    "    )\n",
    "    # Train for one iteration\n",
    "    trainer.train()\n",
    "    trainer.save(\"/tmp/rllib_checkpoint\")\n",
    "    return \"/tmp/rllib_checkpoint/checkpoint_000001/checkpoint-1\"\n",
    "\n",
    "\n",
    "checkpoint_path = train_ppo_model()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d716d0bd",
   "metadata": {},
   "source": [
    "As you can see, the code block is hidden, but you can expand it by click on the \"+\" button.\n",
    "\n",
    "### Tags for your notebook\n",
    "\n",
    "What makes this work is the `:tags: [hide-cell]` directive in the `code-cell`.\n",
    "The reason we suggest starting with `.md` files is that it's much easier to add tags to them, as you've just seen.\n",
    "You can also add tags to `.ipynb` files, but you'll need to start a notebook server for that first, which may\n",
    "not want to do to contribute a piece of documentation.\n",
    "\n",
    "Apart from `hide-cell`, you also have `hide-input` and `hide-output` tags that hide the input and output of a cell.\n",
    "Also, if you need code that gets executed in the notebook, but you don't want to show it in the documentation,\n",
    "you can use the `remove-cell`, `remove-input`, and `remove-output` tags in the same way.\n",
    "\n",
    "### Testing notebooks\n",
    "\n",
    "Removing cells can be particularly interesting for compute-intensive notebooks.\n",
    "We want you to contribute notebooks that use _realistic_ values, not just toy examples.\n",
    "At the same time we want our notebooks to be tested by our CI system, and running them should not take too long.\n",
    "What you can do to address this is to have notebook cells with the parameters you want the users to see first:\n",
    "\n",
    "````markdown\n",
    "```{code-cell} python3\n",
    "num_workers = 8\n",
    "num_gpus = 2\n",
    "```\n",
    "````\n",
    "\n",
    "which will render as follows in the browser:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8412103e",
   "metadata": {
    "vscode": {
     "languageId": "python"
    }
   },
   "outputs": [],
   "source": [
    "num_workers = 8\n",
    "num_gpus = 2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1d8cc54e",
   "metadata": {},
   "source": [
    "But then in your notebook you follow that up with a _removed_ cell that won't get rendered, but has much smaller\n",
    "values and make the notebook run faster:\n",
    "\n",
    "````markdown\n",
    "```{code-cell} python3\n",
    ":tags: [remove-cell]\n",
    "num_workers = 0\n",
    "num_gpus = 0\n",
    "```\n",
    "````\n",
    "\n",
    "### Converting markdown notebooks to ipynb\n",
    "\n",
    "Once you're finished writing your example, you can convert it to an `.ipynb` notebook using `jupytext`:\n",
    "\n",
    "```shell\n",
    "jupytext your-example.md --to ipynb\n",
    "```\n",
    "\n",
    "In the same way, you can convert `.ipynb` notebooks to `.md` notebooks with `--to myst`.\n",
    "And if you want to convert your notebook to a Python file, e.g. to test if your whole script runs without errors,\n",
    "you can use `--to py` instead.\n",
    "\n",
    "## Where to go from here?\n",
    "\n",
    "There are many other ways to contribute to Ray other than documentation.\n",
    "See {ref}`our contributor guide <getting-involved>` for more information."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}