The cluster configuration is defined within a YAML file that will be used by the Cluster Launcher to launch the head node, and by the Autoscaler to launch worker nodes. Once the cluster configuration is defined, you will need to use the :ref:`Ray CLI <ray-cli>` to perform any operations such as starting and stopping the cluster.
The ``available_nodes_types`` object's keys represent the names of the different node types.
Deleting a node type from ``available_node_types`` and updating with :ref:`ray up<ray-up-doc>` will cause the autoscaler to scale down all nodes of that type.
In particular, changing the key of a node type object will
result in removal of nodes corresponding to the old key; nodes with the new key name will then be
created according to cluster configuration and Ray resource demands.
A YAML object which conforms to the EC2 ``create_instances`` API in `the AWS docs <https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ec2.html#EC2.ServiceResource.create_instances>`_.
A YAML object as defined in `the deployment template <https://docs.microsoft.com/en-us/azure/templates/microsoft.compute/virtualmachines>`_ whose resources are defined in `the Azure docs <https://docs.microsoft.com/en-us/azure/templates/>`_.
The number of nodes allowed to be pending as a multiple of the current number of nodes. For example, if set to 1.0, the cluster can grow in size by at most 100% at any time, so if the cluster currently has 20 nodes, at most 20 pending launches are allowed. Note that although the autoscaler will scale down to `min_workers` (which could be 0), it will always scale up to 5 nodes at a minimum when scaling up.
In rare cases when Docker is not available on the system by default (e.g., bad AMI), add the following commands to :ref:`initialization_commands <cluster-configuration-initialization-commands>` to install it.
The key for one of the node types in :ref:`available_node_types <cluster-configuration-available-node-types>`. This node type will be used to launch the head node.
If the field ``head_node_type`` is changed and an update is executed with :ref:`ray up<ray-up-doc>`, the currently running head node will
be considered outdated. The user will receive a prompt asking to confirm scale-down of the outdated head node, and the cluster will restart with a new
head node. Changing the :ref:`node_config<cluster-configuration-node-config>` of the :ref:`node_type<cluster-configuration-node-types-type>` with key ``head_node_type`` will also result in cluster restart after a user prompt.
A list of paths to the files or directories to copy from the head node to the worker nodes. The same path on the head node will be copied to the worker node. This behavior is a subset of the file_mounts behavior, so in the vast majority of cases one should just use :ref:`file_mounts <cluster-configuration-file-mounts>`.
***Required:** No
***Importance:** Low
***Type:** List of String
***Default:**``[]``
.._cluster-configuration-rsync-exclude:
``rsync_exclude``
~~~~~~~~~~~~~~~~~
A list of patterns for files to exclude when running ``rsync up`` or ``rsync down``. The filter is applied on the source directory only.
Example for a pattern in the list: ``**/.git/**``.
***Required:** No
***Importance:** Low
***Type:** List of String
***Default:**``[]``
.._cluster-configuration-rsync-filter:
``rsync_filter``
~~~~~~~~~~~~~~~~
A list of patterns for files to exclude when running ``rsync up`` or ``rsync down``. The filter is applied on the source directory and recursively through all subdirectories.
Example for a pattern in the list: ``.gitignore``.
***Required:** No
***Importance:** Low
***Type:** List of String
***Default:**``[]``
.._cluster-configuration-initialization-commands:
``initialization_commands``
~~~~~~~~~~~~~~~~~~~~~~~~~~~
A list of commands that will be run before the :ref:`setup commands <cluster-configuration-setup-commands>`. If Docker is enabled, these commands will run outside the container and before Docker is setup.
***Required:** No
***Importance:** Medium
***Type:** List of String
***Default:**``[]``
.._cluster-configuration-setup-commands:
``setup_commands``
~~~~~~~~~~~~~~~~~~
A list of commands to run to set up nodes. These commands will always run on the head and worker nodes and will be merged with :ref:`head setup commands <cluster-configuration-head-setup-commands>` for head and with :ref:`worker setup commands <cluster-configuration-worker-setup-commands>` for workers.
- Setup commands should ideally be *idempotent* (i.e., can be run multiple times without changing the result); this allows Ray to safely update nodes after they have been created. You can usually make commands idempotent with small modifications, e.g. ``git clone foo`` can be rewritten as ``test -e foo || git clone foo`` which checks if the repo is already cloned first.
- Setup commands are run sequentially but separately. For example, if you are using anaconda, you need to run ``conda activate env && pip install -U ray`` because splitting the command into two setup commands will not work.
- Ideally, you should avoid using setup_commands by creating a docker image with all the dependencies preinstalled to minimize startup time.
-**Tip**: if you also want to run apt-get commands during setup add the following list of commands:
..code-block:: yaml
setup_commands:
- sudo pkill -9 apt-get || true
- sudo pkill -9 dpkg || true
- sudo dpkg --configure -a
.._cluster-configuration-head-setup-commands:
``head_setup_commands``
~~~~~~~~~~~~~~~~~~~~~~~
A list of commands to run to set up the head node. These commands will be merged with the general :ref:`setup commands <cluster-configuration-setup-commands>`.
***Required:** No
***Importance:** Low
***Type:** List of String
***Default:**``[]``
.._cluster-configuration-worker-setup-commands:
``worker_setup_commands``
~~~~~~~~~~~~~~~~~~~~~~~~~
A list of commands to run to set up the worker nodes. These commands will be merged with the general :ref:`setup commands <cluster-configuration-setup-commands>`.
***Required:** No
***Importance:** Low
***Type:** List of String
***Default:**``[]``
.._cluster-configuration-head-start-ray-commands:
``head_start_ray_commands``
~~~~~~~~~~~~~~~~~~~~~~~~~~~
Commands to start ray on the head node. You don't need to change this.
The default Docker image to pull in the head and worker nodes. This can be overridden by the :ref:`head_image <cluster-configuration-head-image>` and :ref:`worker_image <cluster-configuration-worker-image>` fields. If neither `image` nor (:ref:`head_image <cluster-configuration-head-image>` and :ref:`worker_image <cluster-configuration-worker-image>`) are specified, Ray will not use Docker.
***Required:** Yes (If Docker is in use.)
***Importance:** High
***Type:** String
The Ray project provides Docker images on `DockerHub <https://hub.docker.com/u/rayproject>`_. The repository includes following images:
*``rayproject/ray-ml:latest-gpu``: CUDA support, includes ML dependencies.
*``rayproject/ray:latest-gpu``: CUDA support, no ML dependencies.
*``rayproject/ray-ml:latest``: No CUDA support, includes ML dependencies.
*``rayproject/ray:latest``: No CUDA support, no ML dependencies.
.._cluster-configuration-head-image:
``docker.head_image``
~~~~~~~~~~~~~~~~~~~~~
Docker image for the head node to override the default :ref:`docker image <cluster-configuration-image>`.
***Required:** No
***Importance:** Low
***Type:** String
.._cluster-configuration-worker-image:
``docker.worker_image``
~~~~~~~~~~~~~~~~~~~~~~~
Docker image for the worker nodes to override the default :ref:`docker image <cluster-configuration-image>`.
***Required:** No
***Importance:** Low
***Type:** String
.._cluster-configuration-container-name:
``docker.container_name``
~~~~~~~~~~~~~~~~~~~~~~~~~
The name to use when starting the Docker container.
***Required:** Yes (If Docker is in use.)
***Importance:** Low
***Type:** String
***Default:** ray_container
.._cluster-configuration-pull-before-run:
``docker.pull_before_run``
~~~~~~~~~~~~~~~~~~~~~~~~~~
If enabled, the latest version of image will be pulled when starting Docker. If disabled, ``docker run`` will only pull the image if no cached version is present.
***Required:** No
***Importance:** Medium
***Type:** Boolean
***Default:**``True``
.._cluster-configuration-run-options:
``docker.run_options``
~~~~~~~~~~~~~~~~~~~~~~
The extra options to pass to ``docker run``.
***Required:** No
***Importance:** Medium
***Type:** List of String
***Default:**``[]``
.._cluster-configuration-head-run-options:
``docker.head_run_options``
~~~~~~~~~~~~~~~~~~~~~~~~~~~
The extra options to pass to ``docker run`` for head node only.
***Required:** No
***Importance:** Low
***Type:** List of String
***Default:**``[]``
.._cluster-configuration-worker-run-options:
``docker.worker_run_options``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The extra options to pass to ``docker run`` for worker nodes only.
If enabled, Ray will not automatically specify the size ``/dev/shm`` for the started container and the runtime's default value (64MiB for Docker) will be used.
If ``--shm-size=<>`` is manually added to ``run_options``, this is *automatically* set to ``True``, meaning that Ray will defer to the user-provided value.
The path to an existing private key for Ray to use. If not configured, Ray will create a new private keypair (default behavior). If configured, the key must be added to the project-wide metadata and ``KeyName`` has to be defined in the :ref:`node configuration <cluster-configuration-node-config>`.
The path to an existing private key for Ray to use. If not configured, Ray will create a new private keypair (default behavior). If configured, the key must be added to the project-wide metadata and ``KeyName`` has to be defined in the :ref:`node configuration <cluster-configuration-node-config>`.
If enabled, nodes will be *stopped* when the cluster scales down. If disabled, nodes will be *terminated* instead. Stopped nodes launch faster than terminated nodes.
The configuration to be used to launch the nodes on the cloud service provider. Among other things, this will specify the instance type to be launched.
The resources that a node type provides, which enables the autoscaler to automatically select the right type of nodes to launch given the resource demands of the application. The resources specified will be automatically passed to the ``ray start`` command for the node via an environment variable. If not provided, Autoscaler can automatically detect them only for AWS/Kubernetes cloud providers. For more information, see also the `resource demand scheduler <https://github.com/ray-project/ray/blob/master/python/ray/autoscaler/_private/resource_demand_scheduler.py>`_
In some cases, adding special nodes without any resources may be desirable. Such nodes can be used as a driver which connects to the cluster to launch jobs. In order to manually add a node to an autoscaled cluster, the *ray-cluster-name* tag should be set and *ray-node-type* tag should be set to unmanaged. Unmanaged nodes can be created by setting the resources to ``{}`` and the :ref:`maximum workers <cluster-configuration-node-min-workers>` to 0. The Autoscaler will not attempt to start, stop, or update unmanaged nodes. The user is responsible for properly setting up and cleaning up unmanaged nodes.
The maximum number of workers to have in the cluster for this node type regardless of utilization. This takes precedence over :ref:`minimum workers <cluster-configuration-node-min-workers>`. By default, the number of workers of a node type is unbounded, constrained only by the cluster-wide :ref:`max_workers <cluster-configuration-max-workers>`. (Prior to Ray 1.3.0, the default value for this field was 0.)
A list of commands to run to set up worker nodes of this type. These commands will replace the general :ref:`worker setup commands <cluster-configuration-worker-setup-commands>` for the node.
The memory in bytes allocated for python worker heap memory on the node. If not configured, Autoscaler will automatically detect the amount of RAM on the node for AWS/Kubernetes and allocate 70% of it for the heap.
The memory in bytes allocated for the object store on the node. If not configured, Autoscaler will automatically detect the amount of RAM on the node for AWS/Kubernetes and allocate 30% of it for the object store.
It is possible to use `TPU VMs <https://cloud.google.com/tpu/docs/users-guide-tpu-vm>`_ on GCP. Currently, `TPU pods <https://cloud.google.com/tpu/docs/system-architecture-tpu-vm#pods>`_ (TPUs other than v2-8 and v3-8) are not supported.
Before using a config with TPUs, ensure that the `TPU API is enabled for your GCP project <https://cloud.google.com/tpu/docs/users-guide-tpu-vm#enable_the_cloud_tpu_api>`_.
