[C++ API] add C++ API docs (#17743)

doc/source/actors.rst

@@ -88,6 +88,44 @@ Creating an actor
       // Create an actor with a factory method.
       Ray.actor(CounterFactory::createCounter).remote();
 
+  .. group-tab:: C++
+
+    You can convert a standard C++ class into a Ray actor class as follows:
+
+    .. code-block:: c++
+
+      // A regular C++ class.
+      class Counter {
+
+      private:
+        int value = 0;
+
+      public:
+        int Increment() {
+          value += 1;
+          return value;
+        }
+
+        int GetCounter() {
+          return value;
+        }
+
+        void Reset(int new_value) {
+          value = new_value;
+        }
+      };
+
+      // Factory function of Counter class.
+      Counter* CreateCounter() {
+          return new Counter();
+        }
+
+      RAY_REMOTE(&Counter::Increment, &Counter::GetCounter, 
+                 &Counter::Reset, CreateCounter);
+
+      // Create an actor with a factory method.
+      ray::Actor(CreateCounter).Remote();
+
 When the above actor is instantiated, the following events happen.
 
 1. A node in the cluster is chosen and a worker process is created on that node
@@ -130,6 +168,15 @@ Methods of the actor can be called remotely.
     counterActor.task(Counter::reset, 10).remote();
     Assert.assertEquals((int) counterActor.task(Counter::increment).remote().get(), 11);
 
+  .. code-tab:: c++
+
+    ray::ActorHandle<Counter> counter_actor = ray::Actor(CreateCounter).Remote();
+    // Call an actor method with a return value
+    assert(*counter_actor.Task(&Counter::Increment).Remote().Get(), 1);
+    // Call an actor method without return value. In this case, the return type of `Remote()` is void.
+    counter_actor.Task(&Counter::Reset).Remote(10);
+    assert(*counter_actor.Task(&Counter::Increment).Remote().Get(), 11);
+
 .. _actor-resource-guide:
 
 Resources with Actors
@@ -161,6 +208,20 @@ You can specify that an actor requires CPUs or GPUs in the decorator. While Ray
 
       Ray.actor(GpuActor::new).setResource("CPU", 2.0).setResource("GPU", 0.5).remote();
 
+  .. group-tab:: C++
+
+    .. In C++, we always specify resources when creating actors. There's no annotation available to act like the Python decorator ``@ray.remote(...)``.
+
+    .. code-block:: c++
+
+      class GpuActor {
+        static GpuActor* CreateGpuActor() {
+          return new GpuActor();
+        }
+      }
+
+      ray::Actor(&GpuActor::CreateGpuActor).SetResource("CPU", 2.0).SetResource("GPU", 0.5).Remote();
+
 When an ``GPUActor`` instance is created, it will be placed on a node that has
 at least 1 GPU, and the GPU will be reserved for the actor for the duration of
 the actor's lifetime (even if the actor is not executing tasks). The GPU
@@ -195,6 +256,16 @@ have these resources (see `configuration instructions
 
     Ray.actor(GpuActor::new).setResource("Resource2", 1.0).remote();
 
+  .. code-tab:: c++
+
+    class GpuActor {
+      static GpuActor* CreateGpuActor() {
+        return new GpuActor();
+      }
+    }
+
+    ray::Actor(&GpuActor::CreateGpuActor).SetResource("Resource2", 1.0).Remote();
+
 
 If you need to instantiate many copies of the same actor with varying resource
 requirements, you can do so as follows.
@@ -223,6 +294,18 @@ requirements, you can do so as follows.
     ActorHandle<Counter> a3 = Ray.actor(Counter::new).setResource("CPU", 3.0)
       .setResource("Custom3", 1.0).remote();
 
+  .. code-tab:: c++
+
+    class Counter {
+      ...
+    }
+
+    auto a1 = ray::Actor(&GpuActor::CreateGpuActor).SetResource("CPU", 1.0)
+      .SetResource("Custom1", 1.0).Remote();
+    auto a2 = ray::Actor(&GpuActor::CreateGpuActor).SetResource("CPU", 2.0)
+      .SetResource("Custom2", 1.0).Remote();
+    auto a3 = ray::Actor(&GpuActor::CreateGpuActor).SetResource("CPU", 3.0)
+      .SetResource("Custom3", 1.0).Remote();
 
 Note that to create these actors successfully, Ray will need to be started with
 sufficient CPU resources and the relevant custom resources.
@@ -248,6 +331,10 @@ Automatic termination
 
     Terminating an actor automatically when the initial actor handle goes out of scope hasn't been implemented in Java yet.
 
+  .. group-tab:: C++
+
+    Terminating an actor automatically when the initial actor handle goes out of scope hasn't been implemented in C++ yet.
+
 Manual termination within the actor
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
@@ -274,6 +361,16 @@ This will kill the actor process and release resources associated/assigned to th
     can be waited on to wait for the actor to exit (calling ``ObjectRef::get`` on it will
     throw a ``RayActorException``).
 
+  .. group-tab:: C++
+    .. code-block:: c++
+
+      ray::ExitActor();
+
+    Garbage collection for actors haven't been implemented yet, so this is currently the
+    only way to terminate an actor gracefully. The ``ObjectRef`` resulting from the task
+    can be waited on to wait for the actor to exit (calling ``ObjectRef::Get`` on it will
+    throw a ``RayActorException``).
+
 Note that this method of termination will wait until any previously submitted
 tasks finish executing and then exit the process gracefully with sys.exit.
 
@@ -291,6 +388,10 @@ You can terminate an actor forcefully.
 
     actorHandle.kill();
 
+  .. code-tab:: c++
+
+    actor_handle.Kill();
+
 This will call the exit syscall from within the actor, causing it to exit
 immediately and any pending tasks to fail.
 
@@ -308,6 +409,12 @@ immediately and any pending tasks to fail.
     shutdown hooks installed in the actor using ``Runtime.addShutdownHook(...)`` will
     not be called.
 
+  .. group-tab:: C++
+
+    This will not go through the normal
+    C++ std::exit teardown logic, so any exit handlers installed in the actor using
+    ``std::atexit`` will not be called.
+
 Passing Around Actor Handles
 ----------------------------
 
@@ -336,6 +443,15 @@ Actor handles can be passed into other tasks. We can define remote functions (or
       }
     }
 
+  .. code-tab:: c++
+
+      void Foo(ray::ActorHandle<Counter> counter) {
+        for (int i = 0; i < 1000; i++) {
+          std::this_thread::sleep_for(std::chrono::milliseconds(100));
+          counter.Task(&Counter::Increment).Remote();
+        }
+      }
+
 If we instantiate an actor, we can pass the handle around to various tasks.
 
 .. tabs::
@@ -366,6 +482,21 @@ If we instantiate an actor, we can pass the handle around to various tasks.
       System.out.println(counter.task(Counter::getCounter).remote().get());
     }
 
+  .. code-tab:: c++
+
+    auto counter = ray::Actor(CreateCounter).Remote();
+
+    // Start some tasks that use the actor.
+    for (int i = 0; i < 3; i++) {
+      ray::Task(Foo, counter).Remote();
+    }
+
+    // Print the counter value.
+    for (int i = 0; i < 10; i++) {
+      std::this_thread::sleep_for(std::chrono::seconds(1));
+      std::cout << *counter.Task(&Counter::GetCounter).Remote().Get() << std::endl;
+    }
+
 Named Actors
 ------------
 
@@ -415,6 +546,18 @@ exist. See :ref:`actor-lifetimes` for more details.
       Optional<ActorHandle<Counter>> counter = Ray.getActor("some_name_in_job");
       Assert.assertTrue(counter.isPresent());
 
+  .. group-tab:: C++
+
+    .. code-block:: c++
+
+      // Create an actor with a globally unique name
+      ActorHandle<Counter> counter = ray::Actor(CreateCounter).SetGlobalName("some_name").Remote();
+
+      ...
+
+      // Retrieve the actor later somewhere
+      boost::optional<ray::ActorHandle<Counter>> counter = ray::GetGlobalActor("some_name");
+
 .. note::
 
      Named actors are only accessible in the same namespace. 
@@ -475,6 +618,10 @@ Actor Lifetimes
 
     Customizing lifetime of an actor hasn't been implemented in Java yet.
 
+  .. group-tab:: C++
+
+    Customizing lifetime of an actor hasn't been implemented in C++ yet.
+
 Actor Pool
 ----------
 
@@ -499,6 +646,10 @@ Actor Pool
 
     Actor pool hasn't been implemented in Java yet.
 
+  .. group-tab:: C++
+
+    Actor pool hasn't been implemented in C++ yet.
+
 
 FAQ: Actors, Workers and Resources
 ----------------------------------
@@ -539,3 +690,7 @@ Concurrency within an actor
   .. group-tab:: Java
 
     Actor-level concurrency hasn't been implemented in Java yet.
+
+  .. group-tab:: C++
+
+    Actor-level concurrency hasn't been implemented in C++ yet.

doc/source/cluster/cloud.rst

@@ -418,6 +418,18 @@ To run a distributed Ray program, you'll need to execute your program on the sam
 
     .. note:: Specifying ``auto`` as the address hasn't been implemented in Java yet. You need to provide the actual address. You can find the address of the server from the output of the ``ray up`` command.
 
+  .. group-tab:: C++
+
+    You need to add the ``RAY_ADDRESS`` env var to your command line (like ``RAY_ADDRESS=...``).
+
+    To connect your program to the Ray cluster, run it like this:
+
+        .. code-block:: bash
+
+            RAY_ADDRESS=<address> ./<binary> <args>
+
+    .. note:: Specifying ``auto`` as the address hasn't been implemented in C++ yet. You need to provide the actual address. You can find the address of the server from the output of the ``ray up`` command.
+
 
 .. note:: A common mistake is setting the address to be a cluster node while running the script on your laptop. This will not work because the script needs to be started/executed on one of the Ray nodes.
 

doc/source/starting-ray.rst

@@ -51,6 +51,12 @@ Calling ``ray.init()`` (without any ``address`` args) starts a Ray runtime on yo
       }
     }
 
+  .. code-tab:: c++
+
+    #include <ray/api.h>
+    // Other Ray APIs will not work until `ray::Init()` is called.
+    ray::Init()
+
 When the process calling ``ray.init()`` terminates, the Ray runtime will also terminate. To explicitly stop or restart Ray, use the shutdown API.
 
 .. tabs::
@@ -74,6 +80,13 @@ When the process calling ``ray.init()`` terminates, the Ray runtime will also te
       }
     }
 
+  .. code-tab:: c++
+
+    #include <ray/api.h>
+    ray::Init()
+    ... // ray program
+    ray::Shutdown()
+
 .. tabs::
   .. group-tab:: Python
 
@@ -106,6 +119,22 @@ When the process calling ``ray.init()`` terminates, the Ray runtime will also te
         }
       }
 
+  .. group-tab:: C++
+
+    To check if Ray is initialized, you can call ``ray::IsInitialized()``:
+
+    .. code-block:: c++
+
+      #include <ray/api.h>
+
+      int main(int argc, char **argv) {
+        ray::Init();
+        assert(ray::IsInitialized());
+
+        ray::Shutdown();
+        assert(!ray::IsInitialized());
+      }
+
 See the `Configuration <configure.html>`__ documentation for the various ways to configure Ray.
 
 .. _start-ray-cli:
@@ -158,6 +187,21 @@ You can connect to this Ray runtime by starting a driver process on the same nod
         -Dray.address=<address> \
         <classname> <args>
 
+  .. group-tab:: C++
+
+    .. code-block:: c++
+
+      #include <ray/api.h>
+
+      int main(int argc, char **argv) {
+        ray::Init();
+        ...
+      }
+
+    .. code-block:: bash
+
+      RAY_ADDRESS=<address> ./<binary> <args>
+
 
 You can connect other nodes to the head node, creating a Ray cluster by also calling ``ray start`` on those nodes. See :ref:`manual-cluster` for more details. Calling ``ray.init(address="auto")`` on any of the cluster machines will connect to the ray cluster.
 
@@ -204,6 +248,16 @@ By default, Ray will parallelize its workload and run tasks on multiple processe
     
     .. note:: If you just want to run your Java code in local mode, you can run it without Ray or even Python installed.
 
+  .. group-tab:: C++
+
+    .. code-block:: c++
+
+      RayConfig config;
+      config.local_mode = true;
+      ray::Init(config);
+
+    .. note:: If you just want to run your C++ code in local mode, you can run it without Ray or even Python installed.
+
 Note that there are some known issues with local mode. Please read :ref:`these tips <local-mode-tips>` for more information.
 
 

doc/source/walkthrough.rst

@@ -28,6 +28,10 @@ Installation
 
     Note: To run your Ray Java application, you need to install Ray Python with `pip install -U ray` first. (For Ray Java snapshot versions, install nightly Ray Python wheels.) The versions of Ray Java and Ray Python must match.
 
+  .. group-tab:: C++
+
+    To run this walkthrough, install Ray with ``pip install -U ray[cpp]``. For the latest wheels (for a snapshot of ``master``), you can use these instructions at :ref:`install-nightlies`.
+
 Starting Ray
 ------------
 
@@ -62,6 +66,19 @@ You can start Ray on a single machine by adding this to your code.
       }
     }
 
+  .. code-tab:: c++
+
+    // Run `ray cpp --show-library-path` to find headers and libraries.
+    #include <ray/api.h>
+
+    int main(int argc, char **argv) {
+      // Start Ray runtime. If you're connecting to an existing cluster, you can set
+      // the `RAY_ADDRESS` env var.
+      ray::Init();
+      ...
+    }
+
+
 Ray will then be able to utilize all cores of your machine. Find out how to configure the number of cores Ray will use at :ref:`configuring-ray`.
 
 To start a multi-node Ray cluster, see the :ref:`cluster setup page <cluster-index>`.
@@ -142,6 +159,38 @@ Ray enables arbitrary functions to be executed asynchronously. These asynchronou
         Ray.task(MyRayApp::slowFunction).remote();
       }
 
+  .. group-tab:: C++
+
+    .. code:: c++
+      
+      // A regular C++ function.
+      int MyFunction() {
+        return 1;
+      }
+      // Register as a remote function by `RAY_REMOTE`.
+      RAY_REMOTE(MyFunction);
+
+      // Invoke the above method as a Ray remote function.
+      // This will immediately return an object ref (a future) and then create
+      // a task that will be executed on a worker process.
+      auto res = ray::Task(MyFunction).Remote();
+
+      // The result can be retrieved with ``ray::ObjectRef::Get``.
+      assert(*res.Get() == 1);
+
+      int SlowFunction() {
+        std::this_thread::sleep_for(std::chrono::seconds(10));
+        return 1;
+      }
+      RAY_REMOTE(SlowFunction);
+
+      // Invocations of Ray remote functions happen in parallel.
+      // All computation is performed in the background, driven by Ray's internal event loop.
+      for(int i = 0; i < 4; i++) {
+        // This doesn't block.
+        ray::Task(SlowFunction).Remote();
+      }
+
 .. _ray-object-refs:
 
 Passing object refs to remote functions
@@ -179,6 +228,20 @@ Passing object refs to remote functions
     ObjectRef<Integer> objRef2 = Ray.task(MyRayApp::functionWithAnArgument, objRef1).remote();
     Assert.assertTrue(objRef2.get() == 2);
 
+  .. code-tab:: c++
+
+    static int FunctionWithAnArgument(int value) {
+      return value + 1;
+    }
+    RAY_REMOTE(FunctionWithAnArgument);
+
+    auto obj_ref1 = ray::Task(MyFunction).Remote();
+    assert(*obj_ref1.Get() == 1);
+
+    // You can pass an object ref as an argument to another Ray remote function.
+    auto obj_ref2 = ray::Task(FunctionWithAnArgument, obj_ref1).Remote();
+    assert(*obj_ref2.Get() == 2);
+
 Note the following behaviors:
 
   -  The second task will not be executed until the first task has finished
@@ -206,6 +269,15 @@ this default behavior by passing in specific resources.
 
     Set Java system property: ``-Dray.resources=CPU:8,GPU:4,Custom:2``.
 
+
+  .. code-tab:: c++
+
+    RayConfig config;
+    config.num_cpus = 8;
+    config.num_gpus = 4;
+    config.resources = {{"Custom", 2}};
+    ray::Init(config);
+
 Ray also allows specifying a task's resources requirements (e.g., CPU, GPU, and custom resources).
 The task will only run on a machine if there are enough resources
 available to execute the task.
@@ -223,6 +295,11 @@ available to execute the task.
     // Specify required resources.
     Ray.task(MyRayApp::myFunction).setResource("CPU", 1.0).setResource("GPU", 4.0).remote();
 
+  .. code-tab:: c++
+
+    // Specify required resources.
+    ray::Task(MyFunction).SetResource("CPU", 1.0).SetResource("GPU", 4.0).Remote();
+
 .. note::
 
     * If you do not specify any resources, the default is 1 CPU resource and
@@ -259,6 +336,11 @@ Below are more examples of resource specifications:
     // Ray aslo supports fractional and custom resources.
     Ray.task(MyRayApp::myFunction).setResource("GPU", 0.5).setResource("Custom", 1.0).remote();
 
+  .. code-tab:: c++
+
+    // Ray aslo supports fractional and custom resources.
+    ray::Task(MyFunction).SetResource("GPU", 0.5).SetResource("Custom", 1.0).Remote();
+
 Multiple returns
 ~~~~~~~~~~~~~~~~
 
@@ -279,6 +361,10 @@ Multiple returns
 
     Java remote functions doesn't support returning multiple objects.
 
+  .. group-tab:: C++
+
+    C++ remote functions doesn't support returning multiple objects.
+
 Cancelling tasks
 ~~~~~~~~~~~~~~~~
 
@@ -307,6 +393,10 @@ Cancelling tasks
 
     Task cancellation hasn't been implemented in Java yet.
 
+  .. group-tab:: C++
+
+    Task cancellation hasn't been implemented in C++ yet.
+
 .. _objects-in-ray:
 
 Objects in Ray
@@ -336,6 +426,12 @@ Object refs can be created in multiple ways.
     int y = 1;
     ObjectRef<Integer> objectRef = Ray.put(y);
 
+  .. code-tab:: c++
+
+    // Put an object in Ray's object store.
+    int y = 1;
+    ray::ObjectRef<int> object_ref = ray::Put(y);
+
 .. note::
 
     Remote objects are immutable. That is, their values cannot be changed after
@@ -384,16 +480,35 @@ If the current node's object store does not contain the object, the object is do
 
       // Get the value of one object ref.
       ObjectRef<Integer> objRef = Ray.put(1);
-      Assert.assertTrue(object.get() == 1);
+      Assert.assertTrue(objRef.get() == 1);
 
       // Get the values of multiple object refs in parallel.
-      List<ObjectRef<Integer>> objRefs = new ArrayList<>();
+      List<ObjectRef<Integer>> objectRefs = new ArrayList<>();
       for (int i = 0; i < 3; i++) {
 	objectRefs.add(Ray.put(i));
       }
       List<Integer> results = Ray.get(objectRefs);
       Assert.assertEquals(results, ImmutableList.of(0, 1, 2));
 
+  .. group-tab:: C++
+
+    .. code-block:: c++
+
+      // Get the value of one object ref.
+      ray::ObjectRef<int> obj_ref = ray::Put(1);
+      assert(*obj_ref.Get() == 1);
+
+      // Get the values of multiple object refs in parallel.
+      std::vector<ray::ObjectRef<int>> obj_refs;
+      for (int i = 0; i < 3; i++) {
+        obj_refs.emplace_back(ray::Put(i));
+      }
+      auto results = ray::Get(obj_refs);
+      assert(results.size() == 3);
+      assert(*results[0] == 0);
+      assert(*results[1] == 1);
+      assert(*results[2] == 2);
+
 After launching a number of tasks, you may want to know which ones have
 finished executing. This can be done with ``wait`` (:ref:`ray-wait-ref`). The function
 works as follows.
@@ -409,6 +524,10 @@ works as follows.
     System.out.println(waitResult.getReady());  // List of ready objects.
     System.out.println(waitResult.getUnready());  // list of unready objects.
   
+  .. code-tab:: c++
+
+    ray::WaitResult<int> wait_result = ray::Wait(object_refs, /*num_objects=*/0, /*timeout_ms=*/1000);
+
 Object Spilling
 ---------------
 
@@ -463,6 +582,38 @@ Actors extend the Ray API from functions (tasks) to classes. An actor is essenti
       // as the argument.
       ActorHandle<Counter> counter = Ray.actor(Counter::new).remote();
 
+  .. group-tab:: C++
+
+    ``ray::Actor`` is used to create actors from regular C++ classes.
+
+    .. code-block:: c++
+
+      // A regular C++ class.
+      class Counter {
+
+      private:
+          int value = 0;
+
+      public:
+        int Increment() {
+          value += 1;
+          return value;
+        }
+      };
+
+      // Factory function of Counter class.
+      static Counter *CreateCounter() {
+          return new Counter();
+      };
+
+      RAY_REMOTE(&Counter::Increment, CreateCounter);
+
+      // Create an actor from this class.
+      // `ray::Actor` takes a factory method that can produce
+      // a `Counter` object. Here, we pass `Counter`'s factory function
+      // as the argument.
+      auto counter = ray::Actor(CreateCounter).Remote();
+
 Specifying required resources
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
@@ -482,6 +633,11 @@ You can specify resource requirements in actors too (see the `Actors section
     // Specify required resources for an actor.
     Ray.actor(Counter::new).setResource("CPU", 2.0).setResource("GPU", 0.5).remote();
 
+  .. code-tab:: c++
+
+    // Specify required resources for an actor.
+    ray::Actor(CreateCounter).SetResource("CPU", 2.0).SetResource("GPU", 0.5).Remote();
+
 
 Calling the actor
 ~~~~~~~~~~~~~~~~~
@@ -500,9 +656,15 @@ value.
   .. code-tab:: java
 
     // Call the actor.
-    ObjectRef<Integer> objectRef = counter.task(Counter::increment).remote();
+    ObjectRef<Integer> objectRef = counter.task(&Counter::increment).remote();
     Assert.assertTrue(objectRef.get() == 1);
 
+  .. code-tab:: c++
+
+    // Call the actor.
+    auto object_ref = counter.Task(&Counter::increment).Remote();
+    assert(*object_ref.Get() == 1);
+
 Methods called on different actors can execute in parallel, and methods called on the same actor are executed serially in the order that they are called. Methods on the same actor will share state with one another, as shown below.
 
 .. tabs::
@@ -547,4 +709,36 @@ Methods called on different actors can execute in parallel, and methods called o
     // prints [2, 3, 4, 5, 6]
     System.out.println(Ray.get(objectRefs));
 
+  .. code-tab:: c++
+
+    // Create ten Counter actors.
+    std::vector<ray::ActorHandle<Counter>> counters;
+    for (int i = 0; i < 10; i++) {
+      counters.emplace_back(ray::Actor(CreateCounter).Remote());
+    }
+
+    // Increment each Counter once and get the results. These tasks all happen in
+    // parallel.
+    std::vector<ray::ObjectRef<int>> object_refs;
+    for (ray::ActorHandle<Counter> counter_actor : counters) {
+      object_refs.emplace_back(counter_actor.Task(&Counter::Increment).Remote());
+    }
+    // prints 1, 1, 1, 1, 1, 1, 1, 1, 1, 1
+    auto results = ray::Get(object_refs);
+    for (const auto &result : results) {
+      std::cout << *result;
+    }
+
+    // Increment the first Counter five times. These tasks are executed serially
+    // and share state.
+    object_refs.clear();
+    for (int i = 0; i < 5; i++) {
+      object_refs.emplace_back(counters[0].Task(&Counter::Increment).Remote());
+    }
+    // prints 2, 3, 4, 5, 6
+    results = ray::Get(object_refs);
+    for (const auto &result : results) {
+      std::cout << *result;
+    }
+
 To learn more about Ray Actors, see the `Actors section <actors.html>`__.