hiro/ray
1
0
Fork 0
mirror of https://github.com/vale981/ray synced 2025-03-06 02:21:39 -05:00
Code Issues Projects Releases Packages Wiki Activity Actions

Update profiling instructions for raylet (#3311)

Browse source
This commit is contained in:
Philipp Moritz 2018-11-13 14:48:33 -08:00 • committed by Robert Nishihara
parent 97f423781b
commit d4fad222e1
2 changed files with 14 additions and 42 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

50
doc/source/profiling.rst
View file

@ -14,54 +14,20 @@ symbolize on Mac OS have failed.
sudo apt-get install google-perftools libgoogle-perftools-dev
Changes to compilation and linking
----------------------------------
Let's say we want to profile the ``plasma_manager``. Change the link
instruction in ``src/plasma/CMakeLists.txt`` from
.. code-block:: cmake
target_link_libraries(plasma_manager common ${PLASMA_STATIC_LIB} ray_static ${ARROW_STATIC_LIB} -lpthread)
to additionally include ``-lprofiler``:
.. code-block:: cmake
target_link_libraries(plasma_manager common ${PLASMA_STATIC_LIB} ray_static ${ARROW_STATIC_LIB} -lpthread -lprofiler)
Additionally, add ``-g -ggdb`` to ``CMAKE_C_FLAGS`` and ``CMAKE_CXX_FLAGS`` to
enable the debug symbols. (Keeping ``-O3`` seems okay.)
Recompile.
Launching the to-profile binary
-------------------------------
In various places, instead of launching the target binary via
``plasma_manager <args>``, it must be launched with
If you want to launch Ray in profiling mode, define the following variables:
.. code-block:: bash
LD_PRELOAD=/usr/lib/libprofiler.so CPUPROFILE=/tmp/pprof.out plasma_manager <args>
export RAYLET_PERFTOOLS_PATH=/usr/lib/x86_64-linux-gnu/libprofiler.so
export RAYLET_PERFTOOLS_LOGFILE=/tmp/pprof.out
In practice, this means modifying ``python/ray/plasma/plasma.py`` so that the
manager is launched with a command that passes a ``modified_env`` into
``Popen``.
.. code-block:: python
modified_env = os.environ.copy()
modified_env["LD_PRELOAD"] = "/usr/lib/libprofiler.so"
modified_env["CPUPROFILE"] = "/tmp/pprof.out"
process = subprocess.Popen(command,
stdout=stdout_file,
stderr=stderr_file,
env=modified_env)
The file ``/tmp/pprof.out`` will be empty until you let the binary run the
target workload for a while and then ``kill`` it.
target workload for a while and then ``kill`` it via ``ray stop`` or by
letting the driver exit.
Visualizing the CPU profile
---------------------------
@ -72,14 +38,14 @@ zoomable ``.svg`` image displaying the call graph annotated with hot paths.
.. code-block:: bash
# Use the appropriate path.
PLASMA_MANAGER=ray/python/ray/core/src/plasma/plasma_manager
RAYLET=ray/python/ray/core/src/ray/raylet/raylet
google-pprof -svg $PLASMA_MANAGER /tmp/pprof.out > /tmp/pprof.svg
google-pprof -svg $RAYLET /tmp/pprof.out > /tmp/pprof.svg
# Then open the .svg file with Chrome.
# If you realize the call graph is too large, use -focus=<some function> to zoom
# into subtrees.
google-pprof -focus=epoll_wait -svg $PLASMA_MANAGER /tmp/pprof.out > /tmp/pprof.svg
google-pprof -focus=epoll_wait -svg $RAYLET /tmp/pprof.out > /tmp/pprof.svg
Here's a snapshot of an example svg output, taken from the official
documentation:

6
python/ray/services.py
View file

@ -971,6 +971,12 @@ def start_raylet(redis_address,
["valgrind", "--tool=callgrind"] + command,
stdout=stdout_file,
stderr=stderr_file)
elif "RAYLET_PERFTOOLS_PATH" in os.environ:
modified_env = os.environ.copy()
modified_env["LD_PRELOAD"] = os.environ["RAYLET_PERFTOOLS_PATH"]
modified_env["CPUPROFILE"] = os.environ["RAYLET_PERFTOOLS_LOGFILE"]
pid = subprocess.Popen(
command, stdout=stdout_file, stderr=stderr_file, env=modified_env)
else:
pid = subprocess.Popen(command, stdout=stdout_file, stderr=stderr_file)