mirror of
https://github.com/vale981/ray
synced 2025-03-06 18:41:40 -05:00
ebf4070d88
* Ray documentation - created new section 'Profiling for Ray Users', opposed to current Profiling section for Ray developers. Completed three sections 'A Basic Profiling Example', 'Timing Performance Using Python's Timestamps', and 'Profiling Using An External Profiler (Line_Profiler).' Left to-do two sections on CProfile and Ray Timeline Visualization.' * Ray documentation - Fixed rst codeblock linebreaks in 'User Profiling' * Ray documentation - For User Profiling, added section on cProfile * Ray documentation - For User Profiling, completed Ray Timeline Visualization section, including graphical images * Ray documentation - made User Profiling timeline image larger, minor wording edits * Ray documentation - minor wording edits to User Profiling * Ray documentation - User Profiling- fixed broken link * Minor wording changes requested by Philipp Moritz addressed. Still need to address (1) compressing the image files, (2) correcting ex 3 to not be remote, and (3) using cProfile on an actor * Ray documentation - For user-profiling.rst, revised example 3 to show a semi-parallelized example. Compressed timeline example image to be under 50 KB, removed view timeline GUI image. Updated timeline example image to reflect revised example 3. cProfile actor example left * Ray documentation - in user-profiling.rst, added a new example including actors in the cProfile section * Ray documentation - For user-profiling.rst, added section header for the Ray actor cProfile example * Update user-profiling.rst * Update user-profiling.rst * 4 space indentation * Update user-profiling.rst * Update user-profiling.rst * Update user-profiling.rst * corrections
94 lines
3 KiB
ReStructuredText
94 lines
3 KiB
ReStructuredText
Profiling for Ray Developers
|
|
============================
|
|
|
|
This document details, for Ray developers, how to use ``pprof`` to profile Ray
|
|
binaries.
|
|
|
|
Installation
|
|
------------
|
|
|
|
These instructions are for Ubuntu only. Attempts to get ``pprof`` to correctly
|
|
symbolize on Mac OS have failed.
|
|
|
|
.. code-block:: bash
|
|
|
|
sudo apt-get install google-perftools libgoogle-perftools-dev
|
|
|
|
Changes to compilation and linking
|
|
----------------------------------
|
|
|
|
Let's say we want to profile the ``plasma_manager``. Change the link
|
|
instruction in ``src/plasma/CMakeLists.txt`` from
|
|
|
|
.. code-block:: cmake
|
|
|
|
target_link_libraries(plasma_manager common ${PLASMA_STATIC_LIB} ray_static ${ARROW_STATIC_LIB} -lpthread)
|
|
|
|
to additionally include ``-lprofiler``:
|
|
|
|
.. code-block:: cmake
|
|
|
|
target_link_libraries(plasma_manager common ${PLASMA_STATIC_LIB} ray_static ${ARROW_STATIC_LIB} -lpthread -lprofiler)
|
|
|
|
Additionally, add ``-g -ggdb`` to ``CMAKE_C_FLAGS`` and ``CMAKE_CXX_FLAGS`` to
|
|
enable the debug symbols. (Keeping ``-O3`` seems okay.)
|
|
|
|
Recompile.
|
|
|
|
Launching the to-profile binary
|
|
-------------------------------
|
|
|
|
In various places, instead of launching the target binary via
|
|
``plasma_manager <args>``, it must be launched with
|
|
|
|
.. code-block:: bash
|
|
|
|
LD_PRELOAD=/usr/lib/libprofiler.so CPUPROFILE=/tmp/pprof.out plasma_manager <args>
|
|
|
|
In practice, this means modifying ``python/ray/plasma/plasma.py`` so that the
|
|
manager is launched with a command that passes a ``modified_env`` into
|
|
``Popen``.
|
|
|
|
.. code-block:: python
|
|
|
|
modified_env = os.environ.copy()
|
|
modified_env["LD_PRELOAD"] = "/usr/lib/libprofiler.so"
|
|
modified_env["CPUPROFILE"] = "/tmp/pprof.out"
|
|
|
|
process = subprocess.Popen(command,
|
|
stdout=stdout_file,
|
|
stderr=stderr_file,
|
|
env=modified_env)
|
|
|
|
The file ``/tmp/pprof.out`` will be empty until you let the binary run the
|
|
target workload for a while and then ``kill`` it.
|
|
|
|
Visualizing the CPU profile
|
|
---------------------------
|
|
|
|
The output of ``pprof`` can be visualized in many ways. Here we output it as a
|
|
zoomable ``.svg`` image displaying the call graph annotated with hot paths.
|
|
|
|
.. code-block:: bash
|
|
|
|
# Use the appropriate path.
|
|
PLASMA_MANAGER=ray/python/ray/core/src/plasma/plasma_manager
|
|
|
|
google-pprof -svg $PLASMA_MANAGER /tmp/pprof.out > /tmp/pprof.svg
|
|
# Then open the .svg file with Chrome.
|
|
|
|
# If you realize the call graph is too large, use -focus=<some function> to zoom
|
|
# into subtrees.
|
|
google-pprof -focus=epoll_wait -svg $PLASMA_MANAGER /tmp/pprof.out > /tmp/pprof.svg
|
|
|
|
Here's a snapshot of an example svg output, taken from the official
|
|
documentation:
|
|
|
|
.. image:: http://goog-perftools.sourceforge.net/doc/pprof-test-big.gif
|
|
|
|
References
|
|
----------
|
|
|
|
- The `pprof documentation <http://goog-perftools.sourceforge.net/doc/cpu_profiler.html>`_.
|
|
- A `Go version of pprof <https://github.com/google/pprof>`_.
|
|
- The `gperftools <https://github.com/gperftools/gperftools>`_, including libprofiler, tcmalloc, and other goodies.
|