Task/actor/object summary
Tasks: Group by the func name. In the future, we will also allow to group by task_group.
Actors: Group by actor class name. In the future, we will also allow to group by actor_group.
Object: Group by callsite. In the future, we will allow to group by reference type or task state.
Enable checking of the ray core module, excluding serve, workflows, and tune, in ./ci/lint/check_api_annotations.py. This required moving many files to ray._private and associated fixes.
Improve observability for general objects and lineage reconstruction by adding a "Status" field to `ray memory`. The value of the field can be:
```
// The task is waiting for its dependencies to be created.
WAITING_FOR_DEPENDENCIES = 1;
// All dependencies have been created and the task is scheduled to execute.
SCHEDULED = 2;
// The task finished successfully.
FINISHED = 3;
```
In addition, tasks that failed or that needed to be re-executed due to lineage reconstruction will have a field listing the attempt number. Example output:
```
IP Address | PID | Type | Call Site | Status | Size | Reference Type | Object Ref
192.168.4.22 | 279475 | Driver | (task call) ... | Attempt #2: FINISHED | 10000254.0 B | LOCAL_REFERENCE | c2668a65bda616c1ffffffffffffffffffffffff0100000001000000
```
Currently `wait_until_succeeded_without_exception` is used in the dashboard, and it returns True/False. Unfortunately, there are lots of code that doesn't assert on this method (which means things are not actually tested).
* Better formatting when terminal size doesn't support tabular
* Summary now displays size of reference types
* Add unit conversion support (e.g. b, kb, mb, gb)
* Format and test
* Add ability to specify the number of sorted entries
* Linting
* Clean up group summary, move import defaultdict, comment num entries counter, n
* Format and lint
* Minor improvements in Ray Core Walkthrough as seen in https://github.com/ray-project/ray/issues/12472
* Define node_stats() to return NodeStats object from cluster
* Add --group-by and --sort-by capabilities to ray memory script
* Resolve merge conflict
* Add helper functions for group by and sorting type in memory_utils.py
* Reformat
* Format
* Compartmentalize memory script into get_memory_summary and get_store_stats_summary
* Modify unit tests in test_mem_stat
* Lint and format
* Test cases for group_by sort_by
* Lint and format
* Fix actor handle failing test case
* Update test_memstat.py
* Resolve merge conflicts
* Adjust ray memory output based on terminal size
* Formatting and linting
* Use constant for callsite length
* Switch from OS to shutil for querying terminal size (official python support)
* Linting and formatting
* Lint and format
* Resolve lint issue in walkthrough.rst
* Revert to python 3.6
* Delete visitor.py
It was accidentally included in most recent commit
* Delete .eggs
It was accidentally included in most recent commit
* Resolve test_object_spilling.py test case
* Add stats only argument
* revert changes on this file
* Remove package-lock.json
* Add back npm installation
* Sync package-lock.json
* Linting and formatting
* Sync with package-lock
* Sync with package-lock pt 2
* Update documentation in https://docs.ray.io/en/master/memory-management.html
* Add include_memory_info as argument for node_stats
* Switch object ref and call site positions
* Linting and formatting
* Change from MiB to B
* Change from stats-only to store-true
* Add memory test case
* Add memory test case
* Lint and format
* Correct test in memstat
* Change line wrap and stats only to flags
* Clarify --stats-only and --no-format in ray memory
* --stats-only description modified
Co-authored-by: Micah Yong <micahyong@Micahs-MacBook-Pro.local>
* Add API support for the logical view and machine view, which lean on datacenter in common.
* Update dashboard/datacenter.py
Co-authored-by: fyrestone <fyrestone@outlook.com>
* Update dashboard/modules/logical_view/logical_view_head.py
Co-authored-by: fyrestone <fyrestone@outlook.com>
* Address PR comments
* lint
* Add dashboard tests to CI build
* Fix integration issues
* lint
Co-authored-by: Max Fitton <max@semprehealth.com>
Co-authored-by: fyrestone <fyrestone@outlook.com>
* Add actor and memory utility functions needed by upcoming logical view and memory view APIs
* Add a method to allow printing Dict custom class and add support for hot-reloading local dev environment.
* Address PR comments
* Add unit tests from test metrics to branch for new memory_utils module
* Add note about sorting / grouping
* lint
Co-authored-by: Max Fitton <max@semprehealth.com>
