Commit graph

107 commits

Author SHA1 Message Date
Ian Rodney
7b1c5dbe0a
[Hotfix][Lint] Pin other ESlint Deps (#15816) 2021-05-14 09:18:43 -07:00
fyrestone
56c309416e
[Job submission] Basic job submission structure (#15103) 2021-05-12 15:08:20 +08:00
Ashwin Hegde
4d8ed6dd5c
#13890 [new-dashboard] add object store memory column (#15697) 2021-05-11 15:36:16 -05:00
Ian Rodney
90ce25cb35
[dashboard] Avoid global min_workers (#15660) 2021-05-10 15:47:51 -07:00
Ian Rodney
c50490ccef
[Lint] Pin Prettier to 2.3.0 (#15721) 2021-05-10 11:46:29 -07:00
Ian Rodney
11b5c6c702
[HotFix][Lint] Fix Lint because of Prettier update (#15720) 2021-05-10 09:51:41 -07:00
SongGuyang
b8ff86adb9
Add objectStore stats to dashboard API. (#15677) 2021-05-10 11:32:14 -05:00
Amog Kamsetty
ebc44c3d76
[CI] Upgrade flake8 to 3.9.1 (#15527)
* formatting

* format util

* format release

* format rllib/agents

* format rllib/env

* format rllib/execution

* format rllib/evaluation

* format rllib/examples

* format rllib/policy

* format rllib utils and tests

* format streaming

* more formatting

* update requirements files

* fix rllib type checking

* updates

* update

* fix circular import

* Update python/ray/tests/test_runtime_env.py

* noqa
2021-05-03 14:23:28 -07:00
Ian Rodney
546e5f6f13
[API] Remove non-API top Level function imports (#15440) 2021-04-27 12:33:59 -07:00
fyrestone
43de7f48a7
Fix reported dashboard ip when using 0.0.0.0 (#15506) 2021-04-27 23:48:22 +08:00
Dmitri Gekhtman
410f768046
[Kubernetes] [Dashboard] Remove disk data from dashboard when running on K8s. (#14676) 2021-04-05 17:16:20 -07:00
Micah Yong
b3089b31f2
[RFC] Ray memory improvements: format and summary (#14520)
* Better formatting when terminal size doesn't support tabular

* Summary now displays size of reference types

* Add unit conversion support (e.g. b, kb, mb, gb)

* Format and test

* Add ability to specify the number of sorted entries

* Linting

* Clean up group summary, move import defaultdict, comment num entries counter, n

* Format and lint
2021-03-28 21:03:06 -07:00
fyrestone
52cfa1cdd7
Fix load code from local (#12102) 2021-03-24 11:49:58 +08:00
Clark Zinzow
1a9ba19012
[Core] Adds deprecation decorator and fixes privatization of a few APIs. (#14811) 2021-03-22 10:31:50 -07:00
Ian Rodney
eb12033612
[Code Cleanup] Switch to use ray.util.get_node_ip_address() (#14741)
Co-authored-by: Richard Liaw <rliaw@berkeley.edu>
2021-03-18 13:10:57 -07:00
Lixin Wei
72d87093b9
[Core] Make Actor DEAD and Save Exceptions in GCS When Error Happens in Constructor (#14211) 2021-03-17 12:50:28 -07:00
Kathryn Zhou
01dda99b8c
Export cluster statistics to Prometheus (#14612) 2021-03-15 19:28:13 -07:00
Dmitri Gekhtman
6babd1928c
[Kubernetes][dashboard][minor] Fix uptime (#14655) 2021-03-12 18:30:13 -06:00
Dmitri Gekhtman
a90cffe26c
[dashboard][k8s] Better CPU reporting when running on K8s (#14593) 2021-03-12 12:02:15 -06:00
Clark Zinzow
5a788474aa
[Core] First pass at privatizing non-public Python APIs. (#14607)
* async_compat

* utils

* cluster_utils

* compat

* function_manager

* import_thread

* memory_monitor

* monitor, log_monitor, ray_process_reaper

* metrics_agent

* parameter

* prometheus_exporter

* ray_logging

* signature
2021-03-10 22:47:28 -08:00
Dmitri Gekhtman
4a7d9e71bb
[dashboard][kubernetes] Show container's memory info on K8s, not the physical host's. (#14499)
* random doc typo

* more reasonable memory output

* no if

* get rid of comment
2021-03-08 18:59:41 -08:00
fyrestone
3616424f10
Disable dashboard tune module if pandas version is incorrect (#14381) 2021-03-08 20:40:59 -06:00
fyrestone
2da58bb021
[Dashboard] Fix reporter agent (#14378) 2021-03-08 13:12:34 -06:00
SangBin Cho
a04ab9b472
[Core] Fix ray memory bug (#14452)
* ray memory bug

* Fix ray memory issue.

* done.
2021-03-03 09:20:00 -08:00
SangBin Cho
09fd38ede1
[Multi node shuffle] More efficient ray memory --stats-only (#14423)
* Done.

* Fix all the issues.
2021-03-01 23:14:06 -08:00
Eric Liang
9db000ff2c
Auto report object store memory usage; remove some deprecated code (#14260) 2021-03-01 13:19:44 -08:00
Micah Yong
db0c16824c
[Dashboard][CLI] Ray memory parity with dashboard 2 (#13444)
* Minor improvements in Ray Core Walkthrough as seen in https://github.com/ray-project/ray/issues/12472

* Define node_stats() to return NodeStats object from cluster

* Add --group-by and --sort-by capabilities to ray memory script

* Resolve merge conflict

* Add helper functions for group by and sorting type in memory_utils.py

* Reformat

* Format

* Compartmentalize memory script into get_memory_summary and get_store_stats_summary

* Modify unit tests in test_mem_stat

* Lint and format

* Test cases for group_by sort_by

* Lint and format

* Fix actor handle failing test case

* Update test_memstat.py

* Resolve merge conflicts

* Adjust ray memory output based on terminal size

* Formatting and linting

* Use constant for callsite length

* Switch from OS to shutil for querying terminal size (official python support)

* Linting and formatting

* Lint and format

* Resolve lint issue in walkthrough.rst

* Revert to python 3.6

* Delete visitor.py

It was accidentally included in most recent commit

* Delete .eggs

It was accidentally included in most recent commit

* Resolve test_object_spilling.py test case

* Add stats only argument

* revert changes on this file

* Remove package-lock.json

* Add back npm installation

* Sync package-lock.json

* Linting and formatting

* Sync with package-lock

* Sync with package-lock pt 2

* Update documentation in https://docs.ray.io/en/master/memory-management.html

* Add include_memory_info as argument for node_stats

* Switch object ref and call site positions

* Linting and formatting

* Change from MiB to B

* Change from stats-only to store-true

* Add memory test case

* Add memory test case

* Lint and format

* Correct test in memstat

* Change line wrap and stats only to flags

* Clarify --stats-only and --no-format in ray memory

* --stats-only description modified

Co-authored-by: Micah Yong <micahyong@Micahs-MacBook-Pro.local>
2021-03-01 09:27:22 -08:00
Kathryn Zhou
456d9aab47
Add Cypress test for Ray Dashboard (#14253) 2021-02-24 20:41:52 -08:00
niole
488f63efe3
[Dashboard] Make requests sent by the dashboard reverse proxy compatible (#14012) 2021-02-24 18:31:59 -08:00
fyrestone
5e76a51d56
[Dashboard] Select port in dashboard (#13763)
* Dashboard select port; Fix dashboard may hangs when exit

* Add test case

* Fix

* Fix test_stats_collector.py::test_get_all_node_details

* Refine dashboard error messages

* Refine code

* Refine code

* Show last 10 lines of dashboard log if start dashboard failed

* Fix ValueError: too many values to unpack (expected 2) when getsockname

* Fix test_multi_node_3.py::test_calling_start_ray_head may fail

* Fix Windows CI

* Disable dashboard in C++ test

* Refine code

* Fix issue 7084

Co-authored-by: 刘宝 <po.lb@antfin.com>
2021-02-23 16:27:48 -08:00
Kathryn Zhou
d6521be7ef
Export GPU metrics, CPU count, and additional Memory metrics to Prometheus (#14170) 2021-02-22 10:04:18 -08:00
Kathryn Zhou
f6b5e838fe
Add disk and network metrics to Prometheus and fix dashboard (#14144) 2021-02-17 10:27:14 -08:00
Simon Mo
33316d4f8f
Revert "Export additional metrics to Prometheus (#14061)" (#14134)
This reverts commit 82539f2da4.
2021-02-16 12:49:12 -08:00
Kathryn Zhou
82539f2da4
Export additional metrics to Prometheus (#14061) 2021-02-14 23:16:26 -08:00
Alex Wu
02938f3a21
[hotfix] Disable dashboard agent windows (#14062) 2021-02-11 17:54:55 -08:00
Dominic Ming
4b60c388ef
[Dashboard] fix new dashboard entrance and some table problem (#13790) 2021-01-30 10:42:16 +08:00
Dominic Ming
752da83bb7
[Dashboard] Add the new dashboard code and prompt users to try it (#11667) 2021-01-29 15:22:26 +08:00
Tao Wang
56ee6ef55f
[GCS]only update states related fields when publish actor table data (#13448) 2021-01-28 11:12:57 +08:00
Clark Zinzow
2d34e95c93
Don't gather check_parent_task on Windows, since it's undefined. (#13700) 2021-01-27 09:19:58 -08:00
Amog Kamsetty
d96a9fa192
Revert "Revert "[dashboard] Fix RAY_RAYLET_PID KeyError on Windows (#12948)" (#13572)" (#13685)
This reverts commit c4a710369b.
2021-01-25 10:35:25 -08:00
Ameer Haj Ali
b7dd7ddb52
deprecate useless fields in the cluster yaml. (#13637)
* prepare for head node

* move command runner interface outside _private

* remove space

* Eric

* flake

* min_workers in multi node type

* fixing edge cases

* eric not idle

* fix target_workers to consider min_workers of node types

* idle timeout

* minor

* minor fix

* test

* lint

* eric v2

* eric 3

* min_workers constraint before bin packing

* Update resource_demand_scheduler.py

* Revert "Update resource_demand_scheduler.py"

This reverts commit 818a63a2c86d8437b3ef21c5035d701c1d1127b5.

* reducing diff

* make get_nodes_to_launch return a dict

* merge

* weird merge fix

* auto fill instance types for AWS

* Alex/Eric

* Update doc/source/cluster/autoscaling.rst

* merge autofill and input from user

* logger.exception

* make the yaml use the default autofill

* docs Eric

* remove test_autoscaler_yaml from windows tests

* lets try changing the test a bit

* return test

* lets see

* edward

* Limit max launch concurrency

* commenting frac TODO

* move to resource demand scheduler

* use STATUS UP TO DATE

* Eric

* make logger of gc freed refs debug instead of info

* add cluster name to docker mount prefix directory

* grrR

* fix tests

* moving docker directory to sdk

* move the import to prevent circular dependency

* smallf fix

* ian

* fix max launch concurrency bug to assume failing nodes as pending and consider only load_metric's connected nodes as running

* small fix

* deflake test_joblib

* lint

* placement groups bypass

* remove space

* Eric

* first ocmmit

* lint

* exmaple

* documentation

* hmm

* file path fix

* fix test

* some format issue in docs

* modified docs

* joblib strikes again on windows

* add ability to not start autoscaler/monitor

* a

* remove worker_default

* Remove default pod type from operator

* Remove worker_default_node_type from rewrite_legacy_yaml_to_availble_node_types

* deprecate useless fields

Co-authored-by: Ameer Haj Ali <ameerhajali@ameers-mbp.lan>
Co-authored-by: Alex Wu <alex@anyscale.io>
Co-authored-by: Alex Wu <itswu.alex@gmail.com>
Co-authored-by: Eric Liang <ekhliang@gmail.com>
Co-authored-by: Ameer Haj Ali <ameerhajali@Ameers-MacBook-Pro.local>
Co-authored-by: root <root@ip-172-31-56-188.us-west-2.compute.internal>
Co-authored-by: Dmitri Gekhtman <dmitri.m.gekhtman@gmail.com>
2021-01-23 12:06:51 -08:00
Amog Kamsetty
c4a710369b
Revert "[dashboard] Fix RAY_RAYLET_PID KeyError on Windows (#12948)" (#13572)
This reverts commit ef6d859e9b.
2021-01-22 14:10:24 -06:00
Tao Wang
aa5d7a5e6c
[Dashboard]Don't set node actors when node_id of actor is Nil (#13573)
* Don't set node actors when node_id of actor is Nil

* add test per comment
2021-01-21 20:18:34 -08:00
Xianyang Liu
4ecd29ea2b
[dashboard] Fixes dashboard issues when environments have set http_proxy (#12598)
* fixes ray start with http_proxy

* format

* fixes

* fixes

* increase timeout

* address comments
2021-01-21 20:10:01 -08:00
Simon Mo
dac8b3d58a
[CI] Enable Dashboard tests for master (#13425) 2021-01-15 09:43:34 -08:00
Simon Mo
321bbe1ffb
[Dashboard] Fix GPU resource rendering issue (#13388) 2021-01-14 12:23:21 -08:00
fyrestone
4853aa96cb
[Dashboard] Fix missing actor pid (#13229) 2021-01-13 16:45:12 +08:00
fyrestone
a6d135a072
[Dashboard] Add GET /log_proxy API (#13165) 2021-01-08 11:45:07 +08:00
SangBin Cho
32dc5676b4
[Metrics] Record per node and raylet cpu / mem usage (#12982)
* Record per node and raylet cpu / mem usage

* Add comments.

* Addressed code review.
2021-01-05 21:57:21 -08:00
Edward Oakes
ef6d859e9b
[dashboard] Fix RAY_RAYLET_PID KeyError on Windows (#12948) 2020-12-31 10:54:40 -06:00