Edward Oakes
10947c83b3
[runtime_env] Make pip installs incremental ( #20341 )
...
Uses a direct `pip install` instead of creating a conda env to make pip installs incremental to the cluster environment.
Separates the handling of `pip` and `conda` dependencies.
The new `pip` approach still works if only the base Ray is installed on the cluster and the user specifies libraries like "ray[serve]" in the `pip` field. The mechanism is as follows:
- We don't actually want to reinstall ray via pip, since this could lead to version mismatch issues. Instead, we want to use the Ray that's already installed in the cluster.
- So if "ray" was included by the user in the pip list, remove it
- If a library "ray[serve]" or "ray[tune, rllib]" was included in the pip list, remove it and replace it by its dependencies (e.g. "uvicorn", "requests", ..)
Co-authored-by: architkulkarni <arkulkar@gmail.com>
Co-authored-by: architkulkarni <architkulkarni@users.noreply.github.com>
2021-12-14 15:55:18 -08:00
Guyang Song
53630ee03b
Revert "Revert "[runtime env] redefine runtime env to protobuf"" and fix windows compiling ( #20692 )
...
- Fix windows compiling and revert https://github.com/ray-project/ray/pull/20641
- Seems the pr https://github.com/ray-project/ray/pull/20670 can solve the windows compiling issue.
2021-11-24 09:01:01 -08:00
Alex Wu
9388d28233
Revert "[runtime env] redefine runtime env to protobuf" ( #20641 )
...
Reverts #19511
Breaks windows compilation
2021-11-22 13:11:30 -08:00
Guyang Song
ad56b9b432
[runtime env] redefine runtime env to protobuf ( #19511 )
2021-11-20 16:54:42 +08:00
Yi Cheng
e54d3117a4
[gcs] Update all redis kv usage in python except function table ( #20014 )
...
## Why are these changes needed?
This is part of redis removal project. In this PR all direct usage of redis got removed except function table.
Function table will be migrated in the next PR
## Related issue number
#19443
2021-11-10 20:24:53 -08:00
architkulkarni
c5175073b2
[runtime env] Add garbage collection for conda envs ( #20072 )
2021-11-04 23:13:34 -05:00
architkulkarni
bcb63961d9
[runtime env] Add plugin name to internal URI format and add GC for py_modules ( #20009 )
2021-11-04 10:16:14 -05:00
chenk008
57363995f3
[runtime env] Move container related code to runtime env ( #19067 )
2021-10-29 16:31:11 -07:00
Edward Oakes
b2e12dc43b
[runtime_env] Add basic support for python modules ( #19651 )
2021-10-27 17:56:46 -05:00
chenk008
3780a73b45
[Core] Add worker resource info to runtime env ( #18804 )
2021-10-08 10:37:29 -07:00
Edward Oakes
1fa81673bd
[runtime_env] Clean up validation logic ( #18984 )
...
Splits the runtime_env parsing/validation and overriding into two separate codepaths. Adds unit testing for both.
2021-10-07 14:24:41 -05:00
Simon Mo
9b2a368c8c
[Runtime Env] Implement basic runtime env plugin mechanism ( #19044 )
2021-10-01 17:22:54 -07:00
architkulkarni
fbf5f5d56b
[runtime env] [Serve] Fix error when uris field is None ( #18874 )
2021-09-24 14:07:17 -05:00
Edward Oakes
7736cdd91d
[dashboard] Rename "new_dashboard" -> "dashboard" ( #18214 )
2021-09-15 11:17:15 -05:00
Edward Oakes
111a31d6a1
[runtime_env] Make Ray client server setup go through the runtime_env agent ( #18478 )
2021-09-13 14:16:35 -05:00
Edward Oakes
c482779da2
[runtime_env] Improve file-not-found msg in deletion ( #18496 )
2021-09-13 11:32:22 -05:00
Edward Oakes
2fcfea10b3
[runtime_env] Move URI deletion logic to the agent, remove util worker code ( #18471 )
2021-09-10 00:13:32 -07:00
Edward Oakes
f0555f88d6
[runtime_env] Move worker process startup logic to context ( #18341 )
2021-09-08 17:08:27 -05:00
Edward Oakes
1f6705d35d
[runtime_env] Centralize runtime_env logic into ray._private.runtime_env submodule ( #18310 )
2021-09-03 10:19:00 -05:00
Edward Oakes
5d122cf7b7
[runtime_env] Move working dir setup to the agent ( #18170 )
2021-08-31 17:22:49 -05:00
architkulkarni
ac9a1a20df
[core] [runtime_env] Use per-env async lock in agent ( #17542 )
...
Co-authored-by: Ed Oakes <ed.nmi.oakes@gmail.com>
2021-08-06 11:11:37 -05:00
Edward Oakes
7007c6271d
[runtime_env] Gracefully fail tasks when an environment fails to be set up ( #17249 )
2021-07-28 15:25:02 -05:00
Simon Mo
4a4210a083
Support streaming output of runtime env setup to logger/driver ( #17306 )
2021-07-27 16:39:15 -07:00
architkulkarni
bcb3a6789b
[Core] [runtime env] Cache created runtime envs ( #17342 )
2021-07-26 14:37:40 -05:00
SongGuyang
e74d9d3ded
[runtime env] Download runtime env(conda) in agent instead of setup_worker ( #16525 )
2021-06-25 19:39:05 +08:00
SongGuyang
874e947d6f
[runtime env] support create or delete runtime envs in agent ( #15904 )
2021-06-09 20:22:25 +08:00
