Commit graph

183 commits

Author SHA1 Message Date
mwtian
cb8dc5c94e
Fix unused import warning in streaming.proto (#19912)
## Why are these changes needed?
This generates a warning when calling `protoc` on the proto.

## Related issue number
2021-10-31 13:29:51 -07:00
Jiajun Yao
805ce453dd
[Java] Remove auto-generated pom.xml files. (#19475) 2021-10-19 17:35:37 +08:00
Gagandeep Singh
d226cbf21a
Added StartupToken to idenitfy a process at startup (#19014)
* Added StartupToken to idenitfy a process at startup

* Applied linting formats

* Addressed reviews

* Fixing worker_pool_test

* Fixed worker_pool_test

* Applied linting formatting

* Added documentation for StartupToken

* Fixed linting

* Reordered initialisation of WorkerPool members

* Fixed Python docs

* Fixing bugs in cluster_mode_test

* Fixing Java tests

* Create and set shim process after verifying startup_token

* shim_process.GetId() -> worker_shim_pid

* Improvements in startup token and modifying java files

* update io_ray_runtime_RayNativeRuntime.h

* Fixed java tests by adding startup-token to conf

* Applied linting

* Increased arg count for startup_token

* Attempt to fix streaming tests

* Type correction

* applied linting

* Corrected index of startup token arg

* Modified, mock_worker.cc to accept startup tokens

* Applied linting

* Applied linting changes from CI

* Removed override from worker.h

* Applied linting from scripts/format.sh

* Addressed reviews and applied scripts/format.sh

* Applied linting script from ci/travis

* Removed unrequired methods from public scope

* Applied linting
2021-10-15 15:13:13 -07:00
Guyang Song
ab55b808c5
[runtime env] move worker env to runtime env in Java (#19060) 2021-10-11 17:25:09 +08:00
gjoliver
635010d460
Update build rules and patches for darwin_arm64 platform. (#19037)
* Update build rules and patches for darwin_arm64 platform.

Changes include:

Update nelhage/rules_boost package from current version (08/5/2020) to 5/27/2021 version.
Remove rules_boost-undefine-boost_fallthrough.patch, since BOOST_FALLTHROUGH seems to be defined now.
Minor changes to rules_boost-windows-linkopts.patch to use default condition to add -lpthread flag for all platforms.
Add darwin_arm64 config to BUILD files for lib civetweb pulled in via prometheu dependency.

* upgrade boost to 1.74.0 from 1.71.0 to match the udpated build file for windows.

* Fix ray_cpp_pkg

* Use boost/bind/bind.hpp

boost/bind.hpp and global namespace placeholders are deprecated.

* lint

* Use absl::bind_front when possible. Otherwise, NOLINT

* lint

* lint

* lint

* lint

* more lint

* final lint

* trigger build
2021-10-09 18:48:35 -07:00
SangBin Cho
0ef0d9a77d
Revert "[core] Assign tasks to the first available worker (#18167)" (#19180)
This reverts commit 545db13800.
2021-10-07 10:38:37 -07:00
Stephanie Wang
545db13800
[core] Assign tasks to the first available worker (#18167)
* Convert worker pool to queue

* Start up to backlog size more workers

* fixes

* Prestart workers according to num available CPUs

* lint

* x

* Update src/ray/raylet/worker_pool.h

Co-authored-by: Eric Liang <ekhliang@gmail.com>

* Update src/ray/raylet/worker_pool.h

Co-authored-by: Eric Liang <ekhliang@gmail.com>

* dedicated workers

* Fix tests

* x

* fix

* asan

* asan

* Workers can only exec tasks with same job ID

* size_t for runtime env hash, fix unit tests

* include job ID in runtime env hash, remove from worker registration msg

* x

* conflict

* debug

* Schedule and dispatch periodically, skip if no new tasks

* Update src/ray/common/task/task_spec.h

Co-authored-by: Eric Liang <ekhliang@gmail.com>

* Update src/ray/raylet/scheduling/cluster_task_manager.h

Co-authored-by: Eric Liang <ekhliang@gmail.com>

* Update src/ray/raylet/worker_pool.h

Co-authored-by: Eric Liang <ekhliang@gmail.com>

Co-authored-by: Eric Liang <ekhliang@gmail.com>
2021-10-05 13:45:50 -07:00
Lingxuan Zuo
a67b9ee8d7
Remove custom resource from streaming (#18490) 2021-09-12 12:20:59 -07:00
Yi Cheng
7126d01c91
[core] upgrade gtest (#18288)
* up

* up

* format

* up

* flaky fix

* format

* up

* up

* format

* add debug

* up

* up

* up

* up

* up

* format

* fix

* format

* up

* up

* format
2021-09-08 11:15:34 -07:00
Lingxuan Zuo
46b941b702
[Streaming] Support streaming metric reporter (#17981)
* Streaming support metric reporter

* fix lint

* fix bazel format lint

* fix lint

* metric deps lint

* lint

* and comments for runtime reporter

* unordered_map instead

* comments

* fix visibility flag

* deps local .so target

* make stats public visibility

* stats lib in public

* add antgroup team tag
2021-09-08 14:36:00 +08:00
Stephanie Wang
d43d297d9a
[core] Attach call site to ObjectRefs, print on error (#17971)
* Attach call site to ObjectRef

* flag

* Fix build

* build

* build

* build

* x

* x

* skip on windows

* lint
2021-09-01 15:29:05 -07:00
Eric Liang
1adce7da4e
Revert "Auto discover dashboard agent port (#17855)" (#18217)
This reverts commit 53ddb551d5.
2021-08-30 10:46:37 -07:00
Zhi Lin
d3786ac131
Bump Java version to 2.0.0-SNAPSHOT (#15394)
* bump java version to 2.0.0-SNAPSHOT

* update
2021-08-30 12:25:30 +08:00
fyrestone
53ddb551d5
Auto discover dashboard agent port (#17855) 2021-08-30 12:06:28 +08:00
Stephanie Wang
b8fe776638
[core] Fix inlined nested ids (#17834)
* test

* Use ObjectRef instead of ObjectID in nested refs

* java

* doc

* java

* build

* build

* x

* lint

* simplify

* fix
2021-08-20 08:58:29 -07:00
Eric Liang
661ac4e37b
Remove last traces of ref-counting flag (#17932) 2021-08-19 21:08:13 -07:00
Clark Zinzow
d958457d07
[Core] Second pass at privatizing APIs. (#17885)
* gcs_utils

* resource_spec

* profiling

* ray_perf and ray_cluster_perf

* test_utils
2021-08-18 20:56:33 -07:00
Simon Mo
b573864928
[CI] Add test owners (#17893) 2021-08-18 18:38:31 -07:00
Hao Chen
0858f0e4f2
Change core worker C++ namespace to ray::core (#17610) 2021-08-08 23:34:25 +08:00
Tao Wang
d98ec7fc4d
Remove libray_redis_module (#17283) 2021-07-25 23:15:29 -07:00
lantian-xu
daf37b7621
[Streaming] Fix illegal cast when rollbacking. (#17195)
Co-authored-by: yz54123 <57480840+yz54123@users.noreply.github.com>
2021-07-23 13:08:34 +08:00
Siyuan (Ryans) Zhuang
8efc04a8a6
[Core] Actor namespace (#17178)
* set actor namespace in Python on creation

* get actor with namespace in Python

* update message
2021-07-19 21:51:04 -07:00
Amog Kamsetty
503b748d64
Revert "Revert "[Java] upgrade jar deps to fix cves" (#16889)" (#16899)
This reverts commit f2308a0cdf.
2021-07-06 14:00:50 -07:00
Amog Kamsetty
f2308a0cdf
Revert "[Java] upgrade jar deps to fix cves" (#16889)
This reverts commit 25666fff81.
2021-07-06 10:33:31 -07:00
chaokunyang
25666fff81
[Java] upgrade jar deps to fix cves (#16870) 2021-07-06 09:04:21 -07:00
Qing Wang
3272997b0d
[Java] Upgrade some deps to fix CVEs (#16650) 2021-06-24 10:56:20 -07:00
architkulkarni
194c5e3a96
[Core] Cache workers by runtime_env in worker pool (#15782)
* pass RuntimeEnv in task spec as opaque string

* lint

* set correct empty value for json: "{}" not ""

* add comment for field in proto

* fix worker pool test by checking both "" and "{}"

* add RAY_CHECK todo

* make dict empty if all values null

* remove unnecessary ser/de

* fix

* address comments

* add WorkerCacheKey with hash function

* clean up

* add naive impl., dedicated workers never killed

* put dedicated workers in idle_of_all_languages

* pipe env hash from worker.py -> Worker

* fully pipe through hash, basic cache test passing

* use int type for runtime env hash

* convert Worker env hash type from size_t to int

* fix

* add method to MockWorker to fix cpp tests

* make compatible with java streaming test

* restore old dynamic_options code to fix java test

* address comments

* add comment about sorting before hash

* add comments for private members of WorkerCacheKey
2021-05-18 00:19:27 -07:00
Amog Kamsetty
ebc44c3d76
[CI] Upgrade flake8 to 3.9.1 (#15527)
* formatting

* format util

* format release

* format rllib/agents

* format rllib/env

* format rllib/execution

* format rllib/evaluation

* format rllib/examples

* format rllib/policy

* format rllib utils and tests

* format streaming

* more formatting

* update requirements files

* fix rllib type checking

* updates

* update

* fix circular import

* Update python/ray/tests/test_runtime_env.py

* noqa
2021-05-03 14:23:28 -07:00
Siyuan (Ryans) Zhuang
87c79553e9
[Core] Remove code paths that contains plasma store executable (#14950)
* remove plasma store executable & never used tests

* set default behavior

* fix tests
2021-03-28 21:22:14 -07:00
Siyuan (Ryans) Zhuang
b92531918e
Make use of C++14 'make_unique' (#14663) 2021-03-15 03:00:52 -07:00
Kai Yang
92f1e0902e
[Java] Fix return of java doc (#13601) 2021-01-21 23:57:20 +08:00
chaokunyang
33089c44e2
Fix streaming ci failure (#12830) 2020-12-30 10:45:52 +08:00
chaokunyang
d1dd3410c8
[Java] Format ray java code (#13056) 2020-12-29 10:36:16 +08:00
Yi Cheng
40032541dc
[core] Introduce fetch_local to ray.wait (#12526) 2020-12-16 23:44:28 -08:00
Lixin Wei
867d2a8aa3
[Streaming] Add more documents. (#12746)
* add doc

draft

draft

draft

draft

draft

fix

fix

fix

fix

fix

fix

fix

Update README.md

fix

fix

fix

* md to rst

* fix

* fix

* fix

* jpg modified

* add getting envolved

* jpg modified

* Update README.rst

* fix

* fix
2020-12-11 20:36:17 +08:00
Sumanth Ratna
932837eb4c
[streaming] Remove unused imports in streaming CI tests (#12722) 2020-12-10 16:27:06 -06:00
Kai Yang
e3b5deb741
[Multi-tenancy] Delete flag enable_multi_tenancy and remove old code path (#10573) 2020-12-10 19:01:40 +08:00
Philipp Moritz
73a1a232b9
Ray debugger stepping between tasks (#12075) 2020-12-06 21:50:18 -08:00
Kai Yang
21fcee28f9
[Java] Simplify Ray.init() by invoking ray start internally (#10762) 2020-12-04 14:33:45 +08:00
chaokunyang
bea0031491
fix linux wheel build (#9896) 2020-11-17 15:49:42 +08:00
chaokunyang
1979ea9c0a
fix disable javadoc lint (#11907) 2020-11-11 13:40:50 +08:00
Kai Yang
904f48ebd9
[Core] Multi-tenancy: Pass job ID from Raylet to worker via env variable (#11829)
* Pass job ID from Raylet to worker via env variable

* fix

* fix

* fix

* lint

* fix

* fix test_object_spilling

* address comments

* lint

* fix
2020-11-09 11:02:15 -08:00
Lingxuan Zuo
2f3ec4ef75
add streaming data writer unit tests (#11387) 2020-10-16 14:42:44 +08:00
Lingxuan Zuo
c84a9b457c
[Streaming] add barrier helper tests (#11107) 2020-10-13 09:55:55 +08:00
chaokunyang
0737e78445
[Java] upgrade common-collections version (#10613) 2020-10-10 11:16:12 +08:00
Kai Yang
0c254878db
[Java] Upgrade TestNG to 7.3.0 (#11135) 2020-09-30 14:38:52 +08:00
chaokunyang
842861b4fc
[Streaming] refine streaming tests sleep condition (#10991) 2020-09-24 17:06:55 +08:00
Alex Wu
9a07c7b963
[1.0] Remove args from ray start (#10659) 2020-09-18 16:41:23 -07:00
chaokunyang
322f12600f
bump java version to 1.1.0 (#10795) 2020-09-16 12:05:11 +08:00
Hao Chen
1a5cfe0b79
[Java] rename config ray.redis.address to ray.address (#10772) 2020-09-15 11:13:19 +08:00