Commit graph

410 commits

Author SHA1 Message Date
Kai Yang
a98cd0670e
[Java] Improve JNI performance when submitting and executing tasks (#9032) 2020-07-10 17:51:07 +08:00
Zhuohan Li
8a76f4cbb5
[Core] put small objects in memory store (#8972)
* remove the put in memory store

* put small objects directly in memory store

* cast data type

* fix another place that uses Put to spill to plasma store

* fix multiple tests related to memory limits

* partially fix test_metrics

* remove not functioning codes

* fix core_worker_test

* refactor put to plasma codes

* add a flag for the new feature

* add flag to more places

* do a warmup round for the plasma store

* lint

* lint again

* fix warmup store

* Update _raylet.pyx

Co-authored-by: Eric Liang <ekhliang@gmail.com>
2020-07-09 15:39:40 -07:00
Ian Rodney
9172f8c3a6
[core] Store Internal Config in GCS (#8921) 2020-07-08 11:22:08 -05:00
Stephanie Wang
b42d6a1ddc
[core] Refactor task arguments and attach owner address (#9152)
* Add intended worker ID to GetObjectStatus, tests

* Remove TaskID owner_id

* lint

* Add owner address to task args

* Make TaskArg a virtual class, remove multi args

* Set owner address for task args

* merge

* Fix tests

* Fix

* build

* update

* build

* java

* Move code

* build

* Revert "Fix Google log directory again (#9063)"

This reverts commit 275da2e400.

* Fix free

* x

* build

* Fix java

* Revert "Revert "Fix Google log directory again (#9063)""

This reverts commit 4a326fcb148ca09a35bc7de11d89df10edbb56e7.

* lint
2020-07-06 21:25:14 -07:00
mehrdadn
7135cb2aec
Fix .exe file extensions (#9197)
Co-authored-by: Mehrdad <noreply@github.com>
2020-07-02 15:29:34 -05:00
chaokunyang
b5fafe1fce
[Java] fix java test (#9079) 2020-06-22 16:56:47 +08:00
Zhilei Chen
d8a9247448
Remove gcs_service_disabled ci jobs and code (#8854) 2020-06-19 11:32:27 +08:00
chaokunyang
cb6f337372
[Java] Refine python function (#8943) 2020-06-16 16:22:49 +08:00
chaokunyang
dfa4768fc6
[Java] Refactor java api (#8858) 2020-06-12 10:49:01 +08:00
chaokunyang
700d81fa20
[Java] Remove java api sub package from test module (#8853) 2020-06-11 14:59:45 +08:00
chaokunyang
31a4d07bc4
[Java] Rename java ObjectRef/ActorHandle (#8799) 2020-06-09 11:40:43 +08:00
chaokunyang
4a6c4003fd
[Java] fix serializer test location (#8730) 2020-06-05 15:02:08 +08:00
Siyuan (Ryans) Zhuang
ea05ebe89e
Ship plasma store with Ray (#7901) 2020-06-03 17:44:34 -07:00
Hao Chen
dc3e98890f
Improve Java logging (#8640) 2020-06-02 16:15:20 +08:00
Lingxuan Zuo
4cbbc15ca7
[GCS] Global state accessor from node resource table (#8658) 2020-06-02 14:01:00 +08:00
fangfengbin
35eeec5647
Add C++ global state for actor table (#8501)
* add global state actors

* fix code style

* fix GcsActorManagerTest bug

* rebase master

* add jni code

* add get checkpoint id code

* add debug code

* add debug code

* change log level

* fix compile bug

* return null in jni

* fix crash bug

* change import seq

Co-authored-by: 灵洵 <fengbin.ffb@antfin.com>
Co-authored-by: Hao Chen <chenh1024@gmail.com>
2020-05-29 21:10:42 +08:00
Lingxuan Zuo
e594524ed3
[GCS] global state query node info table from GCS. (#8498) 2020-05-28 16:39:13 +08:00
Lingxuan Zuo
bd4fbcd7fc
Global state accessor jni (#8637) 2020-05-27 17:43:47 +08:00
Kai Yang
26cffb9c7c
Fix shutdown hook in worker mode (#8098) 2020-05-26 15:23:44 +08:00
Edward Oakes
a76434ccde
Add ability to specify worker and driver ports (#8071) 2020-05-20 15:31:13 -05:00
Edward Oakes
16f48078d9
Remove use of ObjectID transport flag (#7699) 2020-05-17 11:29:49 -05:00
Max Fitton
00325eb2b2
Rename max_reconstructions to max_restarts and use -1 for infinite (#8274)
Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com>
2020-05-14 10:30:29 -05:00
Kai Yang
1d5bceddf0
fix java UT about multi-threading (#8014) 2020-04-27 15:11:22 +08:00
mehrdadn
f15618033d
Remove --no-transfer-progress as it appears to be unsupported (#8066)
Co-authored-by: Mehrdad <noreply@github.com>
2020-04-17 16:30:37 +02:00
mehrdadn
ba00c29b67
Factor out Travis 'install' sections for use with GitHub Actions (#7988) 2020-04-15 08:10:22 -07:00
Qing Wang
dfb0ad0d3e
[Java] Fix Java CI exit code issue (#8028) 2020-04-15 15:28:52 +08:00
ZhuSenlin
4a81793ba5
GCS-Based actor management implementation (#6763)
* add gcs actor manager

* fix test_metrics.py

* fix TestTaskInfo

* fix comment

* fix comment

* fix comment

* fix comment

* fix comment

* fix comment

* fix compile error

* fix merge error

Co-authored-by: senlin.zsl <senlin.zsl@antfin.com>
2020-04-13 09:48:48 -07:00
Qing Wang
98bfcd53bc
[Java] Rename group id and package name. (#7864)
* Initial

* Change streaming's

* Fix

* Fix

* Fix org_ray

* Fix cpp file name

* Fix streaming

* Fix

* Fix

* Fix testlistening

* Fix missing sth in python

* Fix

* Fix

* Fix SPI

* Fix

* Fix complation

* Fix

* Fix CI

* Fix checkstyle

Fix checkstyle

* Fix streaming tests

* Fix streaming CI

* Fix streaming checkstyle.

* Fix build

* Fix bazel dep

* Fix

* Fix ray checkstyle

* Fix streaming checkstyle

* Fix bazel checkstyle
2020-04-12 17:59:34 +08:00
mehrdadn
3061067039
Fix bug in java/test.sh (#7952)
Co-authored-by: Mehrdad <noreply@github.com>
2020-04-11 23:56:14 -07:00
mehrdadn
4aa68b82fa
[CI] Various Improvements to Travis Scripts (#7956)
* Delete LINT section of install-ray.sh since it appears unused

* Delete install.sh since it appears unused

* Delete run_test.sh since it appears unused

* Put environment variables on separate lines in .travis.yml

* Move --jobs 50 out of install-ray.sh

* Delete upgrade-syn.sh since it appears unused

* Move CI bazel flags to .bazelrc via --config

* Make installations quieter

* Get rid of verbose Maven messages

* Install Bazel system-wide for CI so that there's no need to update PATH

* Recognize Windows as valid platform

Co-authored-by: Mehrdad <noreply@github.com>
2020-04-10 13:26:28 -07:00
fyrestone
fc6259a656
Cross language serialization for primitive types (#7711)
* Cross language serialization for Java and Python

* Use strict types when Python serializing

* Handle recursive objects in Python; Pin msgpack >= 0.6.0, < 1.0.0

* Disable gc for optimizing msgpack loads

* Fix merge bug

* Java call Python use returnType; Fix ClassLoaderTest

* Fix RayMethodsTest

* Fix checkstyle

* Fix lint

* prepare_args raises exception if try to transfer a non-deserializable object to another language

* Fix CrossLanguageInvocationTest.java, Python msgpack treat float as double

* Minor fixes

* Fix compile error on linux

* Fix lint in java/BUILD.bazel

* Fix test_failure

* Fix lint

* Class<?> to Class<T>; Refine metadata bytes.

* Rename FST to Fst; sort java dependencies

* Change Class<?>[] to Optional<Class<?>>; sort requirements in setup.py

* Improve CrossLanguageInvocationTest

* Refactor MessagePackSerializer.java

* Refactor MessagePackSerializer.java; Refine CrossLanguageInvocationTest.java

* Remove unnecessary dependencies for Java; Add getReturnType() for RayFunction in Java

* Fix bug

* Remove custom cross language type support

* Replace Serializer.Meta with MutableBoolean

* Remove @SuppressWarnings support from checkstyle.xml; Add null test in CrossLanguageInvocationTest.java

* Refine MessagePackSerializer.pack

* Ray.get support RayObject as input

* Improve comments and error info

* Remove classLoader argument from serializer

* Separate msgpack from pickle5 in Python

* Pair<byte[], MutableBoolean> to Pair<byte[], Boolean>

* Remove public static <T> T get(RayObject<T> object), use RayObject.get() instead

* Refine test

* small fixes

Co-authored-by: 刘宝 <po.lb@antfin.com>
Co-authored-by: Hao Chen <chenh1024@gmail.com>
2020-04-08 21:10:57 +08:00
Kai Yang
48b48cc8c2
Support multiple core workers in one process (#7623) 2020-04-07 11:01:47 +08:00
fangfengbin
e196fcdbaf
Add gcs_service_enabled function to avoid getting environment variable directly (#7742) 2020-03-26 22:02:53 +08:00
fangfengbin
bf866de6fd
Enable GCS Service by default (#7541) 2020-03-24 14:20:23 +08:00
Robert Nishihara
ee8c9ff732
Remove six and cloudpickle from setup.py. (#7694) 2020-03-23 11:42:05 -07:00
Robert Nishihara
1a0c9228d0
Remove pytest from setup.py and other minor changes. (#7700) 2020-03-23 08:46:56 -07:00
fyrestone
a1ae935839
Java call Python use structured function descriptors (#7634) 2020-03-20 17:29:45 +08:00
Hao Chen
7678418210
[Java] Fix the issue that the cached value in RayObject is serialized (#7613) 2020-03-17 22:07:41 +08:00
Kai Yang
6b888b0247
[Java] Make both RayActor and RayPyActor inheriting from BaseActor (#7462) 2020-03-17 21:45:56 +08:00
fyrestone
7697ea2be2
Java call Python actor method use actor.call (#7614) 2020-03-17 14:52:43 +08:00
Kai Yang
630e48967d
[Java] Allow passing internal config from raylet to Java worker (#7532) 2020-03-15 12:03:38 +08:00
Qing Wang
d6365c2586
[Java] Enable stress test. (#7596) 2020-03-13 21:02:13 +08:00
Kai Yang
d6e8f47065
Add a flag to disable reconstruction for a killed actor (#7346) 2020-03-13 19:10:21 +08:00
Qing Wang
575c89cf47
[Java] Pass large object by reference (#7595) 2020-03-13 18:38:03 +08:00
Qing Wang
f4656d8cc3
[Java] Enable direct call by default. (#7408)
* WIP

* Address comments.

* Linting

* Fix

* Fix

* Fix test

* Fix

* Fix single process ci

* Fix ut

* Update java/test/src/main/java/org/ray/api/test/PlasmaFreeTest.java

* Address comments

* Fix linting

* Minor update comments.

* Fix streaming CI
2020-03-13 12:25:30 +08:00
Kai Yang
932a749fa9
Fix the java_worker_options parameter (#7537)
* fix Java CI

* Minor fix

* move json.loads out of build_java_worker_command

* lint

* fix cross language test
2020-03-12 10:44:23 +08:00
Stephanie Wang
fdb528514b
[core] Ref counting for actor handles (#7434)
* tmp

* Move Exit handler into CoreWorker, exit once owner's ref count goes to 0

* fix build

* Remove __ray_terminate__ and add test case for distributed ref counting

* lint

* Remove unused

* Fixes for detached actor, duplicate actor handles

* Remove unused

* Remove creation return ID

* Remove ObjectIDs from python, set references in CoreWorker

* Fix crash

* Fix memory crash

* Fix tests

* fix

* fixes

* fix tests

* fix java build

* fix build

* fix

* check status

* check status
2020-03-10 17:45:07 -07:00
fangfengbin
fa785a2ad2
ServiceBasedGcsClient support detect gcs server availability and retry (#7292) 2020-03-10 21:01:07 +08:00
Qing Wang
7a33a6ea3c
[Java] Enable skipped direct call cases (#7363)
* Comment out

* Refine

* Revert
2020-03-06 16:22:08 +08:00
Hao Chen
fe7820fec9
[Java] New Java actor API (#7414) 2020-03-04 22:39:23 +08:00