Commit graph

269 commits

Author SHA1 Message Date
Qing Wang
7bda5edc16 Fix Java CI failure (#4995) 2019-06-19 11:36:21 +08:00
Hao Chen
2bf92e02e2
[gRPC] Use gRPC for inter-node-manager communication (#4968) 2019-06-17 19:00:50 +08:00
Qing Wang
e6baffba56
[Java] Add inner class Builder to build call options. (#4956)
* Add Builder class

* format

* Refactor by IDE

* Remove uncessary dependency
2019-06-10 23:52:08 +08:00
Qing Wang
671c0f769e
[Java] Fix serializing issues of RaySerializer (#4887)
* Fix

* Address comment.
2019-06-08 22:56:00 +08:00
Yuhong Guo
1a39fee9c6
Refactor ID Serial 1: Separate ObjectID and TaskID from UniqueID (#4776)
* Enable BaseId.

* Change TaskID and make python test pass

* Remove unnecessary functions and fix test failure and change TaskID to
16 bytes.

* Java code change draft

* Refine

* Lint

* Update java/api/src/main/java/org/ray/api/id/TaskId.java

Co-Authored-By: Hao Chen <chenh1024@gmail.com>

* Update java/api/src/main/java/org/ray/api/id/BaseId.java

Co-Authored-By: Hao Chen <chenh1024@gmail.com>

* Update java/api/src/main/java/org/ray/api/id/BaseId.java

Co-Authored-By: Hao Chen <chenh1024@gmail.com>

* Update java/api/src/main/java/org/ray/api/id/ObjectId.java

Co-Authored-By: Hao Chen <chenh1024@gmail.com>

* Address comment

* Lint

* Fix SINGLE_PROCESS

* Fix comments

* Refine code

* Refine test

* Resolve conflict
2019-05-22 14:46:30 +08:00
Qing Wang
081708bdef [Java] Dynamic resource API in Java (#4824) 2019-05-21 17:13:48 +08:00
Hao Chen
643f62dc43
[Java][Bazel] Refine auto-generated pom files (#4780) 2019-05-16 11:19:31 +08:00
Ruifang Chen
c578be23a5 [Bazel] Modifying WORKSPACE file, so that you can make the project used as a thirdparty project (#4711)
* Modifying WORKSPACE file, so that you can make the project used as a thirdparty

* Modifying WORKSPACE file, so that you can make the project used as a thirdparty

* add some files

* modify some repositories

* modify the name of 'ray_deps_build_all'
2019-04-28 22:02:49 -07:00
Qing Wang
4dd628a837 [Java] Do not expose GcsClient to normal users (#4675) 2019-04-24 10:50:35 +08:00
Hao Chen
d52b080081
[Java] Avoid unnecessary memory copy and addd a benchmark (#4611) 2019-04-14 00:17:04 +08:00
Romil Bhardwaj
0f42f87ebc Updating zero capacity resource semantics (#4555) 2019-04-12 16:53:57 -07:00
Wang Qing
5cfbfe5df6 [Java] Implement GcsClient (#4601) 2019-04-12 22:44:47 +08:00
Wang Qing
fe07a5b4b1 Add delete_creating_tasks option for internal.free() (#4588)
* add delete creating task objects.

* format code style

* Fix lint

* add tests add address comments.

* Refine test

* Refine java test

* Fix CI

* Refine

* Fix lint

* Fix CI
2019-04-12 13:38:31 +08:00
bibabolynn
ab55a1f93a [Java] Clean up outdated dependencies (#4489) 2019-03-28 14:33:45 +08:00
Robert Nishihara
a22bf1e511 Minor aesthetic changes to python file. (#4492) 2019-03-28 10:59:29 +08:00
bibabolynn
7a9d1546d4 [java] Fix getWorker and add create multi actors test (#4472) 2019-03-26 20:26:13 +08:00
Ruifang Chen
59d74d5e92 [Java] Build Java code with Bazel (#4284) 2019-03-22 14:30:05 +08:00
Hao Chen
d03999d01e
Cross-language invocation Part 1: Java calling Python functions and actors (#4166) 2019-03-21 13:34:21 +08:00
Hao Chen
f8d12b0418
[Java] Package native dependencies into jar (#4367) 2019-03-15 12:38:40 +08:00
Hao Chen
f0465bc68c
[Java] Refine tests and fix single-process mode (#4265) 2019-03-07 09:59:13 +08:00
Wang Qing
a116b7f646 [Java] Add runtime context (#4194) 2019-03-05 20:25:29 +08:00
bibabolynn
c73d5086f3 [Java] Single-process mode (#4245) 2019-03-05 13:50:20 +08:00
Hao Chen
a99676e39b [Java] lint unused imports (#4100) 2019-02-20 12:37:04 -08:00
Hao Chen
de17443dc2
Propagate backend error to worker (#4039) 2019-02-16 11:39:15 +08:00
Hao Chen
f31a79f3f7
Implement actor checkpointing (#3839)
* Implement Actor checkpointing

* docs

* fix

* fix

* fix

* move restore-from-checkpoint to HandleActorStateTransition

* Revert "move restore-from-checkpoint to HandleActorStateTransition"

This reverts commit 9aa4447c1e3e321f42a1d895d72f17098b72de12.

* resubmit waiting tasks when actor frontier restored

* add doc about num_actor_checkpoints_to_keep=1

* add num_actor_checkpoints_to_keep to Cython

* add checkpoint_expired api

* check if actor class is abstract

* change checkpoint_ids to long string

* implement java

* Refactor to delay actor creation publish until checkpoint is resumed

* debug, lint

* Erase from checkpoints to restore if task fails

* fix lint

* update comments

* avoid duplicated actor notification log

* fix unintended change

* add actor_id to checkpoint_expired

* small java updates

* make checkpoint info per actor

* lint

* Remove logging

* Remove old actor checkpointing Python code, move new checkpointing code to FunctionActionManager

* Replace old actor checkpointing tests

* Fix test and lint

* address comments

* consolidate kill_actor

* Remove __ray_checkpoint__

* fix non-ascii char

* Loosen test checks

* fix java

* fix sphinx-build
2019-02-13 19:39:02 +08:00
Wang Qing
c523bc04ad Enable redis password in Java worker (#3943)
* Support Java redis password

* Fix

* Refine

* Fix lint.
2019-02-12 13:11:25 +08:00
Wang Qing
bc438ca73b [Java] Refine Java config item (#4014)
* Refine

* Address comment.
2019-02-11 23:55:40 +08:00
Yuhong Guo
3a66d47a3a
Remove RAY_CHECK from JNI code (#3978)
* Remove RAY_CHECK in JNI

* Try to add mvn test to test the exception.

* Refine

* Address comments
2019-02-09 18:10:22 +08:00
bibabolynn
728031a972 [java] when put an object in plasma store, ignore "object alreay exists" exception (#3687)
* distinct plasma client exception

* Update ObjectStoreProxy.java

* Update and rename PlasmaArrowTest.java to PlasmaStoreTest.java

* store put

* Use testng to replace junit to fix test failure
2019-02-09 18:03:17 +08:00
Ion
f987572795 Inline objects (#3756)
* added store_client_ to object_manager and node_manager

* half through...

* all code in, and compiling! Nothing tested though...

* something is working ;-)

* added a few more comments

* now, add only one entry to the in GCS for inlined objects

* more comments

* remove a spurious todo

* some comment updates

* add test

* added support for meta data for inline objects

* avoid some copies

* Initialize plasma client in tests

* Better comments. Enable configuring nline_object_max_size_bytes.

* Update src/ray/object_manager/object_manager.cc

Co-Authored-By: istoica <istoica@cs.berkeley.edu>

* Update src/ray/raylet/node_manager.cc

Co-Authored-By: istoica <istoica@cs.berkeley.edu>

* Update src/ray/raylet/node_manager.cc

Co-Authored-By: istoica <istoica@cs.berkeley.edu>

* fiexed comments

* fixed various typos in comments

* updated comments in object_manager.h and object_manager.cc

* addressed all comments...hopefully ;-)

* Only add eviction entries for objects that are not inlined

* fixed a bunch of comments

* Fix test

* Fix object transfer dump test

* lint

* Comments

* Fix test?

* Fix test?

* lint

* fix build

* Fix build

* lint

* Use const ref

* Fixes, don't let object manager hang

* Increase object transfer retry time for travis?

* Fix test

* Fix test?

* Add internal config to java, fix PlasmaFreeTest
2019-02-07 10:32:39 -08:00
Wang Qing
e1c68a0881 Enable including Java worker for ray start command (#3838) 2019-02-04 16:23:43 +08:00
Wang Qing
3cf59855af [Java] Replace junit with testNG (#3768) 2019-01-14 17:49:17 +08:00
Hao Chen
1bb20badec [Java] Fix bug when actor creation task fails (#3740)
* [Java] Fix bug when actor creation task fails

* remove imports
2019-01-14 11:09:15 +08:00
Wang Qing
8674606e26 Support to auto-generate Java files from flatbuffer (#3749)
* auto gen flatbuffers for Java

* Add auto_gen_tool.py

* Refine

* Add a comment

* address comments.

* Address comments.

* Addressed

* Refine

* Address comments

* Fix typo

* Add exception

* Address comments.

* Refine

* Fix lint

* Fix

* Fix lint and address comment.

* Fix lint error
2019-01-13 11:39:23 -08:00
Wang Qing
a0cf8ee5a8 Refine Java worker code (#3735) 2019-01-12 22:45:33 +08:00
Wang Qing
692fdc6bc3 [Java] Allow actor handle to be serialized without forking (#3686) 2019-01-06 00:29:08 +08:00
Wang Qing
c59b506c6e [Java] Support calling Ray APIs from multiple threads (#3646) 2018-12-28 17:44:31 +08:00
Wang Qing
a971b73bbe [Java] Fix the issue when waiting an empty list or a null pointer (#3632) 2018-12-26 11:29:29 +08:00
Wang Qing
8393df2516 Use BaseTest to instead of TestListener. (#3577) 2018-12-21 16:29:16 -08:00
bibabolynn
e65b8f18f4 [java] change RayLog.core to org.slf4j.Logger (#3579) 2018-12-21 15:58:32 +08:00
bibabolynn
7fd24e384b [java] Pass large args by reference (#3504) 2018-12-14 23:32:35 +08:00
Hao Chen
e7b51cbd1b [xray] Implement Actor Reconstruction (#3332)
* Implement Actor Reconstruction

* fix

* fix actor handle __del__

* fix lint

* add comment

* Remove actorCreationDummyObjectId

* address comments

* fix

* address comments

* avoid copy

* change log to debug

* fix error name
2018-12-13 21:28:58 -08:00
Hao Chen
abd37df41e Add stress test for Java worker (#3424) 2018-12-01 16:11:09 -08:00
Robert Nishihara
658c14282c Remove legacy Ray code. (#3121)
* Remove legacy Ray code.

* Fix cmake and simplify monitor.

* Fix linting

* Updates

* Fix

* Implement some methods.

* Remove more plasma manager references.

* Fix

* Linting

* Fix

* Fix

* Make sure class IDs are strings.

* Some path fixes

* Fix

* Path fixes and update arrow

* Fixes.

* linting

* Fixes

* Java fixes

* Some java fixes

* TaskLanguage -> Language

* Minor

* Fix python test and remove unused method signature.

* Fix java tests

* Fix jenkins tests

* Remove commented out code.
2018-10-26 13:36:58 -07:00
Wang Qing
b410ee0d29 [Java] Support dynamically defining resources when submitting task. (#3070)
## What do these changes do?
Before this PR, if we want to specify some resources, we must do as following codes:
```java
@RayRemote(Resources={ResourceItem("CPU", 10)})
public static void f1() {
// do sth
}

@RayRemote(Resources={ResourceItem("CPU", 10)})
class Demo {
// sth
}
```
Unfortunately, it's no way for us to create another actor or task with different resources required.

After this PR, the thing will be:
```java
ActorCreationOptions option = new ActorCreationOptions(); 
option.resources.put("CPU", 4.0);
RayActor<Echo> echo1 = Ray.createActor(Echo::new, option);
option.resources.put("Res-A", 4.0);
RayActor<Echo> echo2 = Ray.createActor(Echo::new, option);


//if we don't specify resource,  the resources will be `{"cpu":0.0}` by default.
Ray.call(Echo::echo, echo2, 100);
```


## Related issue number
N/A
2018-10-19 06:22:32 -07:00
Wang Qing
4a2ed47b6c [Java] Improve some Java code (#3040)
This PR improves some java codes,  and removes some duplicated code.
2018-10-10 17:30:23 -07:00
Wang Qing
ef1f2fde95 Fix the uniqueId toString format. (#3035) 2018-10-08 13:12:14 -07:00
Wang Qing
84bf5fc8f3 [Java] Load driver resources from local path. (#3001)
## What do these changes do?
1. Add a configuration item `driver.resource-path`.
2. Load driver resources from the local path which is specified in the `ray.conf`.

Before this change, we should add all driver resources(like user's jar package, dependencies package and config files) into `classpath`.

After this change, we should add the driver resources into the mount path which we can configure it in `ray.conf`, and we shouldn't configure `classpath` for driver resources any more.

## Related issue number
N/A
2018-10-08 21:05:26 +01:00
Wang Qing
fcef4edd46 [Java] Fix the required-resources issue of actor member function in Java worker. (#3002)
This fixes a bug in which Java actor methods inherit the resource requirements of the actor creation task.
2018-10-01 12:56:36 -07:00
Hao Chen
4ffe1e3556 [Java] Fix: task spec's resource map should contain CPU (#2987) 2018-09-28 14:23:38 -05:00