Philipp Moritz
ccee77aafd
fix node_failures.py ( #5167 )
2019-07-11 11:40:13 -07:00
Zhijun Fu
1649f1370e
[direct call] changes raylet to push tasks to worker ( #5140 )
...
* refactor grpc server
* format
* change GetTask() to PushTask()
* change PushTask to AssignTask
* format
* add resource_ids
* move done_callback to server call
* remove SetTaskHandler and initialize it in task receiver's constructor
* format
* resolve comments
* update
* update
* Update src/ray/core_worker/core_worker.cc
Co-Authored-By: Stephanie Wang <swang@cs.berkeley.edu>
* resolve comments
* format
* Update src/ray/core_worker/transport/raylet_transport.cc
Co-Authored-By: Hao Chen <chenh1024@gmail.com>
* resolve comments
* resolve comments
* fix build
* format
* fix
* format
* noop
2019-07-11 11:01:32 -07:00
Hao Chen
fd835d107e
Move task to common module and add checks in getter methods ( #5147 )
2019-07-11 17:07:04 +08:00
Qing Wang
f2293243cc
[ID Refactor] Shorten the length of JobID to 4 bytes ( #5110 )
...
* WIP
* Fix
* Add jobid test
* Fix
* Add python part
* Fix
* Fix tes
* Remove TODOs
* Fix C++ tests
* Lint
* Fix
* Fix exporting functions in multiple ray.init
* Fix java test
* Fix lint
* Fix linting
* Address comments.
* FIx
* Address and fix linting
* Refine and fix
* Fix
* address
* Address comments.
* Fix linting
* Fix
* Address
* Address comments.
* Address
* Address
* Fix
* Fix
* Fix
* Fix lint
* Fix
* Fix linting
* Address comments.
* Fix linting
* Address comments.
* Fix linting
* address comments.
* Fix
2019-07-11 14:25:16 +08:00
Kai Yang
43b6513d19
[GCS] Move node resource info from client table to resource table ( #5050 )
2019-07-11 13:17:19 +08:00
Philipp Moritz
e6a81d40a5
[stability] Make task result for RemoveTask optional ( #5146 )
...
* make task result for RemoveTask optional
* lint
* update
* update
* update
* rename
* lint
2019-07-10 13:33:41 -07:00
Joey Jiang
e55c8ca165
Fix crash because of the reference to deleted variable in grpc server call ( #5158 )
2019-07-10 14:06:21 +08:00
Joey Jiang
5733690aa6
Add success and fail callback of grpc sending reply ( #5141 )
2019-07-09 17:03:57 +08:00
Hao Chen
8a30b93e42
Define common data structures with protobuf. ( #5121 )
2019-07-08 22:41:37 +08:00
Joey Jiang
274233962f
Remove unused connection file in object manager ( #5123 )
2019-07-08 10:59:36 +08:00
Philipp Moritz
c5253cc300
Add job table to state API ( #5076 )
2019-07-06 00:05:48 -07:00
Zhijun Fu
54d5969cea
[grpc] Add grpc server to worker ( #5054 )
...
* refactor grpc server
* format
* change GetTask() to PushTask()
* change PushTask to AssignTask
* format
* update
* fix test
* format
* Update src/ray/rpc/worker_client.h
Co-Authored-By: Hao Chen <chenh1024@gmail.com>
* Update BUILD.bazel
* Update src/ray/core_worker/task_execution.cc
Co-Authored-By: Stephanie Wang <swang@cs.berkeley.edu>
* update
* format
* address comments
* format
* Update src/ray/rpc/worker/worker_server.h
Co-Authored-By: Stephanie Wang <swang@cs.berkeley.edu>
* Update src/ray/protobuf/worker.proto
Co-Authored-By: Stephanie Wang <swang@cs.berkeley.edu>
* format
* fix
* format
2019-07-04 20:16:42 +08:00
Stephanie Wang
71d4637b75
[core worker] Refactor CoreWorker member classes ( #5062 )
...
* Move store client mutex inside CoreWorkerPlasmaStoreProvider
* Move PlasmaClient inside CoreWorkerStoreProvider
* Remove CoreWorkerObjectInterface's ref to CoreWorker
* Remove WorkerLanguage
* Remove CoreWorkerTaskInterface's ref to CoreWorker
* Remove CoreWorkerTaskExecutionInterface's ref to CoreWorker
* lint
* move comment
* Fix build
* Fix build
2019-07-02 15:30:30 -07:00
Kai Yang
1cf7728f35
[Core worker] Serialize ActorHandle in core worker. Make ActorHandle thread safe. ( #5034 )
...
* Serialize ActorHandle in core worker. Make ActorHandle thread safe.
* Address comments
* Address comments
* Address comments
* Address comments
* lint
* Address comments
* Address comments
* Address comments
* Address comments
* Minor update
* Address comments
* lint
2019-07-02 16:48:43 +08:00
Qing Wang
247f95b3ff
Refine RegisterClientRequest message to make it clearer. ( #5057 )
...
* transfor driver task id Explicitly
* Refins
* Fix and add comment.
* add more
* Fix
* Fix
* Add comments
* Fix
2019-07-02 14:26:19 +08:00
Simon Mo
6c4c1d444d
Update VersionKey in stats ( #5070 )
2019-06-30 18:23:12 +08:00
Kai Yang
4ccb7b05cc
[Core worker] Add metadata support in object interface ( #5031 )
2019-06-28 11:35:03 -07:00
Hao Chen
cefbb0c94c
Fix driver id in TaskInfo ( #5055 )
2019-06-28 12:56:48 +08:00
Kai Yang
a39982e676
[Core worker] Task execution passes TaskInfo struct to executor ( #5032 )
2019-06-28 10:59:45 +08:00
Joey Jiang
d6bbbdef35
Use gRPC to handle communication and data transmission between object manager ( #4996 )
2019-06-28 10:56:34 +08:00
Qing Wang
62e4b591e3
[ID Refactor] Rename DriverID to JobID ( #5004 )
...
* WIP
WIP
WIP
Rename Driver -> Job
Fix complition
Fix
Rename in Java
In py
WIP
Fix
WIP
Fix
Fix test
Fix
Fix C++ linting
Fix
* Update java/runtime/src/main/java/org/ray/runtime/config/RayConfig.java
Co-Authored-By: Stephanie Wang <swang@cs.berkeley.edu>
* Update src/ray/core_worker/core_worker.cc
Co-Authored-By: Stephanie Wang <swang@cs.berkeley.edu>
* Address comments
* Fix
* Fix CI
* Fix cpp linting
* Fix py lint
* FIx
* Address comments and fix
* Address comments
* Address
* Fix import_threading
2019-06-28 00:44:51 +08:00
Hao Chen
469ae41013
Fix memory leak in rpc ServerCall and ClientCall ( #5046 )
2019-06-27 13:19:47 +08:00
Stephanie Wang
1a8d0af814
Remove debug check for uncommitted lineage ( #5038 )
2019-06-26 11:21:00 -07:00
Zhijun Fu
bb8e75b532
[grpc] refactor rpc server to support multiple io services ( #5023 )
2019-06-25 19:08:09 -07:00
Hao Chen
0131353d42
[gRPC] Migrate gcs data structures to protobuf ( #5024 )
2019-06-25 14:31:19 -07:00
Qing Wang
e33d0eac68
Add dynamic worker options for worker command. ( #4970 )
...
* Add fields for fbs
* WIP
* Fix complition errors
* Add java part
* FIx
* Fix
* Fix
* Fix lint
* Refine API
* address comments and add test
* Fix
* Address comment.
* Address comments.
* Fix linting
* Refine
* Fix lint
* WIP: address comment.
* Fix java
* Fix py
* Refin
* Fix
* Fix
* Fix linting
* Fix lint
* Address comments
* WIP
* Fix
* Fix
* minor refine
* Fix lint
* Fix raylet test.
* Fix lint
* Update src/ray/raylet/worker_pool.h
Co-Authored-By: Hao Chen <chenh1024@gmail.com>
* Update java/runtime/src/main/java/org/ray/runtime/AbstractRayRuntime.java
Co-Authored-By: Hao Chen <chenh1024@gmail.com>
* Address comments.
* Address comments.
* Fix test.
* Update src/ray/raylet/worker_pool.h
Co-Authored-By: Hao Chen <chenh1024@gmail.com>
* Address comments.
* Address comments.
* Fix
* Fix lint
* Fix lint
* Fix
* Address comments.
* Fix linting
2019-06-23 18:08:33 +08:00
Hao Chen
2bf92e02e2
[gRPC] Use gRPC for inter-node-manager communication ( #4968 )
2019-06-17 19:00:50 +08:00
Qing Wang
b08765a08b
Fix a crash when unknown worker registering to raylet ( #4992 )
2019-06-17 13:34:23 +08:00
Zhijun Fu
37abdb283f
[Core worker] add store & task provider ( #4966 )
2019-06-14 18:35:32 +08:00
Hao Chen
3c92b2ee4d
Upgrade CI clang-format to 6.0 ( #4976 )
2019-06-14 14:52:32 +08:00
Stephanie Wang
89ca5eeb29
Flush all tasks from local lineage cache after a node failure ( #4964 )
2019-06-12 11:13:39 -07:00
Zhijun Fu
472c36ed1e
[core worker] add task submission & execution interface ( #4922 )
2019-06-12 10:10:12 +08:00
Philipp Moritz
ebb3b3b928
[core] Use int64_t instead of int to keep track of fractional resources ( #4959 )
2019-06-10 23:49:04 -07:00
Philipp Moritz
1e2b649580
Use proper session directory for debug_string.txt ( #4960 )
2019-06-10 23:46:37 -07:00
Robert Nishihara
a82e8118a0
Fix resource bookkeeping bug with acquiring unknown resource. ( #4945 )
2019-06-07 21:07:27 -07:00
Stephanie Wang
873d45b467
Flush lineage cache on task submission instead of execution ( #4942 )
2019-06-07 11:35:18 -07:00
Yuhong Guo
5eff47b657
[C++] Add hash table to Redis-Module ( #4911 )
2019-06-07 16:11:37 +08:00
Robert Nishihara
c3f8fc1c44
Update version number in documentation after release 0.7.0 -> 0.7.1 and 0.8.0.dev0 -> 0.8.0.dev1. ( #4941 )
2019-06-06 17:22:45 -07:00
Hao Chen
d106283769
Better organize ray_common module ( #4898 )
2019-06-04 23:19:09 -07:00
Zhijun Fu
b674c4a5ba
[Core Worker] implement ObjectInterface and add test framework ( #4899 )
2019-06-03 19:59:43 +08:00
Yuhong Guo
0066d7cf2a
Hotfix for change of from_random to FromRandom ( #4909 )
2019-05-31 16:41:31 +08:00
Yuhong Guo
1f0809e2b4
Refactor ID Serial 2: change all ID functions to CamelCase ( #4896 )
2019-05-31 11:31:18 +08:00
Hao Chen
2912a7cb86
Initial high-level code structure of CoreWorker. ( #4875 )
2019-05-30 02:43:17 -07:00
Qing Wang
b7c284aaa3
Refactor redis callback handling ( #4841 )
...
* Add CallbackReply
* Fix
* fix linting by format.sh
* Fix linting
* Address comments.
* Fix
2019-05-30 11:54:30 +08:00
Yuhong Guo
fa0892f285
Replace ReturnIds with NumReturns in TaskInfo to reduce the size ( #4854 )
...
* Refine TaskInfo
* Fix
* Add a test to print task info size
* Lint
* Refine
2019-05-28 13:30:41 +08:00
Yuhong Guo
1a39fee9c6
Refactor ID Serial 1: Separate ObjectID and TaskID from UniqueID ( #4776 )
...
* Enable BaseId.
* Change TaskID and make python test pass
* Remove unnecessary functions and fix test failure and change TaskID to
16 bytes.
* Java code change draft
* Refine
* Lint
* Update java/api/src/main/java/org/ray/api/id/TaskId.java
Co-Authored-By: Hao Chen <chenh1024@gmail.com>
* Update java/api/src/main/java/org/ray/api/id/BaseId.java
Co-Authored-By: Hao Chen <chenh1024@gmail.com>
* Update java/api/src/main/java/org/ray/api/id/BaseId.java
Co-Authored-By: Hao Chen <chenh1024@gmail.com>
* Update java/api/src/main/java/org/ray/api/id/ObjectId.java
Co-Authored-By: Hao Chen <chenh1024@gmail.com>
* Address comment
* Lint
* Fix SINGLE_PROCESS
* Fix comments
* Refine code
* Refine test
* Resolve conflict
2019-05-22 14:46:30 +08:00
Qing Wang
081708bdef
[Java] Dynamic resource API in Java ( #4824 )
2019-05-21 17:13:48 +08:00
Stephanie Wang
cb1a195ca2
Queue tasks in the raylet in between async callbacks ( #4766 )
...
* Add a SWAP TaskQueue so that we can keep track of tasks that are temporarily dequeued
* Fix bug where tasks that fail to be forwarded don't appear to be local by adding them to SWAP queue
* cleanups
* updates
* updates
2019-05-15 10:23:25 -07:00
Stephanie Wang
1622fc21fc
Fatal check if object store dies ( #4763 )
2019-05-13 11:59:12 -07:00
Romil Bhardwaj
004440f526
Dynamic Custom Resources - create and delete resources ( #3742 )
2019-05-11 20:06:04 +08:00
