mehrdadn
b8cfdba752
Bazelify hiredis ( #6203 )
2019-11-29 15:32:45 -08:00
Stephanie Wang
f6a0408173
Track pending tasks with TaskManager ( #6259 )
...
* TaskStateManager to track and complete pending tasks
* Convert actor transport to use task state manager
* Refactor direct actor transport to use TaskStateManager
* rename
* Unit test
* doc
* IsTaskPending
* Fix?
* Shared ptr
* HUH?
* Update src/ray/core_worker/task_manager.cc
Co-Authored-By: Zhijun Fu <37800433+zhijunfu@users.noreply.github.com>
* Revert "HUH?"
This reverts commit f80f0ba204ff4da5e0b03191fa0d5a4d9f552434.
* Fix memory issue
* oops
2019-11-25 16:37:26 -08:00
Eric Liang
53641f1f74
Move more unit tests to bazel ( #6250 )
...
* move more unit tests to bazel
* move to avoid conflict
* fix lint
* fix deps
* seprate
* fix failing tests
* show tests
* ignore mismatch
* try combining bazel runs
* build lint
* remove tests from install
* fix test utils
* better config
* split up
* exclusive
* fix verbosity
* fix tests class
* cleanup
* remove flaky
* fix metrics test
* Update .travis.yml
* no retry flaky
* split up actor
* split basic test
* split up trial runner test
* split stress
* fix basic test
* fix tests
* switch to pytest runner for main
* make microbench not fail
* move load code to py3
* test is no longer package
* bazel to end
2019-11-24 11:43:34 -08:00
Ion
68ac08332b
Initial commit of new cluster resource scheduler ( #6178 )
2019-11-22 11:14:46 -08:00
mehrdadn
ba86c75c21
Patch Cython in grpc to use our COPTS ( #6223 )
2019-11-21 15:32:48 -08:00
Simon Mo
29ba6bfc64
Basic Async Actor Call ( #6183 )
...
* Start trying to figure out where to put fibers
* Pass is_async flag from python to context
* Just running things in fiber works
* Yield implemented, need some debugging to make it work
* It worked!
* Remove debug prints
* Lint
* Revert the clang-format
* Remove unnecessary log
* Remove unncessary import
* Add attribution
* Address comment
* Add test
* Missed a merge conflict
* Make test pass and compile
* Address comment
* Rename async -> asyncio
* Move async test to py3 only
* Fix ignore path
2019-11-21 11:56:46 -08:00
Stephanie Wang
c0be9e6738
Resolve dependencies locally before submitting direct actor tasks ( #6191 )
...
* Priority queue in direct actor transport by task number
* Move LocalDependencyResolver out to separate file, share with direct actor transport
* works
* Test case for ordering
* Cleanups
* Remove priority queue
* comment
* Share ClientFactoryFn with direct actor transport
* Unit test
* fix
2019-11-20 16:45:19 -08:00
Ion
1b80675206
Scheduling ids ( #6137 )
2019-11-15 16:04:16 -08:00
Eric Liang
f3f86385d6
Minimal implementation of direct task calls ( #6075 )
2019-11-12 11:45:28 -08:00
Philipp Moritz
ccbcc4bafa
Use GRCP and Bazel 1.0 ( #6002 )
2019-11-08 15:58:28 -08:00
Edward Oakes
ca53af4d0f
Add pending task dependencies to ObjectID ref counting ( #6054 )
2019-11-07 18:37:10 -08:00
Edward Oakes
9820c10a09
Simplify gRPC service definition for the worker ( #6095 )
2019-11-06 13:00:39 -08:00
mehrdadn
e312f3d282
Compatibility issues ( #6071 )
...
* Pass -f - to tar to force stdin on Windows
* Quote paths that may contain spaces (causes issues on Windows)
* Copy over Windows code from Arrow for glog signal handle uninstall
* Add missing COPTS to build rules since we'll need them for Windows compatibility
* Begin adding COPTS for Windows compatibility
* Disable glog on Arrow until we change WIN32 to _WIN32 there
* Missing header files that cause problems on Windows
* WORD typedef conflicts with Windows; remove it
* uint -> unsigned int wherever we're dealing with milliseconds (signed version is already int)
* uint -> unsigned int for enums
* uint -> size_t, wherever we're dealing with sizes or indices into arrays
* Work around Boost 1.68 bug in detecting clang-cl (revert this after upgrading)
* Missing #include <unistd.h>
* Add check for signal handler uninstallation failure
* Linting issue
2019-11-05 00:08:14 -08:00
Eric Liang
fb34928a2a
[minor] Perf optimizations for direct actor task submission ( #6044 )
...
* merge optimizations
* fix
* fix memory err
* optimize
* fix tests
* fix serialization of method handles
* document weakref
* fix check
* bazel format
* disable on 2
2019-11-01 14:41:14 -07:00
Eric Liang
c86f945520
Support pass by ref args in for direct actor calls ( #6040 )
2019-10-31 16:55:10 -07:00
Eric Liang
8ebba202df
[minor] Reduce perf overhead of object ref tracking ( #6041 )
2019-10-29 18:14:51 -07:00
Stephanie Wang
eb41c945a1
Add gRPC endpoint to raylet to expose metrics ( #6005 )
2019-10-26 16:37:39 -07:00
Philipp Moritz
b6e7ed20ce
Fix random numbers on linux wheel build ( #5975 )
2019-10-22 17:52:12 -07:00
Edward Oakes
fc56872012
Send active object IDs to the raylet ( #5803 )
...
* Send active object IDs to the raylet
* comment
* comments
* dedup
* signed int in config
* comments
* Remove object ID from monitor
* Fix test
* re-add check
* fix cast
* check if core worker
* Add comment
* Reservoir sampling
* Fix lint
* Pointer return
* tmp
* Fix merge
* Initialize object ids properly
* Fix lint
2019-10-20 22:05:28 -07:00
Si-Yuan
2fb7d7846f
Initial implementation of Cython pickle5 support ( #5725 )
2019-10-03 09:20:26 -07:00
Mitchell Stern
98dcc1d440
[Dashboard] Add initial version of new dashboard ( #5730 )
2019-09-23 08:50:40 -07:00
Philipp Moritz
a6dd794818
[Projects] Fix template path ( #5716 )
2019-09-16 19:58:54 -07:00
Philipp Moritz
b1aadd863b
Fix project templates in wheel ( #5714 )
2019-09-16 15:21:59 -07:00
Eric Liang
4979b8c4d9
Ordered execution of tasks per actor handle ( #5664 )
2019-09-14 22:31:33 -07:00
Edward Oakes
07c4c6367a
[core worker] Python core worker object interface ( #5272 )
2019-09-12 23:07:46 -07:00
Edward Oakes
0bf79cfbde
Properly short circuit core worker Get() on exception ( #5672 )
2019-09-11 18:38:14 -07:00
micafan
da7bdacea5
support for subscription to an actor ( #5269 )
2019-08-20 20:32:53 +08:00
Philipp Moritz
599cc2be60
Revert raylet to worker GRPC communication back to asio ( #5450 )
2019-08-17 19:11:32 -07:00
Kai Yang
b1aae0e398
[Java worker] Migrate task execution and submission on top of core worker ( #5370 )
2019-08-16 13:52:13 +08:00
Qing Wang
d372f24e3c
[ID Refactor] Refactor ActorID, TaskID and ObjectID ( #5286 )
...
* Refactor ActorID, TaskID on the Java side.
Left a TODO comment
WIP for ObjectID
ADD test
Fix
Add java part
Fix Java test
Fix
Refine test.
Enable test in CI
* Extra a helper function.
* Resolve TODOs
* Fix Python CI
* Fix Java lint
* Update .travis.yml
Co-Authored-By: Stephanie Wang <swang@cs.berkeley.edu>
* Address some comments.
Address some comments.
Add id_specification.rst
Reanme id_specification.rst to id_specification.md
typo
Address zhijun's comments.
Fix test
Address comments.
Fix lint
Address comments
* Fix test
* Address comments.
* Fix build error
* Update src/ray/design_docs/id_specification.md
Co-Authored-By: Stephanie Wang <swang@cs.berkeley.edu>
* Update src/ray/design_docs/id_specification.md
Co-Authored-By: Stephanie Wang <swang@cs.berkeley.edu>
* Update src/ray/design_docs/id_specification.md
Co-Authored-By: Stephanie Wang <swang@cs.berkeley.edu>
* Update src/ray/design_docs/id_specification.md
Co-Authored-By: Stephanie Wang <swang@cs.berkeley.edu>
* Update src/ray/design_docs/id_specification.md
Co-Authored-By: Stephanie Wang <swang@cs.berkeley.edu>
* Address comments
* Update src/ray/common/id.h
Co-Authored-By: Stephanie Wang <swang@cs.berkeley.edu>
* Update src/ray/common/id.h
Co-Authored-By: Stephanie Wang <swang@cs.berkeley.edu>
* Update src/ray/common/id.h
Co-Authored-By: Stephanie Wang <swang@cs.berkeley.edu>
* Update src/ray/design_docs/id_specification.md
Co-Authored-By: Hao Chen <chenh1024@gmail.com>
* Update src/ray/design_docs/id_specification.md
Co-Authored-By: Hao Chen <chenh1024@gmail.com>
* Address comments.
* Address comments.
* Address comments.
* Update C++ part to make sure task id is generated determantic
* WIP
* Fix core worker
* Fix Java part
* Fix comments.
* Add Python side
* Fix python
* Address comments
* Fix linting
* Fix
* Fix C++ linting
* Add JobId() method to TaskID
* Fix linting
* Update src/ray/common/id.h
Co-Authored-By: Hao Chen <chenh1024@gmail.com>
* Update java/api/src/main/java/org/ray/api/id/TaskId.java
Co-Authored-By: Hao Chen <chenh1024@gmail.com>
* Update java/api/src/main/java/org/ray/api/id/TaskId.java
Co-Authored-By: Hao Chen <chenh1024@gmail.com>
* Update java/api/src/main/java/org/ray/api/id/ActorId.java
Co-Authored-By: Hao Chen <chenh1024@gmail.com>
* Address comments
* Add DriverTaskId embeding job id
* Fix tests
* Add python dor_fake_driver_id
* Address comments and fix linting
* Fix CI
2019-08-07 11:04:51 +08:00
Philipp Moritz
e8d9cfc1f1
Ray projects schema and validation ( #5329 )
2019-08-06 14:36:04 -07:00
Zhijun Fu
134c6bd128
[direct call] In memory store ( #5303 )
2019-08-05 13:14:45 -07:00
Zhijun Fu
eb307f93f8
Support direct actor call ( #5183 )
2019-07-30 17:47:17 +08:00
Joey Jiang
40395acadf
[gRPC] Migrate raylet client implementation to grpc ( #5120 )
2019-07-25 14:48:56 +08:00
micafan
b5b8c1d361
[GCS] introduce new gcs client and refactor actor table ( #5058 )
2019-07-19 11:28:34 +08:00
Kai Yang
806524384b
[Java worker] Refactor object store and worker context on top of core worker ( #5079 )
2019-07-16 20:58:02 +08:00
Hao Chen
fd835d107e
Move task to common module and add checks in getter methods ( #5147 )
2019-07-11 17:07:04 +08:00
Hao Chen
8a30b93e42
Define common data structures with protobuf. ( #5121 )
2019-07-08 22:41:37 +08:00
Zhijun Fu
54d5969cea
[grpc] Add grpc server to worker ( #5054 )
...
* refactor grpc server
* format
* change GetTask() to PushTask()
* change PushTask to AssignTask
* format
* update
* fix test
* format
* Update src/ray/rpc/worker_client.h
Co-Authored-By: Hao Chen <chenh1024@gmail.com>
* Update BUILD.bazel
* Update src/ray/core_worker/task_execution.cc
Co-Authored-By: Stephanie Wang <swang@cs.berkeley.edu>
* update
* format
* address comments
* format
* Update src/ray/rpc/worker/worker_server.h
Co-Authored-By: Stephanie Wang <swang@cs.berkeley.edu>
* Update src/ray/protobuf/worker.proto
Co-Authored-By: Stephanie Wang <swang@cs.berkeley.edu>
* format
* fix
* format
2019-07-04 20:16:42 +08:00
Kai Yang
1cf7728f35
[Core worker] Serialize ActorHandle in core worker. Make ActorHandle thread safe. ( #5034 )
...
* Serialize ActorHandle in core worker. Make ActorHandle thread safe.
* Address comments
* Address comments
* Address comments
* Address comments
* lint
* Address comments
* Address comments
* Address comments
* Address comments
* Minor update
* Address comments
* lint
2019-07-02 16:48:43 +08:00
Joey Jiang
d6bbbdef35
Use gRPC to handle communication and data transmission between object manager ( #4996 )
2019-06-28 10:56:34 +08:00
Hao Chen
0131353d42
[gRPC] Migrate gcs data structures to protobuf ( #5024 )
2019-06-25 14:31:19 -07:00
Hao Chen
2bf92e02e2
[gRPC] Use gRPC for inter-node-manager communication ( #4968 )
2019-06-17 19:00:50 +08:00
Zhijun Fu
37abdb283f
[Core worker] add store & task provider ( #4966 )
2019-06-14 18:35:32 +08:00
Zhijun Fu
472c36ed1e
[core worker] add task submission & execution interface ( #4922 )
2019-06-12 10:10:12 +08:00
Yuhong Guo
5eff47b657
[C++] Add hash table to Redis-Module ( #4911 )
2019-06-07 16:11:37 +08:00
Hao Chen
d106283769
Better organize ray_common module ( #4898 )
2019-06-04 23:19:09 -07:00
Zhijun Fu
b674c4a5ba
[Core Worker] implement ObjectInterface and add test framework ( #4899 )
2019-06-03 19:59:43 +08:00
Hao Chen
2912a7cb86
Initial high-level code structure of CoreWorker. ( #4875 )
2019-05-30 02:43:17 -07:00
Romil Bhardwaj
004440f526
Dynamic Custom Resources - create and delete resources ( #3742 )
2019-05-11 20:06:04 +08:00
