Stephanie Wang
66edebce3a
Spillback scheduling for direct task calls ( #6164 )
...
* add dac
* remove cachign
* rename return buffer
* cleanup
* add tests
* add perf
* fix
* flip
* remove
* remove it
* lint
* remove fork safety
* lint
* comments
* s/core/client
* wip
* remove
* fmt
* consistently return direct naming
* basic pass by ref
* fix bugs
* wip
* wip
* wip
* wip
* add test
* works now
* fix constructor
* fix merge
* add todo for perf
* fix single client test
* use lower n
* bazel
* faster
* fix core worker test
* init
* fix tests
* no plasma for direct call
* Update worker.py
* add order test
* fixes
* comments
* remove old assert
* lint
* add test
* Very wip
* wip
* add options for tasks
* add test
* fmt
* add backpressure
* remove idle prof event
* lint
* Fix 0 returns
* Set memcopy threads globally
* add benchmark
* Fix object exists
* Fix reference
* Remove return_buffer
* Add check
* add exit handler
* update benchmarks
* Fix compile error
* Fix NoReturn
* Use is instead of == for NoReturn
* fix
* Remove list comprehension
* Fix core worker test
* comment
* Apply suggestions from code review
Co-Authored-By: Edward Oakes <ed.nmi.oakes@gmail.com>
* fix merge error
* lint
* wip
* fix merge
* wip
* finish
* lint
* task interface
* add file
* add
* wip
* now works!
* updated
* wip
* dep resolution
* remove remote dep handling
* comments
* fix test_multithreading
* fix merge
* fix exit handling
* fix merge
* comments
* get fallback fetch working
* handle contains
* fix typo
* Skeleton for SubmitTask proto
* Update src/ray/common/id.h
Co-Authored-By: Stephanie Wang <swang@cs.berkeley.edu>
* comments
* rename to core worker service
* lint
* fix compile
* wip
* update
* error code
* fix up and rename
* clean up call manager
* comments
* add test and cleanup deserialization
* fix pickle
* fix comments, lint
* test todo
* comments
* use shared ptr
* rename
* Update src/ray/protobuf/gcs.proto
Co-Authored-By: Stephanie Wang <swang@cs.berkeley.edu>
* require transport type for ids; lint
* cleanup
* comments 1
* use worker available for real
* wip
* fix test
* resolve local dependencies test
* add num pending metric
* client factory
* unit test task submission
* wip
* fix bug
* rename
* Pass through node manager port, connect in raylet client
* finish rename
* Switch submit task to grpc
* fix crash
* Check port in use
* fix merge
* comments more
* doc
* Remove default port, set port randomly from driver
* add unique_ptr comment about TaskSpec
* lint
* fix test
* update
* fix lint
* GetMessageMutable should not be const
* iwyu
* fix const
* Update direct_task_transport_test.cc
* fix segfault
* Fix test
* Add RpcAddress, set in actor table data
* fix serialization
* fix lint
* Pass through task caller address
* Fix object manager test
* RpcAddress -> Address
* merge
* Port WorkerLease to grpc
* wip
* fix test
* add mem test
* update
* comments
* fix core worker tests
* fix
* remove old worker lease code
* First pass on spillback
* lint
* crash?
* Debug
* Fix task spec copy, extend test basic
* lint
* Port return worker to grpc
* lint
* Return worker to the correct raylet
* Only request worker if queued tasks
* A bit better failure handling
* Fix unit test
* Add unit test for spillback
* fix
* python test multinode
* update
* updates
* fix
2019-11-17 20:29:32 -08:00
Ion
1b80675206
Scheduling ids ( #6137 )
2019-11-15 16:04:16 -08:00
Edward Oakes
33040d734f
Disable stopgap GC by default ( #6165 )
...
* disable stopgap gc by default
* fix gc testss
2019-11-15 15:42:59 -08:00
Eric Liang
7d33e9949b
Integrate ref count module into local memory store ( #6122 )
2019-11-15 10:52:19 -08:00
Eric Liang
8ff393a7bd
Handle exchange of direct call objects between tasks and actors ( #6147 )
2019-11-14 17:32:04 -08:00
Edward Oakes
2758cd0b34
Make log message debug ( #6166 )
2019-11-14 15:05:36 -08:00
Eric Liang
0a3623ded6
Fix memory store wait ( #6152 )
2019-11-14 10:17:30 -08:00
Stephanie Wang
bbadde57e0
Pass through caller address when submitting a task ( #6143 )
...
* Add RpcAddress, set in actor table data
* Pass through task caller address
* RpcAddress -> Address
* update
* fix
* lint
* fix cc tests
2019-11-14 09:14:08 -08:00
Ujval Misra
e3e3ad4b25
Add timeout param to ray.get ( #6107 )
2019-11-14 00:50:04 -08:00
Edward Oakes
51e76151d6
Use shared_ptr for gcs client in profiler ( #6150 )
2019-11-13 15:24:01 -08:00
Eric Liang
f3f86385d6
Minimal implementation of direct task calls ( #6075 )
2019-11-12 11:45:28 -08:00
Stephanie Wang
35d177f459
Use grpc for communication from worker to local raylet (task submission and direct actor args only) ( #6118 )
...
* Skeleton for SubmitTask proto
* Pass through node manager port, connect in raylet client
* Switch submit task to grpc
* Check port in use
* doc
* Remove default port, set port randomly from driver
* update
* Fix test
* Fix object manager test
2019-11-11 21:17:25 -08:00
Edward Oakes
5780ec1b62
Refresh ObjectIDs in raylet for stopgap GC ( #6109 )
2019-11-10 23:12:59 -08:00
Philipp Moritz
ccbcc4bafa
Use GRCP and Bazel 1.0 ( #6002 )
2019-11-08 15:58:28 -08:00
Philipp Moritz
5a05eaaa54
Fix compilation on master ( #6116 )
2019-11-07 22:38:42 -08:00
Eric Liang
4a28306186
Allow large returns from direct actor calls ( #6088 )
2019-11-07 21:28:55 -08:00
Edward Oakes
ca53af4d0f
Add pending task dependencies to ObjectID ref counting ( #6054 )
2019-11-07 18:37:10 -08:00
Edward Oakes
9820c10a09
Simplify gRPC service definition for the worker ( #6095 )
2019-11-06 13:00:39 -08:00
mehrdadn
e312f3d282
Compatibility issues ( #6071 )
...
* Pass -f - to tar to force stdin on Windows
* Quote paths that may contain spaces (causes issues on Windows)
* Copy over Windows code from Arrow for glog signal handle uninstall
* Add missing COPTS to build rules since we'll need them for Windows compatibility
* Begin adding COPTS for Windows compatibility
* Disable glog on Arrow until we change WIN32 to _WIN32 there
* Missing header files that cause problems on Windows
* WORD typedef conflicts with Windows; remove it
* uint -> unsigned int wherever we're dealing with milliseconds (signed version is already int)
* uint -> unsigned int for enums
* uint -> size_t, wherever we're dealing with sizes or indices into arrays
* Work around Boost 1.68 bug in detecting clang-cl (revert this after upgrading)
* Missing #include <unistd.h>
* Add check for signal handler uninstallation failure
* Linting issue
2019-11-05 00:08:14 -08:00
Edward Oakes
043d1f4094
Return RayObjects to core worker ( #6052 )
2019-11-04 20:27:57 -08:00
Eric Liang
8485304e83
Support concurrent Actor calls in Ray ( #6053 )
2019-11-04 01:14:35 -08:00
Philipp Moritz
1c5446851a
Use Plasma with LRU refreshing integrated ( #6050 )
2019-11-03 16:19:05 -08:00
Eric Liang
fb34928a2a
[minor] Perf optimizations for direct actor task submission ( #6044 )
...
* merge optimizations
* fix
* fix memory err
* optimize
* fix tests
* fix serialization of method handles
* document weakref
* fix check
* bazel format
* disable on 2
2019-11-01 14:41:14 -07:00
Eric Liang
eef4ad3bba
Report census view data as part of raylet node stats ( #6060 )
2019-11-01 14:26:09 -07:00
Simon Mo
7f5b3502da
Implement Detached Actor ( #6036 )
...
* Arg propagation works
* Implement persistent actor
* Add doc
* Initialize is_persistent_
* Rename persistent->detached
* Address comment
* Make test passes
* Address comment
* Python2 compatiblity
* Fix naming, py2
* Lint
2019-11-01 10:28:23 -07:00
Eric Liang
c86f945520
Support pass by ref args in for direct actor calls ( #6040 )
2019-10-31 16:55:10 -07:00
Edward Oakes
16e9dfd2e1
Exit workers when raylet dies unexpectedly ( #6014 )
2019-10-30 20:29:25 -07:00
Eric Liang
8ebba202df
[minor] Reduce perf overhead of object ref tracking ( #6041 )
2019-10-29 18:14:51 -07:00
Eric Liang
b89cac976a
Basic direct actor call support in Python ( #5991 )
2019-10-28 22:09:04 -07:00
Edward Oakes
c1418b04df
Remove CoreWorkerObjectInterface ( #6023 )
2019-10-28 10:48:41 -07:00
Philipp Moritz
80c01617a3
Optimize python task execution ( #6024 )
2019-10-27 00:43:34 -07:00
Stephanie Wang
eb41c945a1
Add gRPC endpoint to raylet to expose metrics ( #6005 )
2019-10-26 16:37:39 -07:00
Eric Liang
a5523466a2
Enable memstore by default ( #6003 )
2019-10-25 21:59:12 -07:00
Edward Oakes
d4055d70e3
Remove CoreWorkerTaskExecutionInterface ( #6009 )
2019-10-25 16:33:44 -07:00
Edward Oakes
e6141a0b8b
Remove UsePush logic from raylet ( #6015 )
2019-10-25 14:52:19 -07:00
Edward Oakes
1ce521a7f3
Remove task context from python worker ( #5987 )
...
Removes duplicated state between the python and C++ workers. Also cleans up the serialization codepaths a bit.
2019-10-25 07:38:33 -07:00
Philipp Moritz
09d05bb3fa
Reduce actor submission python overhead ( #5949 )
2019-10-23 00:11:32 -07:00
Edward Oakes
02931e08f3
[core worker] Python core worker task execution ( #5783 )
...
Executes tasks via the event loop in the C++ core worker. Also properly handles signals (including KeyboardInterrupt), so ctrl-C in a python interactive shell works now (if connecting to an existing cluster).
2019-10-22 20:15:59 -07:00
Philipp Moritz
b6e7ed20ce
Fix random numbers on linux wheel build ( #5975 )
2019-10-22 17:52:12 -07:00
Edward Oakes
fc56872012
Send active object IDs to the raylet ( #5803 )
...
* Send active object IDs to the raylet
* comment
* comments
* dedup
* signed int in config
* comments
* Remove object ID from monitor
* Fix test
* re-add check
* fix cast
* check if core worker
* Add comment
* Reservoir sampling
* Fix lint
* Pointer return
* tmp
* Fix merge
* Initialize object ids properly
* Fix lint
2019-10-20 22:05:28 -07:00
Stephanie Wang
bc4a0de4da
Fix multiple drivers for named actors and add test ( #5956 )
2019-10-20 16:04:21 -07:00
Stephanie Wang
697f765efc
Refactor CoreWorker to remove TaskInterface ( #5924 )
...
* Remove TaskInterface
* Remove Status return value
* Remove CActorHandle, some return values, TaskSubmitter
* lint
* doc
* doc
* fix build
* lint
* Return Status, guarded by annotation, fail tasks for RECONSTRUCTING actors
* fix
* move annotation
* revert
* Fix core worker test
* nits
2019-10-18 00:03:57 -04:00
Stephanie Wang
3ac8592dcf
Remove actor handle IDs ( #5889 )
...
* Remove actor handle ID from main ActorHandle constructor
* Set the actor caller ID when calling submit task instead of in the actor handle
* Remove ActorHandle::Fork, remove actor handle ID from protobuf
* Make inner actor handle const, remove new_actor_handles
* Move caller ID into the common task spec, start refactoring raylet
* Some fixes for forking actor handles
* Store ActorHandle state in CoreWorker, only expose actor ID to Python
* Remove some unused fields
* lint
* doc
* fix merge
* Remove ActorHandleID from python/cpp
* doc
* Fix core worker test
* Move actor table subscription to CoreWorker, reset actor handles on actor failure
* lint
* Remove GCS client from direct actor
* fix tests
* Fix
* Fix tests for raylet codepath
* Fix local mode
* Fix multithreaded test
* Fix AsyncSubscribe issue...
* doc
* fix serve
* Revert bazel
2019-10-17 12:36:34 -04:00
Richard Liaw
20c0cdee4f
[autoscaler] Worker-Head termination + Better Scale-up message ( #5909 )
2019-10-14 10:37:50 -07:00
Edward Oakes
abbfe7392f
Bump dev version to 0.8.0.dev6 ( #5906 )
2019-10-14 11:36:13 +01:00
Philipp Moritz
1100556ba2
Fix linux wheel build ( #5881 )
2019-10-10 16:15:26 -07:00
Eric Liang
1a8ac3db46
Implement fair task queueing to prevent task starvation ( #5851 )
...
* initial commit
* lint
* clarify
* add feature flag
* comment
* add timeout to test
* fix print
* comment
* use id for scheduling class
* lint
* dad warn
* flake
2019-10-08 21:04:25 -07:00
Edward Oakes
08e4e3a153
[core worker] Submit Python actor tasks through core worker ( #5750 )
...
* Submit actor tasks through core worker
* Fix java
* add comment
* Remove task builder
* Check negative
* Increase -> Increment
* pass by reference
* fix signal
* Clean up c++ actor handle
* more cleanup
* Clean up headers
* Fix unique_ptr construction
* Fix java
* Move profiling to c++
* dedup
* fix error
* comments
* fix java
* Fix tests
* wait for actor to exit
* Start after constructor
* ignore java build
* fix comment
* always init logging
* Fix logging
* fix logging issue
* shared_ptr for profiler
* DEBUG -> WARNING
* fix killed_ init
* Fix flaky checkpointing tests
* -v flag for tune tests
* Fix checkpoint test logic
* Fix exception matching
* timeout exception
* Fix test exception info
* Fix import
* fix build
* Fix test
* shared_ptr
2019-10-07 15:42:19 -07:00
Edward Oakes
17c6835c3f
Just die on signal ( #5842 )
2019-10-03 18:21:21 -07:00
Si-Yuan
2fb7d7846f
Initial implementation of Cython pickle5 support ( #5725 )
2019-10-03 09:20:26 -07:00
