Summary:
Able to run 1000 tasks with object dependencies on a set of distributed Raylets.
Raylet Changes:
Finalized ClientConnection class.
Task forwarding.
NM-to-NM heartbeats.
NM resource accounting for tasks.
Simple scheduling policy with task forwarding.
Creating and maintaining NM 2 NM long-lived connections and reusing them for task forwarding.
LineageCache Changes:
LineageCache without cleanup of tasks committed by remote nodes.
Lineage cache writeback and cleanup implementation.
ObjectManager Changes:
Object manager event loop/ClientConnection refactor.
Multithreaded object manager (disabled in this PR).
Testing Changes:
Integration tests for task submission on multiple Raylets.
Stress tests for object manager (with GCS and object store integration).
Co-authored-by: Stephanie Wang <swang@cs.berkeley.edu>
Co-authored-by: Alexey Tumanov <atumanov@gmail.com>
* TABLE_APPEND call
* Convert callbacks back to taking in a string...
* GCS returns flatbuffers, define Log class
* Cleanups
* Modify client table to use the Log interface
* Fix bug where we replied twice from redis
* Fixes
* lint
* Compile and test raylet TaskTable
* Modify GCS tables to handle unique_ptrs from nested flatbuffers
* Add raylet::TaskTable unit tests to replace ObjectTable ones
* Convert ObjectTable to a log
* Convert ObjectTable tests to the Log
* AppendAt Redis and gcs Log command
* unit test for AppendAt
* Add a Log for task reconstruction data
* Add check for unique entries in TABLE_APPEND
* Documentation
* TABLE_APPEND call
* Convert callbacks back to taking in a string...
* GCS returns flatbuffers, define Log class
* Cleanups
* Modify client table to use the Log interface
* Fix bug where we replied twice from redis
* Fixes
* lint
* Compile and test raylet TaskTable
* Modify GCS tables to handle unique_ptrs from nested flatbuffers
* Add raylet::TaskTable unit tests to replace ObjectTable ones
* Convert ObjectTable to a log
* Convert ObjectTable tests to the Log
* TABLE_APPEND call
* Convert callbacks back to taking in a string...
* GCS returns flatbuffers, define Log class
* Cleanups
* Modify client table to use the Log interface
* Fix bug where we replied twice from redis
* Fixes
* lint
* [tune] Added pbt with keras on cifar10 dataset example
* ENH: add gpu resources
* CLN: requires 4 GPUs resource
* CLN: use single quotes
* CLN: don't save model by default
* Add TableRequestNotifications and TableCancelNotifications to Redis modules
* Add RequestNotifications and CancelNotifications to generic GCS Table
* Add tests for subscribing to specific keys
* Remove TODO!
* Return the current value at the key directly from RequestNotifications instead of through publish
* Add unit test for Lookup failure callback
* Modify tests to account for empty subscription response
* Remove ObjectTable notification methods
* Clean up message parsing and doc in redis context
* Use vectors of DataT in all GCS callbacks
* Clean up SubscriptionCallback
* Move Table definitions into tables.cc
* Refactor and document redis modules
* doc
* Fix new GCS build
* Cleanups
* Revert "Fix new GCS build"
This reverts commit 6e3e69090c67ef60aaf22a9cf62be0290d989e96.
* Use vectors for internal callback interface, user-facing interface takes a reference to a single item
* Fix new GCS build
* Add unit test for Lookup failure callback
* Fix compiler errors
* Cleanup
* Publish the entry ID with the notification
* Check that the ID for a notification matches in client tests
* Print error when actor takes too long to start, and refactor error message pushing.
* Print warning every ten seconds.
* Fix linting and tests.
* Fix tests.
* Speed up actor creation task submission by generating IDs deterministically.
* Revert "Speed up actor creation task submission by generating IDs deterministically."
This reverts commit 175d9587302664916ce9db4071185485da8da041.
* Don't generate actor IDs deterministically yet.
* Factor out ID generation method.
* Provide experimental API for changing number of return values and resource requirements at task submission time.
* Remove code duplication and add tests.
* Treat actor creation like a regular task.
* Small cleanups.
* Change semantics of actor resource handling.
* Bug fix.
* Minor linting
* Bug fix
* Fix jenkins test.
* Fix actor tests
* Some cleanups
* Bug fix
* Fix bug.
* Remove cached actor tasks when a driver is removed.
* Add more info to taskspec in global state API.
* Fix cyclic import bug in tune.
* Fix
* Fix linting.
* Fix linting.
* Don't schedule any tasks (especially actor creaiton tasks) on local schedulers with 0 CPUs.
* Bug fix.
* Add test for 0 CPU case
* Fix linting
* Address comments.
* Fix typos and add comment.
* Add assertion and fix test.
