Summary:
Able to run 1000 tasks with object dependencies on a set of distributed Raylets.
Raylet Changes:
Finalized ClientConnection class.
Task forwarding.
NM-to-NM heartbeats.
NM resource accounting for tasks.
Simple scheduling policy with task forwarding.
Creating and maintaining NM 2 NM long-lived connections and reusing them for task forwarding.
LineageCache Changes:
LineageCache without cleanup of tasks committed by remote nodes.
Lineage cache writeback and cleanup implementation.
ObjectManager Changes:
Object manager event loop/ClientConnection refactor.
Multithreaded object manager (disabled in this PR).
Testing Changes:
Integration tests for task submission on multiple Raylets.
Stress tests for object manager (with GCS and object store integration).
Co-authored-by: Stephanie Wang <swang@cs.berkeley.edu>
Co-authored-by: Alexey Tumanov <atumanov@gmail.com>
* Add TableRequestNotifications and TableCancelNotifications to Redis modules
* Add RequestNotifications and CancelNotifications to generic GCS Table
* Add tests for subscribing to specific keys
* Remove TODO!
* Return the current value at the key directly from RequestNotifications instead of through publish
* Add unit test for Lookup failure callback
* Modify tests to account for empty subscription response
* Remove ObjectTable notification methods
* Clean up message parsing and doc in redis context
* Use vectors of DataT in all GCS callbacks
* Clean up SubscriptionCallback
* Move Table definitions into tables.cc
* Refactor and document redis modules
* doc
* Fix new GCS build
* Cleanups
* Revert "Fix new GCS build"
This reverts commit 6e3e69090c67ef60aaf22a9cf62be0290d989e96.
* Use vectors for internal callback interface, user-facing interface takes a reference to a single item
* Fix new GCS build
* Add unit test for Lookup failure callback
* Fix compiler errors
* Cleanup
* Publish the entry ID with the notification
* Check that the ID for a notification matches in client tests
* Print error when actor takes too long to start, and refactor error message pushing.
* Print warning every ten seconds.
* Fix linting and tests.
* Fix tests.
* added deterministic check for objects received in fetch_timeout_handler.
* use receive time, in case something goes wrong after object is received.
* increase timeout for removal.
* indentation fix.
* make log info log debug. clean up debug log.
* undo unecessary changes.
* changed description var.
* shorten line 949.
* incorporate feedback.
* linting; make is_object_received function consts.
* change semantics of received_objects to objects being received.
added checks to both points at which objects are re-requested.
updated object receive initialization accordingly.
* eliminate erase on receive init. check call to request_transfer_from instead of request_transfer.
* updated comments.
* added todo for multiple object transfers.
* linting.
* Define execution dependencies flatbuffer and add to Redis commands
* Convert TaskSpec to TaskExecutionSpec
* Add execution dependencies to Python bindings
* Submitting actor tasks uses execution dependency API instead of dummy argument
* Fix dependency getters and some cleanup for fetching missing dependencies
* C++ convention
* Make TaskExecutionSpec a C++ class
* Convert local scheduler to use TaskExecutionSpec class
* Convert some pointers to references
* Finish conversion to TaskExecutionSpec class
* fix
* Fix
* Fix memory errors?
* Cast flatbuffers GetSize to size_t
* Fixes
* add more retries in global scheduler unit test
* fix linting and cast fbb.GetSize to size_t
* Style and doc
* Fix linting and simplify from_flatbuf.
* Enable scheduling with custom resource labels.
* Fix.
* Minor fixes and ref counting fix.
* Linting
* Use .data() instead of .c_str().
* Fix linting.
* Fix ResourcesTest.testGPUIDs test by waiting for workers to start up.
* Sleep in test so that all tasks are submitted before any completes.
* Plasma client test for plasma abort
* Use ray-project/arrow:abort-objects branch
* Set plasma manager connection cursor to -1 when not in use
* Handle transfer errors between plasma managers, abort unsealed objects
* Add TODO for local scheduler exiting on plasma manager death
* Revert "Plasma client test for plasma abort"
This reverts commit e00fbd58dc4a632f58383549b19fb9057b305a14.
* Upgrade arrow to version with PlasmaClient::Abort
* Fix plasma manager test
* Fix plasma test
* Temporarily use arrow fork for testing
* fix and set arrow commit
* Fix plasma test
* Fix plasma manager test and make write_object_chunk consistent with read_object_chunk
* style
* upgrade arrow
* Object table lookup returns vector of DBClientID instead of address strings
* Add node IP address to DBClient notification
* DB client cache stores entire DB client, convert addresses to std::string
* get cached db client returns the client
* Expose a call to initialize the redis cache
* Local scheduler filters out dead clients during reconstruction
* Remove node ip address from dbclient, use aux_address for plasma managers
* Get entire db client entry when not found in cache
* Fix common tests
* Fix address in tests
* Push error to driver if driver task did the put
* Address Robert's comments and cleanup
* Remove unused Redis command
* Fix db test
* Initial pass at factoring out C++ configuration into a single file.
* Expose config through Python.
* Forward declarations.
* Fixes with Python extensions
* Remove old code.
* Consistent naming for constants.
* Fixes
* Fix linting.
* More linting.
* Whitespace
* rename config -> _config.
* Move config inside a class.
* update naming convention
* Fix linting.
* More linting
* More linting.
* Add in some more constants.
* Fix linting
* Compile global scheduler with -Werror -Wall.
* Compile plasma manager with -Werror -Wall.
* Compile local scheduler with -Werror -Wall.
* Compile common code with -Werror -Wall.
* Signed/unsigned comparisons.
* More signed/unsigned fixes.
* More signed/unsigned fixes and added extern keyword.
* Fix linting.
* Don't check strict-aliasing because Python.h doesn't pass.
* Add timing statement to loop that calls redis_get_cached_db_client because it has been slow in the past.
* Fix linting.
* Refactoring to make manager vectors into std::vector.
* Fix linting.
* Fixes.
* Comment out local scheduler valgrind test.
* Fix free/delete error.
* More free -> delete errors
* One more free -> delete and also clean up callback state in plasma manager.
* Add set -x to run_valgrind scripts.
* Fix valgrind error in CreateLocalSchedulerInfoMessage.
* Replaced utstring with std::string
* Converted transfer_queue to a list
* Converted pending_object_transfers to unordered_map
* Fix free/delete bug and small modifications.
* Implemented local_available_objects as an unordered set
* Implemented fetch_requests as an unordered map
* Fixed bug and changed fetch_requests from pointer to object
* free(PlasmaManagerState *) -> delete PlasmaManagerState *
* removed unnecessary newline
* Make local_available_objects not a pointer.
* Attempt to safely iterate over unordered_map and remove elements.
* Rebase Ray on top of Plasma in Apache Arrow
* add thirdparty building scripts
* use rebased arrow
* fix
* fix build
* fix python visibility
* comment out C tests for now
* fix multithreading
* fix
* reduce logging
* fix plasma manager multithreading
* make sure old and new object IDs can coexist peacefully
* more rebasing
* update
* fixes
* fix
* install pyarrow
* install cython
* fix
* install newer cmake
* fix
* rebase on top of latest arrow
* getting runtest.py run locally (needed to comment out a test for that to work)
* work on plasma tests
* more fixes
* fix local scheduler tests
* fix global scheduler test
* more fixes
* fix python 3 bytes vs string
* fix manager tests valgrind
* fix documentation building
* fix linting
* fix c++ linting
* fix linting
* add tests back in
* Install without sudo.
* Set PKG_CONFIG_PATH in build.sh so that Ray can find plasma.
* Install pkg-config
* Link -lpthread, note that find_package(Threads) doesn't seem to work reliably.
* Comment in testGPUIDs in runtest.py.
* Set PKG_CONFIG_PATH when building pyarrow.
* Pull apache/arrow and not pcmoritz/arrow.
* Fix installation in docker image.
* adapt to changes of the plasma api
* Fix installation of pyarrow module.
* Fix linting.
* Use correct python executable to build pyarrow.
* Don't reconstruct all objects in every fetch request in local scheduler.
* Separate out fetch timer and reconstruction timer.
* Fix bug.
* Bug fix.
* Fix naming convention for global variables.
* Address comments.
* Make reconstruct_counter a static variable.
* Fix linting.
* Redo reconstruct handler using a set of objects to fetch.
* Fix linting.
* Replace set with vector.
* Dispatch regular and actor tasks when resources become available.
* Make actor methods do resource bookkeeping and add test.
* Remove unnecessary field.
* Fix linting.
* Fix actor test.
* Maintain set of actors with pending tasks to speed up task dispatch.
* Exit early from task dispatch if there are no resources available.
* Fix linting.
* Fix error.
* Fix bug related to iterator invalidation.
* When an actor is removed, remove it from the set of actors with pending tasks.
* Replace a local scheduler ut_array with a std::vector.
* Replace vector of sizes in local scheduler with std::pair.
* Remove utarray include.
* Replace utarray with std::vector for reading local scheduler input messages.
* Remove more UT data structures.
* Remove UT includes.
* Fix linting.
* Include stdlib.h to find size_t.
* Remove includes of stdbool.h.
* Replace std::pair with TaskQueueEntry.
* Fix redis tests.
* Reinstate tests.
* Divide large get requests into smaller chunks.
* Divide fetches into smaller chunks.
* Wait longer in worker and manager before reissuing fetch requests if there are many outstanding fetch requests.
* Log warning if a handler in the local scheduler or plasma manager takes more than one second.
* Log fatal error if plasma manager or local scheduler take too long to send heartbeat.
* Fix linting.
* Use int64_t for milliseconds since unix epoch.
* Implement sharding in the Ray core
* Single node Python modifications to do sharding
* Do the sharding in redis.cc
* Pipe num_redis_shards through start_ray.py and worker.py.
* Use multiple redis shards in multinode tests.
* first steps for sharding ray.global_state
* Fix problem in multinode docker test.
* fix runtest.py
* fix some tests
* fix redis shard startup
* fix redis sharding
* fix
* fix bug introduced by the map-iterator being consumed
* fix sharding bug
* shard event table
* update number of Redis clients to be 64K
* Fix object table tests by flushing shards in between unit tests
* Fix local scheduler tests
* Documentation
* Register shard locations in the primary shard
* Add plasma unit tests back to build
* lint
* lint and fix build
* Fix
* Address Robert's comments
* Refactor start_ray_processes to start Redis shard
* lint
* Fix global scheduler python tests
* Fix redis module test
* Fix plasma test
* Fix component failure test
* Fix local scheduler test
* Fix runtest.py
* Fix global scheduler test for python3
* Fix task_table_test_and_update bug, from actor task table submission race
* Fix jenkins tests.
* Retry Redis shard connections
* Fix test cases
* Convert database clients to DBClient struct
* Fix race condition when subscribing to db client table
* Remove unused lines, add APITest for sharded Ray
* Fix
* Fix memory leak
* Suppress ReconstructionTests output
* Suppress output for APITestSharded
* Reissue task table add/update commands if initial command does not publish to any subscribers.
* fix
* Fix linting.
* fix tests
* fix linting
* fix python test
* fix linting
* plasma manager perf: speedup wait with a wait request object map
* removing duplicate == operator in plasma store
* fix serialization test
* code cleanup
* minor cleanup
* factoring out uniqueid hash and equality operators into common
* plasma manager: c++ify the WaitRequest struct
* plasma manager: get rid of the initial object request malloc
* cleanup
* linting
* cleanups and fix compiler warnings
* compiler warnings and linting
* Ignore deleted clients when reading address info from Redis
* Remove self from db_client table when exiting cleanly
* Fix valgrind test
* Do not call plasma_perform_release when disconnecting
* Fix worker blocked bug
* tmp
* Push an error to the driver on ray.put for non-driver tasks
* Fix result table tests
* Fix test, logging
* Address comments
* Fix suppression bug
* Fix redis module test
* Edit error message
* Get values in chunks during reconstruction
* Test case for driver ray.put errors
* Error for evicting ray.put objects from the driver
* Fix tests
* Reduce verbosity
* Documentation
* Failing test case
* Local scheduler exits cleanly after plasma store dies
* Tolerate one plasma store failure
* Tolerate plasma store failures on all nodes except head node
* Plasma manager heartbeats
* Component failure tests
* Don't run the helper for Python testing
* Fix C test
* Fix hanging plasma transfer test
* Fix python3
* Consolidate ClientConnection code
* Fix valgrind test
* fix c test
* We can restart worker nodes!
* Fix flatbuffers bug
* Address comments
* Only register actual workers with the local scheduler
* Fix bug
* Fix segfaults
* Add test case that tests for driver liveness, fix local scheduler bug
* Clean up after tests
* Allocate retry info on the stack
* Send SIGKILL before waiting
* Relax unit test conditions
* Driver liveness test case and documentation
* use flatbuffer messages for local scheduler
* make sure constructor gets called for C++ object ObjectInfoT
* fix typo
* fix Robert's comments
* Small change to actor test.
* fix valgrind error
* linting
* free notification
* fix
* valgrind
* fix valgrind
* fix other bugs
* valgrind fix
* fixes
* more fixes
* Small changes to comments.
* WARN instead of FATAL for object hash mismatches, push error to driver
* Document the callback signature for object_table_add/remove
* Error table
* Wait for all errors in python test
* Fix doc
* Fix state test
* Change tranfer queue to doubly-linked list to speed up append.
* Maintain set of pending transfers to make deduplication easy.
* Fix naming convention for structs in plasma manager.
