* Reconstruct actor state when local schedulers fail.
* Simplify construction of arguments to pass into default_worker.py from local scheduler.
* Remove deprecated ray.actor.
* Simplify actor reconstruction method.
* Fix linting.
* Small fixes.
* Rebase Ray on top of Plasma in Apache Arrow
* add thirdparty building scripts
* use rebased arrow
* fix
* fix build
* fix python visibility
* comment out C tests for now
* fix multithreading
* fix
* reduce logging
* fix plasma manager multithreading
* make sure old and new object IDs can coexist peacefully
* more rebasing
* update
* fixes
* fix
* install pyarrow
* install cython
* fix
* install newer cmake
* fix
* rebase on top of latest arrow
* getting runtest.py run locally (needed to comment out a test for that to work)
* work on plasma tests
* more fixes
* fix local scheduler tests
* fix global scheduler test
* more fixes
* fix python 3 bytes vs string
* fix manager tests valgrind
* fix documentation building
* fix linting
* fix c++ linting
* fix linting
* add tests back in
* Install without sudo.
* Set PKG_CONFIG_PATH in build.sh so that Ray can find plasma.
* Install pkg-config
* Link -lpthread, note that find_package(Threads) doesn't seem to work reliably.
* Comment in testGPUIDs in runtest.py.
* Set PKG_CONFIG_PATH when building pyarrow.
* Pull apache/arrow and not pcmoritz/arrow.
* Fix installation in docker image.
* adapt to changes of the plasma api
* Fix installation of pyarrow module.
* Fix linting.
* Use correct python executable to build pyarrow.
* Don't reconstruct all objects in every fetch request in local scheduler.
* Separate out fetch timer and reconstruction timer.
* Fix bug.
* Bug fix.
* Fix naming convention for global variables.
* Address comments.
* Make reconstruct_counter a static variable.
* Fix linting.
* Redo reconstruct handler using a set of objects to fetch.
* Fix linting.
* Replace set with vector.
* Dispatch regular and actor tasks when resources become available.
* Make actor methods do resource bookkeeping and add test.
* Remove unnecessary field.
* Fix linting.
* Fix actor test.
* Maintain set of actors with pending tasks to speed up task dispatch.
* Exit early from task dispatch if there are no resources available.
* Fix linting.
* Fix error.
* Fix bug related to iterator invalidation.
* When an actor is removed, remove it from the set of actors with pending tasks.
* Replace a local scheduler ut_array with a std::vector.
* Replace vector of sizes in local scheduler with std::pair.
* Remove utarray include.
* Replace utarray with std::vector for reading local scheduler input messages.
* Remove more UT data structures.
* Remove UT includes.
* Fix linting.
* Include stdlib.h to find size_t.
* Remove includes of stdbool.h.
* Replace std::pair with TaskQueueEntry.
* Fix redis tests.
* Reinstate tests.
* Divide large get requests into smaller chunks.
* Divide fetches into smaller chunks.
* Wait longer in worker and manager before reissuing fetch requests if there are many outstanding fetch requests.
* Log warning if a handler in the local scheduler or plasma manager takes more than one second.
* Log fatal error if plasma manager or local scheduler take too long to send heartbeat.
* Fix linting.
* Use int64_t for milliseconds since unix epoch.
* implement restarting workers after certain number of task executions
* Clean up python code.
* Don't start new worker when an actor disconnects.
* Move wait_for_pid_to_exit to test_utils.py.
* Add test.
* Fix linting errors.
* Fix linting.
* Fix typo.
* Implement sharding in the Ray core
* Single node Python modifications to do sharding
* Do the sharding in redis.cc
* Pipe num_redis_shards through start_ray.py and worker.py.
* Use multiple redis shards in multinode tests.
* first steps for sharding ray.global_state
* Fix problem in multinode docker test.
* fix runtest.py
* fix some tests
* fix redis shard startup
* fix redis sharding
* fix
* fix bug introduced by the map-iterator being consumed
* fix sharding bug
* shard event table
* update number of Redis clients to be 64K
* Fix object table tests by flushing shards in between unit tests
* Fix local scheduler tests
* Documentation
* Register shard locations in the primary shard
* Add plasma unit tests back to build
* lint
* lint and fix build
* Fix
* Address Robert's comments
* Refactor start_ray_processes to start Redis shard
* lint
* Fix global scheduler python tests
* Fix redis module test
* Fix plasma test
* Fix component failure test
* Fix local scheduler test
* Fix runtest.py
* Fix global scheduler test for python3
* Fix task_table_test_and_update bug, from actor task table submission race
* Fix jenkins tests.
* Retry Redis shard connections
* Fix test cases
* Convert database clients to DBClient struct
* Fix race condition when subscribing to db client table
* Remove unused lines, add APITest for sharded Ray
* Fix
* Fix memory leak
* Suppress ReconstructionTests output
* Suppress output for APITestSharded
* Reissue task table add/update commands if initial command does not publish to any subscribers.
* fix
* Fix linting.
* fix tests
* fix linting
* fix python test
* fix linting
* Change local scheduler bookkeeping to use GPU IDs.
* Update actor test.
* Add tests for actors and tasks simultaneously using GPUs.
* Add additional task GPU ID test.
* Fix linting.
* Make redis GPU assignment ignore GPU IDs.
* Small fix.
* copy task specifications put into the actor task cache so it won't get overwritten when the scheduler receives the next task
* cleanup
* cleanup and fix
* linting
* fix jenkins test
* fix linting
* Augment test to verify that relevant workers and actors are killed during driver cleanup.
* Fix bug in which we were only killing one worker when a driver exited.
* Fix remove driver test.
* Fix and augment test.
* Clean up state when drivers exit.
* Remove unnecessary field in ActorMapEntry struct.
* Have monitor release GPU resources in Redis when driver exits.
* Enable multiple drivers in multi-node tests and test driver cleanup.
* Make redis GPU allocation a redis transaction and small cleanups.
* Fix multi-node test.
* Small cleanups.
* Make global scheduler take node_ip_address so it appears in the right place in the client table.
* Cleanups.
* Fix linting and cleanups in local scheduler.
* Fix removed_driver_test.
* Fix bug related to vector -> list.
* Fix linting.
* Cleanup.
* Fix multi node tests.
* Fix jenkins tests.
* Add another multi node test with many drivers.
* Fix linting.
* Make the actor creation notification a flatbuffer message.
* Revert "Make the actor creation notification a flatbuffer message."
This reverts commit af99099c8084dbf9177fb4e34c0c9b1a12c78f39.
* Add comment explaining flatbuffer problems.
* Convert more local scheduler data structures to C++ STL.
* Convert vector pointer to vector.
* Convert some of the UT_arrays to std::vector.
* Simplify worker vectors.
* Simplify remote_object and local_object containers.
* Change some unnecessary checks to DCHECK.
* Ignore deleted clients when reading address info from Redis
* Remove self from db_client table when exiting cleanly
* Fix valgrind test
* Do not call plasma_perform_release when disconnecting
* Switch to using C++ lists for task queues
* Init and free methods for TaskQueueEntry
* Switch from utarray to c++ vector for TaskQueueEntry
* Get rid of some pointers
* Back to O(1) deletion from waiting_task_queue
* Fix comments
* Cut code
* Non const iterators
* Fix Alexey's comments
* Fix worker blocked bug
* tmp
* Push an error to the driver on ray.put for non-driver tasks
* Fix result table tests
* Fix test, logging
* Address comments
* Fix suppression bug
* Fix redis module test
* Edit error message
* Get values in chunks during reconstruction
* Test case for driver ray.put errors
* Error for evicting ray.put objects from the driver
* Fix tests
* Reduce verbosity
* Documentation
* Failing test case
* Local scheduler exits cleanly after plasma store dies
* Tolerate one plasma store failure
* Tolerate plasma store failures on all nodes except head node
* Plasma manager heartbeats
* Component failure tests
* Don't run the helper for Python testing
* Fix C test
* Fix hanging plasma transfer test
* Fix python3
* Consolidate ClientConnection code
* Fix valgrind test
* fix c test
* We can restart worker nodes!
* Fix flatbuffers bug
* Address comments
* Only register actual workers with the local scheduler
* Fix bug
* Fix segfaults
* Add test case that tests for driver liveness, fix local scheduler bug
* Clean up after tests
* Allocate retry info on the stack
* Send SIGKILL before waiting
* Relax unit test conditions
* Driver liveness test case and documentation
* use flatbuffer messages for local scheduler
* make sure constructor gets called for C++ object ObjectInfoT
* fix typo
* fix Robert's comments
* Small change to actor test.
* fix valgrind error
* linting
* free notification
* fix
* valgrind
* fix valgrind
* fix other bugs
* valgrind fix
* fixes
* more fixes
* Small changes to comments.
* WARN instead of FATAL for object hash mismatches, push error to driver
* Document the callback signature for object_table_add/remove
* Error table
* Wait for all errors in python test
* Fix doc
* Fix state test
* Clean up plasma subscribers on EPIPE
First pass at a monitoring script - monitor can detect local scheduler death
Clean up task table upon local scheduler death in monitoring script
Don't schedule to dead local schedulers in global scheduler
Have global scheduler update the db clients table, monitor script cleans up state
Documentation
Monitor script should scan tables before beginning to read from subscription channel
Fix for python3
Redirect monitor output to redis logs, fix hanging in multinode tests
* Publish auxiliary addresses as part of db_client deletion notifications
* Fix test case?
* Small changes.
* Use SCAN instead of KEYS
* Address comments
* Address more comments
* Free redis module strings
