* Reconstruct actor state when local schedulers fail.
* Simplify construction of arguments to pass into default_worker.py from local scheduler.
* Remove deprecated ray.actor.
* Simplify actor reconstruction method.
* Fix linting.
* Small fixes.
* Rebase Ray on top of Plasma in Apache Arrow
* add thirdparty building scripts
* use rebased arrow
* fix
* fix build
* fix python visibility
* comment out C tests for now
* fix multithreading
* fix
* reduce logging
* fix plasma manager multithreading
* make sure old and new object IDs can coexist peacefully
* more rebasing
* update
* fixes
* fix
* install pyarrow
* install cython
* fix
* install newer cmake
* fix
* rebase on top of latest arrow
* getting runtest.py run locally (needed to comment out a test for that to work)
* work on plasma tests
* more fixes
* fix local scheduler tests
* fix global scheduler test
* more fixes
* fix python 3 bytes vs string
* fix manager tests valgrind
* fix documentation building
* fix linting
* fix c++ linting
* fix linting
* add tests back in
* Install without sudo.
* Set PKG_CONFIG_PATH in build.sh so that Ray can find plasma.
* Install pkg-config
* Link -lpthread, note that find_package(Threads) doesn't seem to work reliably.
* Comment in testGPUIDs in runtest.py.
* Set PKG_CONFIG_PATH when building pyarrow.
* Pull apache/arrow and not pcmoritz/arrow.
* Fix installation in docker image.
* adapt to changes of the plasma api
* Fix installation of pyarrow module.
* Fix linting.
* Use correct python executable to build pyarrow.
* Replace a local scheduler ut_array with a std::vector.
* Replace vector of sizes in local scheduler with std::pair.
* Remove utarray include.
* Replace utarray with std::vector for reading local scheduler input messages.
* Remove more UT data structures.
* Remove UT includes.
* Fix linting.
* Include stdlib.h to find size_t.
* Remove includes of stdbool.h.
* Replace std::pair with TaskQueueEntry.
* Fix redis tests.
* Reinstate tests.
* added log_table function and a test
* fixed log_files and added task_profiles
* fixed formatting
* fixed linting errors
* fixes
* removed file
* more fixes
* hopefully fixed
* Small changes.
* Fix linting.
* Fix bug in log monitor.
* Small changes.
* Fix bug in travis.
* Including data_size and hash in the ResultTableReply.
* Included data_size and hash info in object_table.
* Fixed bugs in ray_redis_module.cc.
* Removing commented out code.
* Fixes
* Freed hash and data_size strings after using, and checked if they're null along with task_id and is_put.
* Changed it so that data_size is set correctly.
* Removed iostream import.
* Included a check to ensure that the Redis string to long long conversion was successful.
* Included separate data_size and hash null checks.
* Fixed bug.
* Made linting changes.
* Another linting error.
* Slight simplication.
* Log fatal error if plasma manager or local scheduler take too long to send heartbeat.
* Fix linting.
* Use int64_t for milliseconds since unix epoch.
* Implement sharding in the Ray core
* Single node Python modifications to do sharding
* Do the sharding in redis.cc
* Pipe num_redis_shards through start_ray.py and worker.py.
* Use multiple redis shards in multinode tests.
* first steps for sharding ray.global_state
* Fix problem in multinode docker test.
* fix runtest.py
* fix some tests
* fix redis shard startup
* fix redis sharding
* fix
* fix bug introduced by the map-iterator being consumed
* fix sharding bug
* shard event table
* update number of Redis clients to be 64K
* Fix object table tests by flushing shards in between unit tests
* Fix local scheduler tests
* Documentation
* Register shard locations in the primary shard
* Add plasma unit tests back to build
* lint
* lint and fix build
* Fix
* Address Robert's comments
* Refactor start_ray_processes to start Redis shard
* lint
* Fix global scheduler python tests
* Fix redis module test
* Fix plasma test
* Fix component failure test
* Fix local scheduler test
* Fix runtest.py
* Fix global scheduler test for python3
* Fix task_table_test_and_update bug, from actor task table submission race
* Fix jenkins tests.
* Retry Redis shard connections
* Fix test cases
* Convert database clients to DBClient struct
* Fix race condition when subscribing to db client table
* Remove unused lines, add APITest for sharded Ray
* Fix
* Fix memory leak
* Suppress ReconstructionTests output
* Suppress output for APITestSharded
* Reissue task table add/update commands if initial command does not publish to any subscribers.
* fix
* Fix linting.
* fix tests
* fix linting
* fix python test
* fix linting
* copy task specifications put into the actor task cache so it won't get overwritten when the scheduler receives the next task
* cleanup
* cleanup and fix
* linting
* fix jenkins test
* fix linting
* Clean up state when drivers exit.
* Remove unnecessary field in ActorMapEntry struct.
* Have monitor release GPU resources in Redis when driver exits.
* Enable multiple drivers in multi-node tests and test driver cleanup.
* Make redis GPU allocation a redis transaction and small cleanups.
* Fix multi-node test.
* Small cleanups.
* Make global scheduler take node_ip_address so it appears in the right place in the client table.
* Cleanups.
* Fix linting and cleanups in local scheduler.
* Fix removed_driver_test.
* Fix bug related to vector -> list.
* Fix linting.
* Cleanup.
* Fix multi node tests.
* Fix jenkins tests.
* Add another multi node test with many drivers.
* Fix linting.
* Make the actor creation notification a flatbuffer message.
* Revert "Make the actor creation notification a flatbuffer message."
This reverts commit af99099c8084dbf9177fb4e34c0c9b1a12c78f39.
* Add comment explaining flatbuffer problems.
* plasma manager perf: speedup wait with a wait request object map
* removing duplicate == operator in plasma store
* fix serialization test
* code cleanup
* minor cleanup
* factoring out uniqueid hash and equality operators into common
* plasma manager: c++ify the WaitRequest struct
* plasma manager: get rid of the initial object request malloc
* cleanup
* linting
* cleanups and fix compiler warnings
* compiler warnings and linting
* Ignore deleted clients when reading address info from Redis
* Remove self from db_client table when exiting cleanly
* Fix valgrind test
* Do not call plasma_perform_release when disconnecting
* Switch to using C++ lists for task queues
* Init and free methods for TaskQueueEntry
* Switch from utarray to c++ vector for TaskQueueEntry
* Get rid of some pointers
* Back to O(1) deletion from waiting_task_queue
* Fix comments
* Cut code
* Non const iterators
* Fix Alexey's comments
* Fix worker blocked bug
* tmp
* Push an error to the driver on ray.put for non-driver tasks
* Fix result table tests
* Fix test, logging
* Address comments
* Fix suppression bug
* Fix redis module test
* Edit error message
* Get values in chunks during reconstruction
* Test case for driver ray.put errors
* Error for evicting ray.put objects from the driver
* Fix tests
* Reduce verbosity
* Documentation
* Failing test case
* Local scheduler exits cleanly after plasma store dies
* Tolerate one plasma store failure
* Tolerate plasma store failures on all nodes except head node
* Plasma manager heartbeats
* Component failure tests
* Don't run the helper for Python testing
* Fix C test
* Fix hanging plasma transfer test
* Fix python3
* Consolidate ClientConnection code
* Fix valgrind test
* fix c test
* We can restart worker nodes!
* Fix flatbuffers bug
* Address comments
* Only register actual workers with the local scheduler
* Fix bug
* Fix segfaults
* Add test case that tests for driver liveness, fix local scheduler bug
* Clean up after tests
* Allocate retry info on the stack
* Send SIGKILL before waiting
* Relax unit test conditions
* Driver liveness test case and documentation
* use flatbuffer messages for local scheduler
* make sure constructor gets called for C++ object ObjectInfoT
* fix typo
* fix Robert's comments
* Small change to actor test.
* fix valgrind error
* linting
* free notification
* fix
* valgrind
* fix valgrind
* fix other bugs
* valgrind fix
* fixes
* more fixes
* Small changes to comments.
* Compile the Ray redis module with C++.
* Redo parsing of object table notifications with flatbuffers.
* Update redis module python tests.
* Redo parsing of task table notifications with flatbuffers.
* Fix linting.
* Redo parsing of db client notifications with flatbuffers.
* Redo publishing of local scheduler heartbeats with flatbuffers.
* Fix linting.
* Remove usage of fixed-width formatting of scheduling state in channel name.
* Reply with flatbuffer object to task table queries, also simplify redis string to flatbuffer string conversion.
* Fix linting and tests.
* fix
* cleanup
* simplify logic in ReplyWithTask
* WARN instead of FATAL for object hash mismatches, push error to driver
* Document the callback signature for object_table_add/remove
* Error table
* Wait for all errors in python test
* Fix doc
* Fix state test
* Clean up plasma subscribers on EPIPE
First pass at a monitoring script - monitor can detect local scheduler death
Clean up task table upon local scheduler death in monitoring script
Don't schedule to dead local schedulers in global scheduler
Have global scheduler update the db clients table, monitor script cleans up state
Documentation
Monitor script should scan tables before beginning to read from subscription channel
Fix for python3
Redirect monitor output to redis logs, fix hanging in multinode tests
* Publish auxiliary addresses as part of db_client deletion notifications
* Fix test case?
* Small changes.
* Use SCAN instead of KEYS
* Address comments
* Address more comments
* Free redis module strings
* adding flatbuffers and migrating flatcc to flatbuffers for plasma
* variable name changes in plasma_protocol and plasma flatbuffers schema
* quick fix
* cleanups and remove flatcc
* more cleanup
* add doc
* linting
* fix linting
* fix mac os x build
* linting
* cleanup
* c++ fix for plasma flatbuffers
* Remove flatcc from CMakeLists.txt.
* linting; trigger travis
* Availability after a killed worker
* Workers exit cleanly
* Memory cleanup in photon C tests
* Worker failure in multinode
* Consolidate worker cleanup handlers
* Update the result table before handling a task submission
* KILL_WORKER_TIMEOUT -> KILL_WORKER_TIMEOUT_MILLISECONDS
* Log a warning instead of crashing if no result table entry found
* Implement actor field for tasks
* Implement actor management in local scheduler.
* initial python frontend for actors
* import actors on worker
* IPython code completion and tests
* prepare creating actors through local schedulers
* add actor id to PyTask
* submit actor calls to local scheduler
* starting to integrate
* simple fix
* Fixes from rebasing.
* more work on python actors
* Improve local scheduler actor handlers.
* Pass actor ID to local scheduler when connecting a client.
* first working version of actors
* fixing actors
* fix creating two copies of the same actor
* fix actors
* remove sleep
* get rid of export synchronization
* update
* insert actor methods into the queue in the right order
* remove print statements
* make it compile again after rebase
* Minor updates.
* fix python actor ids
* Pass actor_id to start_worker.
* add test
* Minor changes.
* Update actor tests.
* Temporary plan for import counter.
* Temporarily fix import counters.
* Fix some tests.
* Fixes.
* Make actor creation non-blocking.
* Fix test?
* Fix actors on Python 2.
* fix rare case.
* Fix python 2 test.
* More tests.
* Small fixes.
* Linting.
* Revert tensorflow version to 0.12.0 temporarily.
* Small fix.
* Enhance inheritance test.
* attribute-based heterogeneity-awareness in global scheduler and photon
* minor post-rebase fix
* photon: enforce dynamic capacity constraint on task dispatch
* globalsched: cap the number of times we try to schedule a task in round robin
* propagating ability to specify resource capacity to ray.init
* adding resources to remote function export and fetch/register
* globalsched: remove unused functions; update cached photon resource capacity (until next photon heartbeat)
* Add some integration tests.
* globalsched: cleanup + factor out constraint checking
* lots of style
* task_spec_required_resource: global refactor
* clang format
* clang format + comment update in photon
* clang format photon comment
* valgrind
* reduce verbosity for Travis
* Add test for scheduler load balancing.
* addressing comments
* refactoring global scheduler algorithm
* Minor cleanups.
* Linting.
* Fix array_test.py and linting.
* valgrind fix for photon tests
* Attempt to fix stress tests.
* fix hashmap free
* fix hashmap free comment
* memset photon resource vectors to 0 in case they get used before the first heartbeat
* More whitespace changes.
* Undo whitespace error I introduced.
* Refactor local scheduler to remove worker indices.
* Change scheduling state enum to int in all function signatures.
* Bug fix, don't use pointers into a resizable array.
* Remove total_num_workers.
* Fix tests.
