* First pass at reconstruction in the worker
Modify reconstruction stress testing to start Plasma service before rest of Ray cluster
TODO about reconstructing ray.puts
Fix ray.put error for double creates
Distinguish between empty entry and no entry in object table
Fix test case
Fix Python test
Fix tests
* Only call reconstruct on objects we have not yet received
* Address review comments
* Fix reconstruction for Python3
* remove unused code
* Address Robert's comments, stress tests are crashing
* Test and update the task's scheduling state to suppress duplicate
reconstruction requests.
* Split result table into two lookups, one for task ID and the other as a
test-and-set for the task state
* Fix object table tests
* Fix redis module result_table_lookup test case
* Multinode reconstruction tests
* Fix python3 test case
* rename
* Use new start_redis
* Remove unused code
* lint
* indent
* Address Robert's comments
* Use start_redis from ray.services in state table tests
* Remove unnecessary memset
* Split local scheduler task queue into waiting and dispatch queue
* Fix memory leak
* Add a new task scheduling status for when a task has been queued locally
* Fix global scheduler test case and add task status doc
* Documentation
* Address Philipp's comments
* Move tasks back to the waiting queue if their dependencies become unavailable
* Update existing task table entries instead of overwriting
* Send message from plasma client to get plasma store capacity.
* Release objects from plasma client if they are too large.
* Use doubly-linked list instead of ring buffer for plasma client release history.
* Address comments.
* Fix problem with slicing PlasmaBuffer objects.
* Fix crash in plasma manager during transfer.
* Formatting.
* Make plasma client cache larger and make caching test not throw exceptions on Travis.
* Switch to using redis modules for task table.
* Switch to using redis modules for the task table.
* Fix some tests.
* Fix naming and remove code duplication.
* Remove duplication in redis modules and add more cleanups.
* Address comments.
* global scheduler with object transfer cost awareness -- upstream rebase
* debugging global scheduler: multiple subscriptions
* global scheduler: utarray push bug fix; tasks change state to SCHEDULED
* change global scheduler test to be an integraton test
* unit and integration tests are passing for global scheduler
* improve global scheduler test: break up into several
* global scheduler checkpoint: fix photon object id bug in test
* test with timesync between object and task notifications; TODO: handle OoO object+task notifications in GS
* fallback to base policy if no object dependencies are cached (may happen due to OoO object+task notification arrivals
* clean up printfs; handle a missing LS in LS cache
* Minor changes to Python test and factor out some common code.
* refactoring handle task waiting
* addressing comments
* log_info -> log_debug
* Change object ID printing.
* PRId64 merge
* Python 3 fix.
* PRId64.
* Python 3 fix.
* resurrect differentiation between no args and missing object info; spacing
* Valgrind fix.
* Run all global scheduler tests in valgrind.
* clang format
* Comments and documentation changes.
* Minor cleanups.
* fix whitespace
* Fix.
* Documentation fix.
* Add function for driver to get address info from Redis.
* Use Redis address instead of Redis port.
* Configure Redis to run in unprotected mode.
* Add method for starting Ray processes on non-head node.
* Pass in correct node ip address to start_plasma_manager.
* Script for starting Ray processes.
* Handle the case where an object already exists in the store. Maybe this should also compare the object hashes.
* Have driver get info from Redis when start_ray_local=False.
* Fix.
* Script for killing ray processes.
* Catch some errors when the main_loop in a worker throws an exception.
* Allow redirecting stdout and stderr to /dev/null.
* Wrap start_ray.py in a shell script.
* More helpful error messages.
* Fixes.
* Wait for redis server to start up before configuring it.
* Allow seeding of deterministic object ID generation.
* Small change.
* adding object broadcast channel; published on each object table add
* publishing data size to the bcast channel
* bug fix: objectkey
* update object tests to test for data size: C + py
* remove debug
* clang format
* Minor changes.
* Fix error.
* merging with Robert's comments
* clang format for the object table test upgrade
* Task table redis module implementation
* Publish tasks and take in individual fields as args, not task object
* Scheduling state integer has width 1, error on illegal put
* Unit tests for task table and more documentation
* Task table subscribe, fix publish topics and address Philipp and Alexey's comments
* Helper function to create prefixed strings
* Factor out the table prefixes in the test cases
* Add RAY.CONNECT Redis command.
* Add RAY.GET_CLIENT_ADDRESS command.
* Build and clean Redis in common Makefile.
* Use custom Redis module in Ray and use custom CONNECT and GET_CLIENT_ADDRESS commands.
* Fixes.
* Remove mapping from redis client ID to ray db client ID.
* Fix.
* passing plasma ip:port association with photon through redis to global scheduler
* Fix test.
* sanity-checking aux_address inside db_connect_extended
* clang format
* fix photon tests
* clang format photon tests
* First attempt at fixing psubscribe. psubscribe_success_test will fail
* psubscribe test
* SUBSCRIBE returns the number of subscriptions, not success
* Comment out failing test.
* Object reconstruction in Photon and C test cases for Photon
* Fix hanging test case on mac
* Remove unnecessary event from photon tests
* make photon_disconnect not leak file descriptors
* fix some of the memory errors
* Fix valgrind
* lint
* Address Robert's comments and add test case for object reconstruction suppression
* Remove OWNER
* initial redis module
* temp commit
* temp commit
* temp commit
* Empty object table functions and broken object_table_lookup
* fix segfault and clean up code
* cleanup and tests
* try to ignore redismodule.h
* check if data_size is integer
* Minor changes to redis-module tests.
* try to exclude redismodule from clang-format
* try something different
* fix clang-format and tests
* sleep a bit
* Result table
* fix redis_module tests
* fix tests and add tests for result table
* more tests
* randomize ports
* Minor changes.
* More fixes.
* rebase Alexey's PR on top
* rebase on master
* fix test failure waiting for plasma manager to exit
* clang format
* addressing comments
* Minor formatting and naming fixes.
* factoring out object_info for general use by several Ray components
* addressing comments
* Replace SHA256 task hash with MD5
Add object hash to object table (always overwrites)
Support for table operations that span multiple asynchronous Redis
commands
Add a new object location in a transaction, using Redis's optimistic
concurrency
Use Redis GETSET instead of transactions and Python frontend code for object hashing
Remove spurious log message
Fix for object_table_add
Revert "Replace SHA256 task hash with MD5"
This reverts commit e599de473c8dad9189ccb0600429534b469b76a2.
Revert to sha256
Test case for illegal puts
Use SETNX to set object hashes
Initialize digest with zeros
Initialize plasma_request with zeros
* Fixes
* replace SHA256 with a faster hash in the object store
* Fix valgrind
* Address Robert's comments
* Check that plasma_compute_object_hash succeeds.
* Don't run test_illegal_put test with valgrind because it causes an intentional crash which causes valgrind to complain.
* Debugging after rebase.
* handling Robert's comments
* Fix bugs after rebase.
* final fixes for Stephanie's PR
* fix
* change plasma object notifications to carry a struct of information
* factoring out object_info for general use by several Ray components
* fixing a bug in python test
* addressing comments
* handling Robert's comments
* clang format
* Fix valgrind.
* Implement new plasma client API.
* Formatting fixes.
* Make tests work again.
* Make tests run.
* Comment style.
* Fix bugs with fetch tests.
* Introduce fetch1 flag.
* Remove timer only if present.
* Formatting fixes.
* Don't access object after free.
* Formatting fixes.
* Minor change.
* refactoring plasma datastructures
* Change plasma_request and plasma_reply to use only arrays of object requests.
* some more fixes
* Remove unnecessary methods.
* Trivial.
* fixes
* use plasma_send_reply in return_from_wait1
* Lint.
* Give more informative error message when we do not know how to serialize a class.
* Check that passing arguments to remote functions and getting them does not change their values.
* fix serialization bug
* fix tests for common module
* Formatting.
* Bug fix in init_pickle_module signature.
* Use pickle with HIGHEST_PROTOCOL.
* Add Python and Redis submodules, and remove old third-party modules
* Update VS projects (WARNING: references files that do not exist yet)
* Update code & add shims for APIs except AF_UNIX/{send,recv}msg()
* Minor style changes.
