Commit graph

566 commits

Author SHA1 Message Date
Robert Nishihara
efe8a295ea Add basic LRU eviction for the plasma store. (#26)
* Basic functionality for LRU eviction.

* Test eviction.

* Factor out eviction policy.

* Move delete_object into eviction policy.

* Replace array of released objects with an LRU cache (hash table + doubly linked list).

* Finish rebase on master.

* Move actual object deletion away from eviction policy and into plasma store.

* Small fixes.

* Fixes.

* Make remove_object_from_lru_cache always remove the object.

* Minor formatting and comments.

* Pass in allowed memory as argument to Plasma store.

* Small fix.
2016-11-05 21:34:11 -07:00
Philipp Moritz
90a2aa4bf7 Various performance improvements (#24)
* switch from array to linked list for photon queue

* performance optimizations

* fix tests

* various fixes
2016-11-04 00:41:20 -07:00
Robert Nishihara
1e2b3ceac9 Fix whitespace. (#27) 2016-11-03 17:36:55 -07:00
Ujval Misra
b370a1df57 Merge sealed_objects and open_objects into a single hashmap (#25)
* Merge sealed_objects and open_objects into a single hashmap

* Entry contains enum that determines whether it is open or closed

* Removed unused variable.

* Applied Robert's patch

* Fixed styling.
2016-11-03 17:33:54 -07:00
Robert Nishihara
5dfd76e0f7 Causes build scripts to fail immediately if a single command fails. (#23) 2016-11-02 20:56:25 -07:00
Robert Nishihara
681ec570ba Remove unnecessary pip installs. (#21)
* Small cleanups in worker.py.

* Remove dependencies on subprocess32, graphviz, protobuf, and ipython.

* Retry starting the plasma manager if the port is in use.

* Whitespace

* Move start_plasma_manager into plasma.py.
2016-11-02 16:40:37 -07:00
Robert Nishihara
072f442c1f Update worker.py and services.py to use plasma and the local scheduler. (#19)
* Update worker code and services code to use plasma and the local scheduler.

* Cleanups.

* Fix bug in which threads were started before the worker mode was set. This caused remote functions to be defined on workers before the worker knew it was in WORKER_MODE.

* Fix bug in install-dependencies.sh.

* Lengthen timeout in failure_test.py.

* Cleanups.

* Cleanup services.start_ray_local.

* Clean up random name generation.

* Cleanups.
2016-11-02 00:39:35 -07:00
Philipp Moritz
2068587af8 Fix socket bind collisions in manager_tests, race condition bind/listen/subscribe and memory leaks (#18)
* Fix socket bind collisions in manager_tests

* bind manager sockets before connecting to the store

* fix memory leak in tests

* fix valgrind early process termination

* fix bind/listen/subscribe race condition

* fix photon

* fix other tests

* make it that all of common is tested

* fix clang-format

* fix
2016-11-02 00:09:04 -07:00
Robert Nishihara
d56c1a0b9c Change license to Apache 2 (#20) 2016-11-01 23:19:06 -07:00
Robert Nishihara
47851eeccc Build Ray with setup.py. (#14)
* Build Ray with setup.py.

* Building photon extensions with cmake.

* Fix formatting in photon_extension.c

* Pip install with sudo in Travis.

* Fix plasma __init__.py.

* Rename and remove some files.
2016-10-31 17:08:03 -07:00
Philipp Moritz
695f23b2e0 try again to start plasma manager if there is a port collision (#16) 2016-10-31 15:00:15 -07:00
Philipp Moritz
780bbd170a fix test (#15) 2016-10-31 13:42:56 -07:00
Robert Nishihara
a0049ffa48 Wait until data is ready to read in manager_tests. (#13)
* Wait until data is ready to read in manager_tests.

* remove race conditions from plasma manager test and make CHECK and CHECKM print a backgrace
2016-10-29 19:25:43 -07:00
Philipp Moritz
e6319986ec Implement plasma.wait (#12)
* implement wait

* fix wait documentation

* fixes

* More fixes.

* more fixes

* Update test to check if redis-server is present.
2016-10-29 17:30:34 -07:00
Ion
ee3718c80c Ion and Philipp's table retries (#10)
* Ion and Philipp's table retries

* Refactor the retry struct:
- Rename it from retry_struct to retry_info
- Retry information contains the failure callback, not the retry callback
- All functions take in retry information as an arg instead of its expanded fields

* Rename cb -> callback

* Remove prints

* Fix compiler warnings

* Change some CHECKs to greatest ASSERTs

* Key outstanding callbacks hash table with timer ID instead of callback data pointer

* Use the new retry API for table commands

* Memory cleanup in plasma unit tests

* fix Robert's comments

* add valgrind for common
2016-10-29 15:22:33 -07:00
Philipp Moritz
84c581cf47 Let local clients connect with local manager via unix domain sockets (#11)
* let local clients connect with local manager via unix domain sockets

* fix clang-format

* fix tests

* fix stephanie's comments
2016-10-28 21:26:54 -07:00
Robert Nishihara
09a3ff7173 Pip install numbuf. (#8) 2016-10-28 14:30:20 -07:00
Robert Nishihara
c94f961366 Merge pull request #9 from ray-project/port
Unit testing for the plasma manager.
2016-10-28 12:42:45 -07:00
Stephanie Wang
f3a43179c8 Unit testing for the plasma manager. 2016-10-28 11:56:22 -07:00
Robert Nishihara
5c50d97670 Update .gitignore file. (#7) 2016-10-28 11:40:08 -07:00
Robert Nishihara
43aca17b59 Merge pull request #6 from ray-project/subscribe_no_polling
[Plasma] Subscribe without polling
2016-10-27 17:56:16 -07:00
Philipp Moritz
c9d0ca49eb add documentation 2016-10-27 16:30:29 -07:00
Philipp Moritz
b4b462809f fix valgrind 2016-10-27 15:09:50 -07:00
Philipp Moritz
e26fb46b02 subscribe without polling 2016-10-27 13:21:16 -07:00
Robert Nishihara
6ed641177d Remove unnecessary files. (#4) 2016-10-26 23:24:40 -07:00
Robert Nishihara
6f75c738b5 [WIP] Fix valgrind tests. (#5)
* Make tests fail when valgrind finds a memory leak.

* Properly clean up scheduler state.

* Remove unnecessary malloc.
2016-10-26 23:23:46 -07:00
Robert Nishihara
f83a98d71a Changes to make tests pass on Travis. (#3)
* Remove hiredis submodule.

* Squashed 'src/common/thirdparty/hiredis/' content from commit acd1966

git-subtree-dir: src/common/thirdparty/hiredis
git-subtree-split: acd1966bf7f5e1be74b426272635c672def78779

* Make Plasma tests pass.

* Make Photon tests pass.

* Compile and test with Travis.

* Deactive fetch test so that the tests pass.
2016-10-25 22:39:21 -07:00
Philipp Moritz
ed550ea8af Merge pull request #1 from ray-project/switch
Move plasma, common, and photon into the Ray repository.
2016-10-25 14:33:51 -07:00
Robert Nishihara
8c48410e82 Merge remote-tracking branch 'r1remote/moveout' into switch 2016-10-25 14:29:00 -07:00
Robert Nishihara
02d4050499 Rearrange plasma files to prepare for merging into Ray. 2016-10-25 14:27:45 -07:00
Robert Nishihara
6a5ddae59f Merge remote-tracking branch 'r1remote/moveout' into switch 2016-10-25 14:19:24 -07:00
Robert Nishihara
ad55166472 Rearrange local scheduler files to prepare to merge into Ray. 2016-10-25 14:16:23 -07:00
Robert Nishihara
5165b41fa7 Merge remote-tracking branch 'r1remote/moveout' into switch 2016-10-25 14:02:32 -07:00
Robert Nishihara
1915539c5f Rearrange files to prepare to merge into Ray. 2016-10-25 13:59:47 -07:00
Robert Nishihara
7c1b2f702f Remove git submodules and C++ files. 2016-10-25 12:51:30 -07:00
Robert Nishihara
db6375701e Prevent ObjectIDs from being pickled. (#42) 2016-10-21 15:42:29 -07:00
Robert Nishihara
d19f7ad853 Implement plasma_release. (#42)
* Implement plasma_release, enabling clients to indicate when they are no longer using an object.

* Call plasma_release for each plasma_create call, and call plasma_release from the manager.

* Fixes.

* Create client specific contexts for plasma_store callbacks.

* More changes.

* Fixes.

* Cleanup lists of clients using objects when a client disconnects.

* Make names parallel.
2016-10-21 00:47:34 -07:00
Robert Nishihara
63ec244784 Connect local scheduler to Plasma. (#11)
* Receive notifications about sealed objects from Plasma, and schedule tasks only when the dependencies are available locally.

* Fix formatting.

* Use version of Plasma with fix.

* Fix.

* Factor out the scheduling algorithm and use worker_index instead of the client socket to identify workers

* Fixes

* clang-format

* fix remaining linter errors
2016-10-18 18:27:43 -07:00
Stephanie Wang
ddfbd70dad Plasma fetch (#36)
* Fetch objects from remote plasma instances. Includes:
  - Configurable number of retries, with a timeout per retry
  - Support for multiple-object fetch
  - Test cases (integration only)

Update ray common

Remove attempts to retry object table lookup during fetch

lint

Fix a couple valgrind errors

Fix valgrind errors

Address Robert and Philipp's comments.

* Add fix from ray common and some TODOs.

* Update ray common

* Update ray common again

* Remove unused file
2016-10-18 18:20:59 -07:00
Stephanie Wang
da3a3127e0 Move get_client_id to db.h (#40) 2016-10-18 15:12:41 -07:00
Stephanie Wang
e57b87928c Fixes for implementing Plasma fetch (#39)
* Add ability to pass callback context to object table lookup

* Propagate errors during socket writes up to caller.

* Use recv and MSG_WAITALL flag instead of looping read

* Error checking in write_bytes

* Method to listen on a network port

* Revert "Use recv and MSG_WAITALL flag instead of looping read"

This reverts commit 32d9333bc6a185729aadb4b41b70b3d7f150a9c2.

* Some documentation

* Clearer documentation

* Fix bug where database clients were getting assigned the same ID

* Regression test for unique client IDs
2016-10-18 12:38:30 -07:00
Robert Nishihara
f189ca746b Send a one byte message in send_fd because otherwise it sometimes hangs on Linux. (#41) 2016-10-15 11:44:51 -07:00
Robert Nishihara
5ad8e145ae Allow clients to subscribe to notifications about sealed objects. (#40)
* Create libplasma_client.a in Makefile.

* Implement plasma subscribe.

* Fixes and tests.

* Buffer notifications in the Plasma store when the socket send buffer is full.

* Fix formatting.

* Turn off -Werror in Makefile.

* Fixes.

* Fix formatting.
2016-10-14 19:27:17 -07:00
Robert Nishihara
0a44145906 Fix the resetting of reusable variables on the driver and cache functions to run on all workers. (#446)
* Properly reset reusable variables on the driver when remote functions are run locally on the driver.

* Cache functions to run on all workers that occur before ray.init is called.
2016-10-12 22:17:22 -07:00
Stephanie Wang
7e23b87a21 Merge pull request #38 from ray-project/fixwarnings
Fix warnings
2016-10-12 15:43:01 -07:00
Robert Nishihara
1c3aaf7189 Update documentation (#445)
* Update documentation for serialization.

* Update documentation for reusable variables.

* Update documentation for using Ray with TensorFlow. This change is to allow code blocks to be copied and pasted into a Python interpreter.

* Fix documentation for hyperparameter optimization example.
2016-10-12 15:41:00 -07:00
Philipp Moritz
182985015c Make warnings errors 2016-10-12 13:20:57 -07:00
Robert Nishihara
1adafee6d3 Fix signature of create_object. (#39) 2016-10-12 12:58:15 -07:00
Philipp Moritz
832888d473 fix formating 2016-10-12 01:10:05 -07:00
Philipp Moritz
50fb53fd91 fix compiler warnings on macOS 2016-10-12 00:56:04 -07:00