Commit graph

1250 commits

Author SHA1 Message Date
Robert Nishihara
681ec570ba Remove unnecessary pip installs. (#21)
* Small cleanups in worker.py.

* Remove dependencies on subprocess32, graphviz, protobuf, and ipython.

* Retry starting the plasma manager if the port is in use.

* Whitespace

* Move start_plasma_manager into plasma.py.
2016-11-02 16:40:37 -07:00
Robert Nishihara
072f442c1f Update worker.py and services.py to use plasma and the local scheduler. (#19)
* Update worker code and services code to use plasma and the local scheduler.

* Cleanups.

* Fix bug in which threads were started before the worker mode was set. This caused remote functions to be defined on workers before the worker knew it was in WORKER_MODE.

* Fix bug in install-dependencies.sh.

* Lengthen timeout in failure_test.py.

* Cleanups.

* Cleanup services.start_ray_local.

* Clean up random name generation.

* Cleanups.
2016-11-02 00:39:35 -07:00
Philipp Moritz
2068587af8 Fix socket bind collisions in manager_tests, race condition bind/listen/subscribe and memory leaks (#18)
* Fix socket bind collisions in manager_tests

* bind manager sockets before connecting to the store

* fix memory leak in tests

* fix valgrind early process termination

* fix bind/listen/subscribe race condition

* fix photon

* fix other tests

* make it that all of common is tested

* fix clang-format

* fix
2016-11-02 00:09:04 -07:00
Robert Nishihara
47851eeccc Build Ray with setup.py. (#14)
* Build Ray with setup.py.

* Building photon extensions with cmake.

* Fix formatting in photon_extension.c

* Pip install with sudo in Travis.

* Fix plasma __init__.py.

* Rename and remove some files.
2016-10-31 17:08:03 -07:00
Philipp Moritz
695f23b2e0 try again to start plasma manager if there is a port collision (#16) 2016-10-31 15:00:15 -07:00
Philipp Moritz
780bbd170a fix test (#15) 2016-10-31 13:42:56 -07:00
Robert Nishihara
a0049ffa48 Wait until data is ready to read in manager_tests. (#13)
* Wait until data is ready to read in manager_tests.

* remove race conditions from plasma manager test and make CHECK and CHECKM print a backgrace
2016-10-29 19:25:43 -07:00
Philipp Moritz
e6319986ec Implement plasma.wait (#12)
* implement wait

* fix wait documentation

* fixes

* More fixes.

* more fixes

* Update test to check if redis-server is present.
2016-10-29 17:30:34 -07:00
Ion
ee3718c80c Ion and Philipp's table retries (#10)
* Ion and Philipp's table retries

* Refactor the retry struct:
- Rename it from retry_struct to retry_info
- Retry information contains the failure callback, not the retry callback
- All functions take in retry information as an arg instead of its expanded fields

* Rename cb -> callback

* Remove prints

* Fix compiler warnings

* Change some CHECKs to greatest ASSERTs

* Key outstanding callbacks hash table with timer ID instead of callback data pointer

* Use the new retry API for table commands

* Memory cleanup in plasma unit tests

* fix Robert's comments

* add valgrind for common
2016-10-29 15:22:33 -07:00
Philipp Moritz
84c581cf47 Let local clients connect with local manager via unix domain sockets (#11)
* let local clients connect with local manager via unix domain sockets

* fix clang-format

* fix tests

* fix stephanie's comments
2016-10-28 21:26:54 -07:00
Stephanie Wang
f3a43179c8 Unit testing for the plasma manager. 2016-10-28 11:56:22 -07:00
Philipp Moritz
c9d0ca49eb add documentation 2016-10-27 16:30:29 -07:00
Philipp Moritz
b4b462809f fix valgrind 2016-10-27 15:09:50 -07:00
Philipp Moritz
e26fb46b02 subscribe without polling 2016-10-27 13:21:16 -07:00
Robert Nishihara
6f75c738b5 [WIP] Fix valgrind tests. (#5)
* Make tests fail when valgrind finds a memory leak.

* Properly clean up scheduler state.

* Remove unnecessary malloc.
2016-10-26 23:23:46 -07:00
Robert Nishihara
f83a98d71a Changes to make tests pass on Travis. (#3)
* Remove hiredis submodule.

* Squashed 'src/common/thirdparty/hiredis/' content from commit acd1966

git-subtree-dir: src/common/thirdparty/hiredis
git-subtree-split: acd1966bf7f5e1be74b426272635c672def78779

* Make Plasma tests pass.

* Make Photon tests pass.

* Compile and test with Travis.

* Deactive fetch test so that the tests pass.
2016-10-25 22:39:21 -07:00
Robert Nishihara
8c48410e82 Merge remote-tracking branch 'r1remote/moveout' into switch 2016-10-25 14:29:00 -07:00
Robert Nishihara
02d4050499 Rearrange plasma files to prepare for merging into Ray. 2016-10-25 14:27:45 -07:00
Robert Nishihara
6a5ddae59f Merge remote-tracking branch 'r1remote/moveout' into switch 2016-10-25 14:19:24 -07:00
Robert Nishihara
ad55166472 Rearrange local scheduler files to prepare to merge into Ray. 2016-10-25 14:16:23 -07:00
Robert Nishihara
1915539c5f Rearrange files to prepare to merge into Ray. 2016-10-25 13:59:47 -07:00
Robert Nishihara
d19f7ad853 Implement plasma_release. (#42)
* Implement plasma_release, enabling clients to indicate when they are no longer using an object.

* Call plasma_release for each plasma_create call, and call plasma_release from the manager.

* Fixes.

* Create client specific contexts for plasma_store callbacks.

* More changes.

* Fixes.

* Cleanup lists of clients using objects when a client disconnects.

* Make names parallel.
2016-10-21 00:47:34 -07:00
Stephanie Wang
ddfbd70dad Plasma fetch (#36)
* Fetch objects from remote plasma instances. Includes:
  - Configurable number of retries, with a timeout per retry
  - Support for multiple-object fetch
  - Test cases (integration only)

Update ray common

Remove attempts to retry object table lookup during fetch

lint

Fix a couple valgrind errors

Fix valgrind errors

Address Robert and Philipp's comments.

* Add fix from ray common and some TODOs.

* Update ray common

* Update ray common again

* Remove unused file
2016-10-18 18:20:59 -07:00
Robert Nishihara
f189ca746b Send a one byte message in send_fd because otherwise it sometimes hangs on Linux. (#41) 2016-10-15 11:44:51 -07:00
Robert Nishihara
5ad8e145ae Allow clients to subscribe to notifications about sealed objects. (#40)
* Create libplasma_client.a in Makefile.

* Implement plasma subscribe.

* Fixes and tests.

* Buffer notifications in the Plasma store when the socket send buffer is full.

* Fix formatting.

* Turn off -Werror in Makefile.

* Fixes.

* Fix formatting.
2016-10-14 19:27:17 -07:00
Robert Nishihara
1adafee6d3 Fix signature of create_object. (#39) 2016-10-12 12:58:15 -07:00
Philipp Moritz
94ad12ff64 Get valgrind in place for plasma (#35)
* plasma store: refactor and get valgrind in place

* fix valgrind errors
2016-10-11 17:58:14 -07:00
Ujval Misra
5a0725ce94 Increase dlmalloc threshold along with granularity (#33)
* Increase allocation granularity dynamically with each MMAP call

* Fewer MMAP calls required when workload contains several objects.

* Delay hitting the per-process file descriptor constraint.

* Change type of GRANULARITY_MULTIPLIER

* Make granularity update more concise.

* Increase dlmalloc threshold along with granularity

* Eventually resolve issue of objects being allocated their own file if larger than dlmalloc threshold

* Avoid dlmalloc threshold and granularity integer overflow

* Update granularity directly without invoking dlmallopt

* Set the threshold to a fixed size (MAX_SIZE_T)

* Removed trailing whitespace
2016-10-03 18:35:13 -07:00
Philipp Moritz
eabfa9ab6f Stephanie's plasma refactor (#31)
* Add Ray common as a submodule

* Convert to Ray common event loop

* Hide plasma manager state

* Interface changes

* Minor fixes: change LOG_INFO calls to LOG_DEBUG, comments, lint

* Turn off DEBUG by default and make Travis happy

* Allow processes time to clean up during Python tests

* Debugging travis...

* Plasma managers have long-lived connections per manager, not per object

* fix valgrind invalid read and cleanup

* make valgrind happy

* update store API

* put in place manager API

* fixed race condition while sending commands to plasma manager and store -- path sent by Phillip

* clang-format

* Revert "fixed race condition while sending commands to plasma manager and store -- path sent by Phillip"

This reverts commit 79e0f6e6d84f2a309b53155955b65c26c75af071.

* Use reliable socket read/writes from Ray common

* Merge data_connection and plasma_manager_connection structs

* small updates

* restore tests
2016-10-03 18:29:18 -07:00
Ujval Misra
eb71c2e84a Increase allocation granularity dynamically with each MMAP call (#32)
* Increase allocation granularity dynamically with each MMAP call

* Fewer MMAP calls required when workload contains several objects.

* Delay hitting the per-process file descriptor constraint.

* Change type of GRANULARITY_MULTIPLIER

* Make granularity update more concise.
2016-10-01 16:03:33 -07:00
Richard Shin
9c223a1e48 Increase dlmalloc's granularity to 1 << 25 (#30)
* Increase dlmalloc's granularity to 1 << 10

* Prevent trimming in dlmalloc
2016-09-29 23:46:04 -07:00
Robert Nishihara
227eab3b5a Use doxygen for generating documentation (#29)
* Auto-generated doxygen config file.

* Customize doxygen config file for Plasma.

* Format plasma_client.h and plasma.h for doxygen.
2016-09-28 18:59:00 -07:00
Robert Nishihara
d41566a499 Implement delete and contains for plasma client and store. (#28)
* Implement delete and contains for plasma client and store.

* Fix style, free object, add tests.
2016-09-23 15:07:50 -07:00
Philipp Moritz
0198a0d299 make plasma robust wrt to number of open file descriptors (#25) 2016-09-15 15:39:33 -07:00
Robert Nishihara
13560bdb6b Fix offset in get_malloc_mapinfo. (#24)
* Fix offset in get_malloc_mapinfo.

* Don't add offset inside mmap_record

* make clang-format happy
2016-09-14 17:45:18 -07:00
Robert Nishihara
72361c9b44 Add metadata handling (#23)
* Automatic whitespace fixes.

* Add metadata handling.

* Make create take a buffer instead of a string for the metadata.

* Small fixes.
2016-09-14 14:20:34 -07:00
Robert Nishihara
96a59200d3 Fix some bugs (#22)
* Bug fixes.

* Remove plasma_reply_type.

* Fix formatting.

* Speed up tests a little.

* Small fixes.
2016-09-13 16:45:44 -07:00
Philipp Moritz
494903c5d4 fix compiler warnings (#17) 2016-09-10 17:47:37 -07:00
Richard Shin
d52bf7d146 Use dlmalloc to manage shared memory (#15)
* Use dlmalloc to manage shared memory

* add stresstest
2016-09-10 16:39:24 -07:00
Richard Shin
04737f3f56 Run clang-format and check in Travis CI (#14)
* Run clang-format and add pre-commit hook for it.

* Modify .travis.yml to check

* Try to fix problems with .travis.yml

* Try to fix .travis.yml yet again

* Update .clang-format to Philipp's preferences

* Don't allow lint to fail in Travis

* Remove git-hooks directory

* Improve clang-format failure output

* Fix clang-format error

* Report which commit clang-format is comparing against, and add whitespace error

* Handle non-PR Travis in clang-format, and add another error

* Check $TRAVIS_PULL_REQUEST correctly and add another error

* Fix syntax error in check-git-clang-format-output.sh

* Add whitespace error

* Remove extra whitespace, add clang-format to README
2016-09-08 15:28:27 -07:00
Philipp Moritz
a62c0f8fac Allow arbitrary number of connections (#13)
* refactor plasma to use an event loop

* unify comment style

* Clean up Makefile flags.

* Randomize socket names in tests so multiple copies of the tests can be run in parallel without conflict.
2016-09-07 20:19:37 -07:00
Philipp Moritz
ad1a8454d5 remove C struct duplication and python plasma manager 2016-09-05 15:37:17 -07:00
Philipp Moritz
e1627319b3 clean up tests 2016-08-22 15:35:47 -07:00
Robert Nishihara
3d6d1e8fe5 Test plasma manager. 2016-08-18 10:04:00 -07:00
Philipp Moritz
23327a18e0 add plasma manager 2016-08-17 13:01:56 -07:00
Philipp Moritz
36af79427d Merge pull request #9 from pcmoritz/test
Basic Python unit tests.
2016-08-16 17:53:51 -07:00
Robert Nishihara
b6b17f3ac3 Retry if plasma client fails to connect to plasma store. 2016-08-16 17:52:18 -07:00
Philipp Moritz
6db8d1c15a fix error handling 2016-08-16 15:58:14 -07:00
Philipp Moritz
7a18347b5c cleanup 2016-08-16 15:00:01 -07:00
Philipp Moritz
97087b0791 initial version of object store 2016-08-14 01:39:44 -07:00