Commit graph

544 commits

Author SHA1 Message Date
Philipp Moritz
b4b462809f fix valgrind 2016-10-27 15:09:50 -07:00
Philipp Moritz
e26fb46b02 subscribe without polling 2016-10-27 13:21:16 -07:00
Robert Nishihara
6ed641177d Remove unnecessary files. (#4) 2016-10-26 23:24:40 -07:00
Robert Nishihara
6f75c738b5 [WIP] Fix valgrind tests. (#5)
* Make tests fail when valgrind finds a memory leak.

* Properly clean up scheduler state.

* Remove unnecessary malloc.
2016-10-26 23:23:46 -07:00
Robert Nishihara
f83a98d71a Changes to make tests pass on Travis. (#3)
* Remove hiredis submodule.

* Squashed 'src/common/thirdparty/hiredis/' content from commit acd1966

git-subtree-dir: src/common/thirdparty/hiredis
git-subtree-split: acd1966bf7f5e1be74b426272635c672def78779

* Make Plasma tests pass.

* Make Photon tests pass.

* Compile and test with Travis.

* Deactive fetch test so that the tests pass.
2016-10-25 22:39:21 -07:00
Philipp Moritz
ed550ea8af Merge pull request #1 from ray-project/switch
Move plasma, common, and photon into the Ray repository.
2016-10-25 14:33:51 -07:00
Robert Nishihara
8c48410e82 Merge remote-tracking branch 'r1remote/moveout' into switch 2016-10-25 14:29:00 -07:00
Robert Nishihara
02d4050499 Rearrange plasma files to prepare for merging into Ray. 2016-10-25 14:27:45 -07:00
Robert Nishihara
6a5ddae59f Merge remote-tracking branch 'r1remote/moveout' into switch 2016-10-25 14:19:24 -07:00
Robert Nishihara
ad55166472 Rearrange local scheduler files to prepare to merge into Ray. 2016-10-25 14:16:23 -07:00
Robert Nishihara
5165b41fa7 Merge remote-tracking branch 'r1remote/moveout' into switch 2016-10-25 14:02:32 -07:00
Robert Nishihara
1915539c5f Rearrange files to prepare to merge into Ray. 2016-10-25 13:59:47 -07:00
Robert Nishihara
7c1b2f702f Remove git submodules and C++ files. 2016-10-25 12:51:30 -07:00
Robert Nishihara
db6375701e Prevent ObjectIDs from being pickled. (#42) 2016-10-21 15:42:29 -07:00
Robert Nishihara
d19f7ad853 Implement plasma_release. (#42)
* Implement plasma_release, enabling clients to indicate when they are no longer using an object.

* Call plasma_release for each plasma_create call, and call plasma_release from the manager.

* Fixes.

* Create client specific contexts for plasma_store callbacks.

* More changes.

* Fixes.

* Cleanup lists of clients using objects when a client disconnects.

* Make names parallel.
2016-10-21 00:47:34 -07:00
Robert Nishihara
63ec244784 Connect local scheduler to Plasma. (#11)
* Receive notifications about sealed objects from Plasma, and schedule tasks only when the dependencies are available locally.

* Fix formatting.

* Use version of Plasma with fix.

* Fix.

* Factor out the scheduling algorithm and use worker_index instead of the client socket to identify workers

* Fixes

* clang-format

* fix remaining linter errors
2016-10-18 18:27:43 -07:00
Stephanie Wang
ddfbd70dad Plasma fetch (#36)
* Fetch objects from remote plasma instances. Includes:
  - Configurable number of retries, with a timeout per retry
  - Support for multiple-object fetch
  - Test cases (integration only)

Update ray common

Remove attempts to retry object table lookup during fetch

lint

Fix a couple valgrind errors

Fix valgrind errors

Address Robert and Philipp's comments.

* Add fix from ray common and some TODOs.

* Update ray common

* Update ray common again

* Remove unused file
2016-10-18 18:20:59 -07:00
Stephanie Wang
da3a3127e0 Move get_client_id to db.h (#40) 2016-10-18 15:12:41 -07:00
Stephanie Wang
e57b87928c Fixes for implementing Plasma fetch (#39)
* Add ability to pass callback context to object table lookup

* Propagate errors during socket writes up to caller.

* Use recv and MSG_WAITALL flag instead of looping read

* Error checking in write_bytes

* Method to listen on a network port

* Revert "Use recv and MSG_WAITALL flag instead of looping read"

This reverts commit 32d9333bc6a185729aadb4b41b70b3d7f150a9c2.

* Some documentation

* Clearer documentation

* Fix bug where database clients were getting assigned the same ID

* Regression test for unique client IDs
2016-10-18 12:38:30 -07:00
Robert Nishihara
f189ca746b Send a one byte message in send_fd because otherwise it sometimes hangs on Linux. (#41) 2016-10-15 11:44:51 -07:00
Robert Nishihara
5ad8e145ae Allow clients to subscribe to notifications about sealed objects. (#40)
* Create libplasma_client.a in Makefile.

* Implement plasma subscribe.

* Fixes and tests.

* Buffer notifications in the Plasma store when the socket send buffer is full.

* Fix formatting.

* Turn off -Werror in Makefile.

* Fixes.

* Fix formatting.
2016-10-14 19:27:17 -07:00
Robert Nishihara
0a44145906 Fix the resetting of reusable variables on the driver and cache functions to run on all workers. (#446)
* Properly reset reusable variables on the driver when remote functions are run locally on the driver.

* Cache functions to run on all workers that occur before ray.init is called.
2016-10-12 22:17:22 -07:00
Stephanie Wang
7e23b87a21 Merge pull request #38 from ray-project/fixwarnings
Fix warnings
2016-10-12 15:43:01 -07:00
Robert Nishihara
1c3aaf7189 Update documentation (#445)
* Update documentation for serialization.

* Update documentation for reusable variables.

* Update documentation for using Ray with TensorFlow. This change is to allow code blocks to be copied and pasted into a Python interpreter.

* Fix documentation for hyperparameter optimization example.
2016-10-12 15:41:00 -07:00
Philipp Moritz
182985015c Make warnings errors 2016-10-12 13:20:57 -07:00
Robert Nishihara
1adafee6d3 Fix signature of create_object. (#39) 2016-10-12 12:58:15 -07:00
Philipp Moritz
832888d473 fix formating 2016-10-12 01:10:05 -07:00
Philipp Moritz
50fb53fd91 fix compiler warnings on macOS 2016-10-12 00:56:04 -07:00
Philipp Moritz
6290cab750 fix compiler warning for linux 2016-10-12 00:23:40 -07:00
Philipp Moritz
94ad12ff64 Get valgrind in place for plasma (#35)
* plasma store: refactor and get valgrind in place

* fix valgrind errors
2016-10-11 17:58:14 -07:00
Robert Nishihara
7f515113fa Generate return object IDs in the task constructor. (#36) 2016-10-07 15:20:56 -07:00
Robert Nishihara
e9a336a344 Fix incorrect free. (#9) 2016-10-07 12:31:57 -07:00
Robert Nishihara
18934c3a45 Make photon client into a C extension. (#8)
* Make photon client into a C extension.

* Fix formatting.

* Rename extension from PhotonClient to Photon.

* Update common submodule.

* Fix Makefile to compile with fPIC.

* Update common submodule.

* Compile C extension against common.

* Fix formatting.

* Remove unnecessary include.

* Update common submodule and rename Photon -> PhotonClient.

* Drop global interpretor lock during get_task.
2016-10-07 11:00:46 -07:00
Robert Nishihara
7be1a93d64 Move common C extension module into a different C file. (#35)
* Move common C extension module into a different C file so that the actual definitions can be more easily included in other C extensions.

* Rename common_extension_module -> common_module.
2016-10-06 19:16:09 -07:00
Robert Nishihara
90a6a99b03 Clean up task interface and add testing. (#34)
* Update c extensions and python tests.

* Updates.

* Documentation fix.

* Move c extensions into common_extension.c
2016-10-06 16:25:04 -07:00
mehrdadn
c6ec36865d Update Windows patches for GRPC (#444) 2016-10-06 11:50:40 -07:00
Robert Nishihara
a048ad954a Merge pull request #7 from ray-project/valgrind
add valgrind check
2016-10-05 23:23:52 -07:00
Philipp Moritz
e8e4aa6d8e add valgrind check 2016-10-05 22:58:50 -07:00
Philipp Moritz
75441a180d add valgrind tests (#33)
* add valgrind

* install valgrind
2016-10-05 16:09:40 -07:00
Robert Nishihara
a3e01c8a45 Merge pull request #6 from ray-project/submit-redis
Submit task to redis
2016-10-05 15:17:06 -07:00
Robert Nishihara
0f97855333 More fixes. 2016-10-05 14:11:02 -07:00
Philipp Moritz
a7a963445d fixes 2016-10-05 13:30:10 -07:00
Robert Nishihara
4204500d23 Fix formatting. (#32) 2016-10-05 09:17:08 -07:00
Philipp Moritz
67677c3c92 update documentation and common 2016-10-04 17:13:21 -07:00
Philipp Moritz
4329afbd53 rename TASK_* -> TASK_STATUS_* (#31) 2016-10-04 16:59:44 -07:00
Philipp Moritz
872e68b5b0 submit task to redis 2016-10-04 16:57:16 -07:00
Philipp Moritz
8e044535e2 Python API for constructing tasks (#28)
* Python API for constructing tasks

* Fixes.
2016-10-04 15:56:24 -07:00
Robert Nishihara
64c5e0880e Let workers get tasks from local scheduler. (#5)
* Restructure to have separate client and scheduler files. Shared stuff is in photon.h.

* Let workers get tasks from local scheduler.
2016-10-04 12:55:10 -07:00
Philipp Moritz
da5ec3b5e0 add documentation for tasks (#30) 2016-10-04 12:11:52 -07:00
Ujval Misra
5a0725ce94 Increase dlmalloc threshold along with granularity (#33)
* Increase allocation granularity dynamically with each MMAP call

* Fewer MMAP calls required when workload contains several objects.

* Delay hitting the per-process file descriptor constraint.

* Change type of GRANULARITY_MULTIPLIER

* Make granularity update more concise.

* Increase dlmalloc threshold along with granularity

* Eventually resolve issue of objects being allocated their own file if larger than dlmalloc threshold

* Avoid dlmalloc threshold and granularity integer overflow

* Update granularity directly without invoking dlmallopt

* Set the threshold to a fixed size (MAX_SIZE_T)

* Removed trailing whitespace
2016-10-03 18:35:13 -07:00