Commit graph

772 commits

Author SHA1 Message Date
Philipp Moritz
eabfa9ab6f Stephanie's plasma refactor (#31)
* Add Ray common as a submodule

* Convert to Ray common event loop

* Hide plasma manager state

* Interface changes

* Minor fixes: change LOG_INFO calls to LOG_DEBUG, comments, lint

* Turn off DEBUG by default and make Travis happy

* Allow processes time to clean up during Python tests

* Debugging travis...

* Plasma managers have long-lived connections per manager, not per object

* fix valgrind invalid read and cleanup

* make valgrind happy

* update store API

* put in place manager API

* fixed race condition while sending commands to plasma manager and store -- path sent by Phillip

* clang-format

* Revert "fixed race condition while sending commands to plasma manager and store -- path sent by Phillip"

This reverts commit 79e0f6e6d84f2a309b53155955b65c26c75af071.

* Use reliable socket read/writes from Ray common

* Merge data_connection and plasma_manager_connection structs

* small updates

* restore tests
2016-10-03 18:29:18 -07:00
Stephanie Wang
f4037ad19f Plasma fixes (#29)
* Allow CHECK to take in a message

* Turn off debug statements by default

* format arguments to CHECK message

* UT list

* Socket methods that block until full message is read/written

* Address Robert and Philipp's comments

* Formatting
2016-10-03 17:55:57 -07:00
Ujval Misra
eb71c2e84a Increase allocation granularity dynamically with each MMAP call (#32)
* Increase allocation granularity dynamically with each MMAP call

* Fewer MMAP calls required when workload contains several objects.

* Delay hitting the per-process file descriptor constraint.

* Change type of GRANULARITY_MULTIPLIER

* Make granularity update more concise.
2016-10-01 16:03:33 -07:00
Richard Shin
9c223a1e48 Increase dlmalloc's granularity to 1 << 25 (#30)
* Increase dlmalloc's granularity to 1 << 10

* Prevent trimming in dlmalloc
2016-09-29 23:46:04 -07:00
Philipp Moritz
e21e9f68df API for task log and scheduled task (#25)
* API revision

* update

* make status a bitmap

* update api

* tests working

* new task log APIs

* update APIs

* write binary structures to redis

* update tests

* fix clang-format

* Fix formatting.
2016-09-29 21:12:06 -07:00
Robert Nishihara
227eab3b5a Use doxygen for generating documentation (#29)
* Auto-generated doxygen config file.

* Customize doxygen config file for Plasma.

* Format plasma_client.h and plasma.h for doxygen.
2016-09-28 18:59:00 -07:00
Philipp Moritz
ff8018db75 First pass at local scheduler (#2)
* draft of local scheduler

* API

* update APIs

* fix

* update

* Rename halo -> photon.

* Add build directory.

* Update common submodule.

* More renaming.

* Fix python ctypes.

* Compile in travis.

* Process generic messages and not just tasks.

* Move free outside of switch.

* Formatting and address comments.

* Remove event loop from local scheduler state.

* Use accept_client from common.

* Use bind_ipc_sock from common.

* Fix tests.

* Update common submodule.

* Fix formatting.
2016-09-27 19:11:09 -07:00
Robert Nishihara
084220b0e7 Allow reading/writing generic message types, not just tasks. (#24)
* Allow reading/writing generic message types, not just tasks.

* Allow messages of length 0 to be read/written, and handle closed sockets.

* Address comments.

* Simplify accept_client.

* Allow ports to be reused in bind_ipc_sock.
2016-09-27 18:51:35 -07:00
Philipp Moritz
21d6ceba01 Merge pull request #4 from ray-project/trav
Add basic travis file and lint check.
2016-09-27 10:56:35 -07:00
Robert Nishihara
2776d6a37c Fix permissions. 2016-09-27 00:19:21 -07:00
Robert Nishihara
57214b3a11 Fixes. 2016-09-27 00:15:54 -07:00
Robert Nishihara
184242a85c Add basic travis file and lint check. 2016-09-27 00:13:28 -07:00
Richard Liaw
8925580fb6 nit for filter (#443) 2016-09-26 20:58:19 -07:00
Richard Liaw
d22321da7a Changes to run TRPO (#442)
Filter/new updates

final keras changes

some changes
2016-09-26 20:20:45 -07:00
Robert Nishihara
631de92170 Build redis before libcommon. (#20) 2016-09-26 00:12:11 -07:00
Robert Nishihara
9a6991116f Small fix in test. (#441) 2016-09-25 23:08:27 -07:00
Robert Nishihara
9f26c2d793 Merge pull request #3 from ray-project/rename
Update README.md
2016-09-25 22:51:00 -07:00
Philipp Moritz
675ea40171 Update README.md 2016-09-25 22:49:55 -07:00
Philipp Moritz
db8c0acc71 Refactor state database (#22)
* make db_connect return the connection

* rename db_conn -> db_handle

* more renaming

* clang-format

* free the db_handle
2016-09-25 21:52:06 -07:00
Philipp Moritz
1e08629013 fix submit_task for task queue (#21) 2016-09-25 16:51:24 -07:00
Philipp Moritz
7907992609 [WIP] Event loop refactoring (#19)
* task queue tests and extensions

* event loop refactor

* fix formating
2016-09-23 22:53:58 -07:00
Philipp Moritz
e1b8711a01 Redis logging (#17)
* Redis logging

* Rearrange logging interfaces

* Fix test case

* Changes to logging interface and test case for logging

* Fixes

* Fix memory leaks

* Add interface method to destroy logger

* is_local -> is_direct

* Merge fix
2016-09-23 17:10:15 -07:00
Robert Nishihara
d41566a499 Implement delete and contains for plasma client and store. (#28)
* Implement delete and contains for plasma client and store.

* Fix style, free object, add tests.
2016-09-23 15:07:50 -07:00
Philipp Moritz
7a079547b0 task queue tests and extensions (#18)
* task queue tests and extensions

* clean up test
2016-09-22 23:15:45 -07:00
Stephanie Wang
313241e303 Asynchronous Redis IPC (#14)
* Asynchronous Redis IPC

* make valgrind happy

* cleanup
2016-09-20 22:40:35 -07:00
Philipp Moritz
6c6f2d0473 Implement object table API (#16) 2016-09-20 17:02:56 -07:00
Robert Nishihara
de6ec47f9e Add a recursion depth for serialization to prevent infinite loops. (#440) 2016-09-19 17:17:42 -07:00
Robert Nishihara
7055c6f793 Limit recursion depth for serializing objects to avoid infinite loops. (#17) 2016-09-19 14:54:56 -07:00
Philipp Moritz
d11161bb01 make static libraries (#13) 2016-09-18 20:47:53 -07:00
Philipp Moritz
37f035dbd0 implementing reading and writing tasks (#11) 2016-09-18 18:06:42 -07:00
Philipp Moritz
c238ae4aa0 do not re-download and rebuild redis if it already exists (#10) 2016-09-18 13:57:27 -07:00
Philipp Moritz
b18f214d55 Make it possible to read and write data that is not null-terminated (#9)
* Make it possible to read and write data that is not null-terminated

* formating
2016-09-18 13:35:43 -07:00
Stephanie Wang
ff11ee21ef Convert to streaming sockets (#7)
* Convert to streaming sockets

* Formatting
2016-09-17 15:15:18 -07:00
Robert Nishihara
91f16a3df0 Migrate repositories to ray-project. (#438)
* Migrate repositories to ray-project.

* Update numbuf to the migrated version.
2016-09-17 00:52:05 -07:00
Robert Nishihara
4fc42c3e1e Fix in documentation link. (#439) 2016-09-17 00:45:22 -07:00
Philipp Moritz
77d5365cc0 Merge pull request #1 from ray-project/name
Update README.md
2016-09-17 00:36:52 -07:00
Robert Nishihara
9e4203c18c Update README.md 2016-09-17 00:30:31 -07:00
Robert Nishihara
0527736490 Migrate repository to ray-project. (#15) 2016-09-17 00:14:35 -07:00
Philipp Moritz
0b7d81cae6 API for creating task specifications (#5)
* API for creating task specifications

* fixes

* add more checks and improve comments
2016-09-17 00:03:10 -07:00
Robert Nishihara
52d953bde6 Document serialization in the object store. (#436) 2016-09-16 23:49:59 -07:00
Robert Nishihara
228525eec7 Add documentation for reusable variables. (#437) 2016-09-16 23:10:34 -07:00
Robert Nishihara
4863a5155c Cleanup setting and getting of tensorflow weights. (#385)
* Cleanup setting and getting of tensorflow weights.

* Add documentation for using TensorFlow.

* Group get_weights and set_weights in a function.

* Update readme.
2016-09-16 23:05:14 -07:00
Philipp Moritz
4521e5f183 Initial commit 2016-09-16 18:26:57 -07:00
Stephanie Wang
73f4b96253 Sockets (#3)
* Socket methods to be used by an event loop

* Git ignore build files

* File renames

* Some fixes

* Fixes

* Fixes

* Memory leakage fix
2016-09-15 16:28:52 -07:00
Robert Nishihara
1aa89a4ae6 Update numbuf to properly handle Python floats. (#435) 2016-09-15 15:44:11 -07:00
Philipp Moritz
0198a0d299 make plasma robust wrt to number of open file descriptors (#25) 2016-09-15 15:39:33 -07:00
Robert Nishihara
d469366c75 Serialize Python floats properly as doubles. (#14) 2016-09-15 15:11:51 -07:00
Robert Nishihara
292656013a Suppress exceptions in the error logging thread when program exits. (#432) 2016-09-15 13:48:23 -07:00
Robert Nishihara
a5c026c033 Remove unused serialize/deserialize code in raylib. (#434) 2016-09-15 13:46:38 -07:00
Robert Nishihara
13560bdb6b Fix offset in get_malloc_mapinfo. (#24)
* Fix offset in get_malloc_mapinfo.

* Don't add offset inside mmap_record

* make clang-format happy
2016-09-14 17:45:18 -07:00