Commit graph

560 commits

Author SHA1 Message Date
Robert Nishihara
90a6a99b03 Clean up task interface and add testing. (#34)
* Update c extensions and python tests.

* Updates.

* Documentation fix.

* Move c extensions into common_extension.c
2016-10-06 16:25:04 -07:00
mehrdadn
c6ec36865d Update Windows patches for GRPC (#444) 2016-10-06 11:50:40 -07:00
Robert Nishihara
a048ad954a Merge pull request #7 from ray-project/valgrind
add valgrind check
2016-10-05 23:23:52 -07:00
Philipp Moritz
e8e4aa6d8e add valgrind check 2016-10-05 22:58:50 -07:00
Philipp Moritz
75441a180d add valgrind tests (#33)
* add valgrind

* install valgrind
2016-10-05 16:09:40 -07:00
Robert Nishihara
a3e01c8a45 Merge pull request #6 from ray-project/submit-redis
Submit task to redis
2016-10-05 15:17:06 -07:00
Robert Nishihara
0f97855333 More fixes. 2016-10-05 14:11:02 -07:00
Philipp Moritz
a7a963445d fixes 2016-10-05 13:30:10 -07:00
Robert Nishihara
4204500d23 Fix formatting. (#32) 2016-10-05 09:17:08 -07:00
Philipp Moritz
67677c3c92 update documentation and common 2016-10-04 17:13:21 -07:00
Philipp Moritz
4329afbd53 rename TASK_* -> TASK_STATUS_* (#31) 2016-10-04 16:59:44 -07:00
Philipp Moritz
872e68b5b0 submit task to redis 2016-10-04 16:57:16 -07:00
Philipp Moritz
8e044535e2 Python API for constructing tasks (#28)
* Python API for constructing tasks

* Fixes.
2016-10-04 15:56:24 -07:00
Robert Nishihara
64c5e0880e Let workers get tasks from local scheduler. (#5)
* Restructure to have separate client and scheduler files. Shared stuff is in photon.h.

* Let workers get tasks from local scheduler.
2016-10-04 12:55:10 -07:00
Philipp Moritz
da5ec3b5e0 add documentation for tasks (#30) 2016-10-04 12:11:52 -07:00
Ujval Misra
5a0725ce94 Increase dlmalloc threshold along with granularity (#33)
* Increase allocation granularity dynamically with each MMAP call

* Fewer MMAP calls required when workload contains several objects.

* Delay hitting the per-process file descriptor constraint.

* Change type of GRANULARITY_MULTIPLIER

* Make granularity update more concise.

* Increase dlmalloc threshold along with granularity

* Eventually resolve issue of objects being allocated their own file if larger than dlmalloc threshold

* Avoid dlmalloc threshold and granularity integer overflow

* Update granularity directly without invoking dlmallopt

* Set the threshold to a fixed size (MAX_SIZE_T)

* Removed trailing whitespace
2016-10-03 18:35:13 -07:00
Philipp Moritz
eabfa9ab6f Stephanie's plasma refactor (#31)
* Add Ray common as a submodule

* Convert to Ray common event loop

* Hide plasma manager state

* Interface changes

* Minor fixes: change LOG_INFO calls to LOG_DEBUG, comments, lint

* Turn off DEBUG by default and make Travis happy

* Allow processes time to clean up during Python tests

* Debugging travis...

* Plasma managers have long-lived connections per manager, not per object

* fix valgrind invalid read and cleanup

* make valgrind happy

* update store API

* put in place manager API

* fixed race condition while sending commands to plasma manager and store -- path sent by Phillip

* clang-format

* Revert "fixed race condition while sending commands to plasma manager and store -- path sent by Phillip"

This reverts commit 79e0f6e6d84f2a309b53155955b65c26c75af071.

* Use reliable socket read/writes from Ray common

* Merge data_connection and plasma_manager_connection structs

* small updates

* restore tests
2016-10-03 18:29:18 -07:00
Stephanie Wang
f4037ad19f Plasma fixes (#29)
* Allow CHECK to take in a message

* Turn off debug statements by default

* format arguments to CHECK message

* UT list

* Socket methods that block until full message is read/written

* Address Robert and Philipp's comments

* Formatting
2016-10-03 17:55:57 -07:00
Ujval Misra
eb71c2e84a Increase allocation granularity dynamically with each MMAP call (#32)
* Increase allocation granularity dynamically with each MMAP call

* Fewer MMAP calls required when workload contains several objects.

* Delay hitting the per-process file descriptor constraint.

* Change type of GRANULARITY_MULTIPLIER

* Make granularity update more concise.
2016-10-01 16:03:33 -07:00
Richard Shin
9c223a1e48 Increase dlmalloc's granularity to 1 << 25 (#30)
* Increase dlmalloc's granularity to 1 << 10

* Prevent trimming in dlmalloc
2016-09-29 23:46:04 -07:00
Philipp Moritz
e21e9f68df API for task log and scheduled task (#25)
* API revision

* update

* make status a bitmap

* update api

* tests working

* new task log APIs

* update APIs

* write binary structures to redis

* update tests

* fix clang-format

* Fix formatting.
2016-09-29 21:12:06 -07:00
Robert Nishihara
227eab3b5a Use doxygen for generating documentation (#29)
* Auto-generated doxygen config file.

* Customize doxygen config file for Plasma.

* Format plasma_client.h and plasma.h for doxygen.
2016-09-28 18:59:00 -07:00
Philipp Moritz
ff8018db75 First pass at local scheduler (#2)
* draft of local scheduler

* API

* update APIs

* fix

* update

* Rename halo -> photon.

* Add build directory.

* Update common submodule.

* More renaming.

* Fix python ctypes.

* Compile in travis.

* Process generic messages and not just tasks.

* Move free outside of switch.

* Formatting and address comments.

* Remove event loop from local scheduler state.

* Use accept_client from common.

* Use bind_ipc_sock from common.

* Fix tests.

* Update common submodule.

* Fix formatting.
2016-09-27 19:11:09 -07:00
Robert Nishihara
084220b0e7 Allow reading/writing generic message types, not just tasks. (#24)
* Allow reading/writing generic message types, not just tasks.

* Allow messages of length 0 to be read/written, and handle closed sockets.

* Address comments.

* Simplify accept_client.

* Allow ports to be reused in bind_ipc_sock.
2016-09-27 18:51:35 -07:00
Philipp Moritz
21d6ceba01 Merge pull request #4 from ray-project/trav
Add basic travis file and lint check.
2016-09-27 10:56:35 -07:00
Robert Nishihara
2776d6a37c Fix permissions. 2016-09-27 00:19:21 -07:00
Robert Nishihara
57214b3a11 Fixes. 2016-09-27 00:15:54 -07:00
Robert Nishihara
184242a85c Add basic travis file and lint check. 2016-09-27 00:13:28 -07:00
Richard Liaw
8925580fb6 nit for filter (#443) 2016-09-26 20:58:19 -07:00
Richard Liaw
d22321da7a Changes to run TRPO (#442)
Filter/new updates

final keras changes

some changes
2016-09-26 20:20:45 -07:00
Robert Nishihara
631de92170 Build redis before libcommon. (#20) 2016-09-26 00:12:11 -07:00
Robert Nishihara
9a6991116f Small fix in test. (#441) 2016-09-25 23:08:27 -07:00
Robert Nishihara
9f26c2d793 Merge pull request #3 from ray-project/rename
Update README.md
2016-09-25 22:51:00 -07:00
Philipp Moritz
675ea40171 Update README.md 2016-09-25 22:49:55 -07:00
Philipp Moritz
db8c0acc71 Refactor state database (#22)
* make db_connect return the connection

* rename db_conn -> db_handle

* more renaming

* clang-format

* free the db_handle
2016-09-25 21:52:06 -07:00
Philipp Moritz
1e08629013 fix submit_task for task queue (#21) 2016-09-25 16:51:24 -07:00
Philipp Moritz
7907992609 [WIP] Event loop refactoring (#19)
* task queue tests and extensions

* event loop refactor

* fix formating
2016-09-23 22:53:58 -07:00
Philipp Moritz
e1b8711a01 Redis logging (#17)
* Redis logging

* Rearrange logging interfaces

* Fix test case

* Changes to logging interface and test case for logging

* Fixes

* Fix memory leaks

* Add interface method to destroy logger

* is_local -> is_direct

* Merge fix
2016-09-23 17:10:15 -07:00
Robert Nishihara
d41566a499 Implement delete and contains for plasma client and store. (#28)
* Implement delete and contains for plasma client and store.

* Fix style, free object, add tests.
2016-09-23 15:07:50 -07:00
Philipp Moritz
7a079547b0 task queue tests and extensions (#18)
* task queue tests and extensions

* clean up test
2016-09-22 23:15:45 -07:00
Stephanie Wang
313241e303 Asynchronous Redis IPC (#14)
* Asynchronous Redis IPC

* make valgrind happy

* cleanup
2016-09-20 22:40:35 -07:00
Philipp Moritz
6c6f2d0473 Implement object table API (#16) 2016-09-20 17:02:56 -07:00
Robert Nishihara
de6ec47f9e Add a recursion depth for serialization to prevent infinite loops. (#440) 2016-09-19 17:17:42 -07:00
Philipp Moritz
d11161bb01 make static libraries (#13) 2016-09-18 20:47:53 -07:00
Philipp Moritz
37f035dbd0 implementing reading and writing tasks (#11) 2016-09-18 18:06:42 -07:00
Philipp Moritz
c238ae4aa0 do not re-download and rebuild redis if it already exists (#10) 2016-09-18 13:57:27 -07:00
Philipp Moritz
b18f214d55 Make it possible to read and write data that is not null-terminated (#9)
* Make it possible to read and write data that is not null-terminated

* formating
2016-09-18 13:35:43 -07:00
Stephanie Wang
ff11ee21ef Convert to streaming sockets (#7)
* Convert to streaming sockets

* Formatting
2016-09-17 15:15:18 -07:00
Robert Nishihara
91f16a3df0 Migrate repositories to ray-project. (#438)
* Migrate repositories to ray-project.

* Update numbuf to the migrated version.
2016-09-17 00:52:05 -07:00
Robert Nishihara
4fc42c3e1e Fix in documentation link. (#439) 2016-09-17 00:45:22 -07:00