Commit graph

652 commits

Author SHA1 Message Date
Robert Nishihara
02e8dd245d Update gitignore. (#94) 2016-12-07 11:54:16 -08:00
Robert Nishihara
b3c05655a0 Enable fetching objects from remote object stores. (#87)
* Fetch missing dependencies from local scheduler.

* Factor out global scheduler policy state.

* Use object_table_subscribe instead of object_table_lookup.

* Fix bug in which timer was being created twice for a single fetch request.

* Free old manager vector.
2016-12-06 15:47:31 -08:00
Philipp Moritz
03324caffc Put object store memory on /dev/shm on linux (#89)
* put object store memory on /dev/shm on linux

* fix

* fix mac
2016-12-06 00:31:47 -08:00
Robert Nishihara
dd39008532 Fix bug in which we were running out of file descriptors. (#91) 2016-12-05 23:45:49 -08:00
Philipp Moritz
ba53e4a43a Change object table subscribe to also return payload (#88)
* implement object table subscribe that also returns payload

* fix

* fix valgrind

* fix ray test

* fix clang-format

* fix

* fix
2016-12-05 00:26:53 -08:00
Robert Nishihara
b5ed2f063d Allow starting multiple local schedulers. (#86) 2016-12-04 17:08:16 -08:00
Robert Nishihara
35b9dedb48 Remove scheduler_info. (#84) 2016-12-04 15:51:03 -08:00
Robert Nishihara
2a3e9267f8 Non-blocking fetch implementation. (#83)
* Non-blocking fetch implementation.

* Make fetch tests more robust to timing issues.

* Bug fix when ignoring transferred objects.

* Fix.

* Documentation fixes.
2016-12-03 19:09:05 -08:00
Wapaul1
9a513363f9 Init_table_callback now takes ownership of passed in data (#80)
* temp commit

* Stuff

* Ownership is now taken by init table callback

* Fixed lint errors

* Fixed travis warnings

* Fixed spacing

* add .gitkeep

* fix global scheduler

* Whitespace.
2016-12-03 13:49:09 -08:00
Robert Nishihara
61755bc168 Replace direct calls to recv with read_bytes. (#78)
* Replace direct calls to recv with read_bytes.

* Fixes.

* Formatting.
2016-12-02 18:51:50 -08:00
Robert Nishihara
de2dd1f506 Allow fatal errors to produce core dumps. (#79) 2016-12-02 15:10:37 -08:00
Stephanie Wang
dadbcae17d Cache plasma object information on the plasma client (#73)
* Cache plasma object information on the plasma client

* Plasma client uses cached object references to get the object data
2016-12-01 16:46:17 -08:00
Ion
f89be9699c Introduce non-blocking Plasma API. (#71)
* Implement new plasma client API.

* Formatting fixes.

* Make tests work again.

* Make tests run.

* Comment style.

* Fix bugs with fetch tests.

* Introduce fetch1 flag.

* Remove timer only if present.

* Formatting fixes.

* Don't access object after free.

* Formatting fixes.

* Minor change.

* refactoring plasma datastructures

* Change plasma_request and plasma_reply to use only arrays of object requests.

* some more fixes

* Remove unnecessary methods.

* Trivial.

* fixes

* use plasma_send_reply in return_from_wait1

* Lint.
2016-12-01 02:15:21 -08:00
Philipp Moritz
58e8bbcb34 Fix bug in serializing arguments of tasks that are more complex objects (#72)
* Give more informative error message when we do not know how to serialize a class.

* Check that passing arguments to remote functions and getting them does not change their values.

* fix serialization bug

* fix tests for common module

* Formatting.

* Bug fix in init_pickle_module signature.

* Use pickle with HIGHEST_PROTOCOL.
2016-11-30 23:21:53 -08:00
atumanov
1499834be1 handling partial/interrupted I/O (#69)
* read_bytes: handle EINTR + easier flow

* addressing whitespace + style comments

* handling partial i/o: write_bytes; use [read|write]_bytes in plasma send/receive
2016-11-29 23:04:13 -08:00
Philipp Moritz
a2692ead6e add errno logging for fatal checks (#68) 2016-11-29 22:40:01 -08:00
mehrdadn
4d327c571a Make Redis script executable (#66) 2016-11-29 14:33:20 -08:00
Robert Nishihara
2c639c05ef When searching for Python with cmake, try custom tricks before using default cmake behavior. (#67)
* Change CMakeLists.txt.

* remove dead code
2016-11-29 14:32:54 -08:00
Philipp Moritz
c7073d623b Object table subscribe with new semantics (#62)
* new plasma subscribe implementation

* object table subscribe with test

* clang-format

* fix

* fix test

* fix tests

* fix clang-format

* add check

* final clang-format

* final fixes

* fix clang-format
2016-11-27 21:26:23 -08:00
Robert Nishihara
bc1d7db926 Re-enable fetch tests. (#63)
* Bring back fetch tests.

* Zero initialize success array in PyPlasma_fetch.

* Fix bug in fetch in case where the object ID doesn't have any managers in the object table.

* Temporarily disallow calling fetch with multiple copies of the same object ID.

* Fix.

* Factor out code for checking if list of object IDs are all distinct.

* Remove commented out code.

* Fix.
2016-11-24 16:27:17 -08:00
Wapaul1
800bf8deb3 Redis Server started in Tests now shutdowns at end of tests (#61)
Redis Server started in tests now shutdowns at end of tests.
2016-11-23 21:47:53 -08:00
Philipp Moritz
8147b62964 move methods related to plasma requests and replies to plasma.c (#60) 2016-11-23 21:13:19 -08:00
Robert Nishihara
1f9fce8e53 Remove semicolon in macro. (#59) 2016-11-23 00:54:07 -08:00
mehrdadn
7237ec4124 Windows compatibility (#57)
* Add Python and Redis submodules, and remove old third-party modules

* Update VS projects (WARNING: references files that do not exist yet)

* Update code & add shims for APIs except AF_UNIX/{send,recv}msg()

* Minor style changes.
2016-11-22 17:04:24 -08:00
Robert Nishihara
a93c6b7596 Quick workaround for ray.wait bug. (#42) 2016-11-22 13:31:22 -08:00
mehrdadn
35ce5f8001 Enable more warnings and fix usleep() issue (#55)
* Enable more warnings and fix usleep() issue

* Periods after comments.
2016-11-22 00:29:27 -08:00
mehrdadn
3714984094 Remove Redis version from Linux scripts (#56)
* Remove Redis version from Linux scripts

* Add documentation.
2016-11-21 15:02:40 -08:00
Robert Nishihara
4b00c029ac Remove numbuf from requirements for setup.py. (#54)
* Remove numbuf from requirements for setup.py.

* Update documentation.
2016-11-21 14:30:17 -08:00
Philipp Moritz
da6bcfcc70 Statically link arrow libraries when building numbuf (#53)
* Fix numbuf compilation on Linux

* Statically link arrow libraries when building numbuf.

* Last fixes.
2016-11-21 02:14:13 -08:00
Robert Nishihara
9ed56c23db Fix pip install hanging by moving C tests out of build.sh. (#52)
* Remove C tests from build.sh.

* Run C tests in Travis.
2016-11-20 21:02:54 -08:00
Robert Nishihara
f267da6aa8 Improve case on Mac where cmake fails to find Python with the custom search path. (#49)
* On Mac OS X, if cmake fails to find Python with custom search path, default to using find_package.

* Fix formatting in CMakeLists.txt.

* Set CUSTOM_PYTHON_EXECUTABLE variable on mac if the custom search path fails.

* Require exact Python version match.

* Use cmake mode keyword for logging messages.

* Fix.

* Find python libraries in cmake by searching near the include directories.

* Update cmakelists for numbuf.
2016-11-19 19:00:42 -08:00
Robert Nishihara
c9757a6fd0 Merge pull request #48 from ray-project/numbuf-integration
Numbuf integration
2016-11-19 18:10:53 -08:00
Philipp Moritz
d64f69f215 switch from submodule to cloning arrow, travis fixes & Robert's comments 2016-11-19 17:38:36 -08:00
Philipp Moritz
05e0226047 clang-fixes 2016-11-19 14:27:08 -08:00
Philipp Moritz
9d6da1281e add clang-format for numbuf (arrow style) 2016-11-19 12:56:11 -08:00
Philipp Moritz
0b888c38f1 Merge numbuf. 2016-11-19 12:30:01 -08:00
Robert Nishihara
c8c3983195 Use sizeof(field) instead of sizeof(type) and other fixes. (#47)
* Use sizeof(field) instead of sizeof(type) and other fixes.

* Fix formatting.

* Bug fix.

* Zero-initialize structs. There are many more instances of these that I haven't changed yet.

* Bug fix.

* Revert from atexit to signaling to fix valgrind tests.

* Address Philipp's comments.
2016-11-19 12:19:49 -08:00
Philipp Moritz
c3ab68e88c Rebase numbuf to latest arrow (#23)
* rebase to latest arrow

* fix arrow linking

* finish rebase

* point arrow to pcmoritz's arrow

* fix

* fix macOS

* fix
2016-11-19 02:27:21 -08:00
Robert Nishihara
d77b685a90 Global scheduler skeleton (#45)
* Initial scheduler commit

* global scheduler

* add global scheduler

* Implement global scheduler skeleton.

* Formatting.

* Allow local scheduler to be started without a connection to redis so that we can test it without a global scheduler.

* Fail if there are no local schedulers when the global scheduler receives a task.

* Initialize uninitialized value and formatting fix.

* Generalize local scheduler table to db client table.

* Remove code duplication in local scheduler and add flag for whether a task came from the global scheduler or not.

* Queue task specs in the local scheduler instead of tasks.

* Simple global scheduler tests, including valgrind.

* Factor out functions for starting processes.

* Fixes.
2016-11-18 19:57:51 -08:00
Wapaul1
08707f9408 Integration of Webui with Ray (#32)
* Initial integration of webui with ray

* Re-organized calling of build-webui in setup.py

* Fixed Lint comments on js code

* Fixed more lint issues

* Fixed various issues

* Fixed directory in services.py

* Small changes.

* Changes to match lint
2016-11-17 22:33:29 -08:00
Robert Nishihara
c3db225cbe Patch arrow to work with xcode 8. This will be unecessary when we rebase on an updated arrow. (#22) 2016-11-16 19:43:26 -08:00
Stephanie Wang
7babe0d22f Logging level (#38)
* Set logging levels in Makefile using -DRAY_COMMON_LOG_LEVEL=level

* Lower level of some LOG_ERROR messages, log the name of the table operation on failure

* Address rest of Robert's comments

* Fix spurious log message
2016-11-15 20:33:29 -08:00
Philipp Moritz
986ed5c9e8 Plasma C extensions (#34)
* switch plasma from ctypes to python C API

* clang-format

* various fixes
2016-11-13 16:23:28 -08:00
Robert Nishihara
ad6a401740 Get rid of old git submodule init calls. (#41) 2016-11-13 14:55:28 -08:00
Robert Nishihara
43aba4fc6d Install python package dependencies through setup.py. (#39)
* Install python package dependencies through setup.py.

* Do pip installs without sudo.

* Clearer instructions for running tests.

* Wording.
2016-11-12 19:37:20 -08:00
Robert Nishihara
488f881d70 Install numpy before building numbuf in setup.py. (#20) 2016-11-12 19:36:47 -08:00
Philipp Moritz
d26fbcebbe switch to release mode (#19) 2016-11-12 19:35:46 -08:00
Robert Nishihara
28a80ca058 Remove out-of-date documentation. (#40) 2016-11-12 19:34:22 -08:00
Robert Nishihara
5c89a7ab4e Fix bug in which gym environment could not be imported because of module naming conflicts. (#37) 2016-11-11 19:02:40 -08:00
Philipp Moritz
b1083e92e9 fix bug in wait (#35) 2016-11-11 17:05:40 -08:00