Commit graph

667 commits

Author SHA1 Message Date
Robert Nishihara
ddba1df802 Start working toward Python3 compatibility. (#117) 2016-12-11 12:25:31 -08:00
Robert Nishihara
3d083c8b58 Build arrow without tests. (#85) 2016-12-10 21:25:52 -08:00
Robert Nishihara
0f7091099d Throw an exception if a Ray method is called from a thread that isn't the main thread. (#97) 2016-12-10 21:24:50 -08:00
Robert Nishihara
9474d03912 Switch to updated Plasma API and consolidate wait and fetch implementations. (#116)
* Consolidate wait implementations.

* Consolidate fetch implementations.

* Share callback between wait and fetch to address issue in which only one callback can be run for a given subscribe channel.

* Reactivate manager tests.

* Remove more code.

* Add some documentation.
2016-12-10 21:22:05 -08:00
Robert Nishihara
86973059de Switch to new wait implementation. (#113)
* Duplicate wait1 implementation and seperate out wait datastructures.

* Address Philipp's comments.

* Temporarily address test failure problem by increasing timeout and reducing load in tests.

* Update stress tests to include distributed wait.
2016-12-09 19:26:11 -08:00
Robert Nishihara
6441571d31 Introduce some stress tests. (#106)
* Retry first connection to redis in db_connect.

* Declare usleep.

* Formatting.

* Introduce some stress tests.
2016-12-09 17:49:31 -08:00
Robert Nishihara
c740b165f4 Retry first connection to redis in db_connect. (#112)
* Retry first connection to redis in db_connect.

* Declare usleep.

* Formatting.
2016-12-09 17:21:49 -08:00
Robert Nishihara
46d0e6bdfb In tests, check that processes are still alive before killing them to catch crashes. (#110) 2016-12-09 13:04:08 -08:00
Robert Nishihara
5819e90962 Remove all object files when doing make clean. (#109) 2016-12-09 11:00:00 -08:00
Alexey Tumanov
0abbf5a113 End-to-end object size information passthrough (#105)
* rebase Alexey's PR on top

* rebase on master

* fix test failure waiting for plasma manager to exit

* clang format

* addressing comments

* Minor formatting and naming fixes.
2016-12-09 00:51:44 -08:00
Stephanie Wang
61904c4c3e Object hashes (#104)
* factoring out object_info for general use by several Ray components

* addressing comments

* Replace SHA256 task hash with MD5

Add object hash to object table (always overwrites)

Support for table operations that span multiple asynchronous Redis
commands

Add a new object location in a transaction, using Redis's optimistic
concurrency

Use Redis GETSET instead of transactions and Python frontend code for object hashing

Remove spurious log message

Fix for object_table_add

Revert "Replace SHA256 task hash with MD5"

This reverts commit e599de473c8dad9189ccb0600429534b469b76a2.

Revert to sha256

Test case for illegal puts

Use SETNX to set object hashes

Initialize digest with zeros

Initialize plasma_request with zeros

* Fixes

* replace SHA256 with a faster hash in the object store

* Fix valgrind

* Address Robert's comments

* Check that plasma_compute_object_hash succeeds.

* Don't run test_illegal_put test with valgrind because it causes an intentional crash which causes valgrind to complain.

* Debugging after rebase.

* handling Robert's comments

* Fix bugs after rebase.

* final fixes for Stephanie's PR

* fix
2016-12-08 20:57:08 -08:00
Robert Nishihara
4a62a3c5d7 Cause plasma client tests failures to show up on travis. (#99)
* Cause plasma client tests failures to show up on travis.

* Don't run flaky test.
2016-12-08 19:32:24 -08:00
atumanov
1c946b2f6a Factoring out object_info structure for use in several Ray components (#101)
* change plasma object notifications to carry a struct of information

* factoring out object_info for general use by several Ray components

* fixing a bug in python test

* addressing comments

* handling Robert's comments

* clang format

* Fix valgrind.
2016-12-08 19:14:10 -08:00
atumanov
88206417cb unifying plasma seal path through the store to mgr to redis (#96) 2016-12-07 17:25:40 -08:00
Aleks Kamko
b695f2b7e4 drop GIL for plasma_get (#95) 2016-12-07 17:01:14 -08:00
Robert Nishihara
02e8dd245d Update gitignore. (#94) 2016-12-07 11:54:16 -08:00
Robert Nishihara
b3c05655a0 Enable fetching objects from remote object stores. (#87)
* Fetch missing dependencies from local scheduler.

* Factor out global scheduler policy state.

* Use object_table_subscribe instead of object_table_lookup.

* Fix bug in which timer was being created twice for a single fetch request.

* Free old manager vector.
2016-12-06 15:47:31 -08:00
Philipp Moritz
03324caffc Put object store memory on /dev/shm on linux (#89)
* put object store memory on /dev/shm on linux

* fix

* fix mac
2016-12-06 00:31:47 -08:00
Robert Nishihara
dd39008532 Fix bug in which we were running out of file descriptors. (#91) 2016-12-05 23:45:49 -08:00
Philipp Moritz
ba53e4a43a Change object table subscribe to also return payload (#88)
* implement object table subscribe that also returns payload

* fix

* fix valgrind

* fix ray test

* fix clang-format

* fix

* fix
2016-12-05 00:26:53 -08:00
Robert Nishihara
b5ed2f063d Allow starting multiple local schedulers. (#86) 2016-12-04 17:08:16 -08:00
Robert Nishihara
35b9dedb48 Remove scheduler_info. (#84) 2016-12-04 15:51:03 -08:00
Robert Nishihara
2a3e9267f8 Non-blocking fetch implementation. (#83)
* Non-blocking fetch implementation.

* Make fetch tests more robust to timing issues.

* Bug fix when ignoring transferred objects.

* Fix.

* Documentation fixes.
2016-12-03 19:09:05 -08:00
Wapaul1
9a513363f9 Init_table_callback now takes ownership of passed in data (#80)
* temp commit

* Stuff

* Ownership is now taken by init table callback

* Fixed lint errors

* Fixed travis warnings

* Fixed spacing

* add .gitkeep

* fix global scheduler

* Whitespace.
2016-12-03 13:49:09 -08:00
Robert Nishihara
61755bc168 Replace direct calls to recv with read_bytes. (#78)
* Replace direct calls to recv with read_bytes.

* Fixes.

* Formatting.
2016-12-02 18:51:50 -08:00
Robert Nishihara
de2dd1f506 Allow fatal errors to produce core dumps. (#79) 2016-12-02 15:10:37 -08:00
Stephanie Wang
dadbcae17d Cache plasma object information on the plasma client (#73)
* Cache plasma object information on the plasma client

* Plasma client uses cached object references to get the object data
2016-12-01 16:46:17 -08:00
Ion
f89be9699c Introduce non-blocking Plasma API. (#71)
* Implement new plasma client API.

* Formatting fixes.

* Make tests work again.

* Make tests run.

* Comment style.

* Fix bugs with fetch tests.

* Introduce fetch1 flag.

* Remove timer only if present.

* Formatting fixes.

* Don't access object after free.

* Formatting fixes.

* Minor change.

* refactoring plasma datastructures

* Change plasma_request and plasma_reply to use only arrays of object requests.

* some more fixes

* Remove unnecessary methods.

* Trivial.

* fixes

* use plasma_send_reply in return_from_wait1

* Lint.
2016-12-01 02:15:21 -08:00
Philipp Moritz
58e8bbcb34 Fix bug in serializing arguments of tasks that are more complex objects (#72)
* Give more informative error message when we do not know how to serialize a class.

* Check that passing arguments to remote functions and getting them does not change their values.

* fix serialization bug

* fix tests for common module

* Formatting.

* Bug fix in init_pickle_module signature.

* Use pickle with HIGHEST_PROTOCOL.
2016-11-30 23:21:53 -08:00
atumanov
1499834be1 handling partial/interrupted I/O (#69)
* read_bytes: handle EINTR + easier flow

* addressing whitespace + style comments

* handling partial i/o: write_bytes; use [read|write]_bytes in plasma send/receive
2016-11-29 23:04:13 -08:00
Philipp Moritz
a2692ead6e add errno logging for fatal checks (#68) 2016-11-29 22:40:01 -08:00
mehrdadn
4d327c571a Make Redis script executable (#66) 2016-11-29 14:33:20 -08:00
Robert Nishihara
2c639c05ef When searching for Python with cmake, try custom tricks before using default cmake behavior. (#67)
* Change CMakeLists.txt.

* remove dead code
2016-11-29 14:32:54 -08:00
Philipp Moritz
c7073d623b Object table subscribe with new semantics (#62)
* new plasma subscribe implementation

* object table subscribe with test

* clang-format

* fix

* fix test

* fix tests

* fix clang-format

* add check

* final clang-format

* final fixes

* fix clang-format
2016-11-27 21:26:23 -08:00
Robert Nishihara
bc1d7db926 Re-enable fetch tests. (#63)
* Bring back fetch tests.

* Zero initialize success array in PyPlasma_fetch.

* Fix bug in fetch in case where the object ID doesn't have any managers in the object table.

* Temporarily disallow calling fetch with multiple copies of the same object ID.

* Fix.

* Factor out code for checking if list of object IDs are all distinct.

* Remove commented out code.

* Fix.
2016-11-24 16:27:17 -08:00
Wapaul1
800bf8deb3 Redis Server started in Tests now shutdowns at end of tests (#61)
Redis Server started in tests now shutdowns at end of tests.
2016-11-23 21:47:53 -08:00
Philipp Moritz
8147b62964 move methods related to plasma requests and replies to plasma.c (#60) 2016-11-23 21:13:19 -08:00
Robert Nishihara
1f9fce8e53 Remove semicolon in macro. (#59) 2016-11-23 00:54:07 -08:00
mehrdadn
7237ec4124 Windows compatibility (#57)
* Add Python and Redis submodules, and remove old third-party modules

* Update VS projects (WARNING: references files that do not exist yet)

* Update code & add shims for APIs except AF_UNIX/{send,recv}msg()

* Minor style changes.
2016-11-22 17:04:24 -08:00
Robert Nishihara
a93c6b7596 Quick workaround for ray.wait bug. (#42) 2016-11-22 13:31:22 -08:00
mehrdadn
35ce5f8001 Enable more warnings and fix usleep() issue (#55)
* Enable more warnings and fix usleep() issue

* Periods after comments.
2016-11-22 00:29:27 -08:00
mehrdadn
3714984094 Remove Redis version from Linux scripts (#56)
* Remove Redis version from Linux scripts

* Add documentation.
2016-11-21 15:02:40 -08:00
Robert Nishihara
4b00c029ac Remove numbuf from requirements for setup.py. (#54)
* Remove numbuf from requirements for setup.py.

* Update documentation.
2016-11-21 14:30:17 -08:00
Philipp Moritz
da6bcfcc70 Statically link arrow libraries when building numbuf (#53)
* Fix numbuf compilation on Linux

* Statically link arrow libraries when building numbuf.

* Last fixes.
2016-11-21 02:14:13 -08:00
Robert Nishihara
9ed56c23db Fix pip install hanging by moving C tests out of build.sh. (#52)
* Remove C tests from build.sh.

* Run C tests in Travis.
2016-11-20 21:02:54 -08:00
Robert Nishihara
f267da6aa8 Improve case on Mac where cmake fails to find Python with the custom search path. (#49)
* On Mac OS X, if cmake fails to find Python with custom search path, default to using find_package.

* Fix formatting in CMakeLists.txt.

* Set CUSTOM_PYTHON_EXECUTABLE variable on mac if the custom search path fails.

* Require exact Python version match.

* Use cmake mode keyword for logging messages.

* Fix.

* Find python libraries in cmake by searching near the include directories.

* Update cmakelists for numbuf.
2016-11-19 19:00:42 -08:00
Robert Nishihara
c9757a6fd0 Merge pull request #48 from ray-project/numbuf-integration
Numbuf integration
2016-11-19 18:10:53 -08:00
Philipp Moritz
d64f69f215 switch from submodule to cloning arrow, travis fixes & Robert's comments 2016-11-19 17:38:36 -08:00
Philipp Moritz
05e0226047 clang-fixes 2016-11-19 14:27:08 -08:00
Philipp Moritz
9d6da1281e add clang-format for numbuf (arrow style) 2016-11-19 12:56:11 -08:00