Commit graph

54 commits

Author SHA1 Message Date
Yujie Liu
a8d3c057c1 [JavaWorker] Enable java worker support (#2094)
* Enable java worker support
--------------------------
This commit includes a tailored version of the Java worker implementation from Ant Financial.
The changes for build system, python module, src module and arrow are in other commits, this commit consists of the following modules:
 - java/api: Ray API definition
 - java/common: utilities
 - java/hook: binary rewrite of the Java byte-code for remote execution
 - java/runtime-common: common implementation of the runtime in worker
 - java/runtime-dev: a pure-java mock implementation of the runtime for fast development
 - java/runtime-native: a native implementation of the runtime
 - java/test: various tests

Contributors for this work:
 Guyang Song, Peng Cao, Senlin Zhu,Xiaoying Chu, Yiming Yu, Yujie Liu, Zhenyu Guo

* change the format of java help document from markdown to RST

* update the vesion of Arrow for java worker

* adapt the new version of plasma java client from arrow which use byte[] instead of custom type

* add java worker test to ci

* add the example module for better usage guide
2018-05-26 14:38:50 -07:00
Zongheng Yang
fa97acbc89 Integrate credis with Ray & route task table entries into credis. (#1841) 2018-05-24 23:35:25 -07:00
Philipp Moritz
225608ec66 Update arrow to latest master (#2100) 2018-05-24 00:26:13 -07:00
Philipp Moritz
af88fdefcf Incorporate C++ Buffer management and Seal global threadpool fix from arrow (#1950) 2018-04-25 22:53:44 -07:00
Robert Nishihara
24c944e499 Update arrow to efficiently serialize more types of numpy arrays. (#1889) 2018-04-13 09:41:51 -07:00
Robert Nishihara
d0fffec2d0 Update arrow and parquet-cpp. (#1875)
* Update arrow.

* Fix bug.

* Cherry-pick commit for fixing parquet segfault.

* Update arrow and revert auto-releasing buffer commit.

* Remove parquet cherry-pick.
2018-04-12 16:17:12 -07:00
alonamid
202f9683ea check if arrow build dir exists (#1863) 2018-04-10 14:52:51 -07:00
Robert Nishihara
d8adfbd440 Fail build if pyarrow hasn't successfully been built. (#1721) 2018-03-16 12:02:36 -07:00
Philipp Moritz
5ef0892236 Compile boost from source to fix macOS wheels (#1688) 2018-03-08 23:22:23 -08:00
Philipp Moritz
a9acfab3a6 Start chain replicated GCS with Ray (#1538) 2018-03-07 10:18:58 -08:00
Zhenyu Guo
f1e5789c26 restructure how to organize 3rd party libs (#1630)
* restructure how to organize 3rd party libs

* Minor whitespace changes.

* Fix compilation on Linux.

* Pass around Python executable so that the correct version of Python is used.
2018-03-01 14:29:56 -08:00
mehrdadn
7237ec4124 Windows compatibility (#57)
* Add Python and Redis submodules, and remove old third-party modules

* Update VS projects (WARNING: references files that do not exist yet)

* Update code & add shims for APIs except AF_UNIX/{send,recv}msg()

* Minor style changes.
2016-11-22 17:04:24 -08:00
Robert Nishihara
f83a98d71a Changes to make tests pass on Travis. (#3)
* Remove hiredis submodule.

* Squashed 'src/common/thirdparty/hiredis/' content from commit acd1966

git-subtree-dir: src/common/thirdparty/hiredis
git-subtree-split: acd1966bf7f5e1be74b426272635c672def78779

* Make Plasma tests pass.

* Make Photon tests pass.

* Compile and test with Travis.

* Deactive fetch test so that the tests pass.
2016-10-25 22:39:21 -07:00
Robert Nishihara
7c1b2f702f Remove git submodules and C++ files. 2016-10-25 12:51:30 -07:00
mehrdadn
c6ec36865d Update Windows patches for GRPC (#444) 2016-10-06 11:50:40 -07:00
Robert Nishihara
de6ec47f9e Add a recursion depth for serialization to prevent infinite loops. (#440) 2016-09-19 17:17:42 -07:00
Robert Nishihara
91f16a3df0 Migrate repositories to ray-project. (#438)
* Migrate repositories to ray-project.

* Update numbuf to the migrated version.
2016-09-17 00:52:05 -07:00
Robert Nishihara
1aa89a4ae6 Update numbuf to properly handle Python floats. (#435) 2016-09-15 15:44:11 -07:00
Robert Nishihara
d264713ceb Work around Arrow bug by increasing metadata size. (#415) 2016-09-07 18:46:04 -07:00
Robert Nishihara
11a8914684 Allow users to serialize custom classes. (#393)
* Allow serialization of custom classes.

* Add documentation and test cases, also fix pickle case.

* Don't allow old-style classes.
2016-09-06 13:28:24 -07:00
Philipp Moritz
68cec55a98 Refcount without modifying objects (#407)
* refcount without modifying objects

* add documentation

* Update tests and documentation.

* Remove extraneous code.

* Update numbuf version.
2016-09-04 12:07:52 -07:00
Stephanie Wang
b6872182bf Add hiredis to build (#395)
* Add hiredis to build

* make hiredis build if ray has been cloned before this commit
2016-09-01 20:15:00 -07:00
Philipp Moritz
93e6c9947b update numbuf (#392)
* update numbuf

* Augment serialization tests.
2016-08-25 20:05:48 -07:00
Robert Nishihara
43452b9ab3 Try grpc version 1.0 (#386) 2016-08-23 19:29:18 -07:00
Robert Nishihara
87bb7a8f67 [WIP] Large changes to make the tests pass. (#376)
* Revert "Make tests more informative (#372)"

This reverts commit fd353250c8.

* fix bugs, in particular deactivate worker service on driver and remove condition variables

* changes to minimize the changes in this PR

* switch from faulty mutex synchronization to using atomics

* Increase the default size of the message queues, to accommodate exporting large numbers of remote functions. This is a temporary fix, but not a long term solution.

* Reorganize the scheduler export code to queue up exports. This does not solve the underlying problem yet, but sets up a solution.

* Start a separate thread on driver to print error messages by constantly querying the scheduler. This is a temporary solution because the solution based on starting a worker service for the driver which the scheduler can push error messages to is buggy.

* Fix segfault in taskcapsule destructor.

* Move tests for catching errors into a separate test file.

* Revert "roll back grpc (#368)"

This reverts commit c01ef95d04.
2016-08-15 11:02:54 -07:00
Philipp Moritz
c01ef95d04 roll back grpc (#368) 2016-08-10 16:45:40 -07:00
Philipp Moritz
9903fc0335 Improve error message for serialization of custom objects (#359)
* Improve error message for serialization of custom objects

* gitignore numbuf build directory
2016-08-09 13:38:21 -07:00
Philipp Moritz
8bf877ac1e Serialize and Deserialize unicode (#349) 2016-08-04 21:06:31 -07:00
mehrdadn
1961deeffb Update Windows support (#317) 2016-07-28 13:11:13 -07:00
mehrdadn
82480135a6 numbuf (#316) 2016-07-28 11:42:39 -07:00
mehrdadn
c3695099fc numbuf update (#313) 2016-07-27 23:21:33 -07:00
mehrdadn
65db47b556 Update submodules (#309) 2016-07-27 19:47:14 -07:00
Philipp Moritz
2d929ad938 Only check out submodules that are actually used (#310) 2016-07-27 19:14:40 -07:00
Philipp Moritz
5591aa4665 cleanup serialization code (#291) 2016-07-25 15:47:10 -07:00
Philipp Moritz
4a0f35b042 new arrow serialization code (serialize python objects recursively) (#284) 2016-07-25 13:41:47 -07:00
mehrdadn
c5be9b96de Visual Studio - Project cleanup, warning suppression, and removal of zlib dependency (#272)
* windows_intermodule_singleton warning

* Fix some warnings from Visual Studio

* More Visual Studio project changes

* Replace zlib Visual Studio project with stub; it's not needed

* Un-enable link-time code generation
2016-07-16 13:02:51 -07:00
mehrdadn
69ee25eb51 Fix GRPC project patch, again (#255) 2016-07-10 22:31:12 -07:00
mehrdadn
30023b456a Fix GRPC project patch (#254) 2016-07-10 22:05:50 -07:00
mehrdadn
5db31cd8a2 Update numbuf (#246) 2016-07-09 21:47:53 -07:00
mehrdadn
302591cc0b Edit batch file again (#241) 2016-07-09 21:23:02 -07:00
mehrdadn
fc711abda0 Remove parquet for Windows -- we don't need it yet (#240) 2016-07-08 20:06:21 -07:00
mehrdadn
7ff1580606 Apply patches while considering index as well (#231) 2016-07-08 09:24:25 -07:00
mehrdadn
63f7548763 Fix wrong branch to checkout (and do recursively if possible) (#226) 2016-07-07 00:22:36 -07:00
mehrdadn
eff6924b7c Update third-party patches (#203) 2016-07-05 12:17:09 -07:00
Philipp Moritz
72e1b0fcd6 fix warnings (#207) 2016-07-04 15:40:56 -07:00
Philipp Moritz
3eeb2d315c [WIP] update grpc (#206)
* update grpc

* use GRPC's OpenSSL library
2016-07-04 14:24:28 -07:00
Philipp Moritz
b99ea3ca4e build ray in parallel 2016-06-25 15:35:52 -07:00
Philipp Moritz
8a0b8e8789 Unmap segments properly to avoid memory leaks 2016-06-24 16:56:26 -07:00
mehrdadn
fe918a9f8b Update numbuf to master (#147) 2016-06-22 16:33:04 -07:00
Mehrdad
c230b6b459 Windows support for Visual Studio 2015 2016-06-19 20:51:40 -07:00