Commit graph

149 commits

Author SHA1 Message Date
mehrdadn
74b2e871b7 Tentative workaround for some forks and signals on Windows (#6362)
* Platform shims for Windows

* Tentative workaround for some forks and signals on Windows

* Rewrite WorkerPool::StartProcess by moving spawnvp wrapper to a separate function

* Separate spawnvp the wrappers for POSIX and Windows

* Fix rv use
2019-12-16 16:57:49 -08:00
Chaokun Yang
6272907a57 [Streaming] Streaming data transfer and python integration (#6185) 2019-12-10 20:33:24 +08:00
mehrdadn
17103376c5 Patch arrow for Windows (#6363) 2019-12-05 16:09:21 -08:00
mehrdadn
75cc994e0a Update various build options relating to Windows (#6315)
* Update .bazelrc for Windows compatibility

* Block inclusion of (legacy) WinSock.h to avoid errors

* Suppress warnings for Windows code

* Include boost::asio in includes so that it is passed as -isystem to avoid warnings

* Link with -lpthread only on non-Windows

* Undefine BOOST_FALLTHROUGH, which is unnecessary and causes macro redefinition warnings

* Define RAY_STATIC and ARROW_STATIC to compile for Windows

* Add WinSock import library for Arrow
2019-12-01 15:05:50 -08:00
mehrdadn
b8cfdba752 Bazelify hiredis (#6203) 2019-11-29 15:32:45 -08:00
mehrdadn
cafdaa346f Update glog (#6287) 2019-11-26 14:54:36 -08:00
mehrdadn
5340e5280a Patch prometheus-cpp internally (#6281) 2019-11-26 14:49:24 -08:00
mehrdadn
82d17888e0 Patch grpc for Windows (#6282) 2019-11-26 14:45:21 -08:00
mehrdadn
ca08a8f479 Update grpc to version that fixes typo in third_party/py/python_configure.bzl (#6235)
See https://github.com/grpc/grpc/pull/20774
2019-11-25 15:20:33 -07:00
mehrdadn
94d37eee28 Update Boost via our own rule instead of managing our own fork (#6238) 2019-11-22 16:10:47 -08:00
mehrdadn
05ce789e5b Reorganize ray_deps_setup.bzl to make all the GitHub rules uniform and download ZIP files for everything (#6193)
* Reorganize ray_deps_setup.bzl to make all the GitHub rules uniform

* Rewrite github_repository with explicit keyword-only arguments

Requires Bazel >= 0.29.0: https://github.com/bazelbuild/buildtools/pull/677
2019-11-22 09:59:32 -08:00
mehrdadn
ba86c75c21 Patch Cython in grpc to use our COPTS (#6223) 2019-11-21 15:32:48 -08:00
Philipp Moritz
ccbcc4bafa
Use GRCP and Bazel 1.0 (#6002) 2019-11-08 15:58:28 -08:00
mehrdadn
e312f3d282 Compatibility issues (#6071)
* Pass -f - to tar to force stdin on Windows

* Quote paths that may contain spaces (causes issues on Windows)

* Copy over Windows code from Arrow for glog signal handle uninstall

* Add missing COPTS to build rules since we'll need them for Windows compatibility

* Begin adding COPTS for Windows compatibility

* Disable glog on Arrow until we change WIN32 to _WIN32 there

* Missing header files that cause problems on Windows

* WORD typedef conflicts with Windows; remove it

* uint -> unsigned int wherever we're dealing with milliseconds (signed version is already int)

* uint -> unsigned int for enums

* uint -> size_t, wherever we're dealing with sizes or indices into arrays

* Work around Boost 1.68 bug in detecting clang-cl (revert this after upgrading)

* Missing #include <unistd.h>

* Add check for signal handler uninstallation failure

* Linting issue
2019-11-05 00:08:14 -08:00
Philipp Moritz
1c5446851a
Use Plasma with LRU refreshing integrated (#6050) 2019-11-03 16:19:05 -08:00
Philipp Moritz
894885593c Fix prometheus-cpp failure (#6073) 2019-11-03 15:05:47 -08:00
mehrdadn
e706cb63cc Fix missing double quotes for spaces in paths (#6026) 2019-10-26 20:46:55 -07:00
Stephanie Wang
eb41c945a1 Add gRPC endpoint to raylet to expose metrics (#6005) 2019-10-26 16:37:39 -07:00
Philipp Moritz
0bb922c29f Revert "Use plasma with batched CreateAndSeal implemented (#5864)" (#6022)
This reverts commit 875c84ed63.
2019-10-25 23:02:21 -07:00
Simon Mo
f1d2eb5247
Apply shallow-since and sha256 (#6019) 2019-10-25 19:48:04 -07:00
Danyang Zhuo
875c84ed63 Use plasma with batched CreateAndSeal implemented (#5864) 2019-10-22 21:32:19 -07:00
Edward Oakes
fc56872012
Send active object IDs to the raylet (#5803)
* Send active object IDs to the raylet

* comment

* comments

* dedup

* signed int in config

* comments

* Remove object ID from monitor

* Fix test

* re-add check

* fix cast

* check if core worker

* Add comment

* Reservoir sampling

* Fix lint

* Pointer return

* tmp

* Fix merge

* Initialize object ids properly

* Fix lint
2019-10-20 22:05:28 -07:00
Eric Liang
e2e30ca507 Ray, Tune, and RLlib support for memory, object_store_memory options (#5226) 2019-08-21 23:01:10 -07:00
Richard Liaw
9c00616cdc
Retry and exception for hang on memory store full (#5143) 2019-07-27 01:20:13 -07:00
Kai Yang
806524384b [Java worker] Refactor object store and worker context on top of core worker (#5079) 2019-07-16 20:58:02 +08:00
Philipp Moritz
322b5166ad Update arrow to include user defined status for plasma (#5156) 2019-07-12 22:51:14 -07:00
Hao Chen
8a30b93e42
Define common data structures with protobuf. (#5121) 2019-07-08 22:41:37 +08:00
Joey Jiang
4183303a2f Add bazel build options for plasma to use glog (#5108) 2019-07-05 19:00:19 +08:00
Philipp Moritz
4e82313891 Update to latest arrow (#5011) 2019-06-30 20:36:36 -07:00
Hao Chen
0131353d42 [gRPC] Migrate gcs data structures to protobuf (#5024) 2019-06-25 14:31:19 -07:00
Hao Chen
2bf92e02e2
[gRPC] Use gRPC for inter-node-manager communication (#4968) 2019-06-17 19:00:50 +08:00
Philipp Moritz
64eb7b322c Upgrade arrow to latest master (#4858) 2019-05-28 16:04:16 -07:00
Qing Wang
64a01b2ab6 Update deps commits of opencensus to support building with bzl 0.25.x (#4862)
* Update deps to support bzl 2.5.x

* Fix
2019-05-28 14:29:34 +08:00
Hao Chen
ffd596d5bb
Fix pom file generation (#4800) 2019-05-17 10:56:39 +08:00
Hao Chen
643f62dc43
[Java][Bazel] Refine auto-generated pom files (#4780) 2019-05-16 11:19:31 +08:00
Ruifang Chen
c578be23a5 [Bazel] Modifying WORKSPACE file, so that you can make the project used as a thirdparty project (#4711)
* Modifying WORKSPACE file, so that you can make the project used as a thirdparty

* Modifying WORKSPACE file, so that you can make the project used as a thirdparty

* add some files

* modify some repositories

* modify the name of 'ray_deps_build_all'
2019-04-28 22:02:49 -07:00
Hao Chen
be2cbdf130
[Bazel] Use rules_jvm_external to manage java dependencies (#4615) 2019-04-18 16:53:25 +08:00
Philipp Moritz
1bcb0b94cc Synchronize arrow version and put changes upstream (#4385) 2019-03-27 22:37:07 -07:00
Ruifang Chen
59d74d5e92 [Java] Build Java code with Bazel (#4284) 2019-03-22 14:30:05 +08:00
Yuhong Guo
edb063c3c8 Fix glog problem of no call stack (#4395) 2019-03-18 18:21:21 -07:00
Philipp Moritz
490d896f41 Make sure the right Python interpreter is used (#4334) 2019-03-12 12:21:55 -07:00
Philipp Moritz
4bea25076f [build] Fix bazel glog error (#4279) 2019-03-06 11:03:59 -08:00
alegithub111
67fa0b5c25 Refine JNI bazel script to make it suitable for more systems (#4060)
* Refine JNI bazel script to make it suitable for more systems

* Update BUILD.bazel

the script format has changed

* Update BUILD.plasma

the script format has changed

* Lint bazel/BUILD.plasma  BUILD.bazel
2019-02-20 22:37:41 +08:00
Yuhong Guo
4b0db437ee
Linting Bazel scripts (#4032)
* Use buildifier as bazel script linter

* Checkout golang version in travis

* Using golang-1.8-go in travis

* Add golang apt-repository

* Fix the bazel lint failure example.

* Address comment
2019-02-14 22:16:19 +08:00
ebrevdo
52dfde1cbb Update flatbuffer bazel rule to work with flatbuffer master branch. (#4008) 2019-02-11 10:00:06 -08:00
Yuhong Guo
add8ae7063 Add bazel build for JNI code (#3918)
* Add bazel build for JNI code

* clean

* Add plasma client JNI build process

* refine

* clean linux part

* Add Java Library

* Remove java library

* Generate dylib after build using genrule
2019-02-04 13:03:46 -08:00
Philipp Moritz
f7415b37c5 Build Ray with Bazel (#3867) 2019-01-27 18:32:04 -08:00
Philipp Moritz
20162ce159 Compile raylet cython bindings with bazel (#3842) 2019-01-25 00:57:31 -08:00
Philipp Moritz
0dad4e6a25 Build Raylet with Bazel (#3806) 2019-01-20 12:16:47 -08:00