Commit graph

68 commits

Author SHA1 Message Date
SongGuyang
c195dc8f88
Basic C++ worker implementation (#6125) 2020-03-27 23:01:08 +08:00
mehrdadn
b4030cdbbe
File HANDLE/descriptor translation layer for Windows (#7657)
* Use TCP sockets on Windows with custom HANDLE <-> FD translation layer

* Get Plasma working on Windows

Co-authored-by: Mehrdad <noreply@github.com>
2020-03-23 21:08:25 -07:00
Stephanie Wang
53549314c5
[core] Option to fallback to LRU on OutOfMemory (#7410)
* Add a test for LRU fallback

* Update error message

* Upgrade arrow to master

* Integrate with arrow

* Revert "Bazel mirrors (#7385)"

This reverts commit 44aded5272.

* Don't LRU evict

* Revert "Revert "Bazel mirrors (#7385)""

This reverts commit b6359fea78d1bd3925452ca88ac71e0c9e5c7dd3.

* Add lru_evict flag

* fix internal config

* Fix

* upgrade arrow

* debug

* Set free period in config for lru_evict, override max retries to fix
test

* Fix test?

* fix test

* Revert "debug"

This reverts commit 98f01c63a267f38218f5047b1866e4c1c8280017.

* fix exception str

* Fix ref count test

* Shorten travis test?
2020-03-14 11:28:43 -07:00
mehrdadn
fc76586518
Redis on Windows (#7509)
* Switch hiredis on Windows to that of the Windows port of Redis

* Use boost::asio::ip::tcp::socket::native_handle_type

* Use normal hiredis instead of Windows-specific one

* Finish up using normal hiredis

Co-authored-by: Mehrdad <noreply@github.com>
2020-03-09 18:49:54 -07:00
Stephanie Wang
95bb0c5357
Upgrade plasma to latest version, use synchronous Seal (#7470)
* Upgrade arrow to master

* fix build

* todo

* lint

* Fix hanging test
2020-03-09 10:30:44 -07:00
mehrdadn
44aded5272
Bazel mirrors (#7385)
* Switch to mirrors.bazel.build where possible

* Switch from .zip to .tar.gz for smaller downloads (it's also the default download on UNIX)

* Use direct GitHub URLs in Bazel files for clarity

* Don't pass patches to local_repository

* Remove github_repository()

* Switch to GitHub actions/checkout@v2 which is faster

* Use faster extraction method for LLVm on Windows

* Move LLVM_VERSION_WINDOWS to the shell script since it's not a CI-specific value

* Change GITHUB_TOKEN to GITHUB

* Don't show timestamps for GitHub Actions

* Factor out some options from GitHub Actions

* Tell Bazel to stay on the same volume in GitHun Actions

* Display progress output when downloading toolchains

Co-authored-by: GitHub Web Flow <noreply@github.com>
2020-03-01 14:04:06 -08:00
mehrdadn
5fb5be0ba5
Some bug fixes for Windows (#7374)
* Fix MAP_SHARED check in sys/mman.h

* Fix missing :platform_shims dependency for ray_util

* dlmalloc patch for Arrow
2020-02-28 10:22:32 -08:00
mehrdadn
0efaa9b310
Use Redis for Windows (#7364) 2020-02-28 10:18:56 -08:00
mehrdadn
57b33f1bed
Upgrade Boost (#6899) 2020-02-25 14:33:12 -08:00
mehrdadn
b97b8c2be1
Fix github_repository(path=...) accidentally requiring other fields that shouldn't be necessary (#7275)
Co-authored-by: GitHub Web Flow <noreply@github.com>
2020-02-24 11:39:08 -08:00
mehrdadn
bde575b8dd Revert "Use Boost.Process instead of pid_t (#6510)" (#6909)
This reverts commit fb8e3615d5.
2020-01-26 10:26:44 -06:00
mehrdadn
fb8e3615d5 Use Boost.Process instead of pid_t (#6510)
* Use Boost.Process instead of pid_t

This will let us handle child processes (mostly) uniformly across platforms.
TODO: There is no SIGTERM on Windows; achieving something equivalent is fairly involved.
2020-01-15 20:05:02 -08:00
Chaokun Yang
7bbfa85c66 [Streaming] Streaming data transfer java (#6474) 2019-12-22 10:56:05 +08:00
mehrdadn
f06d7422db Improve handling and caching of GitHub repository archives (#6526)
* Use GitHub mirror to allow caching BoringSSL

* Add archive suffix as an optional parameter
2019-12-20 17:22:35 -08:00
mehrdadn
b0b6b56bb7 Platform shims for Windows (#6548)
* Platform shims for Windows

* Satisfy linter
2019-12-20 12:32:07 -08:00
mehrdadn
1e817f48df Fix Arrow and Windows linking issues in Bazel (#6546)
* Fix Windows system library dependencies in Boost rules

* Fix missing utf8cpp source and header files in arrow target
2019-12-20 11:53:20 -08:00
mehrdadn
ab5a9f0946 Patch hiredis for Windows (#6446) 2019-12-17 15:32:47 -08:00
mehrdadn
7a24144bfd Polish Bazel build scripts (#6424)
* Polish Bazel build scripts

* Remove glog references from streaming_logging.cc

* Move out COPTS and reference them

* Disable streaming on Windows

* Remove -fno-gnu-unique
2019-12-17 02:38:36 -08:00
mehrdadn
9d6e03aba0 Fix use of select() instead of poll() (#6477)
* Fix Arrow poll() patch

- Negative timeout for poll() was not translated to infinite timeout for select()
- Only use select() on Windows, as other systems limit the range of the file descriptors

* Apply poll() -> select() patch to Redis's ae.c as well
2019-12-17 02:33:37 -08:00
mehrdadn
74b2e871b7 Tentative workaround for some forks and signals on Windows (#6362)
* Platform shims for Windows

* Tentative workaround for some forks and signals on Windows

* Rewrite WorkerPool::StartProcess by moving spawnvp wrapper to a separate function

* Separate spawnvp the wrappers for POSIX and Windows

* Fix rv use
2019-12-16 16:57:49 -08:00
Chaokun Yang
6272907a57 [Streaming] Streaming data transfer and python integration (#6185) 2019-12-10 20:33:24 +08:00
mehrdadn
17103376c5 Patch arrow for Windows (#6363) 2019-12-05 16:09:21 -08:00
mehrdadn
75cc994e0a Update various build options relating to Windows (#6315)
* Update .bazelrc for Windows compatibility

* Block inclusion of (legacy) WinSock.h to avoid errors

* Suppress warnings for Windows code

* Include boost::asio in includes so that it is passed as -isystem to avoid warnings

* Link with -lpthread only on non-Windows

* Undefine BOOST_FALLTHROUGH, which is unnecessary and causes macro redefinition warnings

* Define RAY_STATIC and ARROW_STATIC to compile for Windows

* Add WinSock import library for Arrow
2019-12-01 15:05:50 -08:00
mehrdadn
b8cfdba752 Bazelify hiredis (#6203) 2019-11-29 15:32:45 -08:00
mehrdadn
cafdaa346f Update glog (#6287) 2019-11-26 14:54:36 -08:00
mehrdadn
5340e5280a Patch prometheus-cpp internally (#6281) 2019-11-26 14:49:24 -08:00
mehrdadn
82d17888e0 Patch grpc for Windows (#6282) 2019-11-26 14:45:21 -08:00
mehrdadn
ca08a8f479 Update grpc to version that fixes typo in third_party/py/python_configure.bzl (#6235)
See https://github.com/grpc/grpc/pull/20774
2019-11-25 15:20:33 -07:00
mehrdadn
94d37eee28 Update Boost via our own rule instead of managing our own fork (#6238) 2019-11-22 16:10:47 -08:00
mehrdadn
05ce789e5b Reorganize ray_deps_setup.bzl to make all the GitHub rules uniform and download ZIP files for everything (#6193)
* Reorganize ray_deps_setup.bzl to make all the GitHub rules uniform

* Rewrite github_repository with explicit keyword-only arguments

Requires Bazel >= 0.29.0: https://github.com/bazelbuild/buildtools/pull/677
2019-11-22 09:59:32 -08:00
mehrdadn
ba86c75c21 Patch Cython in grpc to use our COPTS (#6223) 2019-11-21 15:32:48 -08:00
Philipp Moritz
ccbcc4bafa
Use GRCP and Bazel 1.0 (#6002) 2019-11-08 15:58:28 -08:00
mehrdadn
e312f3d282 Compatibility issues (#6071)
* Pass -f - to tar to force stdin on Windows

* Quote paths that may contain spaces (causes issues on Windows)

* Copy over Windows code from Arrow for glog signal handle uninstall

* Add missing COPTS to build rules since we'll need them for Windows compatibility

* Begin adding COPTS for Windows compatibility

* Disable glog on Arrow until we change WIN32 to _WIN32 there

* Missing header files that cause problems on Windows

* WORD typedef conflicts with Windows; remove it

* uint -> unsigned int wherever we're dealing with milliseconds (signed version is already int)

* uint -> unsigned int for enums

* uint -> size_t, wherever we're dealing with sizes or indices into arrays

* Work around Boost 1.68 bug in detecting clang-cl (revert this after upgrading)

* Missing #include <unistd.h>

* Add check for signal handler uninstallation failure

* Linting issue
2019-11-05 00:08:14 -08:00
Philipp Moritz
1c5446851a
Use Plasma with LRU refreshing integrated (#6050) 2019-11-03 16:19:05 -08:00
Philipp Moritz
894885593c Fix prometheus-cpp failure (#6073) 2019-11-03 15:05:47 -08:00
mehrdadn
e706cb63cc Fix missing double quotes for spaces in paths (#6026) 2019-10-26 20:46:55 -07:00
Stephanie Wang
eb41c945a1 Add gRPC endpoint to raylet to expose metrics (#6005) 2019-10-26 16:37:39 -07:00
Philipp Moritz
0bb922c29f Revert "Use plasma with batched CreateAndSeal implemented (#5864)" (#6022)
This reverts commit 875c84ed63.
2019-10-25 23:02:21 -07:00
Simon Mo
f1d2eb5247
Apply shallow-since and sha256 (#6019) 2019-10-25 19:48:04 -07:00
Danyang Zhuo
875c84ed63 Use plasma with batched CreateAndSeal implemented (#5864) 2019-10-22 21:32:19 -07:00
Edward Oakes
fc56872012
Send active object IDs to the raylet (#5803)
* Send active object IDs to the raylet

* comment

* comments

* dedup

* signed int in config

* comments

* Remove object ID from monitor

* Fix test

* re-add check

* fix cast

* check if core worker

* Add comment

* Reservoir sampling

* Fix lint

* Pointer return

* tmp

* Fix merge

* Initialize object ids properly

* Fix lint
2019-10-20 22:05:28 -07:00
Eric Liang
e2e30ca507 Ray, Tune, and RLlib support for memory, object_store_memory options (#5226) 2019-08-21 23:01:10 -07:00
Richard Liaw
9c00616cdc
Retry and exception for hang on memory store full (#5143) 2019-07-27 01:20:13 -07:00
Kai Yang
806524384b [Java worker] Refactor object store and worker context on top of core worker (#5079) 2019-07-16 20:58:02 +08:00
Philipp Moritz
322b5166ad Update arrow to include user defined status for plasma (#5156) 2019-07-12 22:51:14 -07:00
Hao Chen
8a30b93e42
Define common data structures with protobuf. (#5121) 2019-07-08 22:41:37 +08:00
Joey Jiang
4183303a2f Add bazel build options for plasma to use glog (#5108) 2019-07-05 19:00:19 +08:00
Philipp Moritz
4e82313891 Update to latest arrow (#5011) 2019-06-30 20:36:36 -07:00
Hao Chen
0131353d42 [gRPC] Migrate gcs data structures to protobuf (#5024) 2019-06-25 14:31:19 -07:00
Hao Chen
2bf92e02e2
[gRPC] Use gRPC for inter-node-manager communication (#4968) 2019-06-17 19:00:50 +08:00