Commit graph

127 commits

Author SHA1 Message Date
SangBin Cho
826f14c824
[Stats] Fix harvestor threads + Fix flaky stats shutdown. (#9745) 2020-07-29 18:57:59 -05:00
Siyuan (Ryans) Zhuang
54a0d8b69e
[Core] Try remove all windows compat shims (#9671)
* try remove compat for arrow

* remove unistd.h

* remove socket compat

* delete arrow windows patch
2020-07-25 12:00:36 -07:00
SangBin Cho
d49b19c24c
[Stats] Improve Stats::Init & Add it to GCS server (#9563) 2020-07-25 10:42:08 -07:00
mehrdadn
dd2cc6eb48
Update hiredis and remove Windows patches (#9289)
Co-authored-by: Mehrdad <noreply@github.com>
2020-07-09 18:45:44 -07:00
Lingxuan Zuo
0562236533
[Stats] Make aggeration window and report interval configurable. (#9235) 2020-07-09 12:44:38 +08:00
mehrdadn
29acf272b7
Build with Visual C++ (#9190)
Co-authored-by: Mehrdad <noreply@github.com>
Co-authored-by: Simon Mo <xmo@berkeley.edu>
2020-07-02 09:34:24 -07:00
mehrdadn
79c4c67ed9
Split hiredis from Redis and use it directly (#9176)
* Split hiredis from Redis and use it directly

Co-authored-by: Mehrdad <noreply@github.com>
2020-06-29 18:16:32 -07:00
mehrdadn
11bda7272b
[windows] Work around msgpack issue on PowerPC64LE (#9140)
Co-authored-by: Mehrdad <noreply@github.com>
2020-06-25 20:36:22 -07:00
Amit Sadaphule
f0b7de7cfe
Fix '//:redis_client' build on RHEL 7.6 ppc64le (#9035)
Co-authored-by: Mehrdad <noreply@github.com>
2020-06-19 16:28:37 -07:00
mehrdadn
07637e5b5b
Upgrade Bazel and add required patches (#8847) 2020-06-12 14:59:22 -05:00
Siyuan (Ryans) Zhuang
ea05ebe89e
Ship plasma store with Ray (#7901) 2020-06-03 17:44:34 -07:00
mehrdadn
ebf060d484
Make more tests run on Windows (#8446)
* Remove worker Wait() call due to SIGCHLD being ignored

* Port _pid_alive to Windows

* Show PID as well as TID in glog

* Update TensorFlow version for Python 3.8 on Windows

* Handle missing Pillow on Windows

* Work around dm-tree PermissionError on Windows

* Fix some lint errors on Windows with Python 3.8

* Simplify torch requirements

* Quiet git clean

* Handle finalizer issues

* Exit with the signal number

* Get rid of wget

* Fix some Windows compatibility issues with tests

Co-authored-by: Mehrdad <noreply@github.com>
2020-05-20 12:25:04 -07:00
mehrdadn
4bdef78e2e
Various CI fixes and cleanup (#8289) 2020-05-05 10:47:49 -07:00
mehrdadn
f86e623095
Fix & improve GitHub Actions CI builds (#7784) 2020-03-30 16:29:54 -07:00
mehrdadn
b4030cdbbe
File HANDLE/descriptor translation layer for Windows (#7657)
* Use TCP sockets on Windows with custom HANDLE <-> FD translation layer

* Get Plasma working on Windows

Co-authored-by: Mehrdad <noreply@github.com>
2020-03-23 21:08:25 -07:00
mehrdadn
fc76586518
Redis on Windows (#7509)
* Switch hiredis on Windows to that of the Windows port of Redis

* Use boost::asio::ip::tcp::socket::native_handle_type

* Use normal hiredis instead of Windows-specific one

* Finish up using normal hiredis

Co-authored-by: Mehrdad <noreply@github.com>
2020-03-09 18:49:54 -07:00
mehrdadn
5fb5be0ba5
Some bug fixes for Windows (#7374)
* Fix MAP_SHARED check in sys/mman.h

* Fix missing :platform_shims dependency for ray_util

* dlmalloc patch for Arrow
2020-02-28 10:22:32 -08:00
mehrdadn
57b33f1bed
Upgrade Boost (#6899) 2020-02-25 14:33:12 -08:00
mehrdadn
83c4e947c7
Make Cython rules more consistent for Bazel (#6840) 2020-02-10 10:45:54 -08:00
mehrdadn
ad4ac9aa70
Add clang-iwyu (#7081)
* Add iwyu

Co-authored-by: GitHub Web Flow <noreply@github.com>
2020-02-07 16:19:46 -08:00
mehrdadn
bde575b8dd Revert "Use Boost.Process instead of pid_t (#6510)" (#6909)
This reverts commit fb8e3615d5.
2020-01-26 10:26:44 -06:00
mehrdadn
fb8e3615d5 Use Boost.Process instead of pid_t (#6510)
* Use Boost.Process instead of pid_t

This will let us handle child processes (mostly) uniformly across platforms.
TODO: There is no SIGTERM on Windows; achieving something equivalent is fairly involved.
2020-01-15 20:05:02 -08:00
mehrdadn
1e817f48df Fix Arrow and Windows linking issues in Bazel (#6546)
* Fix Windows system library dependencies in Boost rules

* Fix missing utf8cpp source and header files in arrow target
2019-12-20 11:53:20 -08:00
mehrdadn
ab5a9f0946 Patch hiredis for Windows (#6446) 2019-12-17 15:32:47 -08:00
mehrdadn
9d6e03aba0 Fix use of select() instead of poll() (#6477)
* Fix Arrow poll() patch

- Negative timeout for poll() was not translated to infinite timeout for select()
- Only use select() on Windows, as other systems limit the range of the file descriptors

* Apply poll() -> select() patch to Redis's ae.c as well
2019-12-17 02:33:37 -08:00
mehrdadn
9948a3779d Simplify patches and make them more robust (#6478)
* Get rid of 'index' lines in patches, which are unnecessary and likely wrong anyway (esp. when there are multiple patches)

* Simplify patches to remove unnecessary context and make them more robust
2019-12-16 19:28:06 -08:00
Chaokun Yang
6272907a57 [Streaming] Streaming data transfer and python integration (#6185) 2019-12-10 20:33:24 +08:00
mehrdadn
17103376c5 Patch arrow for Windows (#6363) 2019-12-05 16:09:21 -08:00
mehrdadn
75cc994e0a Update various build options relating to Windows (#6315)
* Update .bazelrc for Windows compatibility

* Block inclusion of (legacy) WinSock.h to avoid errors

* Suppress warnings for Windows code

* Include boost::asio in includes so that it is passed as -isystem to avoid warnings

* Link with -lpthread only on non-Windows

* Undefine BOOST_FALLTHROUGH, which is unnecessary and causes macro redefinition warnings

* Define RAY_STATIC and ARROW_STATIC to compile for Windows

* Add WinSock import library for Arrow
2019-12-01 15:05:50 -08:00
mehrdadn
b8cfdba752 Bazelify hiredis (#6203) 2019-11-29 15:32:45 -08:00
mehrdadn
cafdaa346f Update glog (#6287) 2019-11-26 14:54:36 -08:00
mehrdadn
5340e5280a Patch prometheus-cpp internally (#6281) 2019-11-26 14:49:24 -08:00
mehrdadn
82d17888e0 Patch grpc for Windows (#6282) 2019-11-26 14:45:21 -08:00
mehrdadn
94d37eee28 Update Boost via our own rule instead of managing our own fork (#6238) 2019-11-22 16:10:47 -08:00
mehrdadn
ba86c75c21 Patch Cython in grpc to use our COPTS (#6223) 2019-11-21 15:32:48 -08:00
Philipp Moritz
010270b3dc
Cleanup left over shell scripts in build process (#6017) 2019-10-26 15:46:46 -07:00
Devin Petersohn
c33d6662ce Remove Modin from Ray wheels. (#5647)
There are several reasons for this:

* We no longer support python2
* There should be only 1 way of installing Modin
  * Issue management on these wheels
* I have never heard of anyone using this feature
* It is rarely kept up to date
* Modin depends on specific versions of Ray because of past API changes
2019-09-05 23:46:27 -07:00
Simon Mo
1f8ae17f60 Silence some installation process for build from source (#5396) 2019-08-07 16:54:48 -07:00
Philipp Moritz
a6a02fccd0 Do not compile redis twice (#5074) 2019-07-01 15:42:54 -07:00
Devin Petersohn
3a72430e87 Bump Modin version to 0.5 (#4752)
* Bump Modin version

* Update Travis pandas version
2019-05-07 01:12:52 -07:00
Philipp Moritz
95254b3d71 Remove the old web UI (#4301) 2019-03-07 23:15:11 -08:00
William Ma
4be3d0c5d3 Update shipped modin to 0.3.1 (#4058) 2019-02-15 15:49:38 -08:00
Devin Petersohn
a1bcd2a4f5 Update Modin to 0.3.0 (#3936) 2019-02-02 23:06:16 -08:00
Robert Nishihara
2887dac427 Use Redis version 5.0.3. (#3886) 2019-01-29 19:19:05 -08:00
Philipp Moritz
e578a38116 Fix TensorFlow and PyTorch compatibility (#3574)
* remove tensorflow workaround
* update docker
* add boost threads
* add date_time, too
* change link order
* cosmetics
2018-12-22 13:25:48 -08:00
Devin Petersohn
3833ba4e4b Bump modin version to 0.2.5 (#3553) 2018-12-17 14:36:47 -05:00
Yuhong Guo
2a4685a08b Add a script to collect built thirdparty libs to avoid download and building again. (#3521) 2018-12-13 23:56:40 -08:00
Devin Petersohn
4d2010a852 Ship Modin with Ray. (#3109) 2018-11-29 20:05:24 +01:00
Yuhong Guo
93ded5a3d5 Update arrow using Plasma with glog (#2913)
* Update Arrow to Plasma with glog and update the building process

* Remove ParquetExternalProject.cmake

* Fix Mac building error in CI

* Use find_package(BISON) instead of hard code

* Revert BISON binary to hard code.

* Remove build_parquet.sh

* Update setup.sh
2018-09-20 13:37:44 -07:00
Hanwei Jin
fbf214e408 update ray cmake build process (#2853)
* use cmake to build ray project, no need to appply build.sh before cmake, fix some abuse of cmake, improve the build performance

* support boost external project, avoid using the system or build.sh boost

* keep compatible with build.sh, remove boost and arrow build from it.

* bugfix: parquet bison version control, plasma_java lib install problem

* bugfix: cmake, do not compile plasma java client if no need

* bugfix: component failures test timeout machenism has problem for plasma manager failed case

* bugfix: arrow use lib64 in centos, travis check-git-clang-format-output.sh does not support other branches except master

* revert some fix

* set arrow python executable, fix format error in component_failures_test.py

* make clean arrow python build directory

* update cmake code style, back to support cmake minimum version 3.4
2018-09-12 11:19:33 -07:00