Commit graph

28 commits

Author SHA1 Message Date
Robert Nishihara
89db7841d2 Update arrow version. (#1512) 2018-02-07 23:05:16 -08:00
Robert Nishihara
5acc98e629 Update arrow with better dataframe serialization and get rid of custo… (#1413)
* Update arrow with better dataframe serialization and get rid of custom dataframe serializers.

* Update plasma client API.

* Fix potential bug.

* Bug fix.

* Update arrow to use deduplicated file descriptors and mutable buffers.

* Fix tests.

* Update commit.

* Update commit.

* Update commit.

* Update commit.

* Update commit

* Update commit back to arrow codebase.'
2018-01-24 10:03:29 -08:00
Robert Nishihara
e970e24ea5 Update arrow, and pass memcopy_threads into put. (#1374) 2017-12-31 13:32:06 -08:00
Philipp Moritz
3d224c4edf Second Part of Internal API Refactor (#1326) 2017-12-26 16:22:04 -08:00
Melih Elibol
4a2d62e7ef fix thirdparty install bug. (#1354) 2017-12-20 23:08:53 -08:00
Philipp Moritz
3c4408cf51 Rebase Ray on Arrow 0.8 (#1323)
* rebase Ray on Arrow 0.8

* rebase on apache repo
2017-12-19 14:24:21 -08:00
Robert Nishihara
94423c0542 Upgrade Arrow with fixes to Plasma eviction policy. (#1228)
* Upgrade Arrow with fixes to Plasma eviction policy.

* Upgrade arrow to have -f flag for plasma store.
2017-11-17 14:41:22 -08:00
Stephanie Wang
c70430f322 Fix bugs in plasma manager transfer (#1188)
* Plasma client test for plasma abort

* Use ray-project/arrow:abort-objects branch

* Set plasma manager connection cursor to -1 when not in use

* Handle transfer errors between plasma managers, abort unsealed objects

* Add TODO for local scheduler exiting on plasma manager death

* Revert "Plasma client test for plasma abort"

This reverts commit e00fbd58dc4a632f58383549b19fb9057b305a14.

* Upgrade arrow to version with PlasmaClient::Abort

* Fix plasma manager test

* Fix plasma test

* Temporarily use arrow fork for testing

* fix and set arrow commit

* Fix plasma test

* Fix plasma manager test and make write_object_chunk consistent with read_object_chunk

* style

* upgrade arrow
2017-11-15 22:32:38 -08:00
Robert Nishihara
97c6369b49 Update arrow to include custom serializer for pytorch and register default serialization handlers. (#1152)
* Update arrow to include custom serializer for pytorch.

* Call pyarrow function for registering default custom serialization handlers.

* Change class ID used in serialization context for object IDs.
2017-10-21 21:24:10 -07:00
Philipp Moritz
684e62e784 upgrade arrow to include numpy bool fix (#1148) 2017-10-20 17:25:15 -07:00
Philipp Moritz
4157bcb80b Improve deserialization performance by rebasing on latest arrow (#1129)
* improve serialization performance by rebasing on latest arrow

* update

* revert worker.py
2017-10-17 14:56:11 -07:00
Philipp Moritz
0684258d2e Update arrow to include pandas serialization (#1102)
* update arrow to include pandas serialization

* update
2017-10-10 22:16:35 -07:00
Robert Nishihara
8f1a73f041 Allow Ray to be built without UI by setting INCLUDE_UI=0. (#1094)
* Allow building Ray without UI by setting INCLUDE_UI=0.

* Fix bash.

* Fix linting.
2017-10-09 23:32:38 -07:00
Mitar
a0d3fb1de1 Fix Arrow's repository URL. (#1072)
Thanks!
2017-10-03 21:40:21 -07:00
Robert Nishihara
0dcf36c91e Switch Arrow commit. (#1068) 2017-10-03 13:56:53 -07:00
Philipp Moritz
57bd1d6ff5 Specialize Serialization for OrderedDict (#1035)
Specialize Serialization for OrderedDict and defaultdict
2017-10-02 17:33:10 -07:00
Philipp Moritz
79e013e876 upgrade to latest arrow to fix XCode 9 problem (#1042) 2017-09-30 16:24:59 -07:00
Eric Liang
b118cef49e [webui] Allow timeline scroll-to-zoom without holding ALT (#993)
* Allow timeline scroll-to-zoom without holding ALT

* Update build_ui.sh

* Update build_ui.sh

* Update build_ui.sh

* Update build_ui.sh

* Retry when getting catapult.
2017-09-29 21:35:12 -07:00
Robert Nishihara
116fe168b5 Download boost 1.65.1 from bintray. (#1019)
* Download boost 1.65.1 from bintray.

* Pass --no-check-certificate to wget.
2017-09-27 13:25:05 -07:00
Robert Nishihara
413140df38 Autogenerate catapult files if they are not already present. (#978)
* Autogenerate catapult files if they are not already present.

* Fix bash syntax.
2017-09-14 12:37:33 -07:00
Robert Nishihara
f3c1248d98 Clone catapult and generate html files during installation. (#956)
* Clone catapult and generate static html during setup.

* Include UI files in installation.

* Fix directory to clone catapult to and fix linting.

* Use absolute path.

* Make sure we find a sufficiently new version of python2 when building wheels.

* Copy the trace_viewer_full.html file to the local directory if it is not present.

* Make sure wheels fail to build if UI is not included.
2017-09-10 13:41:16 -07:00
Philipp Moritz
546ba23ceb Upgrade to latest arrow to include set serialization speedups (#957)
* update arrow to pull in the set serialization speedups

* remove _register_class for set
2017-09-10 00:12:17 -07:00
Philipp Moritz
7030ef366f Rebase Ray on latest arrow (remove numbuf from Ray). (#910)
* remove some stuff

* put get roundtrip working

* fixes

* more fixes

* cleanup

* fix tests

* latest arrow

* fixes

* fix tests

* fix linting

* rebase

* fixes

* fix bug

* bring back libgcc error

* fix linting

* use official arrow repo

* fixes
2017-09-04 22:58:49 -07:00
Robert Nishihara
d8010723d7 Attempt to wget boost up to 20 times during installation. (#927) 2017-09-04 14:42:29 -07:00
Robert Nishihara
be4beb19c1 Changes to build to fix creation of wheels. (#840)
* Pass DPYTHON_EXECUTABLE into cmake for arrow and for ray.

* Add cython to setup.py install_requires.

* Revert custom code for finding python in cmake.

* Correctly find arrow on CentOS.

* In cmake, don't find PythonLibs, just find PYTHON_INCLUDE_DIRS.

* Fix typo.

* Do not use boost shared libraries when building arrow.

* Add six to the setup.py install_requires because it is needed by pyarrow.

* Don't link numbuf against boost_system and boost_filesystem.

* Compile boost when we are on Linux.

* Make numbuf find the correct boost libraries.

* Only use find_package Boost on Linux, suppress output when building boost.

* Changes to wheel building scripts, install cython in mac script.

* Compile flatbuffers ourselves on Linux and pass it in when compiling Arrow.

* Clean up build_flatbuffers.sh and build_boost.sh scripts a little.

* Install cython when building linux wheel.
2017-08-21 17:49:35 -07:00
Robert Nishihara
03f2325780 Package pyarrow along with ray. (#822)
* Rough pass at installing pyarrow along with Ray.

* Remove hardcoded path and try to find correct path automatically.

* Add print.

* Fix linting.

* Copy pyarrow files to a location that we manually add to python path in order to avoid interfering with pre-existing pyarrow installations.

* Move call to build.sh back into build_ext in setup.py.

* Ignore some linting errors.

* Fix problem in which pyarrow files to copy were listed before they were built.

* Fix tests by importing ray before pyarrow.
2017-08-07 21:17:28 -07:00
Robert Nishihara
3071ba0070 Add correct Python executable to Path when building arrow. (#820)
* Tell cmake which python to use when building arrow.

* Pass different path into cmake when building arrow so that cmake finds the right python.

* Add correct python executable to PATH when running cmake for ray.
2017-08-07 14:47:34 -07:00
Philipp Moritz
c3b39b4d86 Pull Plasma from Apache Arrow and remove Plasma store from Ray. (#692)
* Rebase Ray on top of Plasma in Apache Arrow

* add thirdparty building scripts

* use rebased arrow

* fix

* fix build

* fix python visibility

* comment out C tests for now

* fix multithreading

* fix

* reduce logging

* fix plasma manager multithreading

* make sure old and new object IDs can coexist peacefully

* more rebasing

* update

* fixes

* fix

* install pyarrow

* install cython

* fix

* install newer cmake

* fix

* rebase on top of latest arrow

* getting runtest.py run locally (needed to comment out a test for that to work)

* work on plasma tests

* more fixes

* fix local scheduler tests

* fix global scheduler test

* more fixes

* fix python 3 bytes vs string

* fix manager tests valgrind

* fix documentation building

* fix linting

* fix c++ linting

* fix linting

* add tests back in

* Install without sudo.

* Set PKG_CONFIG_PATH in build.sh so that Ray can find plasma.

* Install pkg-config

* Link -lpthread, note that find_package(Threads) doesn't seem to work reliably.

* Comment in testGPUIDs in runtest.py.

* Set PKG_CONFIG_PATH when building pyarrow.

* Pull apache/arrow and not pcmoritz/arrow.

* Fix installation in docker image.

* adapt to changes of the plasma api

* Fix installation of pyarrow module.

* Fix linting.

* Use correct python executable to build pyarrow.
2017-07-31 21:04:15 -07:00