Commit graph

507 commits

Author SHA1 Message Date
adgirish
3c48783a16 [DataFrame] Adding read methods and tests (#1712)
* Adding read methods and tests

* Referencing internal partition method so constructors are more canonical with Pandas

* Fixing to reference from_pandas in utils

* Cleaning up unused imports

* rerunning tests

* fixing flake8

* resolving errors

* Added sql and sas test

* updating

* Temporarily phasing out read_csv code for wrapper while diagnosing, added io tests to travis

* Adding travis

* restoring distributed read csv

* resolving rebases

* lint

* Sampling out HD test

* adding dep

* fix pathing

* Flagging out tests

* resolving read_method issues

* fix build issue

* move additional dependencies to extras

* fixing lint

* removing IO dependencies

* updated requirements doc
2018-04-20 18:33:08 -07:00
Jones Wong
c9a7744e52 [rllib] Contribute DDPG to RLlib (#1877)
*  ongoing ddpg

*  ongoing ddpg converged

*  gpu machine changes

*  tuned

*  tuned ddpg specification

*  ddpg

*  supplement missed optimizer argument clip_rewards in default DQN configuration

*  ddpg supports vision env (atari) now

*  revised according to code review comments

*  added regression test case

*  removed irrelevant files

*  validate ddpg on mountain_car_continuous

*  restore unnecessary slight changes

*  revised according to eric's comments

*  added the requested tests

*  revised accordingly

*  revised accordingly and re-validated

*  formatted by yapf

*  fix lint errors

*  formatted by yapf

*  fix lint errors

*  formatted by yapf

*  fix lint error
2018-04-19 22:36:29 -07:00
Stephanie Wang
aa07f1ce4e [xray] Workers blocked in a ray.get release their resources (#1920)
* [xray] Throttle task dispatch by required resources
* Pass in number of initial workers into raylet command
* Workers blocked in a ray.get release resources
2018-04-18 20:59:58 -07:00
Alexey Tumanov
1c965fcfeb Raylet task dispatch and throttling worker startup (#1912)
* separate task placement and task dispatch; throttle task dispatch with locally available resournces

* keep track of worker's being started/in flight and suppress starting extraneous workers

* cleanup comments

* remove early termination in task dispatch to support zero-resource actor tasks

* info -> debug

* add documentation

* linting

* mock the worker pool for testing

* some linting

* kill all workers in flight; clear the worker pool in dtor

* remove fixed todo

* lint
2018-04-18 10:58:11 -07:00
Omkar Salpekar
0728d4719b [DataFrame] Eval fix (#1903)
* eval now works without assignment - helper function a bit hacky

* removed df.copy() from eval_helper

* one test still failing for qury

* all eval tests passing now

* added check to eval arge verification

* added tests to travis

* added optimization and some comments

* added pd.eval and passes all tests

* added ray dataframe back to test file

* optimizations and code cleanup for eval

* changed position of pandas import in __init__

* fixed linting errors

* fixing eval in __init__.py

* fixed travis file - removed extra tests

* removed test directory from linting exclude for travis
2018-04-18 08:48:32 -07:00
Richard Liaw
f833e4da37
[tune] Polishing docs (#1846) 2018-04-17 09:57:35 -07:00
Eric Liang
7ab890f4a1 [tune] [rllib] Automatically determine RLlib resources and add queueing mechanism for autoscaling (#1848) 2018-04-16 16:58:15 -07:00
Eric Liang
ed8c0f1a38
[tune] Allow fetching pinned objects from trainable functions (#1895)
* updates

* lint

* Update util.py

* Update function_runner.py

* updates
2018-04-16 15:54:38 -07:00
Devin Petersohn
3c817ad908 Add slice functionality (#1832) 2018-04-16 08:50:56 -07:00
Patrick Yang
f505f0642f [DataFrame] Pass read_csv kwargs to _infer_column (#1894)
* pass kwargs to _infer_column

* adding small test for non-comma delim

* fix lint
2018-04-16 08:47:30 -07:00
Robert Nishihara
6ca2c2a609 Allow numpy arrays to be passed by value into tasks (and inlined in the task spec). (#1816)
* Allow numpy arrays and larger objects to be passed by value in task specifications.

* Fix bug.

* Fix bug. Inline all bug numpy object arrays.

* Increase size limit for inlining args in task spec.

* Give numpy init different signatures in Python 2 and Python 3.

* Simplify code.

* Fix test.

* Use import_array1 instead of import_array.
2018-04-15 20:36:01 -07:00
Robert Nishihara
7792032ee3 Fix UI issue for non-json-serializable task arguments. (#1892)
* Fix UI issue for non-json-serializable task arguments.

* Simplify approach.
2018-04-15 13:54:42 -07:00
Robert Nishihara
6f8b81d9e5 Allow multiple raylets to be started on a single machine. (#1904) 2018-04-15 13:51:19 -07:00
Peter Schafhalter
1d605e8f8a [DataFrame] Inherit documentation from Pandas (#1727)
* Added _inherit_docstrings

* DataFrame documentation inherits from Pandas

* Fix formatting

* Replace hasattr and document properties

* Fix rebase

* Override documentation for groupby

* Override documentation for series

* Don't overwrite property docstrings

* Fix property __doc__ for python2
2018-04-12 20:30:19 -07:00
Alexey Tumanov
39cf6ff6e1 raylet command line resource configuration plumbing (#1882)
* raylet command line resource configuration plumbing

* Small changes.
2018-04-12 02:37:15 -07:00
Alexey Tumanov
85d3963172 use raylet for remote ray nodes (#1880) 2018-04-11 22:06:46 -07:00
Eric Liang
4dc04374f6
[rllib] Propagate dim option to deepmind wrappers (#1876)
* updates

* updates
2018-04-11 21:38:06 -07:00
alvkao58
15a668dd12 [RLLib] DDPG (#1685) 2018-04-11 15:08:39 -07:00
Philipp Moritz
74162d1492 Lint Python files with Yapf (#1872) 2018-04-11 10:11:35 -07:00
Omkar Salpekar
a3ddde398c [DataFrame] Fixed repr, info, and memory_usage (#1874)
* working with dataframes with too many rows and columns

* repr works for jupyter notebooks now

* added comments and test file

* added repr test file to .travis.yml

* added back ray.dataframe as pd to test file

* fixed pandas importing issues in test file

* getting the front and back of df more efficiently

* only keeping dataframe tests in travis

* fixing numpy array for row and col lengths issue

* doesn't add dimensions if df is small enough

* implemented memory_usage()

* completed memory_usage - still failing 2 tests

* only failing one test for memory_usage

* all repr and dataframes tests passing now

* fixing error related to python2 in info()

* fixing python2 errors

* fixed linting errosr

* using _arithmetic_helper in memory_usage()

* fixed last lint error

* removed testing-specific code

* adding back travis test

* removing extra tests from travis

* re-added concat test

* fixes with new indexing scheme

* code cleanup

* fully working with new indexing scheme

* added tests for info and memory_usage

* removed test file
2018-04-11 08:07:07 -07:00
Devin Petersohn
806b2c844e Fix getattr compat (#1871) 2018-04-10 21:28:59 -07:00
Patrick Yang
521b549e4a [DataFrame] Encapsulate index and lengths into separate class (#1849)
* baseline impl for index_df.py

* added skeleton for index_df.py

* initial impl index_df

* separate out partition and non-partition impls

* add len function

* drop returns index_df slice of dropped indices

* housecleaning

* Integrate index overhaul

* Rename index df to index metadata

* Fix flake8 issues

* Addressing issues

* fix import issue

* Added metadata passing to constructor
2018-04-10 14:30:20 -07:00
Peter Schafhalter
405b05d58a [DataFrame] Implemented __getattr__ (#1753)
* __getattr__ accesses columns

* Added test
2018-04-10 10:19:33 -07:00
adgirish
efeaacbedc Adding support for concat (#1739)
adding tests

fixing flake8

adding init

flake 8 on test

fixing tests, imports, and flake8

handling for index

adding tests for row, index

added more robust error handling for axis

fixing test failures

cleaning up error sfor 2.7

updating travis

resolving import

fixing flake8

moved import order

Fixing to refactor and delaying implementing ray-pd inner concat

resolving ray-pd concat and from_pandas mutation

Revert "resolving ray-pd concat and from_pandas mutation"

This reverts commit 5db43e4e89e328286532f3ef98a4526575c5d08d.
2018-04-09 21:36:24 -07:00
Philipp Moritz
834e594709 [XRay] Register object store and raylet with the GCS (#1860) 2018-04-09 18:56:33 -07:00
Robert Nishihara
7c9e291b4b In the UI, display task breakdowns by default. (#1857) 2018-04-09 13:24:38 -07:00
Jones Wong
6b85d15bd9 supplement missed optimizer argument clip_rewards in default DQN configuration (#1853) 2018-04-08 17:58:05 -07:00
Robert Nishihara
256389dc59 Use new task spec for computing IDs in raylet code path. (#1830)
* Use new task spec for computing IDs in raylet code path.

* Fix linting.

* Fixes

* Fix test.
2018-04-08 13:31:55 -07:00
Eric Liang
79d37ce240
[rllib] Switch to use lz4 instead of snappy (#1847)
* lz4

* comment

* updates
2018-04-07 14:43:45 -07:00
Eric Liang
e6c00b2b5e
[tune] Add util function to broadcast objects (#1845)
* add util

* Fri Apr  6 15:09:20 PDT 2018

* doc

* Fri Apr  6 15:21:42 PDT 2018

* Fri Apr  6 15:28:07 PDT 2018

* Fri Apr  6 15:28:26 PDT 2018

* Update tune-config.rst

* Update tune-config.rst
2018-04-07 11:37:14 -07:00
Richard Liaw
bc8f62c947
[tune] Fix Median Stopping Rule Verbosity (#1833) 2018-04-06 22:58:13 -07:00
Stephanie Wang
bf194db4bc [xray] Basic actor support (#1835) 2018-04-06 00:17:14 -07:00
Stephanie Wang
cbf3181fd2 [xray] Monitor for Raylet processes (#1831)
* Add raylet monitor script to timeout Raylet heartbeats

* Unit test for removing a different client from the client table

* Set node manager heartbeat according to global config

* Doc and fixes

* Add regression test for client table disconnect, refactor client table

* Fix linting.
2018-04-05 20:45:38 -07:00
Devin Petersohn
0d9a7a3c19 [DataFrame] Update architecture to be more flexible and performant (#1821) 2018-04-05 15:14:33 -07:00
Robert Nishihara
5bde5e75e7 Implement unsafe method for flushing entire object table and task table. (#1824)
* Implement unsafe method for flushing entire object table and task table.

* Add test.

* Fix test.
2018-04-04 18:29:24 -07:00
Richard Liaw
888e70f1be
[tune] HyperOpt Support (v2) (#1763) 2018-04-04 11:08:26 -07:00
Robert Nishihara
fbfbb1c079 [xray] Integrate worker.py with raylet. (#1810)
* Integrate worker with raylet.

* Begin allowing worker to attach to cluster.

* Fix linting and documentation.

* Fix linting.

* Comment tests back in.

* Fix type of worker command.

* Remove xray python files and tests.

* Fix from rebase.

* Add test.

* Copy over raylet executable.

* Small cleanup.
2018-04-03 02:38:56 -07:00
Robert Nishihara
0fc989c6c1 Don't use 127.0.0.1 for local ip address. (#1596)
* Don't use 127.0.0.1 for ip address.

* Update test
2018-04-02 00:34:20 -07:00
Robert Nishihara
d3e974a9a4 Increase ulimit -n in autoscaler examples. (#1769) 2018-04-02 00:32:56 -07:00
Robert Nishihara
23b8793f0e Update documentation and autoscaler to find 0.4.0. (#1789) 2018-04-02 00:28:47 -07:00
Philipp Moritz
40c9b9cd60 Fix the setuptools_scm issue (#1784) 2018-03-31 10:33:40 -07:00
Eric Liang
faaa123046 [rllib] Set num_cpu=None for workers in the default settings (#1793) 2018-03-29 16:33:40 -07:00
Eric Liang
4116c64698
[tune] Remove rllib dep again, and add a test (#1792)
* tune should not depend on rllib

* fix dep test

* Tue Mar 27 16:55:41 PDT 2018

* f401
2018-03-29 15:36:49 -07:00
Robert Nishihara
8d52fe931b Add experimental feature for flushing event logs and logfiles. (#1659)
* Add experimental feature for flushing event logs and logfiles.

* Add documentation.
2018-03-27 11:57:52 -07:00
Robert Nishihara
f69cbd35d4 Bump version to 0.4.0. (#1745) 2018-03-26 22:37:16 -07:00
Robert Nishihara
de3cfa223d Fix monitor.py bottleneck by removing excess Redis queries. (#1786)
* Fix monitor.py bottleneck by removing excess Redis queries.

* Remove unnecessary default value.
2018-03-26 22:30:38 -07:00
Robert Nishihara
1ab0d0ea69 Acquire worker lock when importing actor. (#1783) 2018-03-26 18:31:26 -07:00
Eric Liang
7c4afa4b04 [tune] Fix linting error (#1777) 2018-03-25 23:44:14 -07:00
Yan Facai (颜发才)
6b1e592d5c [tune] Added pbt with keras on cifar10 dataset example (#1729)
* [tune] Added pbt with keras on cifar10 dataset example

* ENH: add gpu resources

* CLN: requires 4 GPUs resource

* CLN: use single quotes

* CLN: don't save model by default
2018-03-25 15:57:23 -07:00
Eric Liang
72595cca0d [tune] Change tune resource request syntax to be less confusing (#1764)
* update

* update examples

* Wed Mar 21 15:19:56 PDT 2018

* Wed Mar 21 15:21:32 PDT 2018

* Update train_a3c.py

* Update train.py

* fix resources accounting
2018-03-23 06:25:01 -07:00