Commit graph

493 commits

Author SHA1 Message Date
Alexey Tumanov
39cf6ff6e1 raylet command line resource configuration plumbing (#1882)
* raylet command line resource configuration plumbing

* Small changes.
2018-04-12 02:37:15 -07:00
Alexey Tumanov
85d3963172 use raylet for remote ray nodes (#1880) 2018-04-11 22:06:46 -07:00
Eric Liang
4dc04374f6
[rllib] Propagate dim option to deepmind wrappers (#1876)
* updates

* updates
2018-04-11 21:38:06 -07:00
alvkao58
15a668dd12 [RLLib] DDPG (#1685) 2018-04-11 15:08:39 -07:00
Philipp Moritz
74162d1492 Lint Python files with Yapf (#1872) 2018-04-11 10:11:35 -07:00
Omkar Salpekar
a3ddde398c [DataFrame] Fixed repr, info, and memory_usage (#1874)
* working with dataframes with too many rows and columns

* repr works for jupyter notebooks now

* added comments and test file

* added repr test file to .travis.yml

* added back ray.dataframe as pd to test file

* fixed pandas importing issues in test file

* getting the front and back of df more efficiently

* only keeping dataframe tests in travis

* fixing numpy array for row and col lengths issue

* doesn't add dimensions if df is small enough

* implemented memory_usage()

* completed memory_usage - still failing 2 tests

* only failing one test for memory_usage

* all repr and dataframes tests passing now

* fixing error related to python2 in info()

* fixing python2 errors

* fixed linting errosr

* using _arithmetic_helper in memory_usage()

* fixed last lint error

* removed testing-specific code

* adding back travis test

* removing extra tests from travis

* re-added concat test

* fixes with new indexing scheme

* code cleanup

* fully working with new indexing scheme

* added tests for info and memory_usage

* removed test file
2018-04-11 08:07:07 -07:00
Devin Petersohn
806b2c844e Fix getattr compat (#1871) 2018-04-10 21:28:59 -07:00
Patrick Yang
521b549e4a [DataFrame] Encapsulate index and lengths into separate class (#1849)
* baseline impl for index_df.py

* added skeleton for index_df.py

* initial impl index_df

* separate out partition and non-partition impls

* add len function

* drop returns index_df slice of dropped indices

* housecleaning

* Integrate index overhaul

* Rename index df to index metadata

* Fix flake8 issues

* Addressing issues

* fix import issue

* Added metadata passing to constructor
2018-04-10 14:30:20 -07:00
Peter Schafhalter
405b05d58a [DataFrame] Implemented __getattr__ (#1753)
* __getattr__ accesses columns

* Added test
2018-04-10 10:19:33 -07:00
adgirish
efeaacbedc Adding support for concat (#1739)
adding tests

fixing flake8

adding init

flake 8 on test

fixing tests, imports, and flake8

handling for index

adding tests for row, index

added more robust error handling for axis

fixing test failures

cleaning up error sfor 2.7

updating travis

resolving import

fixing flake8

moved import order

Fixing to refactor and delaying implementing ray-pd inner concat

resolving ray-pd concat and from_pandas mutation

Revert "resolving ray-pd concat and from_pandas mutation"

This reverts commit 5db43e4e89e328286532f3ef98a4526575c5d08d.
2018-04-09 21:36:24 -07:00
Philipp Moritz
834e594709 [XRay] Register object store and raylet with the GCS (#1860) 2018-04-09 18:56:33 -07:00
Robert Nishihara
7c9e291b4b In the UI, display task breakdowns by default. (#1857) 2018-04-09 13:24:38 -07:00
Jones Wong
6b85d15bd9 supplement missed optimizer argument clip_rewards in default DQN configuration (#1853) 2018-04-08 17:58:05 -07:00
Robert Nishihara
256389dc59 Use new task spec for computing IDs in raylet code path. (#1830)
* Use new task spec for computing IDs in raylet code path.

* Fix linting.

* Fixes

* Fix test.
2018-04-08 13:31:55 -07:00
Eric Liang
79d37ce240
[rllib] Switch to use lz4 instead of snappy (#1847)
* lz4

* comment

* updates
2018-04-07 14:43:45 -07:00
Eric Liang
e6c00b2b5e
[tune] Add util function to broadcast objects (#1845)
* add util

* Fri Apr  6 15:09:20 PDT 2018

* doc

* Fri Apr  6 15:21:42 PDT 2018

* Fri Apr  6 15:28:07 PDT 2018

* Fri Apr  6 15:28:26 PDT 2018

* Update tune-config.rst

* Update tune-config.rst
2018-04-07 11:37:14 -07:00
Richard Liaw
bc8f62c947
[tune] Fix Median Stopping Rule Verbosity (#1833) 2018-04-06 22:58:13 -07:00
Stephanie Wang
bf194db4bc [xray] Basic actor support (#1835) 2018-04-06 00:17:14 -07:00
Stephanie Wang
cbf3181fd2 [xray] Monitor for Raylet processes (#1831)
* Add raylet monitor script to timeout Raylet heartbeats

* Unit test for removing a different client from the client table

* Set node manager heartbeat according to global config

* Doc and fixes

* Add regression test for client table disconnect, refactor client table

* Fix linting.
2018-04-05 20:45:38 -07:00
Devin Petersohn
0d9a7a3c19 [DataFrame] Update architecture to be more flexible and performant (#1821) 2018-04-05 15:14:33 -07:00
Robert Nishihara
5bde5e75e7 Implement unsafe method for flushing entire object table and task table. (#1824)
* Implement unsafe method for flushing entire object table and task table.

* Add test.

* Fix test.
2018-04-04 18:29:24 -07:00
Richard Liaw
888e70f1be
[tune] HyperOpt Support (v2) (#1763) 2018-04-04 11:08:26 -07:00
Robert Nishihara
fbfbb1c079 [xray] Integrate worker.py with raylet. (#1810)
* Integrate worker with raylet.

* Begin allowing worker to attach to cluster.

* Fix linting and documentation.

* Fix linting.

* Comment tests back in.

* Fix type of worker command.

* Remove xray python files and tests.

* Fix from rebase.

* Add test.

* Copy over raylet executable.

* Small cleanup.
2018-04-03 02:38:56 -07:00
Robert Nishihara
0fc989c6c1 Don't use 127.0.0.1 for local ip address. (#1596)
* Don't use 127.0.0.1 for ip address.

* Update test
2018-04-02 00:34:20 -07:00
Robert Nishihara
d3e974a9a4 Increase ulimit -n in autoscaler examples. (#1769) 2018-04-02 00:32:56 -07:00
Robert Nishihara
23b8793f0e Update documentation and autoscaler to find 0.4.0. (#1789) 2018-04-02 00:28:47 -07:00
Philipp Moritz
40c9b9cd60 Fix the setuptools_scm issue (#1784) 2018-03-31 10:33:40 -07:00
Eric Liang
faaa123046 [rllib] Set num_cpu=None for workers in the default settings (#1793) 2018-03-29 16:33:40 -07:00
Eric Liang
4116c64698
[tune] Remove rllib dep again, and add a test (#1792)
* tune should not depend on rllib

* fix dep test

* Tue Mar 27 16:55:41 PDT 2018

* f401
2018-03-29 15:36:49 -07:00
Robert Nishihara
8d52fe931b Add experimental feature for flushing event logs and logfiles. (#1659)
* Add experimental feature for flushing event logs and logfiles.

* Add documentation.
2018-03-27 11:57:52 -07:00
Robert Nishihara
f69cbd35d4 Bump version to 0.4.0. (#1745) 2018-03-26 22:37:16 -07:00
Robert Nishihara
de3cfa223d Fix monitor.py bottleneck by removing excess Redis queries. (#1786)
* Fix monitor.py bottleneck by removing excess Redis queries.

* Remove unnecessary default value.
2018-03-26 22:30:38 -07:00
Robert Nishihara
1ab0d0ea69 Acquire worker lock when importing actor. (#1783) 2018-03-26 18:31:26 -07:00
Eric Liang
7c4afa4b04 [tune] Fix linting error (#1777) 2018-03-25 23:44:14 -07:00
Yan Facai (颜发才)
6b1e592d5c [tune] Added pbt with keras on cifar10 dataset example (#1729)
* [tune] Added pbt with keras on cifar10 dataset example

* ENH: add gpu resources

* CLN: requires 4 GPUs resource

* CLN: use single quotes

* CLN: don't save model by default
2018-03-25 15:57:23 -07:00
Eric Liang
72595cca0d [tune] Change tune resource request syntax to be less confusing (#1764)
* update

* update examples

* Wed Mar 21 15:19:56 PDT 2018

* Wed Mar 21 15:21:32 PDT 2018

* Update train_a3c.py

* Update train.py

* fix resources accounting
2018-03-23 06:25:01 -07:00
Robert Nishihara
10dabce4d7 Remove from X import Y convention in RLlib ES. (#1774) 2018-03-23 05:54:31 -07:00
Christian Barra
13b3df9321 Check if the provider is external before getting the config. (#1743) 2018-03-22 22:59:29 -07:00
Robert Nishihara
c6ad71fc9d Fix bug when connecting another driver in local case. (#1760)
* Allow connecting another driver when using ip address 127.0.0.1.

* Add test.
2018-03-21 11:49:53 -07:00
Eric Liang
b41bdcefa0
[rllib] Update RLlib to work with new actor scheduling behavior (#1754)
* Mon Mar 19 21:23:01 PDT 2018

* Mon Mar 19 21:23:07 PDT 2018

* Mon Mar 19 21:30:49 PDT 2018

* Mon Mar 19 21:32:05 PDT 2018

* Mon Mar 19 21:35:43 PDT 2018

* fix cpu limits

* Mon Mar 19 22:25:07 PDT 2018
2018-03-20 19:29:52 -07:00
Robert Nishihara
4bccabd910 Redirect output of all processes by default. (#1752)
* Redirect output of all processes by default.

* Add separate flag for redirecting worker output.

* Fix tests.
2018-03-20 18:14:54 -07:00
Robert Nishihara
2922e1c388 Add API for getting total cluster resources. (#1736)
* Add API for getting total cluster resources.

* Add test.
2018-03-20 15:57:00 -07:00
Robert Nishihara
4658d0a180 Print error when actor takes too long to start, and refactor error me… (#1747)
* Print error when actor takes too long to start, and refactor error message pushing.

* Print warning every ten seconds.

* Fix linting and tests.

* Fix tests.
2018-03-19 20:24:35 -07:00
Robert Nishihara
f88a2544bf Speed up actor creation task submission by generating IDs with uuid. (#1744)
* Speed up actor creation task submission by generating IDs deterministically.

* Revert "Speed up actor creation task submission by generating IDs deterministically."

This reverts commit 175d9587302664916ce9db4071185485da8da041.

* Don't generate actor IDs deterministically yet.

* Factor out ID generation method.
2018-03-19 19:32:46 -07:00
Robert Nishihara
d78de0d41f Provide experimental API for changing number of return values and res… (#1735)
* Provide experimental API for changing number of return values and resource requirements at task submission time.

* Remove code duplication and add tests.
2018-03-19 15:32:23 -07:00
Richard Liaw
23954e7ce2
[tune] Tune Documentation and expose better API (#1681) 2018-03-19 12:55:10 -07:00
Philipp Moritz
7b493aa4a1 Register credis with redis (#1730) 2018-03-18 14:02:19 -07:00
Christian Barra
070e27ea7a Add external module as a node scaler. (#1703)
* WIP: add external module as a node scaler.

* Fix style.

* Add tests, fix style issues.

* Fix typos.

* Fix test error.

* Fix node provider path.

* Add function to spli pkg from class.

* Add doc.

* Correct documentation.

* Debugging....

* Debugging....

* Add __init__.py to tests.

* add more output for debugging

* Add more test, fix error with import.

* Add a small detail to the documentation.

* Update autoscaler.py
2018-03-17 16:59:13 -07:00
Eric Liang
e3685fca5e
[rllib] remove redundant docs (#1728)
* wip

* more work

* fix apex

* docs

* apex doc

* pool comment

* clean up

* make wrap stack pluggable

* Mon Mar 12 21:45:50 PDT 2018

* clean up comment

* table

* Mon Mar 12 22:51:57 PDT 2018

* Mon Mar 12 22:53:05 PDT 2018

* Mon Mar 12 22:55:03 PDT 2018

* Mon Mar 12 22:56:18 PDT 2018

* Mon Mar 12 22:59:54 PDT 2018

* Update apex_optimizer.py

* Update index.rst

* Update README.rst

* Update README.rst

* comments

* Wed Mar 14 19:01:02 PDT 2018

* Fri Mar 16 15:44:27 PDT 2018
2018-03-17 14:45:04 -07:00
Richard Liaw
9b361115c3
[tune] Added Async HyperBand example (#1709) 2018-03-16 13:25:29 -07:00