Commit graph

239 commits

Author SHA1 Message Date
Alok Singh
42a9233e1d Improve yapf speed and document its usage (#2160)
* Allow yapf to lint individual files

* Add tip for using yapf

* Update doc

* Update script to autoformat changed py files

The new default is for the script to only updated changed files to encourage
using it as a pre-push hook. Travis still checks all since it's not that big an
increase to runtime.

* Exclude formatting thirdparty/autogen py files

* Symlink .travis -> scripts

Hidden directories may get glossed over otherwise.

* .travis -> scripts in docs

They are symlinks to the same thing, but `scripts` is more dev-friendly, while
`.travis` is really only for Travis CI.

* Document different yapf format functions

Most devs will only need `format_changed`, and this is run by default.
`format_changed` should be fast enough in most cases to work as a pre-commit
hook.

* Speed up yapf by only formatting changed files

* Update docs

1. Mention how yapf can be used a pre-commit hook
2. rm `bash`, script is executable

* Update yapf.sh

* Update development.rst

* Update yapf.sh

* Use bash arrays for correct argument splitting

Playing fast and loose with whitespace in bash is a terrible idea.

* Only format non-excluded by default

* Check changes against master

Normally, the remote is called `origin`, but naming it explicit

* Adding missing directory to `format_all`

* Cleanup YAPF code

Remove unused function and move around code to make clearer and adding lines
give cleaner diffs.

* Ensure correct files are autoformatted

* Fix cmd line arg splitting

Each arg has to be in its own set of quotes.

* Diff against mergebase

TIL there's a clean syntax for doing that, but it's too clever to belong in a
shell script.

We use `mapfile -t` to ensure no problems down the line with weird filenames.
2018-06-05 20:22:11 -07:00
songqing
4dd4698564 unify build dir for Python and Java (#2171)
* unify build dir for Python and Java

* enable executables auto installed when just running 'make'

* fix plasma_store copy error

* fix cmake error about copying executables

* lint fix

* recover python/setup.py

* enable to copy optional file automatically

* a small fix of path

* lint fix

* lint fix

* lint fix

* Add comment.
2018-06-01 16:28:27 -07:00
Robert Nishihara
6172f94c04 Implement Python global state API for xray. (#2125)
* Implement global state API for xray.

* Fix object table.

* Fixes for log structure.

* Implement cluster_resources.

* Add driver task to task table.

* Remove python flatbuffers code

* Get some global state API tests running.

* Python linting.

* Fix linting.

* Fix mock modules for doc

* Copy over flatbuffer bindings.

* Fix for tests.

* Linting

* Fix monitor crash.
2018-05-29 16:25:54 -07:00
Robert Nishihara
dc03506108 Update resource documentation (remove outdated limitations). (#2022) 2018-05-25 22:19:47 -07:00
Eric Liang
f37e2e5d2f
[rllib] [doc] Broken link in ddpg doc 2018-05-20 00:10:59 -07:00
Ken Fehling
19b743c84b Fixed attribute name in code example (#2054)
hyperparam_mutations
2018-05-14 01:05:06 -07:00
Ken Fehling
4ff900e131 Added missing comma to code example (#2050) 2018-05-13 19:01:01 -07:00
Aris L
041c37506e Fix error in api.rst. (#2048)
Fix error in api.rst.
2018-05-12 09:35:45 -07:00
Eric Liang
b55f4a7f04 [rllib] Fix broken link in docs (#1967)
* Update README.rst

* Update rllib.rst
2018-04-30 16:02:48 -07:00
Eric Liang
47bc4c3009
[rllib] Add DDPG documentation, rename DDPG2 <=> DDPG (#1946)
* updates

* updates

* updates

* updates

* updates

* updates

* Update rllib.rst

* Update policy-optimizers.rst
2018-04-30 00:18:15 -07:00
Robert Nishihara
3c76461b22 Remove smart_open install. (#1943) 2018-04-23 23:18:09 -07:00
Richard Liaw
f833e4da37
[tune] Polishing docs (#1846) 2018-04-17 09:57:35 -07:00
Eric Liang
7ab890f4a1 [tune] [rllib] Automatically determine RLlib resources and add queueing mechanism for autoscaling (#1848) 2018-04-16 16:58:15 -07:00
Richard Liaw
e82bea40b1 Add better analytics to docs (#1854) 2018-04-10 00:51:44 -07:00
Eric Liang
e6c00b2b5e
[tune] Add util function to broadcast objects (#1845)
* add util

* Fri Apr  6 15:09:20 PDT 2018

* doc

* Fri Apr  6 15:21:42 PDT 2018

* Fri Apr  6 15:28:07 PDT 2018

* Fri Apr  6 15:28:26 PDT 2018

* Update tune-config.rst

* Update tune-config.rst
2018-04-07 11:37:14 -07:00
Richard Liaw
888e70f1be
[tune] HyperOpt Support (v2) (#1763) 2018-04-04 11:08:26 -07:00
Robert Nishihara
fbfbb1c079 [xray] Integrate worker.py with raylet. (#1810)
* Integrate worker with raylet.

* Begin allowing worker to attach to cluster.

* Fix linting and documentation.

* Fix linting.

* Comment tests back in.

* Fix type of worker command.

* Remove xray python files and tests.

* Fix from rebase.

* Add test.

* Copy over raylet executable.

* Small cleanup.
2018-04-03 02:38:56 -07:00
Robert Nishihara
23b8793f0e Update documentation and autoscaler to find 0.4.0. (#1789) 2018-04-02 00:28:47 -07:00
Eric Liang
72595cca0d [tune] Change tune resource request syntax to be less confusing (#1764)
* update

* update examples

* Wed Mar 21 15:19:56 PDT 2018

* Wed Mar 21 15:21:32 PDT 2018

* Update train_a3c.py

* Update train.py

* fix resources accounting
2018-03-23 06:25:01 -07:00
Richard Liaw
23954e7ce2
[tune] Tune Documentation and expose better API (#1681) 2018-03-19 12:55:10 -07:00
Christian Barra
070e27ea7a Add external module as a node scaler. (#1703)
* WIP: add external module as a node scaler.

* Fix style.

* Add tests, fix style issues.

* Fix typos.

* Fix test error.

* Fix node provider path.

* Add function to spli pkg from class.

* Add doc.

* Correct documentation.

* Debugging....

* Debugging....

* Add __init__.py to tests.

* add more output for debugging

* Add more test, fix error with import.

* Add a small detail to the documentation.

* Update autoscaler.py
2018-03-17 16:59:13 -07:00
Eric Liang
e3685fca5e
[rllib] remove redundant docs (#1728)
* wip

* more work

* fix apex

* docs

* apex doc

* pool comment

* clean up

* make wrap stack pluggable

* Mon Mar 12 21:45:50 PDT 2018

* clean up comment

* table

* Mon Mar 12 22:51:57 PDT 2018

* Mon Mar 12 22:53:05 PDT 2018

* Mon Mar 12 22:55:03 PDT 2018

* Mon Mar 12 22:56:18 PDT 2018

* Mon Mar 12 22:59:54 PDT 2018

* Update apex_optimizer.py

* Update index.rst

* Update README.rst

* Update README.rst

* comments

* Wed Mar 14 19:01:02 PDT 2018

* Fri Mar 16 15:44:27 PDT 2018
2018-03-17 14:45:04 -07:00
Robert Nishihara
96913be939 Treat actor creation like a regular task. (#1668)
* Treat actor creation like a regular task.

* Small cleanups.

* Change semantics of actor resource handling.

* Bug fix.

* Minor linting

* Bug fix

* Fix jenkins test.

* Fix actor tests

* Some cleanups

* Bug fix

* Fix bug.

* Remove cached actor tasks when a driver is removed.

* Add more info to taskspec in global state API.

* Fix cyclic import bug in tune.

* Fix

* Fix linting.

* Fix linting.

* Don't schedule any tasks (especially actor creaiton tasks) on local schedulers with 0 CPUs.

* Bug fix.

* Add test for 0 CPU case

* Fix linting

* Address comments.

* Fix typos and add comment.

* Add assertion and fix test.
2018-03-16 11:18:07 -07:00
Eric Liang
882a649f0c
[rllib] [docs] Cleanup RLlib API and make docs consistent with upcoming blog post (#1708)
* wip

* more work

* fix apex

* docs

* apex doc

* pool comment

* clean up

* make wrap stack pluggable

* Mon Mar 12 21:45:50 PDT 2018

* clean up comment

* table

* Mon Mar 12 22:51:57 PDT 2018

* Mon Mar 12 22:53:05 PDT 2018

* Mon Mar 12 22:55:03 PDT 2018

* Mon Mar 12 22:56:18 PDT 2018

* Mon Mar 12 22:59:54 PDT 2018

* Update apex_optimizer.py

* Update index.rst

* Update README.rst

* Update README.rst

* comments

* Wed Mar 14 19:01:02 PDT 2018
2018-03-15 15:57:31 -07:00
Devin Petersohn
c19c2a4e60 [DataFrame] readthedocs page for Pandas on Ray (#1714) 2018-03-13 22:23:50 -07:00
Robert Nishihara
15a4392156 Add instructions for pip installing the latest wheel. (#1672) 2018-03-12 00:52:00 -07:00
Philipp Moritz
5ef0892236 Compile boost from source to fix macOS wheels (#1688) 2018-03-08 23:22:23 -08:00
Eric Liang
d85274a12e [docs] update to expose libraries + landing page (#1642) 2018-03-08 09:18:09 -08:00
Richard Liaw
162d063f0d
[autoscaler/tune] Optional YAML Fields + Fix Pretty Printing for Tune (#1541) 2018-03-04 23:35:58 -08:00
Richard Liaw
061e435411
[rllib] Fix eval.py -> rollout.py (#1650) 2018-03-04 14:59:16 -08:00
Richard Liaw
78716094b5
[tune] Async Hyperband (#1595) 2018-03-04 14:05:56 -08:00
Eric Liang
80d7def9dc
[autoscaler] [tune] More doc fixes (#1560)
* Fri Feb 16 13:53:50 PST 2018

* Sat Feb 17 15:32:08 PST 2018

* Sat Feb 17 15:44:59 PST 2018

* fix

* Sun Feb 18 14:46:24 PST 2018

* Sun Feb 18 14:46:37 PST 2018

* Sun Feb 18 14:55:52 PST 2018

* Sun Feb 18 15:14:32 PST 2018

* Wed Feb 21 17:34:17 PST 2018

* Sun Feb 25 17:51:17 PST 2018

* Sun Feb 25 22:18:40 PST 2018

* Wed Feb 28 13:19:05 PST 2018

* Wed Feb 28 13:22:13 PST 2018

* Wed Feb 28 13:33:29 PST 2018

* Wed Feb 28 13:35:33 PST 2018

* add ex

* Fri Mar  2 12:50:17 PST 2018

* Fri Mar  2 12:54:31 PST 2018
2018-03-03 13:01:49 -08:00
Philipp Moritz
2026c147ec say which port is local and which one is remote (#1591) 2018-02-25 10:19:12 -08:00
Robert Nishihara
5859a2d249 Replace python setup.py install with pip install -e . (#1460) 2018-02-22 11:15:03 -08:00
Robert Nishihara
db4a920bdb Cleanup parquet installation. (#1549)
* Cleanup parquet installation.

* Fix

* Small changes.

* Add brew installs

* Modify paths for compilation of parquet.

* Remove LD_LIBRARY_PATH

* Don't set unnecessary environment variables on Linux.

* Set environment variables for make.

* Brew installs for macos wheels.

* Update

* Pass PARQUET_HOME when building pyarrow.

* Don't exit with error code.
2018-02-20 15:21:32 -08:00
Richard Liaw
1cd2703cac
[autoscaler] Docker Support (#1505) 2018-02-20 00:24:01 -08:00
Eric Liang
09b29c267d [autoscaler] some doc updates (#1550) 2018-02-18 00:53:05 -08:00
William Paul
f2b6a7b58d Polished TensorFlowVariables code and documentation (#566) 2018-02-12 15:38:58 -08:00
Eric Liang
ca0f08d100 [tune] Recover experiments from last checkpoint (#1532) 2018-02-12 14:01:19 -08:00
eugenevinitsky
639df85fda updated multiagent docs (#1523)
* updated multiagent docs

* Update rllib.rst

* Update multiagent_mountaincar_env.py

* Update multiagent_pendulum_env.py
2018-02-11 16:35:03 -08:00
alvkao58
81a4be8f65 [rllib] Added vanilla policy gradient (#1497) 2018-02-10 13:54:51 -08:00
Eric Liang
41007722f9 [tune] improve s3 log sync (#1511) 2018-02-08 00:42:25 -08:00
Eric Liang
b948405532
[tune] clean up population based training prototype (#1478)
* patch up pbt

* Sat Jan 27 01:00:03 PST 2018

* Sat Jan 27 01:04:14 PST 2018

* Sat Jan 27 01:04:21 PST 2018

* Sat Jan 27 01:15:15 PST 2018

* Sat Jan 27 01:15:42 PST 2018

* Sat Jan 27 01:16:14 PST 2018

* Sat Jan 27 01:38:42 PST 2018

* Sat Jan 27 01:39:21 PST 2018

* add pbt

* Sat Jan 27 01:41:19 PST 2018

* Sat Jan 27 01:44:21 PST 2018

* Sat Jan 27 01:45:46 PST 2018

* Sat Jan 27 16:54:42 PST 2018

* Sat Jan 27 16:57:53 PST 2018

* clean up test

* Sat Jan 27 18:01:15 PST 2018

* Sat Jan 27 18:02:54 PST 2018

* Sat Jan 27 18:11:18 PST 2018

* Sat Jan 27 18:11:55 PST 2018

* Sat Jan 27 18:14:09 PST 2018

* review

* try out a ppo example

* some tweaks to ppo example

* add postprocess hook

* Sun Jan 28 15:00:40 PST 2018

* clean up custom explore fn

* Sun Jan 28 15:10:21 PST 2018

* Sun Jan 28 15:14:53 PST 2018

* Sun Jan 28 15:17:04 PST 2018

* Sun Jan 28 15:33:13 PST 2018

* Sun Jan 28 15:56:40 PST 2018

* Sun Jan 28 15:57:36 PST 2018

* Sun Jan 28 16:00:35 PST 2018

* Sun Jan 28 16:02:58 PST 2018

* Sun Jan 28 16:29:50 PST 2018

* Sun Jan 28 16:30:36 PST 2018

* Sun Jan 28 16:31:44 PST 2018

* improve tune doc

* concepts

* update humanoid

* Fri Feb  2 18:03:33 PST 2018

* fix example

* show error file
2018-02-02 23:03:12 -08:00
Eric Liang
8c96c34bb3 [tune] fix doc typo and also make sure to clean "/" from config (#1476) 2018-01-26 21:51:07 -08:00
Zongheng Yang
782b4aeb0f Document how to profile Ray using pprof. (#1464)
* Document how to profile Ray using pprof.

* Link to profiling.rst via main doc site; fix lint

* Some changes and clarifications.
2018-01-25 21:40:52 -08:00
Richard Liaw
f3d2dc0ad4
[tune] Fix Docs (#1469) 2018-01-25 16:39:00 -08:00
Robert Nishihara
ab5d4a6010 Bring cloudpickle inside the repository. (#1445)
* Bring cloudpickle version 0.5.2 inside the repo.

* Use internal copy of cloudpickle everywhere.

* Fix linting.

* Import ordering.

* Change __init__.py.

* Set pickler in serialization context.

* Don't check ray location.
2018-01-25 11:36:37 -08:00
Eric Liang
173f1d629a
[tune] Ray Tune API cleanup (#1454)
Remove rllib dep: trainable is now a standalone abstract class that can be easily subclassed.

Clean up hyperband: fix debug string and add an example.

Remove YAML api / ScriptRunner: this was never really used.

Move ray.init() out of run_experiments(): This provides greater flexibility and should be less confusing since there isn't an implicit init() done there. Note that this is a breaking API change for tune.
2018-01-24 16:55:17 -08:00
Richard Liaw
a7d544424c [tune] Experiment Management API (#1328)
* init for exposing external interface

* revisions

* http server

* small

* simplify

* ui

* fixes

* test

* nit

* nit

* merge

* untested

* nits

* nit

* init tests

* tests

* more tests

* nit

* fix hyperband

* cleanup

* nits

* good stuff

* cleanup

* comments and need to test

* nit

* notebook

* testing

* test and expose server

* server_tests

* docs

* periods

* fix tests

* committing test

* fi
2018-01-24 13:45:10 -08:00
Devin Petersohn
21a916009e Adding documentation for building from current master without cloning. (#1455)
* Updating docs with pip way to build from source

* wording
2018-01-23 14:47:30 -08:00