Commit graph

717 commits

Author SHA1 Message Date
Eric Liang
0aa9373d62
Revert "Removing Pyarrow dependency (#7146)" (#7209)
This reverts commit 2116fd3bca.
2020-02-18 14:12:06 -08:00
Eric Liang
5df801605e
Add ray.util package and move libraries from experimental (#7100) 2020-02-18 13:43:19 -08:00
ijrsvt
2116fd3bca
Removing Pyarrow dependency (#7146) 2020-02-17 18:00:13 -08:00
mehrdadn
3bd82d0bcd
Fix various issues/warnings that come up on Jenkins (#7147)
* Avoid warning about swap being unlimited

Currently we get the following message on Jenkins:
"Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap."

Since we're not limiting swap anyway, we might as well avoid trying to.
https://docs.docker.com/config/containers/resource_constraints/#--memory-swap-details

* Fix escaping in re.search()

* Fix escaping in _noisy_layer()

* Raise a more descriptive error when dashboard data isn't found

* Don't error on dashboard files not being found when webui isn't required

* Change dashboard error to a warning instead
2020-02-17 16:08:55 -08:00
Richard Liaw
94e2fcea2e
[sgd] fp16 (apex) and scheduler support + move examples page (#7061)
* Init fp16

* fp16 and schedulers

* scheduler linking and fp16

* to fp16

* loss scaling and documentation

* more documentation

* add tests, refactor config

* moredocs

* more docs

* fix logo, add test mode, add fp16 flag

* fix tests

* fix scheduler

* fix apex

* improve safety

* fix tests

* fix tests

* remove pin memory default

* rm

* fix

* Update doc/examples/doc_code/raysgd_torch_signatures.py

* fix

* migrate changes from other PR

* ok thanks

* pass

* signatures

* lint'

* Update python/ray/experimental/sgd/pytorch/utils.py

* Apply suggestions from code review

Co-Authored-By: Edward Oakes <ed.nmi.oakes@gmail.com>

* should address most comments

* comments

* fix this ci

* fix tests'

* testmode

Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com>
2020-02-16 19:04:08 -08:00
Eric Liang
b7016504e8
[rllib] Only run one set of tests unless rllib or tune dirs are changed. (#7179)
* full filter

* lint
2020-02-16 08:52:49 -08:00
Sven Mika
2e60f0d4d8
[RLlib] Move all jenkins RLlib-tests into bazel (rllib/BUILD). (#7178)
* commit

* comment
2020-02-15 14:50:44 -08:00
Simon Mo
30de1286bd
Use pip install setup.py (#7158) 2020-02-14 13:53:36 -08:00
Edward Oakes
b81b93a9c0
Convert stress tests to projects (#6495) 2020-02-13 09:19:24 -08:00
Sven Mika
5518a738b3
[RLlib] Fix erroneous use of LinearSchedule (in DDPG's exploration annealing). (#7125)
* Fix erroneous use of LinearSchedule (in DDPG's exploration annealing).
Erase schedules_obsoleted.py.

* Trigger re-test.

* Re-test.
2020-02-12 23:46:49 -08:00
Edward Oakes
275fd343fb
Change CI to properly list python3.6 (#7126) 2020-02-12 11:15:46 -08:00
mehrdadn
e09f63ad65
Fix build errors and add more targets to Windows builds (#6811)
* Fix common.fbs rename (due to apache/arrow/commit/bef9a1c251397311a6415d3dc362ef419d154caa)

* Add missing COPTS

* Use socketpair(AF_INET) if boost::asio::local is unavailable (e.g. on Windows)

* Fix compile bug in service_based_gcs_client_test.cc (fix build breakage in #6686)

* Work around googletest/gmock inability to specify override to avoid -Werror,-Winconsistent-missing-override

* Fix missing override on IsPlasmaBuffer()

* Fix missing libraries for streaming

* Factor out install-toolchains.sh

* Put some Bazel flags into .bazelrc

* Fix jni_md.h missing inclusion

* Add ~/bin to PATH for Bazel

* Change echo $$(date) > $@ to date > $@

* Fix lots of unquoted paths

* Add system() call checks for Windows

Co-authored-by: GitHub Web Flow <noreply@github.com>
2020-02-11 16:49:33 -08:00
Simon Mo
0ddc389830
Fix documentation building with psutil issue (#7077) 2020-02-11 10:00:29 -08:00
Simon Mo
bec92a8946
[Hotfix] Fix flake8 lint failing (#7118) 2020-02-10 19:57:21 -08:00
mehrdadn
ad1848b623
Comment iwyu.sh (#7110)
Co-authored-by: GitHub Web Flow <noreply@github.com>
2020-02-10 15:28:36 -08:00
Sven Mika
6e1c3ea824
[RLlib] Exploration API (+EpsilonGreedy sub-class). (#6974) 2020-02-10 15:22:07 -08:00
Simon Mo
f6c09ff614
Add serve stress test (#7076) 2020-02-10 09:37:39 -08:00
mehrdadn
ad4ac9aa70
Add clang-iwyu (#7081)
* Add iwyu

Co-authored-by: GitHub Web Flow <noreply@github.com>
2020-02-07 16:19:46 -08:00
ijrsvt
0826f95e1c
Including psutil & setproctitle (#7031) 2020-02-05 14:16:58 -08:00
Eric Liang
fbc545c03b
[rllib] Support parallel, parameterized evaluation (#6981)
* eval api

* update

* sync eval filters

* sync fix

* docs

* update

* docs

* update

* link

* nit

* doc updates

* format
2020-02-01 22:12:12 -08:00
SangBin Cho
c9f5def56a
Show lint download commands if tools not installed (#6984) 2020-01-31 10:42:09 -08:00
Ameer Haj Ali
b8135da122
Adding dependencies for scikit-learn in travis (#6969)
* Revert "Revert "Support of scikit-learn with ray joblib backend (#6925)" (#6957)"

This reverts commit 86100bc119.

* adding scikit-learn to dependencies
2020-01-30 09:46:54 -08:00
Amog Kamsetty
11d90d6d0c
Change files tested in Travis by changing git diff from 2-dot to 3-dot (#6960). 2020-01-30 09:26:44 -08:00
Richard Liaw
037aa2b961
[sgd] Refactor PyTorch SGD Documentation. (#6910)
* Refactor documentation and directory structurre

* update loss

* ,ore examples

* fix comments

* more code

* svgs

* formatting

* more_docs

* more writing

* comments ready

* move

* whitespace

* examples

* fix

* bold

* pytorch

* batch

* fix

* fix test

* Apply suggestions from code review

* quarantinegp

* tests/

* fix missing
2020-01-29 08:51:01 -08:00
Sven Mika
446cbdf2e0 [RLlib] Fix issue (bug): LSTM + non-shared vf + PPO + tuple actions (#6890)
* Add `RandomEnv` example to examples folder.
Convert warning into Error message when using an LSTM in a non-shared-vf network (after the warning, the program would crash).

* LINT.

* Fix issue #6884. LSTM + non-shared vf NN + PPO crashes when using a Tuple action space.

* LINT

* Change warning message for Model: shared_vf=False, LSTM=True cases.

* Bug fix.

* Add examples/random_env.py test to Jenkins.
2020-01-24 10:29:35 -08:00
Sven Mika
ae9a3a2237 [RLlib] from_config util method for framework agnostic components; start moving RLlib tests into Bazel. (#6865) 2020-01-22 17:02:58 -08:00
Sven Mika
c957ed58ed [RLlib] Implement PPO torch version. (#6826) 2020-01-20 23:06:50 -08:00
Edward Oakes
b750bd7fc9
Use 2xlarge instances in long running tests (#6802) 2020-01-15 19:47:59 -06:00
mehrdadn
4780b52ea8 Polish clang-format and let it run in a local repo (#6793) 2020-01-15 11:17:49 -08:00
Edward Oakes
fc473e6a08
Use us-west-2 for application stress tests (#6782) 2020-01-13 15:01:03 -06:00
chaokunyang
723fe86882 [Java] Fix building Java with maven (#6764)
* lint

* gen_maven_deps for ray_java_pkg
2020-01-11 14:26:27 +08:00
Sven
60d4d5e1aa Remove future imports (#6724)
* Remove all __future__ imports from RLlib.

* Remove (object) again from tf_run_builder.py::TFRunBuilder.

* Fix 2xLINT warnings.

* Fix broken appo_policy import (must be appo_tf_policy)

* Remove future imports from all other ray files (not just RLlib).

* Remove future imports from all other ray files (not just RLlib).

* Remove future import blocks that contain `unicode_literals` as well.
Revert appo_tf_policy.py to appo_policy.py (belongs to another PR).

* Add two empty lines before Schedule class.

* Put back __future__ imports into determine_tests_to_run.py. Fails otherwise on a py2/print related error.
2020-01-09 00:15:48 -08:00
Edward Oakes
5f843cd998
Clean up stress_testing_config.yaml (#6738)
* Clean up stress_testing_config.yaml

* comment
2020-01-07 17:05:07 -06:00
Sven
f1b56fa5ee PG unify/cleanup tf vs torch and PG functionality test cases (tf + torch). (#6650)
* Unifying the code for PGTrainer/Policy wrt tf vs torch.
Adding loss function test cases for the PGAgent (confirm equivalence of tf and torch).

* Fix LINT line-len errors.

* Fix LINT errors.

* Fix `tf_pg_policy` imports (formerly: `pg_policy`).

* Rename tf_pg_... into pg_tf_... following <alg>_<framework>_... convention, where ...=policy/loss/agent/trainer.
Retire `PGAgent` class (use PGTrainer instead).

* - Move PG test into agents/pg/tests directory.
- All test cases will be located near the classes that are tested and
  then built into the Bazel/Travis test suite.

* Moved post_process_advantages into pg.py (from pg_tf_policy.py), b/c
the function is not a tf-specific one.

* Fix remaining import errors for agents/pg/...

* Fix circular dependency in pg imports.

* Add pg tests to Jenkins test suite.
2020-01-02 16:08:03 -08:00
mehrdadn
f4b29dae9c Perform Bazel install directly in Windows CI (#6653) 2019-12-31 20:48:08 -08:00
Robert Nishihara
d2c6457832
Remove public facing references to --redis-address. (#6631) 2019-12-31 13:21:53 -08:00
Philipp Moritz
735f282494
Use 0.9.0.dev0 as the version tag (#6630) 2019-12-30 10:14:07 -08:00
Robert Nishihara
96f2f8ff10 Stop testing Python 2.7 and building Python 2.7 wheels. (#6601) 2019-12-27 20:47:49 -08:00
Robert Nishihara
eb0813ea35
Re-enable UI tests for wheels. (#6602) 2019-12-26 22:34:56 -08:00
Philipp Moritz
eaee672b7f
Revert "Perform Bazel install directly in Windows CI (#6529)" (#6593)
This reverts commit c5f141013b.
2019-12-24 16:39:07 -08:00
micafan
687de41273 [GCS] refactor the GCS Client Node Interface (#6010) 2019-12-24 20:36:37 +08:00
mehrdadn
c5f141013b Perform Bazel install directly in Windows CI (#6529) 2019-12-22 16:14:51 -08:00
Chaokun Yang
7bbfa85c66 [Streaming] Streaming data transfer java (#6474) 2019-12-22 10:56:05 +08:00
Simon Mo
26ec500ef9
Implement async get for direct actor call (#6339) 2019-12-18 11:50:21 -08:00
Eric Liang
6725a61bda
Release 0.8.0 test logs (#6512) 2019-12-17 15:56:50 -08:00
Eric Liang
1a1324d2a2
Bump version from 0.8.0.dev6 -> 0.9.0.dev (#6508) 2019-12-16 23:57:42 -08:00
Edward Oakes
b1e83d83d1
Print summaries for stress tests (#6498) 2019-12-16 14:14:48 -08:00
Mitchell Stern
1531c21dbd [Dashboard] Add remaining features from old dashboard (#6489)
* [Dashboard] Add remaining features from old dashboard

* Fix linting errors

* Set cluster uptime statistic to N/A

* Use proper singular or plural words for workers column

* Ignore .js, .jsx, .ts, .tsx files in check-git-clang-format-output.sh

* Fix bash quote issue
2019-12-16 11:21:18 -08:00
Richard Liaw
5719a05757
[sgd] Add support for multi-model multi-optimizer training (#6317) 2019-12-15 15:19:45 -08:00
Philipp Moritz
f5d10eea0b
[Projects] Refactor cluster specification (#6488) 2019-12-14 22:43:06 -08:00