Commit graph

83 commits

Author SHA1 Message Date
matthewdeng
caa4ff3783
[train][datasets] update example and remove dask (#20592) 2021-11-21 17:06:44 -08:00
architkulkarni
42085fd3d5
[runtime env] [Doc] Add concepts and basic workflows (#20222)
Address followup comments from https://github.com/ray-project/ray/pull/19863
- Add short "Concepts" section
- Add more section headings to break up the text
- Add "Workflow: Local Files" example
- Add "Workflow: Library development" example
2021-11-19 13:58:50 -08:00
matthewdeng
e22632dabc
[train] wrap BackendExecutor in ray.remote() (#20123)
* [train] wrap BackendExecutor in ray.remote()

* wip

* fix trainer tests

* move CheckpointManager to Trainer

* [tune] move force_on_current_node to ml_utils

* fix import

* force on head node

* init ray

* split test files

* update example

* move tests to ray client

* address comments

* move comment

* address comments
2021-11-13 15:30:44 -08:00
Sven Mika
e5ead6a4b0
[RLlib; Documentation] Minor fixes "rllib in 60s" and per-feature sigils. (#20248) 2021-11-13 22:10:47 +01:00
Kim Pevey
82a5bf68fa
[Docs] Add note for multi-node on Windows (#20184)
* add note for multi-node on Windows

* update message

Co-authored-by: Philipp Moritz <pcmoritz@gmail.com>
2021-11-09 16:02:01 -08:00
Kim Pevey
bbacb6d828
[Docs] Streaming MapReduce: Remove breaking city (#20187) 2021-11-09 11:15:57 -08:00
matthewdeng
78e9ff7c91
[train][datasets] add example for big data training (#20042)
* [train][datasets] add example for big data training

* add title docstring

* lint and dependencies

* add dask_ml requirement
2021-11-05 09:28:48 -07:00
Kim Pevey
3ff4fde0f5
[Doc] Update newsreader example (#19893) 2021-10-29 22:25:40 -07:00
Kim Pevey
8aa61566fa
[Doc] Example docs minor wording fixes (#19890) 2021-10-29 22:15:35 -07:00
Kim Pevey
96480d97d6
[DOC] Minor typos/fixes to Tips for First Timers (#19887)
* fix typos

* some more fixes

Co-authored-by: Philipp Moritz <pcmoritz@gmail.com>
2021-10-29 22:13:15 -07:00
architkulkarni
fdefd875c3
[Doc] [runtime env] Move runtime env section up one level, add inbound links (#19863) 2021-10-29 12:02:39 -05:00
Antoni Baum
e9df253f5d
[CI/docs] Remove [default] from xgboost-ray (#19186)
Co-authored-by: Kai Fricke <kai@anyscale.com>
2021-10-14 16:29:55 +01:00
Wansoo Kim
0f6d4661d7
[tune] Port all MNIST examples to specify data_dir (#19033)
Co-authored-by: Kai Fricke <kai@anyscale.com>
2021-10-12 15:36:06 +01:00
Chen Shen
c740aae54c
[Core][Dataset] adding example for large scale data ingestion (#18998) 2021-10-11 15:37:09 -07:00
matthewdeng
3fbe135a24
[docs] add modin_xgboost and dask_xgboost notebook tutorials (#18775)
* Add xgboost-dask golden notebook

* [examples] add modin-xgboost Jupyter notebook

* Add xgboost dast gn

* update modin notebook to sphinx-gallery compatible python file

* fix build file

* fix test

* fix test

* Add modin notebook anyscale connect test

* Add missing file

* add dask_xgboost notebook

* Add the new modin golden notebook to CI

* fix lint and filter out tests with py37

* Update release/golden_notebook_tests_new/golden_notebook_tests.yaml

Co-authored-by: matthewdeng <matthew.j.deng@gmail.com>

* Add dask, wait for cluster client, remove pytest

* Replace folder

* Fix

* Update dask_xgboost_app_config.yaml

* Update modin_xgboost_app_config.yaml

* comment on filtered out tests

Co-authored-by: Antoni Baum <antoni.baum@protonmail.com>
2021-10-05 09:17:33 -07:00
Jin Dong
7197b26a3c
Update streaming example to use wait 2021-07-30 16:14:30 -07:00
architkulkarni
756a4e7a90
[Core] [runtime env] update tests to use ray.init(runtime_env=...) and add e2e test (#17232) 2021-07-26 11:21:30 -05:00
Vince Jankovics
05c9dfbbda
[RLlib] CV2 to Skimage dependency change (#16841) 2021-07-21 22:24:18 -04:00
Amog Kamsetty
33d798f8fc
[Docs] Add e2e guide on using Pytorch Lightning with Ray (#16484)
Co-authored-by: Richard Liaw <rliaw@berkeley.edu>
2021-06-19 10:04:58 -07:00
architkulkarni
7d029f8e71
[Doc] [Core] [runtime env] Add runtime env doc (#16290) 2021-06-09 20:02:16 -05:00
Richard Liaw
f4b2dd94b2
[tune] Cache MNIST and restore MNIST tests (#15260) 2021-04-13 14:20:26 -07:00
Richard Liaw
56c95075d1
Revert "[tune] enable mnist test v3 (#15198)" (#15242)
This reverts commit d913f32126.
2021-04-12 09:27:55 -07:00
Richard Liaw
d913f32126
[tune] enable mnist test v3 (#15198) 2021-04-09 00:10:12 -07:00
Ian Rodney
eb12033612
[Code Cleanup] Switch to use ray.util.get_node_ip_address() (#14741)
Co-authored-by: Richard Liaw <rliaw@berkeley.edu>
2021-03-18 13:10:57 -07:00
Edward Oakes
72615ae590
[metrics] Improve custom metrics docs, add an example on how to use them (#14690) 2021-03-15 17:37:02 -05:00
Richard Liaw
c2aeccaf14
[tune] revert all mnist tests (#14677)
This reverts commit 3f557348a2.
2021-03-14 23:58:13 -07:00
Richard Liaw
3f557348a2
[tune] re-enable MNIST tests! (#14561) 2021-03-12 13:35:43 -08:00
Richard Liaw
dba533dd84
Disable more torch (#14480)
Signed-off-by: Richard Liaw <rliaw@berkeley.edu>
2021-03-03 15:46:32 -08:00
javi-redondo
b8b2d6410d
[docs] new Ray Cluster documentation (#13839)
Co-authored-by: Javier Redondo <javier@anyscale.com>
Co-authored-by: AmeerHajAli <ameerh@berkeley.edu>
2021-02-15 00:47:14 -08:00
Ameer Haj Ali
b7dd7ddb52
deprecate useless fields in the cluster yaml. (#13637)
* prepare for head node

* move command runner interface outside _private

* remove space

* Eric

* flake

* min_workers in multi node type

* fixing edge cases

* eric not idle

* fix target_workers to consider min_workers of node types

* idle timeout

* minor

* minor fix

* test

* lint

* eric v2

* eric 3

* min_workers constraint before bin packing

* Update resource_demand_scheduler.py

* Revert "Update resource_demand_scheduler.py"

This reverts commit 818a63a2c86d8437b3ef21c5035d701c1d1127b5.

* reducing diff

* make get_nodes_to_launch return a dict

* merge

* weird merge fix

* auto fill instance types for AWS

* Alex/Eric

* Update doc/source/cluster/autoscaling.rst

* merge autofill and input from user

* logger.exception

* make the yaml use the default autofill

* docs Eric

* remove test_autoscaler_yaml from windows tests

* lets try changing the test a bit

* return test

* lets see

* edward

* Limit max launch concurrency

* commenting frac TODO

* move to resource demand scheduler

* use STATUS UP TO DATE

* Eric

* make logger of gc freed refs debug instead of info

* add cluster name to docker mount prefix directory

* grrR

* fix tests

* moving docker directory to sdk

* move the import to prevent circular dependency

* smallf fix

* ian

* fix max launch concurrency bug to assume failing nodes as pending and consider only load_metric's connected nodes as running

* small fix

* deflake test_joblib

* lint

* placement groups bypass

* remove space

* Eric

* first ocmmit

* lint

* exmaple

* documentation

* hmm

* file path fix

* fix test

* some format issue in docs

* modified docs

* joblib strikes again on windows

* add ability to not start autoscaler/monitor

* a

* remove worker_default

* Remove default pod type from operator

* Remove worker_default_node_type from rewrite_legacy_yaml_to_availble_node_types

* deprecate useless fields

Co-authored-by: Ameer Haj Ali <ameerhajali@ameers-mbp.lan>
Co-authored-by: Alex Wu <alex@anyscale.io>
Co-authored-by: Alex Wu <itswu.alex@gmail.com>
Co-authored-by: Eric Liang <ekhliang@gmail.com>
Co-authored-by: Ameer Haj Ali <ameerhajali@Ameers-MacBook-Pro.local>
Co-authored-by: root <root@ip-172-31-56-188.us-west-2.compute.internal>
Co-authored-by: Dmitri Gekhtman <dmitri.m.gekhtman@gmail.com>
2021-01-23 12:06:51 -08:00
Eric Liang
ee6332dbb0
Bump dev branch to 2.0 to avoid endless version bump toil (#13497)
* wip

* fix

* fix
2021-01-15 17:41:17 -08:00
Max Fitton
e077bc4206
[Release] Bump master to 1.2.0 for 1.1.0 release (#12856) 2020-12-15 09:40:26 -08:00
Eric Squires
9f70293700
Remove debug extras from setup.py (#12751) 2020-12-10 16:23:11 -06:00
SangBin Cho
040cf2c13b
[Doc] Placement group doc small update (#12594)
* Modify doc  that wasn't supposed to be merged.

* Addressed coder eview.
2020-12-07 13:58:27 -08:00
SangBin Cho
981df65b91
[Doc] Improve the placement group document (#12507)
* Improve the placement group document.

* Fix grammar.

* Addressed code review.
2020-12-01 13:15:30 -08:00
Eric Liang
9b8218aabd
[docs] Move all /latest links to /master (#11897)
* use master link

* remae

* revert non-ray

* more

* mre
2020-11-10 10:53:28 -08:00
Barak Michener
05c4e3fb2a
[build] Build wheels with manylinux2014 (#11621)
* necessary changes

* Split bazel install

* manylinux2014

* change references to manylinux2014

* Fix lint

* port alex's docker build changes

* fix config issue

* remove extra manylinux2010 requirement script

* revert SHA overwrite

* wip

* incompatible_linklibs

* fix nits
2020-11-03 19:36:32 -08:00
Ian Rodney
4c3f09094a
[docs] redis-port -> port (#10937) 2020-09-23 17:04:13 -07:00
Richard Liaw
8d6fee14c7
[fix] Cite sources for example (#10798) 2020-09-15 14:37:40 -07:00
Richard Liaw
1a4023387d
[docs] slurm + progress_bar example (#10782) 2020-09-15 00:16:36 -07:00
Max Fitton
017737b82b
[Documentation] local_mode doc updates and actor / worker explanation from Slack (#10748)
* wip

* Update local mode docs in all locations

* Update doc/source/actors.rst

Co-authored-by: Richard Liaw <rliaw@berkeley.edu>

* Update doc/source/actors.rst

Co-authored-by: Richard Liaw <rliaw@berkeley.edu>

* Change duplicated text to links to a subtitle for local_mode

* change a reference to be explicit

* Apply suggestions from code review

Co-authored-by: Max Fitton <max@semprehealth.com>
Co-authored-by: Richard Liaw <rliaw@berkeley.edu>
2020-09-14 13:19:38 -07:00
Barak Michener
c6b1ed7f8f
release process: bump version number to 1.1.0.dev0 everywhere (#10686) 2020-09-10 16:00:21 -07:00
Richard Liaw
4d4b262069
[docs] Tips for first-time users (copy from blog) (#10578)
Co-authored-by: Alex Wu <itswu.alex@gmail.com>
2020-09-04 21:44:22 -07:00
Richard Liaw
4b3b46e09f
[docs] Tips for testing Ray applications (#10583) 2020-09-04 18:40:48 -07:00
Barak Michener
8e76796fd0
ci: Redo format.sh --all script & backfill lint fixes (#9956) 2020-08-07 16:49:49 -07:00
Barak Michener
68f3fec744
*: Centralize requirements.txt and unify dependency versions (#9759)
* python_test: fix cython_examples in doc/ and tests/

* update setup.py to parse the bazel version string better

* all: centralize all python deps into stackable requirements files in python/

* format

* Move cython test into the proper package

* Add cross-reference dependency comments for requirements and setup.py

* re-enable version pinning on CI, fix formatting

* fix up torchvision version

* fix case in shell
2020-07-30 11:22:56 -07:00
mehrdadn
56d2cf6479
Shellcheck rewrites (#9597)
* Fix SC2001: See if you can use ${variable//search/replace} instead.

* Fix SC2010: Don't use ls | grep. Use a glob or a for loop with a condition to allow non-alphanumeric filenames.

* Fix SC2012: Use find instead of ls to better handle non-alphanumeric filenames.

* Fix SC2015: Note that A && B || C is not if-then-else. C may run when A is true.

* Fix SC2028: echo may not expand escape sequences. Use printf.

* Fix SC2034: variable appears unused. Verify use (or export if used externally).

* Fix SC2035: Use ./*glob* or -- *glob* so names with dashes won't become options.

* Fix SC2071: > is for string comparisons. Use -gt instead.

* Fix SC2154: variable is referenced but not assigned

* Fix SC2164: Use 'cd ... || exit' or 'cd ... || return' in case cd fails.

* Fix SC2188: This redirection doesn't have a command. Move to its command (or use 'true' as no-op).

* Fix SC2236: Use -n instead of ! -z.

* Fix SC2242: Can only exit with status 0-255. Other data should be written to stdout/stderr.

* Fix SC2086: Double quote to prevent globbing and word splitting.

Co-authored-by: Mehrdad <noreply@github.com>
2020-07-24 17:24:19 -05:00
mehrdadn
b14728d999
Shellcheck quoting (#9596)
* Fix SC2006: Use $(...) notation instead of legacy backticked `...`.

* Fix SC2016: Expressions don't expand in single quotes, use double quotes for that.

* Fix SC2046: Quote this to prevent word splitting.

* Fix SC2053: Quote the right-hand side of == in [[ ]] to prevent glob matching.

* Fix SC2068: Double quote array expansions to avoid re-splitting elements.

* Fix SC2086: Double quote to prevent globbing and word splitting.

* Fix SC2102: Ranges can only match single chars (mentioned due to duplicates).

* Fix SC2140: Word is of the form "A"B"C" (B indicated). Did you mean "ABC" or "A\"B\"C"?

* Fix SC2145: Argument mixes string and array. Use * or separate argument.

* Fix SC2209: warning: Use var=$(command) to assign output (or quote to assign string).

Co-authored-by: Mehrdad <noreply@github.com>
2020-07-21 21:56:41 -05:00
mehrdadn
4f470c3fc1
Shellcheck comments (#9595) 2020-07-21 16:47:09 -05:00
Richard Liaw
7abf7a0109
[docs] Render ActorPool documentation, etc (#9433) 2020-07-13 17:59:22 -07:00