hiro/ray
1
0
Fork 0
mirror of https://github.com/vale981/ray synced 2025-03-09 12:56:46 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions
ray/doc/source/data/package-ref.rst

182 lines
3.8 KiB
ReStructuredText
Raw Normal View History

[Docs] Ray Data docs target state (#21931) Preview: [docs](https://ray--21931.org.readthedocs.build/en/21931/data/dataset.html) The Ray Data project's docs now have a clearer structure and have partly been rewritten/modified. In particular we have - [x] A Getting Started Guide - [x] An explicit User / How-To Guide - [x] A dedicated Key Concepts page - [x] A consistent naming convention in `Ray Data` whenever is is referred to the project. This surfaces quite clearly that, apart from the "Getting Started" sections, we really only have one real example. Once we have more, we can create an "Example" section like many other sub-projects have. This will be addressed in https://github.com/ray-project/ray/issues/21838.
2022-01-27 22:14:36 +01:00
.. _data_api:
[Datasets] [Docs] Datasets library branding + positioning tweaks (#22067)
2022-02-05 18:59:34 -06:00
Ray Datasets API
================
First cut at dataset documentation (#16956)
2021-07-14 23:27:13 -07:00
[Docs] Ray Data docs target state (#21931) Preview: [docs](https://ray--21931.org.readthedocs.build/en/21931/data/dataset.html) The Ray Data project's docs now have a clearer structure and have partly been rewritten/modified. In particular we have - [x] A Getting Started Guide - [x] An explicit User / How-To Guide - [x] A dedicated Key Concepts page - [x] A consistent naming convention in `Ray Data` whenever is is referred to the project. This surfaces quite clearly that, apart from the "Getting Started" sections, we really only have one real example. Once we have more, we can create an "Example" section like many other sub-projects have. This will be addressed in https://github.com/ray-project/ray/issues/21838.
2022-01-27 22:14:36 +01:00
Creating Datasets
-----------------
[data] Fix the ObjectRef type in the dataset docs (#17111) * fix reft * remove exp * fix
2021-07-15 09:50:37 -07:00
.. autofunction:: ray.data.range
[Datasets] Change `range_arrow()` API to `range_table()` (#24704) This PR changes the ray.data.range_arrow() to ray.data.range_table(), making the Arrow representation an implementation detail.
2022-05-17 01:09:45 -07:00
.. autofunction:: ray.data.range_table
Support top-level tensor values in dataset (#17439)
2021-08-01 22:45:21 -07:00
.. autofunction:: ray.data.range_tensor
[data] Fix the ObjectRef type in the dataset docs (#17111) * fix reft * remove exp * fix
2021-07-15 09:50:37 -07:00
.. autofunction:: ray.data.read_csv
.. autofunction:: ray.data.read_json
.. autofunction:: ray.data.read_parquet
Support top-level tensor values in dataset (#17439)
2021-08-01 22:45:21 -07:00
.. autofunction:: ray.data.read_numpy
.. autofunction:: ray.data.read_text
[data] Fix the ObjectRef type in the dataset docs (#17111) * fix reft * remove exp * fix
2021-07-15 09:50:37 -07:00
.. autofunction:: ray.data.read_binary_files
.. autofunction:: ray.data.read_datasource
Move ray.data out of experimental (#17560)
2021-08-04 13:31:10 -07:00
.. autofunction:: ray.data.from_items
Initial implementation of Dataset pipelining and docs (#17309)
2021-07-28 21:12:01 -07:00
.. autofunction:: ray.data.from_arrow
[Datasets] Delineate between ref and raw APIs for the Pandas/Arrow integrations. (#18992)
2021-10-01 13:08:25 -07:00
.. autofunction:: ray.data.from_arrow_refs
[Datasets] Add `from_huggingface` for Hugging Face datasets integration (#24464) Adds a from_huggingface method to Datasets, which allows the conversion of a Hugging Face Dataset to a Ray Dataset. As a Hugging Face Dataset is backed by an Arrow table, the conversion is trivial.
2022-05-06 22:09:28 +02:00
.. autofunction:: ray.data.from_huggingface
[data] Fix the ObjectRef type in the dataset docs (#17111) * fix reft * remove exp * fix
2021-07-15 09:50:37 -07:00
.. autofunction:: ray.data.from_spark
.. autofunction:: ray.data.from_dask
.. autofunction:: ray.data.from_modin
.. autofunction:: ray.data.from_mars
.. autofunction:: ray.data.from_pandas
[Datasets] Delineate between ref and raw APIs for the Pandas/Arrow integrations. (#18992)
2021-10-01 13:08:25 -07:00
.. autofunction:: ray.data.from_pandas_refs
[Datasets] Add from_numpy() and to_numpy() APIs (#18146)
2021-08-27 13:33:11 -07:00
.. autofunction:: ray.data.from_numpy
[Datasets] Make `from_numpy()` more user-friendly. (#23871) `ray.data.from_numpy()` currently expects to be given a list of ndarray futures, instead of handling concrete ndarrays, as expected (and as allowed by other `from_*` APIs, e.g. `from_pandas`). This PR renames the existing `from_numpy` API to `from_numpy_refs`, and exposes `ray.data.from_numpy`, which takes concrete ndarrays (not object references).
2022-04-12 18:37:59 -07:00
.. autofunction:: ray.data.from_numpy_refs
First cut at dataset documentation (#16956)
2021-07-14 23:27:13 -07:00
[SGD] Sgd v2 Dataset Integration (#17626) * wip * wip * wip * draft * disable tf autosharding * wip * wip * wip * wip * add example * wip * wip * wip * use dataset.split * add unit tests * add linear example * concatenate tensors and fix example * WIP tune example * add tensorflow example * wip * random_shuffle_each_window * fault tolerance test * GPU, examples, CI * formatting * fix * Update python/ray/util/sgd/v2/tests/test_trainer.py Co-authored-by: matthewdeng <matthew.j.deng@gmail.com> * wip * type hints * wip * update user guide * fix * fix immediate issues * update example * update * fix tune gpu test * fix resources for smoke test - 1 CPU for dataset tasks * update tests, docs, examples * Apply suggestions from code review Co-authored-by: Clark Zinzow <clarkzinzow@gmail.com> * address comments * add warning * fix tests * minor doc updates * update example in doc * configure tests * Update doc/source/raysgd/v2/user_guide.rst Co-authored-by: Clark Zinzow <clarkzinzow@gmail.com> * Update python/ray/data/dataset.py Co-authored-by: matthewdeng <matthew.j.deng@gmail.com> * fix docstring Co-authored-by: Matthew Deng <matthew.j.deng@gmail.com> Co-authored-by: matthewdeng <matt@anyscale.com> Co-authored-by: Clark Zinzow <clarkzinzow@gmail.com>
2021-10-12 14:03:10 -07:00
.. _dataset-api:
First cut at dataset documentation (#16956)
2021-07-14 23:27:13 -07:00
Dataset API
-----------
[data] Fix the ObjectRef type in the dataset docs (#17111) * fix reft * remove exp * fix
2021-07-15 09:50:37 -07:00
.. autoclass:: ray.data.Dataset
First cut at dataset documentation (#16956)
2021-07-14 23:27:13 -07:00
:members:
[SGD] Sgd v2 Dataset Integration (#17626) * wip * wip * wip * draft * disable tf autosharding * wip * wip * wip * wip * add example * wip * wip * wip * use dataset.split * add unit tests * add linear example * concatenate tensors and fix example * WIP tune example * add tensorflow example * wip * random_shuffle_each_window * fault tolerance test * GPU, examples, CI * formatting * fix * Update python/ray/util/sgd/v2/tests/test_trainer.py Co-authored-by: matthewdeng <matthew.j.deng@gmail.com> * wip * type hints * wip * update user guide * fix * fix immediate issues * update example * update * fix tune gpu test * fix resources for smoke test - 1 CPU for dataset tasks * update tests, docs, examples * Apply suggestions from code review Co-authored-by: Clark Zinzow <clarkzinzow@gmail.com> * address comments * add warning * fix tests * minor doc updates * update example in doc * configure tests * Update doc/source/raysgd/v2/user_guide.rst Co-authored-by: Clark Zinzow <clarkzinzow@gmail.com> * Update python/ray/data/dataset.py Co-authored-by: matthewdeng <matthew.j.deng@gmail.com> * fix docstring Co-authored-by: Matthew Deng <matthew.j.deng@gmail.com> Co-authored-by: matthewdeng <matt@anyscale.com> Co-authored-by: Clark Zinzow <clarkzinzow@gmail.com>
2021-10-12 14:03:10 -07:00
.. _dataset-pipeline-api:
[Datasets] Add missing public APIs to Datasets API docs (#23935)
2022-04-16 11:57:38 -07:00
Block API
---------
.. autoclass:: ray.data.block.BlockExecStats
:members:
.. autoclass:: ray.data.block.BlockMetadata
:members:
.. autoclass:: ray.data.block.BlockAccessor
:members:
DatasetContext API
------------------
.. autoclass:: ray.data.context.DatasetContext
:members:
Initial implementation of Dataset pipelining and docs (#17309)
2021-07-28 21:12:01 -07:00
DatasetPipeline API
-------------------
Move ray.data out of experimental (#17560)
2021-08-04 13:31:10 -07:00
.. autoclass:: ray.data.dataset_pipeline.DatasetPipeline
Initial implementation of Dataset pipelining and docs (#17309)
2021-07-28 21:12:01 -07:00
:members:
Simple block dataset groupBy (#19435)
2021-10-19 19:53:13 -07:00
GroupedDataset API
------------------
.. autoclass:: ray.data.grouped_dataset.GroupedDataset
:members:
[Datasets] Add missing public APIs to Datasets API docs (#23935)
2022-04-16 11:57:38 -07:00
Aggregate API
-------------
.. autoclass:: ray.data.aggregate.AggregateFn
:members:
.. autoclass:: ray.data.aggregate.Count
:members:
.. autoclass:: ray.data.aggregate.Sum
:members:
.. autoclass:: ray.data.aggregate.Max
:members:
.. autoclass:: ray.data.aggregate.Mean
:members:
.. autoclass:: ray.data.aggregate.Std
:members:
[ml] add more preprocessors (#23904) Adding some more common preprocessors: * MaxAbsScaler * RobustScaler * PowerTransformer * Normalizer * FeatureHasher * Tokenizer * HashingVectorizer * CountVectorizer API docs: https://ray--23904.org.readthedocs.build/en/23904/ray-air/getting-started.html Co-authored-by: Kai Fricke <krfricke@users.noreply.github.com>
2022-04-25 13:12:59 -07:00
.. autoclass:: ray.data.aggregate.AbsMax
:members:
Add random access support for Datasets (experimental feature) (#22749) This PR adds experimental support for random access to datasets. A Dataset can be random access enabled by calling `ds.to_random_access_dataset(key, num_workers=N)`. This creates a RandomAccessDataset. RandomAccessDataset partitions the dataset across the cluster by the given sort key, providing efficient random access to records via binary search. A number of worker actors are created, each of which has zero-copy access to the underlying sorted data blocks of the Dataset. Performance-wise, you can expect each worker to provide ~3000 records / second via ``get_async()``, and ~10000 records / second via ``multiget()``. Since Ray actor calls go direct from worker->worker, throughput scales linearly with the number of workers.
2022-03-17 15:01:12 -07:00
RandomAccessDataset API
-----------------------
.. autoclass:: ray.data.random_access_dataset.RandomAccessDataset
:members:
[Datasets] Adds tensor column support (tensors-in-tables) via Pandas/Arrow extension types/arrays. (#18301)
2021-09-08 10:09:01 -07:00
Tensor Column Extension API
---------------------------
.. autoclass:: ray.data.extensions.tensor_extension.TensorDtype
:members:
.. autoclass:: ray.data.extensions.tensor_extension.TensorArray
:members:
.. autoclass:: ray.data.extensions.tensor_extension.ArrowTensorType
:members:
.. autoclass:: ray.data.extensions.tensor_extension.ArrowTensorArray
:members:
First cut at dataset documentation (#16956)
2021-07-14 23:27:13 -07:00
Custom Datasource API
---------------------
[data] Fix the ObjectRef type in the dataset docs (#17111) * fix reft * remove exp * fix
2021-07-15 09:50:37 -07:00
.. autoclass:: ray.data.Datasource
First cut at dataset documentation (#16956)
2021-07-14 23:27:13 -07:00
:members:
[data] Fix the ObjectRef type in the dataset docs (#17111) * fix reft * remove exp * fix
2021-07-15 09:50:37 -07:00
.. autoclass:: ray.data.ReadTask
First cut at dataset documentation (#16956)
2021-07-14 23:27:13 -07:00
:members:
[Data] Add partitioning classes to Data API reference (#24203)
2022-05-23 09:34:41 -07:00
Datasource Partitioning API
---------------------------
.. autoclass:: ray.data.datasource.PartitionStyle
:members:
.. autoclass:: ray.data.datasource.PathPartitionScheme
:members:
.. autoclass:: ray.data.datasource.PathPartitionEncoder
:members:
.. autoclass:: ray.data.datasource.PathPartitionParser
:members:
.. autoclass:: ray.data.datasource.PathPartitionFilter
[Datasets] Add `SimpleTorchDatasource` (#23926) It's difficult to use torchvision datasets with Ray ML. This PR makes it easier to use Torch datasets with Ray Data.
2022-04-28 11:56:45 -07:00
Built-in Datasources
--------------------
.. autoclass:: ray.data.datasource.BinaryDatasource
:members:
.. autoclass:: ray.data.datasource.CSVDatasource
:members:
.. autoclass:: ray.data.datasource.FileBasedDatasource
:members:
.. autoclass:: ray.data.datasource.JSONDatasource
:members:
.. autoclass:: ray.data.datasource.NumpyDatasource
:members:
.. autoclass:: ray.data.datasource.ParquetDatasource
:members:
.. autoclass:: ray.data.datasource.RangeDatasource
:members:
[Data] Add partitioning classes to Data API reference (#24203)
2022-05-23 09:34:41 -07:00
[Datsets] Add SimpleTensorFlowDatasource (#24022) This PR makes it easier to use TensorFlow datasets with Ray Datasets.
2022-04-29 12:15:30 -07:00
.. autoclass:: ray.data.datasource.SimpleTensorFlowDatasource
:members:
[Datasets] Add `SimpleTorchDatasource` (#23926) It's difficult to use torchvision datasets with Ray ML. This PR makes it easier to use Torch datasets with Ray Data.
2022-04-28 11:56:45 -07:00
.. autoclass:: ray.data.datasource.SimpleTorchDatasource
:members:
[Datasets] Expose `TableRow` as public API; minimize copies/type conversions on row-based ops. (#22305) This PR properly exposes `TableRow` as a public API (API docs + the "Public" tag), since it's already exposed to the user in our row-based ops. In addition, the following changes are made: 1. During row-based ops, we also choose a batch format that lines up with the current dataset format in order to eliminate unnecessary copies and type conversions. 2. `TableRow` now derives from `collections.abc.Mapping`, which lets `TableRow` better interop with code expecting a mapping, and includes a few helpful mixins so we only have to implement `__getitem__`, `__iter__`, and `__len__`.
2022-02-14 14:56:17 -06:00
Table Row API
---------------------
.. autoclass:: ray.data.row.TableRow
:members:
[dataset] Implement random_shuffle() and split(equal=True) (#17448)
2021-07-30 09:51:21 -07:00
Utility
-------
.. autofunction:: ray.data.set_progress_bars
Reference in a new issue Copy permalink