ray/doc/source/data/user-guide.rst

.. _data_user_guide :

===========
User Guides
===========

If you’re new to Ray Datasets, we recommend starting with the :ref:`Ray Datasets Quick Start <datasets_getting_started>`.
This user guide will help you navigate the Ray Datasets project and show you how achieve several tasks, for instance
you will learn

- how to load data and preprocess it for machine learning applications,
- how to use Tensors with Ray Datasets,
- how to run Dataset Pipelines in common scenarios,
- and how to tune your Ray Datasets applications for performance.

.. toctree::
    :maxdepth: 2

    creating-datasets
    saving-datasets
    transforming-datasets
    accessing-datasets
    pipelining-compute
    dataset-ml-preprocessing
    dataset-tensor-support
    advanced-pipelines
    random-access
    custom-data
    memory-management
    performance-tips
-												[Docs] Ray Data docs target state (#21931)

Preview: [docs](https://ray--21931.org.readthedocs.build/en/21931/data/dataset.html)

The Ray Data project's docs now have a clearer structure and have partly been rewritten/modified. In particular we have

- [x] A Getting Started Guide
- [x] An explicit User / How-To Guide
- [x] A dedicated Key Concepts page
- [x] A consistent naming convention in `Ray Data` whenever is is referred to the project.

This surfaces quite clearly that, apart from the "Getting Started" sections, we really only have one real example. Once we have more, we can create an "Example" section like many other sub-projects have. This will be addressed in https://github.com/ray-project/ray/issues/21838.
											
										
										
											2022-01-27 22:14:36 +01:00
+								.. _data_user_guide :
 								===========
 								User Guides
 								===========
-												Revamp the Getting Started page for Dataset (#24860)

This is part of the Dataset GA doc fix effort to update/improve the documentation.
This PR revamps the Getting Started page.

What are the changes:
- Focus on basic/core features that are bread-and-butter for users, leave the advanced features out
- Focus on high level introduction, leave the detailed spec out (e.g. what are possible batch_types for map_batches() API)
- Use more realistic (yet still simple) data example that's familiar to people (IRIS dataset in this case)
- Use the same data example throughout to make it context-switch free
- Use runnable code rather than faked
- Reference to the code from doc, instead of inlining them in the doc

Co-authored-by: Ubuntu <ubuntu@ip-172-31-32-136.us-west-2.compute.internal>
Co-authored-by: Eric Liang <ekhliang@gmail.com>
											
										
										
											2022-05-18 13:46:23 -07:00
+								If you’re new to Ray Datasets, we recommend starting with the :ref:`Ray Datasets Quick Start <datasets_getting_started>`.
-												[Datasets] [Docs] Datasets library branding + positioning tweaks (#22067)


											
										
										
											2022-02-05 18:59:34 -06:00
+								This user guide will help you navigate the Ray Datasets project and show you how achieve several tasks, for instance
-												[Docs] Ray Data docs target state (#21931)

Preview: [docs](https://ray--21931.org.readthedocs.build/en/21931/data/dataset.html)

The Ray Data project's docs now have a clearer structure and have partly been rewritten/modified. In particular we have

- [x] A Getting Started Guide
- [x] An explicit User / How-To Guide
- [x] A dedicated Key Concepts page
- [x] A consistent naming convention in `Ray Data` whenever is is referred to the project.

This surfaces quite clearly that, apart from the "Getting Started" sections, we really only have one real example. Once we have more, we can create an "Example" section like many other sub-projects have. This will be addressed in https://github.com/ray-project/ray/issues/21838.
											
										
										
											2022-01-27 22:14:36 +01:00
+								you will learn
 								- how to load data and preprocess it for machine learning applications,
-												[Datasets] [Docs] Datasets library branding + positioning tweaks (#22067)


											
										
										
											2022-02-05 18:59:34 -06:00
+								- how to use Tensors with Ray Datasets,
-												[Docs] Ray Data docs target state (#21931)

Preview: [docs](https://ray--21931.org.readthedocs.build/en/21931/data/dataset.html)

The Ray Data project's docs now have a clearer structure and have partly been rewritten/modified. In particular we have

- [x] A Getting Started Guide
- [x] An explicit User / How-To Guide
- [x] A dedicated Key Concepts page
- [x] A consistent naming convention in `Ray Data` whenever is is referred to the project.

This surfaces quite clearly that, apart from the "Getting Started" sections, we really only have one real example. Once we have more, we can create an "Example" section like many other sub-projects have. This will be addressed in https://github.com/ray-project/ray/issues/21838.
											
										
										
											2022-01-27 22:14:36 +01:00
+								- how to run Dataset Pipelines in common scenarios,
-												[Datasets] [Docs] Datasets library branding + positioning tweaks (#22067)


											
										
										
											2022-02-05 18:59:34 -06:00
+								- and how to tune your Ray Datasets applications for performance.
-												[Docs] Ray Data docs target state (#21931)

Preview: [docs](https://ray--21931.org.readthedocs.build/en/21931/data/dataset.html)

The Ray Data project's docs now have a clearer structure and have partly been rewritten/modified. In particular we have

- [x] A Getting Started Guide
- [x] An explicit User / How-To Guide
- [x] A dedicated Key Concepts page
- [x] A consistent naming convention in `Ray Data` whenever is is referred to the project.

This surfaces quite clearly that, apart from the "Getting Started" sections, we really only have one real example. Once we have more, we can create an "Example" section like many other sub-projects have. This will be addressed in https://github.com/ray-project/ray/issues/21838.
											
										
										
											2022-01-27 22:14:36 +01:00
 								.. toctree::
 								    :maxdepth: 2
-												[Dataset GA doc] Decompose the monolith of Getting Started page (and get them under User Guide) (#23311)

Improve the Dataset documentation for GA.
											
										
										
											2022-03-18 11:25:43 -07:00
+								    creating-datasets
 								    saving-datasets
 								    transforming-datasets
-												Cleanup the DatasetPipeline references in Getting Started; rename Exchanging to Accessing (#23786)


											
										
										
											2022-04-12 17:10:14 -07:00
+								    accessing-datasets
-												[Dataset GA doc] Decompose the monolith of Getting Started page (and get them under User Guide) (#23311)

Improve the Dataset documentation for GA.
											
										
										
											2022-03-18 11:25:43 -07:00
+								    pipelining-compute
-												[Docs] Ray Data docs target state (#21931)

Preview: [docs](https://ray--21931.org.readthedocs.build/en/21931/data/dataset.html)

The Ray Data project's docs now have a clearer structure and have partly been rewritten/modified. In particular we have

- [x] A Getting Started Guide
- [x] An explicit User / How-To Guide
- [x] A dedicated Key Concepts page
- [x] A consistent naming convention in `Ray Data` whenever is is referred to the project.

This surfaces quite clearly that, apart from the "Getting Started" sections, we really only have one real example. Once we have more, we can create an "Example" section like many other sub-projects have. This will be addressed in https://github.com/ray-project/ray/issues/21838.
											
										
										
											2022-01-27 22:14:36 +01:00
+								    dataset-ml-preprocessing
 								    dataset-tensor-support
 								    advanced-pipelines
-												Add random access support for Datasets (experimental feature) (#22749)

This PR adds experimental support for random access to datasets. A Dataset can be random access enabled by calling `ds.to_random_access_dataset(key, num_workers=N)`. This creates a RandomAccessDataset.

RandomAccessDataset partitions the dataset across the cluster by the given sort key, providing efficient random access to records via binary search. A number of worker actors are created, each of which has zero-copy access to the underlying sorted data blocks of the Dataset.

Performance-wise, you can expect each worker to provide ~3000 records / second via ``get_async()``, and ~10000 records / second via ``multiget()``.

Since Ray actor calls go direct from worker->worker, throughput scales linearly with the number of workers.

											
										
										
											2022-03-17 15:01:12 -07:00
+								    random-access
-												[Docs] Ray Data docs target state (#21931)

Preview: [docs](https://ray--21931.org.readthedocs.build/en/21931/data/dataset.html)

The Ray Data project's docs now have a clearer structure and have partly been rewritten/modified. In particular we have

- [x] A Getting Started Guide
- [x] An explicit User / How-To Guide
- [x] A dedicated Key Concepts page
- [x] A consistent naming convention in `Ray Data` whenever is is referred to the project.

This surfaces quite clearly that, apart from the "Getting Started" sections, we really only have one real example. Once we have more, we can create an "Example" section like many other sub-projects have. This will be addressed in https://github.com/ray-project/ray/issues/21838.
											
										
										
											2022-01-27 22:14:36 +01:00
+								    custom-data
-												[data] [docs] Doc audit-- rebalance basic vs advanced materials (#25262)


											
										
										
											2022-06-01 13:50:46 -07:00
+								    memory-management
-												[Docs] Ray Data docs target state (#21931)

Preview: [docs](https://ray--21931.org.readthedocs.build/en/21931/data/dataset.html)

The Ray Data project's docs now have a clearer structure and have partly been rewritten/modified. In particular we have

- [x] A Getting Started Guide
- [x] An explicit User / How-To Guide
- [x] A dedicated Key Concepts page
- [x] A consistent naming convention in `Ray Data` whenever is is referred to the project.

This surfaces quite clearly that, apart from the "Getting Started" sections, we really only have one real example. Once we have more, we can create an "Example" section like many other sub-projects have. This will be addressed in https://github.com/ray-project/ray/issues/21838.
											
										
										
											2022-01-27 22:14:36 +01:00
+								    performance-tips