ray/doc/source/data/custom-data.rst

.. _datasets_custom_datasource:

------------------------
Using Custom Datasources
------------------------

Datasets can read and write in parallel to `custom datasources <package-ref.html#custom-datasource-api>`__ defined in Python.
Once you have implemented `YourCustomDataSource`, you can use it like any other source in Ray Data:

.. code-block:: python

    # Read from a custom datasource.
    ds = ray.data.read_datasource(YourCustomDatasource(), **read_args)

    # Write to a custom datasource.
    ds.write_datasource(YourCustomDatasource(), **write_args)
[Datasets] Overhaul of "Creating Datasets" feature guide. (#24831) This PR is a general overhaul of the "Creating Datasets" feature guide, providing complete coverage of all (public) dataset creation APIs and highlighting features and quirks of the individual APIs, data modalities, storage backends, etc. In order to keep the page from getting too long and keeping it easy to navigate, tabbed views are used heavily. 2022-05-17 16:23:42 -07:00			`.. _datasets_custom_datasource:`

[Docs] Ray Data docs target state (#21931) Preview: [docs](https://ray--21931.org.readthedocs.build/en/21931/data/dataset.html) The Ray Data project's docs now have a clearer structure and have partly been rewritten/modified. In particular we have - [x] A Getting Started Guide - [x] An explicit User / How-To Guide - [x] A dedicated Key Concepts page - [x] A consistent naming convention in `Ray Data` whenever is is referred to the project. This surfaces quite clearly that, apart from the "Getting Started" sections, we really only have one real example. Once we have more, we can create an "Example" section like many other sub-projects have. This will be addressed in https://github.com/ray-project/ray/issues/21838. 2022-01-27 22:14:36 +01:00			`------------------------`
			`Using Custom Datasources`
			`------------------------`

			Datasets can read and write in parallel to `custom datasources <package-ref.html#custom-datasource-api>`__ defined in Python.
			Once you have implemented `YourCustomDataSource`, you can use it like any other source in Ray Data:

			`.. code-block:: python`

			`# Read from a custom datasource.`
			`ds = ray.data.read_datasource(YourCustomDatasource(), **read_args)`

			`# Write to a custom datasource.`
			`ds.write_datasource(YourCustomDatasource(), **write_args)`