mirror of
https://github.com/vale981/ray
synced 2025-03-06 18:41:40 -05:00
015181ab9a
This PR adds experimental support for random access to datasets. A Dataset can be random access enabled by calling `ds.to_random_access_dataset(key, num_workers=N)`. This creates a RandomAccessDataset. RandomAccessDataset partitions the dataset across the cluster by the given sort key, providing efficient random access to records via binary search. A number of worker actors are created, each of which has zero-copy access to the underlying sorted data blocks of the Dataset. Performance-wise, you can expect each worker to provide ~3000 records / second via ``get_async()``, and ~10000 records / second via ``multiget()``. Since Ray actor calls go direct from worker->worker, throughput scales linearly with the number of workers. |
||
|---|---|---|
| .. | ||
| _includes | ||
| _static | ||
| _templates | ||
| cluster | ||
| data | ||
| images | ||
| ray-contribute | ||
| ray-core | ||
| ray-more-libs | ||
| ray-observability | ||
| ray-overview | ||
| ray-references | ||
| raysgd | ||
| rllib | ||
| serve | ||
| train | ||
| tune | ||
| workflows | ||
| _toc.yml | ||
| conf.py | ||
| custom_directives.py | ||
| index.md | ||