mirror of
https://github.com/vale981/ray
synced 2025-03-06 02:21:39 -05:00
015181ab9a
This PR adds experimental support for random access to datasets. A Dataset can be random access enabled by calling `ds.to_random_access_dataset(key, num_workers=N)`. This creates a RandomAccessDataset. RandomAccessDataset partitions the dataset across the cluster by the given sort key, providing efficient random access to records via binary search. A number of worker actors are created, each of which has zero-copy access to the underlying sorted data blocks of the Dataset. Performance-wise, you can expect each worker to provide ~3000 records / second via ``get_async()``, and ~10000 records / second via ``multiget()``. Since Ray actor calls go direct from worker->worker, throughput scales linearly with the number of workers. |
||
|---|---|---|
| .. | ||
| chaos_test | ||
| dask_on_ray | ||
| dataset | ||
| decision_tree | ||
| many_nodes_tests | ||
| placement_group_tests | ||
| shuffle | ||
| stress_tests | ||
| setup_chaos.py | ||