mirror of
https://github.com/vale981/ray
synced 2025-03-09 04:46:38 -04:00
1f509ab331
This adds a per-dataset config object to DataParallelTrainer. These configs define how the Dataset should be read into the DataParallelTrainer. It configures the preprocessing, splitting, and ingest strategy per-dataset. DataParallelTrainers declare default DatasetConfigs for each dataset passed in the ``datasets`` argument. Users have the opportunity to selectively override these configs by passing the ``dataset_config`` argument. Trainers can also define user customizable values (e.g., XGBoostTrainer doesn't support streaming ingest). This PR adds the minimal support for dataset configs. Future PRs will: - Add support for streaming ingest - Move this config from DataParallelTrainer to ml.Trainer |
||
|---|---|---|
| .. | ||
| air_ingest.py | ||
| air_key_concepts.py | ||
| output.txt | ||
| preprocessors.py | ||
| pytorch_starter.py | ||
| tf_starter.py | ||
| xgboost_starter.py | ||