ray/python/requirements/data_processing/requirements_dataset.txt

pickle5; python_version < '3.8'
python-snappy
[dataset] deduct filesystem automatically (#16762) 2021-07-03 00:50:59 -07:00			`pickle5; python_version < '3.8'`
Add support for snappy text decompression #22298 (#22486) Adds a streaming based reading option for Snappy-compressed files. Arrow doesn't support streaming Snappy decompression since the canonical C++ Snappy library doesn't natively support streaming decompression. This PR works around this by doing streaming reads of snappy-compressed files using the streaming decompression API provided in the [python-snappy](https://github.com/andrix/python-snappy) package. This commit supplies a custom datasource that uses Arrow + [python-snappy](https://github.com/andrix/python-snappy) to read and decompress Snappy-compressed files. Co-authored-by: siddharth.goel <siddharth.goel@bytedance.com> Co-authored-by: Chen Shen <scv119@gmail.com> 2022-03-15 16:52:22 -04:00			`python-snappy`