hiro/ray
1
0
Fork 0
mirror of https://github.com/vale981/ray synced 2025-03-10 05:16:49 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions
ray/doc/source/data/images/dataset-shuffle.svg

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

1 line
102 KiB
XML
Raw Normal View History

Split blocks automatically into 500MB chunks on file read and transformation (#20235) This PR adds support for automatic block splitting on read and map transforms, to keep block size bounded to ~500MiB. This avoids potential OOM situations where a map task may consume too much intermediate Python heap memory, or too much object store shared memory for one block.
2021-11-15 22:25:11 -08:00
<svg version="1.1" viewBox="0.0 0.0 747.0866141732283 232.74278215223097" fill="none" stroke="none" stroke-linecap="square" stroke-miterlimit="10" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns="http://www.w3.org/2000/svg"><clipPath id="p.0"><path d="m0 0l747.0866 0l0 232.74278l-747.0866 0l0 -232.74278z" clip-rule="nonzero"/></clipPath><g clip-path="url(#p.0)"><path fill="#ffffff" d="m0 0l747.0866 0l0 232.74278l-747.0866 0z" fill-rule="evenodd"/><path fill="#000000" fill-opacity="0.0" d="m8.186352 191.69292l368.56696 0l0 42.015747l-368.56696 0z" fill-rule="evenodd"/><path fill="#000000" d="m18.186352 214.23792q0 -1.171875 0.3125 -2.078125q0.3125 -0.90625 0.90625 -1.53125q0.59375 -0.625 1.40625 -0.9375q0.828125 -0.328125 1.84375 -0.328125q0.4375 0 0.859375 0.0625q0.421875 0.046875 0.828125 0.15625l0 -3.84375l1.59375 0l0 12.875l-1.421875 0l-0.046875 -1.734375q-0.671875 0.96875 -1.453125 1.4375q-0.765625 0.453125 -1.671875 0.453125q-0.78125 0 -1.375 -0.3125q-0.59375 -0.328125 -1.0 -0.921875q-0.390625 -0.609375 -0.59375 -1.4375q-0.1875 -0.84375 -0.1875 -1.859375zm1.609375 -0.109375q0 1.65625 0.484375 2.484375q0.5 0.8125 1.390625 0.8125q0.59375 0 1.265625 -0.53125q0.671875 -0.546875 1.40625 -1.609375l0 -4.234375q-0.390625 -0.1875 -0.859375 -0.28125q-0.46875 -0.09375 -0.9375 -0.09375q-1.296875 0 -2.03125 0.84375q-0.71875 0.828125 -0.71875 2.609375zm16.351044 1.984375q0 0.484375 -0.171875 0.875q-0.15625 0.375 -0.4375 0.671875q-0.28125 0.296875 -0.65625 0.515625q-0.375 0.203125 -0.796875 0.34375q-0.421875 0.125 -0.875 0.1875q-0.453125 0.0625 -0.875 0.0625q-0.921875 0 -1.703125 -0.078125q-0.78125 -0.078125 -1.53125 -0.265625l0 -1.453125q0.796875 0.234375 1.59375 0.359375q0.796875 0.109375 1.578125 0.109375q1.140625 0 1.6875 -0.3125q0.546875 -0.3125 0.546875 -0.890625q0 -0.234375 -0.09375 -0.421875q-0.078125 -0.203125 -0.3125 -0.375q-0.21875 -0.1875 -0.703125 -0.375q-0.484375 -0.203125 -1.3125 -0.4375q-0.609375 -0.1875 -1.140625 -0.421875q-0.515625 -0.234375 -0.90625 -0.546875q-0.375 -0.328125 -0.59375 -0.75q-0.21875 -0.421875 -0.21875 -1.015625q0 -0.375 0.171875 -0.828125q0.171875 -0.453125 0.59375 -0.84375q0.4375 -0.40625 1.171875 -0.65625q0.734375 -0.265625 1.8125 -0.265625q0.546875 0 1.203125 0.0625q0.65625 0.0625 1.359375 0.203125l0 1.40625q-0.75 -0.171875 -1.421875 -0.25q-0.65625 -0.09375 -1.15625 -0.09375q-0.59375 0 -1.0 0.09375q-0.390625 0.078125 -0.65625 0.25q-0.25 0.15625 -0.359375 0.375q-0.109375 0.203125 -0.109375 0.453125q0 0.25 0.09375 0.453125q0.109375 0.1875 0.359375 0.375q0.265625 0.1875 0.71875 0.375q0.46875 0.1875 1.21875 0.40625q0.8125 0.234375 1.359375 0.5q0.5625 0.25 0.90625 0.578125q0.359375 0.3125 0.5 0.71875q0.15625 0.40625 0.15625 0.90625zm6.6166687 -0.375q0.296875 0 0.578125 0.125q0.28125 0.125 0.484375 0.328125q0.203125 0.203125 0.3125 0.484375q0.125 0.28125 0.125 0.59375q0 0.3125 -0.125 0.59375q-0.109375 0.265625 -0.3125 0.46875q-0.203125 0.203125 -0.484375 0.3125q-0.28125 0.125 -0.578125 0.125q-0.328125 0 -0.609375 -0.125q-0.265625 -0.109375 -0.46875 -0.3125q-0.203125 -0.203125 -0.328125 -0.46875q-0.109375 -0.28125 -0.109375 -0.59375q0 -0.3125 0.109375 -0.59375q0.125 -0.28125 0.328125 -0.484375q0.203125 -0.203125 0.46875 -0.328125q0.28125 -0.125 0.609375 -0.125zm7.0229187 -6.265625l1.453125 0l0.046875 1.6875q0.8125 -0.984375 1.59375 -1.421875q0.796875 -0.4375 1.59375 -0.4375q1.421875 0 2.15625 0.921875q0.734375 0.921875 0.671875 2.734375l-1.59375 0q0.015625 -1.203125 -0.359375 -1.734375q-0.375 -0.546875 -1.109375 -0.546875q-0.3125 0 -0.640625 0.109375q-0.328125 0.109375 -0.671875 0.359375q-0.328125 0.25 -0.71875 0.640625q-0.375 0.390625 -0.8125 0.953125l0 5.875l-1.609375 0l0 -9.140625zm17.616665 4.078125q0 0.34375 -0.015625 0.578125q0 0.21875 -0.03125 0.421875l-6.421871 0q0 1.40625 0.78125 2.15625q0.796875 0.75 2.265625 0.75q0.4062462 0 0.7968712 -0.03125q0.40625 -0.03125 0.78125 -0.078125q0.375 -0.0625 0.71875 -0.125q0.34375 -0.078125 0.625 -0.15625l0 1.296875q-0.640625 0.1875 -1.46875 0.296875q-0.8125 0.109375 -1.6874962 0.109375q-1.171875 0 -2.015625 -0.3125q-0.84375 -0.3125 -1.390625 -0.921875q
Reference in a new issue Copy permalink