hiro/ray
1
0
Fork 0
mirror of https://github.com/vale981/ray synced 2025-03-06 18:41:40 -05:00
Code Issues Projects Releases Packages Wiki Activity Actions
ray/doc/source/data/images/dataset-spill.svg

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

1 line
45 KiB
XML
Raw Normal View History

Split blocks automatically into 500MB chunks on file read and transformation (#20235) This PR adds support for automatic block splitting on read and map transforms, to keep block size bounded to ~500MiB. This avoids potential OOM situations where a map task may consume too much intermediate Python heap memory, or too much object store shared memory for one block.
2021-11-15 22:25:11 -08:00
<svg version="1.1" viewBox="0.0 0.0 526.9370078740158 217.5275590551181" fill="none" stroke="none" stroke-linecap="square" stroke-miterlimit="10" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns="http://www.w3.org/2000/svg"><clipPath id="p.0"><path d="m0 0l526.937 0l0 217.52756l-526.937 0l0 -217.52756z" clip-rule="nonzero"/></clipPath><g clip-path="url(#p.0)"><path fill="#ffffff" d="m0 0l526.937 0l0 217.52756l-526.937 0z" fill-rule="evenodd"/><path fill="#d9d9d9" d="m12.490813 148.86308l0 0c0 -6.38945 95.294014 -11.569122 212.84514 -11.569122c117.55113 0 212.84514 5.1796722 212.84514 11.569122l0 46.27646c0 6.38945 -95.29401 11.569122 -212.84514 11.569122c-117.551125 0 -212.84514 -5.1796722 -212.84514 -11.569122z" fill-rule="evenodd"/><path fill="#000000" fill-opacity="0.0" d="m438.1811 148.86308l0 0c0 6.38945 -95.29401 11.569107 -212.84514 11.569107c-117.551125 0 -212.84514 -5.179657 -212.84514 -11.569107" fill-rule="evenodd"/><path fill="#000000" fill-opacity="0.0" d="m12.490813 148.86308l0 0c0 -6.38945 95.294014 -11.569122 212.84514 -11.569122c117.55113 0 212.84514 5.1796722 212.84514 11.569122l0 46.27646c0 6.38945 -95.29401 11.569122 -212.84514 11.569122c-117.551125 0 -212.84514 -5.1796722 -212.84514 -11.569122z" fill-rule="evenodd"/><path stroke="#000000" stroke-width="1.0" stroke-linejoin="round" stroke-linecap="butt" d="m438.1811 148.86308l0 0c0 6.38945 -95.29401 11.569107 -212.84514 11.569107c-117.551125 0 -212.84514 -5.179657 -212.84514 -11.569107" fill-rule="evenodd"/><path stroke="#000000" stroke-width="1.0" stroke-linejoin="round" stroke-linecap="butt" d="m12.490813 148.86308l0 0c0 -6.38945 95.294014 -11.569122 212.84514 -11.569122c117.55113 0 212.84514 5.1796722 212.84514 11.569122l0 46.27646c0 6.38945 -95.29401 11.569122 -212.84514 11.569122c-117.551125 0 -212.84514 -5.1796722 -212.84514 -11.569122z" fill-rule="evenodd"/><path fill="#000000" fill-opacity="0.0" d="m12.490813 35.047592l0 0c0 -9.625313 7.8028574 -17.42817 17.42817 -17.42817l372.41925 0c4.6222534 0 9.055176 1.8361759 12.323578 5.1045933c3.2684326 3.2684155 5.1046143 7.701338 5.1046143 12.323576l0 69.71059c0 9.625313 -7.802887 17.42817 -17.428192 17.42817l-372.41925 0c-9.625313 0 -17.42817 -7.8028564 -17.42817 -17.42817z" fill-rule="evenodd"/><path stroke="#000000" stroke-width="1.0" stroke-linejoin="round" stroke-linecap="butt" stroke-dasharray="4.0,3.0" d="m12.490813 35.047592l0 0c0 -9.625313 7.8028574 -17.42817 17.42817 -17.42817l372.41925 0c4.6222534 0 9.055176 1.8361759 12.323578 5.1045933c3.2684326 3.2684155 5.1046143 7.701338 5.1046143 12.323576l0 69.71059c0 9.625313 -7.802887 17.42817 -17.428192 17.42817l-372.41925 0c-9.625313 0 -17.42817 -7.8028564 -17.42817 -17.42817z" fill-rule="evenodd"/><path fill="#ffffff" d="m134.21727 30.797377l127.401566 0l0 33.5748l-127.401566 0z" fill-rule="evenodd"/><path stroke="#000000" stroke-width="1.0" stroke-linejoin="round" stroke-linecap="butt" d="m134.21727 30.797377l127.401566 0l0 33.5748l-127.401566 0z" fill-rule="evenodd"/><path fill="#000000" d="m144.6079 54.504776l0 -13.359375l2.65625 0l3.15625 9.453125q0.4375 1.328125 0.640625 1.984375q0.234375 -0.734375 0.703125 -2.140625l3.203125 -9.296875l2.375 0l0 13.359375l-1.703125 0l0 -11.171875l-3.875 11.171875l-1.59375 0l-3.859375 -11.375l0 11.375l-1.703125 0zm21.697052 -1.1875q-0.921875 0.765625 -1.765625 1.09375q-0.828125 0.3125 -1.796875 0.3125q-1.59375 0 -2.453125 -0.78125q-0.859375 -0.78125 -0.859375 -1.984375q0 -0.71875 0.328125 -1.296875q0.328125 -0.59375 0.84375 -0.9375q0.53125 -0.359375 1.1875 -0.546875q0.46875 -0.125 1.453125 -0.25q1.984375 -0.234375 2.921875 -0.5625q0.015625 -0.34375 0.015625 -0.421875q0 -1.0 -0.46875 -1.421875q-0.625 -0.546875 -1.875 -0.546875q-1.15625 0 -1.703125 0.40625q-0.546875 0.40625 -0.8125 1.421875l-1.609375 -0.21875q0.21875 -1.015625 0.71875 -1.640625q0.5 -0.640625 1.453125 -0.984375q0.953125 -0.34375 2.1875 -0.34375q1.25 0 2.015625 0.296875q0.78125 0.28125 1.140625 0.734375q0.375 0.4375 0.515625 1.109375q0.078125 0.421875 0.078125 1.515625l0 2.1875q0 2.28125 0.109375 2.890625q0.109375 0.59375 0.40625 1.15625l-1.70
Reference in a new issue Copy permalink