* working with dataframes with too many rows and columns
* repr works for jupyter notebooks now
* added comments and test file
* added repr test file to .travis.yml
* added back ray.dataframe as pd to test file
* fixed pandas importing issues in test file
* getting the front and back of df more efficiently
* only keeping dataframe tests in travis
* fixing numpy array for row and col lengths issue
* doesn't add dimensions if df is small enough
* implemented memory_usage()
* completed memory_usage - still failing 2 tests
* only failing one test for memory_usage
* all repr and dataframes tests passing now
* fixing error related to python2 in info()
* fixing python2 errors
* fixed linting errosr
* using _arithmetic_helper in memory_usage()
* fixed last lint error
* removed testing-specific code
* adding back travis test
* removing extra tests from travis
* re-added concat test
* fixes with new indexing scheme
* code cleanup
* fully working with new indexing scheme
* added tests for info and memory_usage
* removed test file
* baseline impl for index_df.py
* added skeleton for index_df.py
* initial impl index_df
* separate out partition and non-partition impls
* add len function
* drop returns index_df slice of dropped indices
* housecleaning
* Integrate index overhaul
* Rename index df to index metadata
* Fix flake8 issues
* Addressing issues
* fix import issue
* Added metadata passing to constructor
adding tests
fixing flake8
adding init
flake 8 on test
fixing tests, imports, and flake8
handling for index
adding tests for row, index
added more robust error handling for axis
fixing test failures
cleaning up error sfor 2.7
updating travis
resolving import
fixing flake8
moved import order
Fixing to refactor and delaying implementing ray-pd inner concat
resolving ray-pd concat and from_pandas mutation
Revert "resolving ray-pd concat and from_pandas mutation"
This reverts commit 5db43e4e89e328286532f3ef98a4526575c5d08d.
* Add raylet monitor script to timeout Raylet heartbeats
* Unit test for removing a different client from the client table
* Set node manager heartbeat according to global config
* Doc and fixes
* Add regression test for client table disconnect, refactor client table
* Convert 'Terminate' methods to destructors
* Destroy the Raylet on a SIGTERM
* Clean up workers on a SIGTERM
* Add raylet monitor script to timeout Raylet heartbeats
* Unit test for removing a different client from the client table
* Set node manager heartbeat according to global config
* Doc and fixes
* Add regression test for client table disconnect, refactor client table
* Fix linting.
* Integrate worker with raylet.
* Begin allowing worker to attach to cluster.
* Fix linting and documentation.
* Fix linting.
* Comment tests back in.
* Fix type of worker command.
* Remove xray python files and tests.
* Fix from rebase.
* Add test.
* Copy over raylet executable.
* Small cleanup.
Summary:
Able to run 1000 tasks with object dependencies on a set of distributed Raylets.
Raylet Changes:
Finalized ClientConnection class.
Task forwarding.
NM-to-NM heartbeats.
NM resource accounting for tasks.
Simple scheduling policy with task forwarding.
Creating and maintaining NM 2 NM long-lived connections and reusing them for task forwarding.
LineageCache Changes:
LineageCache without cleanup of tasks committed by remote nodes.
Lineage cache writeback and cleanup implementation.
ObjectManager Changes:
Object manager event loop/ClientConnection refactor.
Multithreaded object manager (disabled in this PR).
Testing Changes:
Integration tests for task submission on multiple Raylets.
Stress tests for object manager (with GCS and object store integration).
Co-authored-by: Stephanie Wang <swang@cs.berkeley.edu>
Co-authored-by: Alexey Tumanov <atumanov@gmail.com>
* TABLE_APPEND call
* Convert callbacks back to taking in a string...
* GCS returns flatbuffers, define Log class
* Cleanups
* Modify client table to use the Log interface
* Fix bug where we replied twice from redis
* Fixes
* lint
* Compile and test raylet TaskTable
* Modify GCS tables to handle unique_ptrs from nested flatbuffers
* Add raylet::TaskTable unit tests to replace ObjectTable ones
* Convert ObjectTable to a log
* Convert ObjectTable tests to the Log
* AppendAt Redis and gcs Log command
* unit test for AppendAt
* Add a Log for task reconstruction data
* Add check for unique entries in TABLE_APPEND
* Documentation
* TABLE_APPEND call
* Convert callbacks back to taking in a string...
* GCS returns flatbuffers, define Log class
* Cleanups
* Modify client table to use the Log interface
* Fix bug where we replied twice from redis
* Fixes
* lint
* Compile and test raylet TaskTable
* Modify GCS tables to handle unique_ptrs from nested flatbuffers
* Add raylet::TaskTable unit tests to replace ObjectTable ones
* Convert ObjectTable to a log
* Convert ObjectTable tests to the Log
* TABLE_APPEND call
* Convert callbacks back to taking in a string...
* GCS returns flatbuffers, define Log class
* Cleanups
* Modify client table to use the Log interface
* Fix bug where we replied twice from redis
* Fixes
* lint
* [tune] Added pbt with keras on cifar10 dataset example
* ENH: add gpu resources
* CLN: requires 4 GPUs resource
* CLN: use single quotes
* CLN: don't save model by default
