* Add regression tests
* Refactor, split RemoveSubmittedTaskReferences into submitted and finished paths
* Add nested return IDs to UpdateFinishedTaskRefs, rename WrapObjectIds
* Basic unit tests pass
* Fix unit test and add an out-of-order regression test
* Add stored_in_objects to ObjectReferenceCount, regression test now passes
* Add an Address to the ReferenceCounter so we can determine ownership
* Set the nested return IDs from the TaskManager
* Add another test
* Simplify
* Update src/ray/core_worker/reference_count_test.cc
Co-Authored-By: Edward Oakes <ed.nmi.oakes@gmail.com>
* Apply suggestions from code review
Co-Authored-By: Edward Oakes <ed.nmi.oakes@gmail.com>
* comments
* Add python test
Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com>
* Skeleton plus a unit test for simple borrower case
* First unit test passes - forward an ID and task returns with 1 submitted task pending on the inner ID
* Invariant for contained_in
* Unit test passes for testing task return without creating a borrower
* Wrap ref count functionality in test case
* Fix bad delete
* Unit test and fix for borrowers creating more borrowers
* Unit test and fix for simple borrowing, but owner sends call after borrower's ref count goes to 0
* Refactor:
- keep a sentinel ref count for task argument IDs
- keep contained_in_borrowed in addition to contained_in_owned
* Unit test for nested IDs passes
* Refactor so that an object ID can only be contained in 1 borrowed ID at a time
* Add check
* Fix
* Unit test (passes) to test nesting object IDs but no borrowers created
* Unit test for nested objects from different owners passes, refactor to unset contained_in when popping refs
* Unit tests for borrowers receiving an ObjectID from multiple sources,
skip adding ownership info if we already have it to handle duplicate
refs
* Unit test for returning object ID passes
* More unit tests for returning object IDs pass
* Add serialized ID tests
* fix serialization issue
* remove swap
* It builds!
* debugging and some fixes:
- register handler for WaitForRefRemoved
- don't create a python reference for arg IDs
- pass in client factory into ReferenceCounter
- fix bad decrement in PopBorrowerRefs
* Fix accounting for serialized IDs:
- don't decrement for IDs on dependency resolution, wait until task finished
- add object IDs that were inlined when building the arguments to the task spec, pin these on the task executor until task finishes
* mu_ -> mutex_
* lint
* fix build
* clear outer_object_id
* add direct call type check
* Fix test for direct call IDs and return IDs for actor calls
* Fix CoreWorkerClient.Addr()
* Remove unneeded lock
* Remove unnecessary ObjectID refs
* Fix worker holding serialized refs test
* Fix hex IDs
* fix
* fix tests
* fix tests
* refactor and cleanups
* lint
* Put inlined Ids in task args and some cleanup
* Add back gc.collect() line for test case
* Refactor and fixes:
- store inlined IDs in RayObject
- allow storing objects with inlined IDs in memory store
- pin objects that were promoted to plasma
* oops
* make sure worker ID is set in address, pass in rpc::Address to CoreWorkerClient
* todos
* cleanups and test builds
* Fix tests
* Add feature flag
* cleanups
* address comments and some cleanups
* cleanup
* fix recursive test
* Comments for tests
* Turn off ref counting by default
* Skip tests
* Fix some bugs for test_array.py, java build
* Don't include nested objects in the ref count when the feature flag is off
* C++ feature flag does not work...
* Remove
* Turn on python tests and add a warning when plasma objects are evicted before being pinned
* Fix build and remove irrelevant test
* Fix for java
* Revert "Fix build and remove irrelevant test"
This reverts commit 056cca9b263ed05b0f9ab2250907338edcbca2d5.
* Fix ray.internal.free
* Fixes and skip some flaky tests
* fix java build
* fix windows build
* Add IDs contained in owned objects
* Update src/ray/protobuf/core_worker.proto
Co-Authored-By: Edward Oakes <ed.nmi.oakes@gmail.com>
* Update src/ray/core_worker/reference_count.cc
Co-Authored-By: Edward Oakes <ed.nmi.oakes@gmail.com>
* Update src/ray/protobuf/core_worker.proto
Co-Authored-By: Edward Oakes <ed.nmi.oakes@gmail.com>
* Update src/ray/protobuf/core_worker.proto
Co-Authored-By: Edward Oakes <ed.nmi.oakes@gmail.com>
* Update src/ray/core_worker/reference_count.h
Co-Authored-By: Edward Oakes <ed.nmi.oakes@gmail.com>
* Update src/ray/core_worker/reference_count.h
Co-Authored-By: Edward Oakes <ed.nmi.oakes@gmail.com>
* Update src/ray/core_worker/reference_count.cc
Co-Authored-By: Edward Oakes <ed.nmi.oakes@gmail.com>
* Apply suggestions from code review
Co-Authored-By: Edward Oakes <ed.nmi.oakes@gmail.com>
* update
* Try to fix ::test_direct_call_serialized_id_eviction
Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com>
* Revert "Revert "Use Boost.Process instead of pid_t (#6510)" (#6909)"
This reverts commit bde575b8dd.
* Process wrapper, using Boost.Process on Windows
- Reverts bde575b8dd.
- Re-applies fb8e3615d5 after some refactoring.
* Remove Boost.Process dependency
* Don't open /proc file on Linux
* Change FATAL to ERROR and modify error message when process doesn't exist
* Fix common.fbs rename (due to apache/arrow/commit/bef9a1c251397311a6415d3dc362ef419d154caa)
* Add missing COPTS
* Use socketpair(AF_INET) if boost::asio::local is unavailable (e.g. on Windows)
* Fix compile bug in service_based_gcs_client_test.cc (fix build breakage in #6686)
* Work around googletest/gmock inability to specify override to avoid -Werror,-Winconsistent-missing-override
* Fix missing override on IsPlasmaBuffer()
* Fix missing libraries for streaming
* Factor out install-toolchains.sh
* Put some Bazel flags into .bazelrc
* Fix jni_md.h missing inclusion
* Add ~/bin to PATH for Bazel
* Change echo $$(date) > $@ to date > $@
* Fix lots of unquoted paths
* Add system() call checks for Windows
Co-authored-by: GitHub Web Flow <noreply@github.com>
* ensure data_ won't be null_ptr when size == 0
* when data_sizes[i] == 0, we should Allocate an empty buffer
* work around for pyarrow.py_buffer
* fix comments
* add null ptr check
* add test for bytes
* lint
