Commit graph

15 commits

Author SHA1 Message Date
atumanov
1c946b2f6a Factoring out object_info structure for use in several Ray components (#101)
* change plasma object notifications to carry a struct of information

* factoring out object_info for general use by several Ray components

* fixing a bug in python test

* addressing comments

* handling Robert's comments

* clang format

* Fix valgrind.
2016-12-08 19:14:10 -08:00
Robert Nishihara
dd39008532 Fix bug in which we were running out of file descriptors. (#91) 2016-12-05 23:45:49 -08:00
Robert Nishihara
2a3e9267f8 Non-blocking fetch implementation. (#83)
* Non-blocking fetch implementation.

* Make fetch tests more robust to timing issues.

* Bug fix when ignoring transferred objects.

* Fix.

* Documentation fixes.
2016-12-03 19:09:05 -08:00
Robert Nishihara
a93c6b7596 Quick workaround for ray.wait bug. (#42) 2016-11-22 13:31:22 -08:00
Robert Nishihara
d77b685a90 Global scheduler skeleton (#45)
* Initial scheduler commit

* global scheduler

* add global scheduler

* Implement global scheduler skeleton.

* Formatting.

* Allow local scheduler to be started without a connection to redis so that we can test it without a global scheduler.

* Fail if there are no local schedulers when the global scheduler receives a task.

* Initialize uninitialized value and formatting fix.

* Generalize local scheduler table to db client table.

* Remove code duplication in local scheduler and add flag for whether a task came from the global scheduler or not.

* Queue task specs in the local scheduler instead of tasks.

* Simple global scheduler tests, including valgrind.

* Factor out functions for starting processes.

* Fixes.
2016-11-18 19:57:51 -08:00
Philipp Moritz
986ed5c9e8 Plasma C extensions (#34)
* switch plasma from ctypes to python C API

* clang-format

* various fixes
2016-11-13 16:23:28 -08:00
Robert Nishihara
90f88af902 Fix bug in which worker import counters were treated incorrectly. (#28)
* Fix bug in which worker import counters were treated incorrectly.

* Fix bug in which cached functions-to-run were double counted as exports. This also runs the functions-to-run on the driver only after ray.init is called.

* Only define reusable variables locally after ray.init has been called.

* Remove flaky reference counting tests. It's not clear that these tests make sense.

* Make numbuf pip install verbose.

* Export cached reusable variables before cached remote functions.

* Fix bug causing the worker to hang sometimes. This happens when the worker is trying to run a task, but it hasn't imported enough imports to run the task, so it continually acquires and releases a lock while checking if it has enough imports. However, for some reason, the import thread is waiting to acquire the same lock and never does so (or takes a very long time to do so). By dropping the lock before sleeping, this makes it easier for other threads to acquire the lock.

* Acquire locks using 'with' statements.

* Fix possible test failure.

* Try to start Redis multiple times with different random ports if the original attempt failed.

* Fix test in which we redefine a remote function.
2016-11-06 22:24:39 -08:00
Philipp Moritz
1147c4d34b Keep objects in cache between tasks (#29)
* fix caching behavior

* fixes
2016-11-06 17:31:14 -08:00
Robert Nishihara
efe8a295ea Add basic LRU eviction for the plasma store. (#26)
* Basic functionality for LRU eviction.

* Test eviction.

* Factor out eviction policy.

* Move delete_object into eviction policy.

* Replace array of released objects with an LRU cache (hash table + doubly linked list).

* Finish rebase on master.

* Move actual object deletion away from eviction policy and into plasma store.

* Small fixes.

* Fixes.

* Make remove_object_from_lru_cache always remove the object.

* Minor formatting and comments.

* Pass in allowed memory as argument to Plasma store.

* Small fix.
2016-11-05 21:34:11 -07:00
Philipp Moritz
90a2aa4bf7 Various performance improvements (#24)
* switch from array to linked list for photon queue

* performance optimizations

* fix tests

* various fixes
2016-11-04 00:41:20 -07:00
Robert Nishihara
681ec570ba Remove unnecessary pip installs. (#21)
* Small cleanups in worker.py.

* Remove dependencies on subprocess32, graphviz, protobuf, and ipython.

* Retry starting the plasma manager if the port is in use.

* Whitespace

* Move start_plasma_manager into plasma.py.
2016-11-02 16:40:37 -07:00
Robert Nishihara
072f442c1f Update worker.py and services.py to use plasma and the local scheduler. (#19)
* Update worker code and services code to use plasma and the local scheduler.

* Cleanups.

* Fix bug in which threads were started before the worker mode was set. This caused remote functions to be defined on workers before the worker knew it was in WORKER_MODE.

* Fix bug in install-dependencies.sh.

* Lengthen timeout in failure_test.py.

* Cleanups.

* Cleanup services.start_ray_local.

* Clean up random name generation.

* Cleanups.
2016-11-02 00:39:35 -07:00
Philipp Moritz
e6319986ec Implement plasma.wait (#12)
* implement wait

* fix wait documentation

* fixes

* More fixes.

* more fixes

* Update test to check if redis-server is present.
2016-10-29 17:30:34 -07:00
Philipp Moritz
84c581cf47 Let local clients connect with local manager via unix domain sockets (#11)
* let local clients connect with local manager via unix domain sockets

* fix clang-format

* fix tests

* fix stephanie's comments
2016-10-28 21:26:54 -07:00
Robert Nishihara
02d4050499 Rearrange plasma files to prepare for merging into Ray. 2016-10-25 14:27:45 -07:00
Renamed from lib/python/plasma.py (Browse further)