Commit graph

667 commits

Author SHA1 Message Date
Philipp Moritz
0b888c38f1 Merge numbuf. 2016-11-19 12:30:01 -08:00
Robert Nishihara
c8c3983195 Use sizeof(field) instead of sizeof(type) and other fixes. (#47)
* Use sizeof(field) instead of sizeof(type) and other fixes.

* Fix formatting.

* Bug fix.

* Zero-initialize structs. There are many more instances of these that I haven't changed yet.

* Bug fix.

* Revert from atexit to signaling to fix valgrind tests.

* Address Philipp's comments.
2016-11-19 12:19:49 -08:00
Philipp Moritz
c3ab68e88c Rebase numbuf to latest arrow (#23)
* rebase to latest arrow

* fix arrow linking

* finish rebase

* point arrow to pcmoritz's arrow

* fix

* fix macOS

* fix
2016-11-19 02:27:21 -08:00
Robert Nishihara
d77b685a90 Global scheduler skeleton (#45)
* Initial scheduler commit

* global scheduler

* add global scheduler

* Implement global scheduler skeleton.

* Formatting.

* Allow local scheduler to be started without a connection to redis so that we can test it without a global scheduler.

* Fail if there are no local schedulers when the global scheduler receives a task.

* Initialize uninitialized value and formatting fix.

* Generalize local scheduler table to db client table.

* Remove code duplication in local scheduler and add flag for whether a task came from the global scheduler or not.

* Queue task specs in the local scheduler instead of tasks.

* Simple global scheduler tests, including valgrind.

* Factor out functions for starting processes.

* Fixes.
2016-11-18 19:57:51 -08:00
Wapaul1
08707f9408 Integration of Webui with Ray (#32)
* Initial integration of webui with ray

* Re-organized calling of build-webui in setup.py

* Fixed Lint comments on js code

* Fixed more lint issues

* Fixed various issues

* Fixed directory in services.py

* Small changes.

* Changes to match lint
2016-11-17 22:33:29 -08:00
Robert Nishihara
c3db225cbe Patch arrow to work with xcode 8. This will be unecessary when we rebase on an updated arrow. (#22) 2016-11-16 19:43:26 -08:00
Stephanie Wang
7babe0d22f Logging level (#38)
* Set logging levels in Makefile using -DRAY_COMMON_LOG_LEVEL=level

* Lower level of some LOG_ERROR messages, log the name of the table operation on failure

* Address rest of Robert's comments

* Fix spurious log message
2016-11-15 20:33:29 -08:00
Philipp Moritz
986ed5c9e8 Plasma C extensions (#34)
* switch plasma from ctypes to python C API

* clang-format

* various fixes
2016-11-13 16:23:28 -08:00
Robert Nishihara
ad6a401740 Get rid of old git submodule init calls. (#41) 2016-11-13 14:55:28 -08:00
Robert Nishihara
43aba4fc6d Install python package dependencies through setup.py. (#39)
* Install python package dependencies through setup.py.

* Do pip installs without sudo.

* Clearer instructions for running tests.

* Wording.
2016-11-12 19:37:20 -08:00
Robert Nishihara
488f881d70 Install numpy before building numbuf in setup.py. (#20) 2016-11-12 19:36:47 -08:00
Philipp Moritz
d26fbcebbe switch to release mode (#19) 2016-11-12 19:35:46 -08:00
Robert Nishihara
28a80ca058 Remove out-of-date documentation. (#40) 2016-11-12 19:34:22 -08:00
Robert Nishihara
5c89a7ab4e Fix bug in which gym environment could not be imported because of module naming conflicts. (#37) 2016-11-11 19:02:40 -08:00
Philipp Moritz
b1083e92e9 fix bug in wait (#35) 2016-11-11 17:05:40 -08:00
Robert Nishihara
336a904404 Implement repr, hash, and richcompare for ObjectIDs. (#33)
* Implement repr, hash, and richcompare for ObjectIDs.

* Addressing comments.

* Partially fix example applications.
2016-11-11 09:18:36 -08:00
Stephanie Wang
9d1e750e8f Merge task table and task log into a single table (#30)
* Merge task table and task log

* Fix test in db tests

* Address Robert's comments and some better error checking

* Add a LOG_FATAL that exits the program
2016-11-10 18:13:26 -08:00
Robert Nishihara
194bdb1d96 Compute task IDs and object IDs deterministically. (#31)
* Put infrastructure in place to compute task IDs and object IDs.

* Fix version number for common library.

* Compute task IDs and object IDs deterministically.

* Address Stephanie's comments.

* Update task documentation.

* Fix formatting.

* Add more tests and checks.

* Fix formatting.

* Enable DCHECKs and change CHECKs to DCHECKs.
2016-11-08 14:46:34 -08:00
Robert Nishihara
90f88af902 Fix bug in which worker import counters were treated incorrectly. (#28)
* Fix bug in which worker import counters were treated incorrectly.

* Fix bug in which cached functions-to-run were double counted as exports. This also runs the functions-to-run on the driver only after ray.init is called.

* Only define reusable variables locally after ray.init has been called.

* Remove flaky reference counting tests. It's not clear that these tests make sense.

* Make numbuf pip install verbose.

* Export cached reusable variables before cached remote functions.

* Fix bug causing the worker to hang sometimes. This happens when the worker is trying to run a task, but it hasn't imported enough imports to run the task, so it continually acquires and releases a lock while checking if it has enough imports. However, for some reason, the import thread is waiting to acquire the same lock and never does so (or takes a very long time to do so). By dropping the lock before sleeping, this makes it easier for other threads to acquire the lock.

* Acquire locks using 'with' statements.

* Fix possible test failure.

* Try to start Redis multiple times with different random ports if the original attempt failed.

* Fix test in which we redefine a remote function.
2016-11-06 22:24:39 -08:00
Philipp Moritz
1147c4d34b Keep objects in cache between tasks (#29)
* fix caching behavior

* fixes
2016-11-06 17:31:14 -08:00
Robert Nishihara
efe8a295ea Add basic LRU eviction for the plasma store. (#26)
* Basic functionality for LRU eviction.

* Test eviction.

* Factor out eviction policy.

* Move delete_object into eviction policy.

* Replace array of released objects with an LRU cache (hash table + doubly linked list).

* Finish rebase on master.

* Move actual object deletion away from eviction policy and into plasma store.

* Small fixes.

* Fixes.

* Make remove_object_from_lru_cache always remove the object.

* Minor formatting and comments.

* Pass in allowed memory as argument to Plasma store.

* Small fix.
2016-11-05 21:34:11 -07:00
Philipp Moritz
90a2aa4bf7 Various performance improvements (#24)
* switch from array to linked list for photon queue

* performance optimizations

* fix tests

* various fixes
2016-11-04 00:41:20 -07:00
Robert Nishihara
1e2b3ceac9 Fix whitespace. (#27) 2016-11-03 17:36:55 -07:00
Ujval Misra
b370a1df57 Merge sealed_objects and open_objects into a single hashmap (#25)
* Merge sealed_objects and open_objects into a single hashmap

* Entry contains enum that determines whether it is open or closed

* Removed unused variable.

* Applied Robert's patch

* Fixed styling.
2016-11-03 17:33:54 -07:00
Robert Nishihara
5dfd76e0f7 Causes build scripts to fail immediately if a single command fails. (#23) 2016-11-02 20:56:25 -07:00
Robert Nishihara
681ec570ba Remove unnecessary pip installs. (#21)
* Small cleanups in worker.py.

* Remove dependencies on subprocess32, graphviz, protobuf, and ipython.

* Retry starting the plasma manager if the port is in use.

* Whitespace

* Move start_plasma_manager into plasma.py.
2016-11-02 16:40:37 -07:00
Robert Nishihara
072f442c1f Update worker.py and services.py to use plasma and the local scheduler. (#19)
* Update worker code and services code to use plasma and the local scheduler.

* Cleanups.

* Fix bug in which threads were started before the worker mode was set. This caused remote functions to be defined on workers before the worker knew it was in WORKER_MODE.

* Fix bug in install-dependencies.sh.

* Lengthen timeout in failure_test.py.

* Cleanups.

* Cleanup services.start_ray_local.

* Clean up random name generation.

* Cleanups.
2016-11-02 00:39:35 -07:00
Philipp Moritz
2068587af8 Fix socket bind collisions in manager_tests, race condition bind/listen/subscribe and memory leaks (#18)
* Fix socket bind collisions in manager_tests

* bind manager sockets before connecting to the store

* fix memory leak in tests

* fix valgrind early process termination

* fix bind/listen/subscribe race condition

* fix photon

* fix other tests

* make it that all of common is tested

* fix clang-format

* fix
2016-11-02 00:09:04 -07:00
Robert Nishihara
d56c1a0b9c Change license to Apache 2 (#20) 2016-11-01 23:19:06 -07:00
Robert Nishihara
47851eeccc Build Ray with setup.py. (#14)
* Build Ray with setup.py.

* Building photon extensions with cmake.

* Fix formatting in photon_extension.c

* Pip install with sudo in Travis.

* Fix plasma __init__.py.

* Rename and remove some files.
2016-10-31 17:08:03 -07:00
Philipp Moritz
695f23b2e0 try again to start plasma manager if there is a port collision (#16) 2016-10-31 15:00:15 -07:00
Philipp Moritz
780bbd170a fix test (#15) 2016-10-31 13:42:56 -07:00
Robert Nishihara
a0049ffa48 Wait until data is ready to read in manager_tests. (#13)
* Wait until data is ready to read in manager_tests.

* remove race conditions from plasma manager test and make CHECK and CHECKM print a backgrace
2016-10-29 19:25:43 -07:00
Philipp Moritz
e6319986ec Implement plasma.wait (#12)
* implement wait

* fix wait documentation

* fixes

* More fixes.

* more fixes

* Update test to check if redis-server is present.
2016-10-29 17:30:34 -07:00
Ion
ee3718c80c Ion and Philipp's table retries (#10)
* Ion and Philipp's table retries

* Refactor the retry struct:
- Rename it from retry_struct to retry_info
- Retry information contains the failure callback, not the retry callback
- All functions take in retry information as an arg instead of its expanded fields

* Rename cb -> callback

* Remove prints

* Fix compiler warnings

* Change some CHECKs to greatest ASSERTs

* Key outstanding callbacks hash table with timer ID instead of callback data pointer

* Use the new retry API for table commands

* Memory cleanup in plasma unit tests

* fix Robert's comments

* add valgrind for common
2016-10-29 15:22:33 -07:00
Philipp Moritz
84c581cf47 Let local clients connect with local manager via unix domain sockets (#11)
* let local clients connect with local manager via unix domain sockets

* fix clang-format

* fix tests

* fix stephanie's comments
2016-10-28 21:26:54 -07:00
Robert Nishihara
09a3ff7173 Pip install numbuf. (#8) 2016-10-28 14:30:20 -07:00
Robert Nishihara
c94f961366 Merge pull request #9 from ray-project/port
Unit testing for the plasma manager.
2016-10-28 12:42:45 -07:00
Stephanie Wang
f3a43179c8 Unit testing for the plasma manager. 2016-10-28 11:56:22 -07:00
Robert Nishihara
5c50d97670 Update .gitignore file. (#7) 2016-10-28 11:40:08 -07:00
Robert Nishihara
d1974afbab Add setup.py and travis. (#8)
* Add setup.py and travis.

* Add script to install dependencies for travis.

* Put Arrow into thirdparty.

* Add .gitmodules file.

* Install boost dependencies.

* Make tests verbose.

* Build arrow and numbuf in setup.py.

* Change build_ext to install.

* Switch develop to install in pip.

* Test

* fix numbuf build and installation

* make import numbuf work

* fix documentation

* fix tests

* quotes
2016-10-27 23:40:10 -07:00
Robert Nishihara
43aca17b59 Merge pull request #6 from ray-project/subscribe_no_polling
[Plasma] Subscribe without polling
2016-10-27 17:56:16 -07:00
Philipp Moritz
c9d0ca49eb add documentation 2016-10-27 16:30:29 -07:00
Philipp Moritz
b4b462809f fix valgrind 2016-10-27 15:09:50 -07:00
Philipp Moritz
e26fb46b02 subscribe without polling 2016-10-27 13:21:16 -07:00
Robert Nishihara
6ed641177d Remove unnecessary files. (#4) 2016-10-26 23:24:40 -07:00
Robert Nishihara
6f75c738b5 [WIP] Fix valgrind tests. (#5)
* Make tests fail when valgrind finds a memory leak.

* Properly clean up scheduler state.

* Remove unnecessary malloc.
2016-10-26 23:23:46 -07:00
Robert Nishihara
f83a98d71a Changes to make tests pass on Travis. (#3)
* Remove hiredis submodule.

* Squashed 'src/common/thirdparty/hiredis/' content from commit acd1966

git-subtree-dir: src/common/thirdparty/hiredis
git-subtree-split: acd1966bf7f5e1be74b426272635c672def78779

* Make Plasma tests pass.

* Make Photon tests pass.

* Compile and test with Travis.

* Deactive fetch test so that the tests pass.
2016-10-25 22:39:21 -07:00
Philipp Moritz
ed550ea8af Merge pull request #1 from ray-project/switch
Move plasma, common, and photon into the Ray repository.
2016-10-25 14:33:51 -07:00
Robert Nishihara
8c48410e82 Merge remote-tracking branch 'r1remote/moveout' into switch 2016-10-25 14:29:00 -07:00