Commit graph

898 commits

Author SHA1 Message Date
Philipp Moritz
08e988aee5 Modernize plasma store (C to C++ changes). (#546) 2017-05-15 01:19:44 -07:00
Eric Liang
e2e9e4ce6f Fix segmentation fault when calling ray.put on a dictionary with object keys (#548)
* fix segfault when serializing dict key

* fix style

* fix test

* Fix linting.
2017-05-15 01:09:13 -07:00
Robert Nishihara
3c5375345f Initial version of evolution strategies example. (#544)
* Initial commit of evolution strategies example.

* Some small simplifications.

* Update example to use new API.

* Add example to documentation.
2017-05-14 17:53:51 -07:00
Robert Nishihara
9f91eb8c91 Change API for remote function declaration, actor instantiation, and actor method invocation. (#541)
* Direction substitution of @ray.remote -> @ray.task.

* Changes to make '@ray.task' work.

* Instantiate actors with Class.remote() instead of Class().

* Convert actor instantiation in tests and examples from Class() to Class.remote().

* Change actor method invocation from object.method() to object.method.remote().

* Update tests and examples to invoke actor methods with .remote().

* Fix bugs in jenkins tests.

* Fix example applications.

* Change @ray.task back to @ray.remote.

* Changes to make @ray.actor -> @ray.remote work.

* Direct substitution of @ray.actor -> @ray.remote.

* Fixes.

* Raise exception if @ray.actor decorator is used.

* Simplify ActorMethod class.
2017-05-14 00:01:20 -07:00
Robert Nishihara
22c6a22f28 Add flatbuffers dependency to setup.py. (#540) 2017-05-11 23:39:34 -07:00
Philipp Moritz
3a6922276a convert malloc.c to STL (#537)
* convert malloc.c to STL

* linting

* cleanup and comments

* address Richard's comments
2017-05-11 11:18:23 -07:00
Philipp Moritz
c1e9496a06 fix problem if old version of arrow is cloned (#538) 2017-05-10 12:16:07 -07:00
Philipp Moritz
3a0e86395e Convert eviction code to STL (#534)
* temp commit

* convert eviction policy to C++

* temp commit

* fix plasma tests

* fix

* linting

* fixes

* fix linting
2017-05-09 21:26:22 -07:00
Robert Nishihara
b4788ae518 Only export actor classes once. (#510)
* Only export actor classes once.

* Fix linting.

* Fixes after rebase.
2017-05-09 19:49:23 -07:00
Philipp Moritz
118fac5619 Remove boost dependencies from Ray (#518)
* remove boost regex

* workaround for boost

* fix

* do not link against boost any more

* rebased on arrow change
2017-05-09 16:17:20 -07:00
Philipp Moritz
e5e2aab5e4 upgrade arrow and fix bug (#530)
* upgrade arrow and fix bug

* fixes suggested by Wes
2017-05-09 13:58:42 -07:00
Robert Nishihara
1f991b6389 Change /tmp/raylogs permissions so multiple users can log there. (#532) 2017-05-09 12:15:31 -07:00
Philipp Moritz
0681107039 add serializing numpy boolean (#529) 2017-05-08 22:24:02 -07:00
Robert Nishihara
f32368bcbe Prevent actors from being placed on removed nodes or nodes with no CPUs. (#527)
* Make note about bug in which actor creation notification message is not received.

* Prevent actors from being created on removed nodes.

* Prevent actors from being created on nodes with no CPUs.

* Fix linting.

* Add test for scheduling actors on local schedulers with no CPUs.

* Improve error message when actors created before ray.init called.
2017-05-08 20:39:43 -07:00
Robert Nishihara
c688a64235 Expose GPU IDs to remote functions. (#496)
* Change local scheduler bookkeeping to use GPU IDs.

* Update actor test.

* Add tests for actors and tasks simultaneously using GPUs.

* Add additional task GPU ID test.

* Fix linting.

* Make redis GPU assignment ignore GPU IDs.

* Small fix.
2017-05-07 13:03:49 -07:00
Robert Nishihara
35dbdcc4f5 Make all export IDs unique. (#522)
* Make all export IDs unique.

* Work around test failure.
2017-05-06 21:17:25 -07:00
Feynman Liang
7825aed230 Fix build-docker.sh bug (#515)
* Build examples by default

The [documentation](http://ray.readthedocs.io/en/latest/install-on-docker.html#build-docker-images)  says that the `examples` image is built by `build-docker.sh` script.

* Update build-docker.sh
2017-05-06 18:57:08 -07:00
Philipp Moritz
1dddd5336a Fix actor bug arising from overwriting task specifications in the local scheduler (#513)
* copy task specifications put into the actor task cache so it won't get overwritten when the scheduler receives the next task

* cleanup

* cleanup and fix

* linting

* fix jenkins test

* fix linting
2017-05-06 17:39:35 -07:00
Robert Nishihara
8532ba4272 Serialize lambdas, sets, and types with pickle by default. (#511)
* Serialize lambdas with pickle by default.

* Serialize sets with pickle by default.

* Serialize types with pickle by default.

* Small update to documentation.

* Update tests.
2017-05-04 00:16:35 -07:00
Alexey Tumanov
7deff73e55 update documentation on how to set ulimit for Redis (#508)
* update documentation on how to set ulimit for Redis

* Move documentation to troubleshooting section.

* Add another section header.

* update redis file descriptor docs: simplify command
2017-05-03 17:24:57 -07:00
Robert Nishihara
245c8ab888 Make sure user seeding does not affect actor ID generation. (#506)
* Make sure user seeding does not affect actor ID generation.

* Fix linting.

* Add test.
2017-05-03 16:29:55 -07:00
Stephanie Wang
e50a23b820 Fix bug with reused file descriptors (#471)
* Fix bug with reused file descriptors

* Remove client connection if write_object_chunk fails

* Handle ECONNRESET on unsuccessful write

* lint

* Back to lowercase

* fix compilation

* fix linting
2017-05-02 19:45:27 -07:00
Robert Nishihara
2bbfc5da8d Dispatch actor tasks when actor connects. (#495) 2017-04-28 17:36:43 -07:00
Robert Nishihara
6d301d9079 Simplify resource bookkeeping in local scheduler. (#494)
* Simplify resource bookkeeping in local scheduler.

* Change ints to doubles.
2017-04-28 12:09:47 -07:00
Robert Nishihara
eea19371b7 Suppress warning about working dying when driver exits. (#492) 2017-04-26 23:52:13 -07:00
Robert Nishihara
1627f89945 Fix problem in which actors and workers running tasks are not killed by driver exit. (#490)
* Augment test to verify that relevant workers and actors are killed during driver cleanup.

* Fix bug in which we were only killing one worker when a driver exited.

* Fix remove driver test.

* Fix and augment test.
2017-04-26 15:13:39 -07:00
Philipp Moritz
b7ace01b5f Convert Plasma client to STL (#486)
* convert mmap table to STL

* update

* fix

* convert objects_in_use

* fix

* convert release_history

* cleanup

* linting

* update

* fix

* linting
2017-04-25 01:25:40 -07:00
Robert Nishihara
0ac125e9b2 Clean up when a driver disconnects. (#462)
* Clean up state when drivers exit.

* Remove unnecessary field in ActorMapEntry struct.

* Have monitor release GPU resources in Redis when driver exits.

* Enable multiple drivers in multi-node tests and test driver cleanup.

* Make redis GPU allocation a redis transaction and small cleanups.

* Fix multi-node test.

* Small cleanups.

* Make global scheduler take node_ip_address so it appears in the right place in the client table.

* Cleanups.

* Fix linting and cleanups in local scheduler.

* Fix removed_driver_test.

* Fix bug related to vector -> list.

* Fix linting.

* Cleanup.

* Fix multi node tests.

* Fix jenkins tests.

* Add another multi node test with many drivers.

* Fix linting.

* Make the actor creation notification a flatbuffer message.

* Revert "Make the actor creation notification a flatbuffer message."

This reverts commit af99099c8084dbf9177fb4e34c0c9b1a12c78f39.

* Add comment explaining flatbuffer problems.
2017-04-24 18:10:21 -07:00
Philipp Moritz
8194b71f32 Convert pending_notifications to STL (#484)
* temp commit

* converted more plasma notifications

* cleanup

* rename

* linting

* fixes

* fixes
2017-04-24 14:41:34 -07:00
Philipp Moritz
892e53d69e Convert plasma client array and object notification queue to STL (#482)
* Conver plasma clients to STL

* use a deque for object notifications in plasma store for perf

* cleanup

* linting

* fix include order
2017-04-24 00:43:48 -07:00
Philipp Moritz
e36de2dad1 Convert object table to STL (#480)
* convert object table to stl

* temp commit

* fix

* comments

* linting
2017-04-23 22:24:05 -07:00
Robert Nishihara
3a2eb1467b Fix failure to propagate error message. (#479) 2017-04-23 16:12:25 -07:00
Alexey Tumanov
a67a107e0e Fix int-type compilation problem on redhat. (#472) 2017-04-19 02:43:33 -07:00
Richard Shin
cf68cf743c Change UniqueID hash function to look at the lowest instead of highest bytes. (#469) 2017-04-18 15:31:49 -07:00
Philipp Moritz
8ac6c59931 Remove n^2 algorithm in plasma get (#466)
Remove n^2 algorithm in plasma get.
2017-04-17 23:37:33 -07:00
Guru Medasani
0189b09581 Fixes Mac OSX installation error (#464)
* changes to address ARROW-826 and ARROW-444

* changes to address ARROW-826 and ARROW-444

* ignoring cmake-build-debug

* additional IDEA ignore files

* additional IDEA ignore files

* remove arrow ipc and arrow io libraries

* add boost dependencies

* fix arrow origin and remove submodule
2017-04-16 15:02:15 -07:00
Robert Nishihara
c802e51d36 Re-enable recursive remote functions in a limited form. (#453)
* Re-enable recursive remote functions in a limited form.

* Fix linting.
2017-04-13 01:47:33 -07:00
Robert Nishihara
dad57e3b62 Convert actor data structures to C++. (#454) 2017-04-12 01:18:16 -07:00
Richard Liaw
94f32db5e6 A3C Polishing (#385)
* number

* gym doesn't have versioning

* Benchmarks

* visualization

* formatting

* small fix for tensorboard

* first pass removing universe dependency

* code

* results polish

* removed extra line

* removed universe dependency

* doc

* remove gym versioning stuff

* changes as suggested

* nit
2017-04-11 22:51:52 -07:00
Richard Liaw
c3a2505ffd Loadbalancing Test issue (#452)
* Limiting number of CPUs in loadbalancing test

* fixes as requested
2017-04-11 22:33:58 -07:00
Robert Nishihara
fb4525f833 Convert some local scheduler data structures to C++ STL. (#445)
* Convert more local scheduler data structures to C++ STL.

* Convert vector pointer to vector.

* Convert some of the UT_arrays to std::vector.

* Simplify worker vectors.

* Simplify remote_object and local_object containers.

* Change some unnecessary checks to DCHECK.
2017-04-10 21:02:36 -07:00
Philipp Moritz
6ffc849d23 Use Arrow Tensors for serializing numpy arrays and get rid of extra memcpy. (#436)
* Use Arrow Tensors for serializing numpy arrays and get rid of extra memcpy

* fix nondeterminism problem

* mark array as immutable

* make arrays contiguous

* fix serialize_list and deseralize_list

* fix numbuf tests

* linting

* add optimization flags

* fixes

* roll back arrow
2017-04-10 01:37:34 -07:00
Robert Nishihara
c9d66555e2 Fix bug in queue_task function in local scheduler. (#443) 2017-04-09 19:34:43 -07:00
Robert Nishihara
f4c1adae17 Unify function signature handling between remote functions and actor … (#441)
* Unify function signature handling between remote functions and actor methods.

* Fixes.

* Fix tests.
2017-04-08 21:34:13 -07:00
Alexey Tumanov
b6c4ae82c0 Increase redis client pubsub buffer size. (#442) 2017-04-08 15:24:07 -07:00
Robert Nishihara
7cd00741b1 Suppress irrelevant Redis connection errors. (#434)
* Suppress error messages in worker import thread when Redis terminates.

* Suppress some warnings from one of the tests.
2017-04-07 23:19:24 -07:00
Robert Nishihara
0eac3ccdd0 Reduce verbosity of component_failures_test.py. (#440) 2017-04-07 23:05:29 -07:00
Robert Nishihara
05fd4c2c37 Changes to local scheduler client protocol. (#435)
* Make local scheduler clients receive reply upon registration.

* Fix tests and linting.
2017-04-07 23:03:37 -07:00
Robert Nishihara
8061b3b596 Revert "Suppress warning in start_ray.sh about leaving child processes running when parent exits. (#429)" (#437)
This reverts commit 85b373a4be.
2017-04-07 17:32:28 -07:00
Alexey Tumanov
6f9225490b Plasma manager performance: speed up wait with a wait request object map (#427)
* plasma manager perf: speedup wait with a wait request object map

* removing duplicate == operator in plasma store

* fix serialization test

* code cleanup

* minor cleanup

* factoring out uniqueid hash and equality operators into common

* plasma manager: c++ify the WaitRequest struct

* plasma manager: get rid of the initial object request malloc

* cleanup

* linting

* cleanups and fix compiler warnings

* compiler warnings and linting
2017-04-07 12:32:12 -07:00