Commit graph

930 commits

Author SHA1 Message Date
Robert Nishihara
8532ba4272 Serialize lambdas, sets, and types with pickle by default. (#511)
* Serialize lambdas with pickle by default.

* Serialize sets with pickle by default.

* Serialize types with pickle by default.

* Small update to documentation.

* Update tests.
2017-05-04 00:16:35 -07:00
Alexey Tumanov
7deff73e55 update documentation on how to set ulimit for Redis (#508)
* update documentation on how to set ulimit for Redis

* Move documentation to troubleshooting section.

* Add another section header.

* update redis file descriptor docs: simplify command
2017-05-03 17:24:57 -07:00
Robert Nishihara
245c8ab888 Make sure user seeding does not affect actor ID generation. (#506)
* Make sure user seeding does not affect actor ID generation.

* Fix linting.

* Add test.
2017-05-03 16:29:55 -07:00
Stephanie Wang
e50a23b820 Fix bug with reused file descriptors (#471)
* Fix bug with reused file descriptors

* Remove client connection if write_object_chunk fails

* Handle ECONNRESET on unsuccessful write

* lint

* Back to lowercase

* fix compilation

* fix linting
2017-05-02 19:45:27 -07:00
Robert Nishihara
2bbfc5da8d Dispatch actor tasks when actor connects. (#495) 2017-04-28 17:36:43 -07:00
Robert Nishihara
6d301d9079 Simplify resource bookkeeping in local scheduler. (#494)
* Simplify resource bookkeeping in local scheduler.

* Change ints to doubles.
2017-04-28 12:09:47 -07:00
Robert Nishihara
eea19371b7 Suppress warning about working dying when driver exits. (#492) 2017-04-26 23:52:13 -07:00
Robert Nishihara
1627f89945 Fix problem in which actors and workers running tasks are not killed by driver exit. (#490)
* Augment test to verify that relevant workers and actors are killed during driver cleanup.

* Fix bug in which we were only killing one worker when a driver exited.

* Fix remove driver test.

* Fix and augment test.
2017-04-26 15:13:39 -07:00
Philipp Moritz
b7ace01b5f Convert Plasma client to STL (#486)
* convert mmap table to STL

* update

* fix

* convert objects_in_use

* fix

* convert release_history

* cleanup

* linting

* update

* fix

* linting
2017-04-25 01:25:40 -07:00
Robert Nishihara
0ac125e9b2 Clean up when a driver disconnects. (#462)
* Clean up state when drivers exit.

* Remove unnecessary field in ActorMapEntry struct.

* Have monitor release GPU resources in Redis when driver exits.

* Enable multiple drivers in multi-node tests and test driver cleanup.

* Make redis GPU allocation a redis transaction and small cleanups.

* Fix multi-node test.

* Small cleanups.

* Make global scheduler take node_ip_address so it appears in the right place in the client table.

* Cleanups.

* Fix linting and cleanups in local scheduler.

* Fix removed_driver_test.

* Fix bug related to vector -> list.

* Fix linting.

* Cleanup.

* Fix multi node tests.

* Fix jenkins tests.

* Add another multi node test with many drivers.

* Fix linting.

* Make the actor creation notification a flatbuffer message.

* Revert "Make the actor creation notification a flatbuffer message."

This reverts commit af99099c8084dbf9177fb4e34c0c9b1a12c78f39.

* Add comment explaining flatbuffer problems.
2017-04-24 18:10:21 -07:00
Philipp Moritz
8194b71f32 Convert pending_notifications to STL (#484)
* temp commit

* converted more plasma notifications

* cleanup

* rename

* linting

* fixes

* fixes
2017-04-24 14:41:34 -07:00
Philipp Moritz
892e53d69e Convert plasma client array and object notification queue to STL (#482)
* Conver plasma clients to STL

* use a deque for object notifications in plasma store for perf

* cleanup

* linting

* fix include order
2017-04-24 00:43:48 -07:00
Philipp Moritz
e36de2dad1 Convert object table to STL (#480)
* convert object table to stl

* temp commit

* fix

* comments

* linting
2017-04-23 22:24:05 -07:00
Robert Nishihara
3a2eb1467b Fix failure to propagate error message. (#479) 2017-04-23 16:12:25 -07:00
Alexey Tumanov
a67a107e0e Fix int-type compilation problem on redhat. (#472) 2017-04-19 02:43:33 -07:00
Richard Shin
cf68cf743c Change UniqueID hash function to look at the lowest instead of highest bytes. (#469) 2017-04-18 15:31:49 -07:00
Philipp Moritz
8ac6c59931 Remove n^2 algorithm in plasma get (#466)
Remove n^2 algorithm in plasma get.
2017-04-17 23:37:33 -07:00
Guru Medasani
0189b09581 Fixes Mac OSX installation error (#464)
* changes to address ARROW-826 and ARROW-444

* changes to address ARROW-826 and ARROW-444

* ignoring cmake-build-debug

* additional IDEA ignore files

* additional IDEA ignore files

* remove arrow ipc and arrow io libraries

* add boost dependencies

* fix arrow origin and remove submodule
2017-04-16 15:02:15 -07:00
Robert Nishihara
c802e51d36 Re-enable recursive remote functions in a limited form. (#453)
* Re-enable recursive remote functions in a limited form.

* Fix linting.
2017-04-13 01:47:33 -07:00
Robert Nishihara
dad57e3b62 Convert actor data structures to C++. (#454) 2017-04-12 01:18:16 -07:00
Richard Liaw
94f32db5e6 A3C Polishing (#385)
* number

* gym doesn't have versioning

* Benchmarks

* visualization

* formatting

* small fix for tensorboard

* first pass removing universe dependency

* code

* results polish

* removed extra line

* removed universe dependency

* doc

* remove gym versioning stuff

* changes as suggested

* nit
2017-04-11 22:51:52 -07:00
Richard Liaw
c3a2505ffd Loadbalancing Test issue (#452)
* Limiting number of CPUs in loadbalancing test

* fixes as requested
2017-04-11 22:33:58 -07:00
Robert Nishihara
fb4525f833 Convert some local scheduler data structures to C++ STL. (#445)
* Convert more local scheduler data structures to C++ STL.

* Convert vector pointer to vector.

* Convert some of the UT_arrays to std::vector.

* Simplify worker vectors.

* Simplify remote_object and local_object containers.

* Change some unnecessary checks to DCHECK.
2017-04-10 21:02:36 -07:00
Philipp Moritz
6ffc849d23 Use Arrow Tensors for serializing numpy arrays and get rid of extra memcpy. (#436)
* Use Arrow Tensors for serializing numpy arrays and get rid of extra memcpy

* fix nondeterminism problem

* mark array as immutable

* make arrays contiguous

* fix serialize_list and deseralize_list

* fix numbuf tests

* linting

* add optimization flags

* fixes

* roll back arrow
2017-04-10 01:37:34 -07:00
Robert Nishihara
c9d66555e2 Fix bug in queue_task function in local scheduler. (#443) 2017-04-09 19:34:43 -07:00
Robert Nishihara
f4c1adae17 Unify function signature handling between remote functions and actor … (#441)
* Unify function signature handling between remote functions and actor methods.

* Fixes.

* Fix tests.
2017-04-08 21:34:13 -07:00
Alexey Tumanov
b6c4ae82c0 Increase redis client pubsub buffer size. (#442) 2017-04-08 15:24:07 -07:00
Robert Nishihara
7cd00741b1 Suppress irrelevant Redis connection errors. (#434)
* Suppress error messages in worker import thread when Redis terminates.

* Suppress some warnings from one of the tests.
2017-04-07 23:19:24 -07:00
Robert Nishihara
0eac3ccdd0 Reduce verbosity of component_failures_test.py. (#440) 2017-04-07 23:05:29 -07:00
Robert Nishihara
05fd4c2c37 Changes to local scheduler client protocol. (#435)
* Make local scheduler clients receive reply upon registration.

* Fix tests and linting.
2017-04-07 23:03:37 -07:00
Robert Nishihara
8061b3b596 Revert "Suppress warning in start_ray.sh about leaving child processes running when parent exits. (#429)" (#437)
This reverts commit 85b373a4be.
2017-04-07 17:32:28 -07:00
Alexey Tumanov
6f9225490b Plasma manager performance: speed up wait with a wait request object map (#427)
* plasma manager perf: speedup wait with a wait request object map

* removing duplicate == operator in plasma store

* fix serialization test

* code cleanup

* minor cleanup

* factoring out uniqueid hash and equality operators into common

* plasma manager: c++ify the WaitRequest struct

* plasma manager: get rid of the initial object request malloc

* cleanup

* linting

* cleanups and fix compiler warnings

* compiler warnings and linting
2017-04-07 12:32:12 -07:00
Robert Nishihara
7af6f462fb Add API for querying global control state. (#431)
* Add API for querying global control state.

* Fix linting.

* Fix errors in Python 2.

* Fix bug in test.

* Fix bug in test.
2017-04-06 23:51:12 -07:00
Robert Nishihara
320109a5bd By default, start a number of workers equal to the number of CPUs. (#430)
* By default, start a number of workers equal to the number of CPUs.

* Fix stress tests.
2017-04-06 00:02:58 -07:00
Robert Nishihara
fa363a5a3a Notify driver when a worker dies while executing a task. (#419)
* Notify driver when a worker dies while executing a task.

* Fix linting.

* Don't push error when local scheduler is cleaning up.
2017-04-06 00:02:39 -07:00
Robert Nishihara
85b373a4be Suppress warning in start_ray.sh about leaving child processes running when parent exits. (#429) 2017-04-05 23:54:22 -07:00
Stephanie Wang
93679df724 Stopped nodes can rejoin immediately (#428)
* Ignore deleted clients when reading address info from Redis

* Remove self from db_client table when exiting cleanly

* Fix valgrind test

* Do not call plasma_perform_release when disconnecting
2017-04-05 23:50:38 -07:00
Philipp Moritz
4043769ba2 Make putting large objects work. (#411)
* putting large objects

* add more checks

* support large objects

* fix test

* fix linting

* upgrade to latest arrow version

* check malloc return code

* print mmap file sizes

* printing

* revert to dlmalloc

* add prints

* more prints

* add printing

* printing

* fix

* update

* fix

* update

* print

* initialization

* temp

* fix

* update

* fix linting

* comment out object_store_full tests

* fix test

* fix test

* evict objects if dlmalloc fails

* fix stresstests

* Fix linting.

* Uncomment large-memory tests.

* Increase memory for docker image for jenkins tests.

* Reduce large memory tests.

* Further reduce large memory tests.
2017-04-05 01:04:05 -07:00
Robert Nishihara
1e84747e13 Remove incorrect check. (#421) 2017-04-03 14:51:53 -07:00
Richard Shin
227c916c25 Convert plasma/plasma_store.cc to use STL (#324)
* Change plasma_store.c to C++ (clobbering existing FlatBuffers usage).

* Convert plasma_store.cc to use STL (with a caveat)

* Fix CMakeLists and mutation-while-iterating problem

* Remove extra extern "C" declarations

* Remove redundant -std=c++11 from plasma/CMakeLists.txt
2017-03-31 22:58:10 -07:00
Robert Nishihara
f1b48f2fd4 Avoid publishing in the task table unnecessarily. (#416) 2017-03-30 13:41:32 -07:00
Stephanie Wang
036b873bf2 Implement local scheduler task queues using C++ data structures (#392)
* Switch to using C++ lists for task queues

* Init and free methods for TaskQueueEntry

* Switch from utarray to c++ vector for TaskQueueEntry

* Get rid of some pointers

* Back to O(1) deletion from waiting_task_queue

* Fix comments

* Cut code

* Non const iterators

* Fix Alexey's comments
2017-03-30 00:40:01 -07:00
Robert Nishihara
8245758ccb Add overview of internals to documentation, improve serialization doc… (#390)
* Add overview of internals to documentation, improve serialization documentation.

* Doc edits

* Small fixes.
2017-03-27 21:52:17 -07:00
Alexey Tumanov
78e1167a42 Parallelize make in build.sh. (#371)
* parallelize build.sh make

* Encode in cmake the dependency of ray_redis_module on autogenerated flatbuffer files.
2017-03-27 20:55:50 -07:00
Robert Nishihara
0925e11c48 Exclude function source from function ID hash in Python interpreter. (#395)
* Exclude function source code from function ID hash in Python interpreter.

* Remove try except block.
2017-03-25 11:31:21 -07:00
Robert Nishihara
054a046b69 Fix installation instructions on Ubuntu and convert md -> rst. (#389) 2017-03-24 17:33:26 -07:00
Alexey Tumanov
a3d58607bf parallelize numbuf memcpy and plasma object hash construction (#366)
* parallelizing memcopy and object hash construction in numbuf/plasma

* clang format

* whitespace

* refactoring compute object hash: get rid of the prefix chunk

* clang format

* Document performance optimization.

* Remove check for 64-byte alignment, since it may not be guaranteed.
2017-03-21 16:17:35 -07:00
Robert Nishihara
ba02fc0eb0 Run flake8 in Travis and make code PEP8 compliant. (#387) 2017-03-21 12:57:54 -07:00
Stephanie Wang
083e7a28ad Push an error to the driver when the workload hangs on ray.put reconstruction (#382)
* Fix worker blocked bug

* tmp

* Push an error to the driver on ray.put for non-driver tasks

* Fix result table tests

* Fix test, logging

* Address comments

* Fix suppression bug

* Fix redis module test

* Edit error message

* Get values in chunks during reconstruction

* Test case for driver ray.put errors

* Error for evicting ray.put objects from the driver

* Fix tests

* Reduce verbosity

* Documentation
2017-03-21 00:16:48 -07:00
Philipp Moritz
4618fd45b1 Port Ray to latest Arrow version (#370)
* rebase on top of latest arrow

* clang-format

* address comments

* fix
2017-03-20 16:31:46 -07:00