hiro/ray
1
0
Fork 0
mirror of https://github.com/vale981/ray synced 2025-03-06 10:31:39 -05:00
Code Issues Projects Releases Packages Wiki Activity Actions
691 commits 227 branches 67 tags 234 MiB
Commit graph

138 commits

Author SHA1 Message Date
Wapaul1
420bcc0477 Remote function returning non-serializable type no longer shuts worker down (#384) 
* Moved put_objects in main_loop to inside of try block

* Added test for failed serialization

* Fixed naming

* Minor
 2016-08-25 15:26:22 -07:00
Robert Nishihara
b1db890cd8 Export GRPC environment variable in Python instead of in setup-env.sh (#382) 
* Export GRPC logging environment variable in Python.

* Don't wipe PYTHONPATH in setup-env.sh
 2016-08-18 15:39:19 -07:00
Robert Nishihara
e06311d415 Automatically add relevant directories to Python paths of workers (#380) 
* Make ray.init set python paths of workers.

* Decouple starting cluster from copying user source code

* also add current directory to path

* Add comments about deallocation.

* Add test for new code path.
 2016-08-16 14:53:55 -07:00
Wapaul1
7246013008 Implement select to enable waiting for a specific number of remote objects to be ready. (#369) 2016-08-15 16:51:59 -07:00
Robert Nishihara
b29fc0c481 Use random string for worker c++ logfile. (#378) 2016-08-15 15:55:34 -07:00
Robert Nishihara
87bb7a8f67 [WIP] Large changes to make the tests pass. (#376) 
* Revert "Make tests more informative (#372)"

This reverts commit fd353250c8.

* fix bugs, in particular deactivate worker service on driver and remove condition variables

* changes to minimize the changes in this PR

* switch from faulty mutex synchronization to using atomics

* Increase the default size of the message queues, to accommodate exporting large numbers of remote functions. This is a temporary fix, but not a long term solution.

* Reorganize the scheduler export code to queue up exports. This does not solve the underlying problem yet, but sets up a solution.

* Start a separate thread on driver to print error messages by constantly querying the scheduler. This is a temporary solution because the solution based on starting a worker service for the driver which the scheduler can push error messages to is buggy.

* Fix segfault in taskcapsule destructor.

* Move tests for catching errors into a separate test file.

* Revert "roll back grpc (#368)"

This reverts commit c01ef95d04.
 2016-08-15 11:02:54 -07:00
Wapaul1
362ffa1f3c Changing hard coded ports for objstore and workers to choose unused ports (#365) 
* let grpc choose unused worker and object store ports

* Add objstore addresses to scheduler info to bring back test
 2016-08-10 19:08:38 -07:00
Robert Nishihara
fbc49410ec properly shutdown worker service thread (#367) 2016-08-10 16:53:24 -07:00
Robert Nishihara
13df8302e6 enable running example apps in cluster mode (#357) 2016-08-08 16:01:13 -07:00
Robert Nishihara
a1e4268d37 Catch errors in importing reusable variables and remote functions (#354) 
* catch errors in importing reusable variables and remote functions

* updates
 2016-08-07 13:53:33 -07:00
Philipp Moritz
8bf877ac1e Serialize and Deserialize unicode (#349) 2016-08-04 21:06:31 -07:00
Robert Nishihara
ac363bf451 Let worker get worker address and object store address from scheduler (#350) 2016-08-04 17:47:08 -07:00
Johann Schleier-Smith
3ee0fd8f34 Update cluster guide (#347) 
* clarify cluster setup instructions

* update multinode documentation, update cluster script, fix minor bug in worker.py

* clarify cluster documentation and fix update_user_code
 2016-08-04 09:14:20 -07:00
Robert Nishihara
de200ff912 better error messages when composing remote functions (#339) 
Better error messages when composing remote functions
 2016-08-03 16:47:13 -07:00
Robert Nishihara
07baf44f26 only display failed task information for new tasks, remove SHELL_MODE (#344) 2016-08-02 17:59:42 -07:00
Robert Nishihara
c27e6c076c Make sure no Python modules mutually import each other. (#334) 2016-08-01 17:55:38 -07:00
Wapaul1
96a70e1316 Changed equality to None in worker.py to use is keyword (#335) 2016-08-01 16:56:54 -07:00
Wapaul1
97b923a750 Changed how ray treats deserialization of custom classes (#333) 2016-08-01 15:38:05 -07:00
Robert Nishihara
98a508d6ca Terminology change Object Reference -> Object ID (#330) 2016-07-31 19:58:03 -07:00
Robert Nishihara
2040372084 unify starting local cluster with attaching to existing cluster (#327) 2016-07-31 19:26:35 -07:00
Robert Nishihara
0e5b858324 change remote function invocation from func() to func.remote() (#328) 2016-07-31 15:25:19 -07:00
Robert Nishihara
352e5e1dd2 use Ray specific logger so logging does not interfere with other python modules that use the logging module (#321) 2016-07-29 12:40:45 -07:00
mehrdadn
4c323d4ecf libraylib -> lib (#311) 2016-07-27 23:23:13 -07:00
Philipp Moritz
dd301e4baa add version string (#304) 2016-07-27 11:29:22 -07:00
Robert Nishihara
975bfe89ad check that we are connected before calling ray commands (#303) 2016-07-27 11:20:37 -07:00
Philipp Moritz
b5215f1e6a make it possible to use directory as user source directory that doesn't contain worker.py (#297) 2016-07-26 18:39:06 -07:00
Robert Nishihara
aa2f618ab7 add directory containing script to python path of workers (#296) 2016-07-26 16:18:39 -07:00
Robert Nishihara
3bae6f136b export remote functions and reusable variables that were defined before connect was called (#292) 2016-07-26 11:40:09 -07:00
Robert Nishihara
8e9f98c5ff Properly import remote functions and reusable variables on workers that register late (#290) 2016-07-25 16:17:17 -07:00
Philipp Moritz
4a0f35b042 new arrow serialization code (serialize python objects recursively) (#284) 2016-07-25 13:41:47 -07:00
mehrdadn
5ff00e0e81 Fix pickling (#289) 2016-07-24 12:53:55 -07:00
Robert Nishihara
03f1830cd0 implement key value store for sharing reusable variables 2016-07-22 18:43:33 -07:00
Robert Nishihara
3bf46ce5ac make remote functions pickleable (#281) 2016-07-19 16:05:45 -07:00
mehrdadn
f2c43bec87 Function serialization (#261) 2016-07-17 22:05:07 -07:00
mehrdadn
06ceecf6f5 Kill child processes when parent dies, and die on fatal errors (#265) 2016-07-14 16:10:26 -07:00
Robert Nishihara
a5d8138f88 mock modules for readthedocs (#264) 2016-07-14 13:57:03 -07:00
Robert Nishihara
80526f7777 add documentation and refactor cluster.py (#238) 2016-07-12 23:54:18 -07:00
Wapaul1
292abaa41c Moved Imagenet loading library to example applications; Changed code to return filenames as well as arrays (#257) 2016-07-11 18:06:58 -07:00
Robert Nishihara
8952ff8cf9 allow cluster script to update worker code on nodes (#243) 2016-07-11 17:58:16 -07:00
Robert Nishihara
191909dd93 adding pylint (#233) 2016-07-08 12:39:11 -07:00
Robert Nishihara
82ab6d68f1 remove recursive option from register_module (#232) 2016-07-08 11:13:03 -07:00
Robert Nishihara
875b20e397 only run cleanup if we've started ray in local mode and actually started the processes (#228) 2016-07-08 00:14:26 -07:00
Robert Nishihara
8e6b7929d6 make services.cleanup happen automatically (#224) 2016-07-07 14:05:25 -07:00
mehrdadn
199b4efd50 int and long should be treated similarly (#220) 2016-07-06 17:31:58 -07:00
Robert Nishihara
e1e09d3a4b catch type checking exceptions 2016-07-06 14:20:57 -07:00
Robert Nishihara
49decce6f6 print python backtrace in error message (#209) 2016-07-06 00:25:02 -07:00
Robert Nishihara
8297522580 don't destruct worker object before all of the object references go out of scope (#212) 2016-07-05 19:13:57 -07:00
Robert Nishihara
0947024ad9 fix bug for functions with no return values and with one return value (#211) 2016-07-05 15:57:05 -07:00
mehrdadn
be26532182 Better code for getting log directory (#198) 
* get_log_file_path instead of LOG_DIRECTORY

* Tabs -> spaces

Whoops

* Log file extension
 2016-07-03 12:58:49 -07:00
Philipp Moritz
2e2bc2c740 Get Sphinx infrastructure in place 2016-07-01 18:21:02 -07:00