mirror of
https://github.com/vale981/ray
synced 2025-03-05 10:01:43 -05:00
Remove old files and remove old documentation for copying files around cluster. (#274)
This commit is contained in:
parent
cb7f6ca9b5
commit
3934d5f6eb
3 changed files with 0 additions and 1131 deletions
|
@ -96,67 +96,3 @@ If the web UI doesn't work, it's possible that the web UI processes were never
|
|||
started (check `ps aux | grep polymer` and `ps aux | grep ray_ui.py`). If they
|
||||
were started, see if you can fetch the web UI from within the head node (try
|
||||
`wget http://localhost:8080` on the head node).
|
||||
|
||||
### Copying Application Files to Other Nodes (Experimental)
|
||||
|
||||
If you're running an application that imports Python files that are present
|
||||
locally but not on the other machines in the cluster, you may first want to copy
|
||||
those files to the other machines. One way to do that is through Ray (this is
|
||||
experimental). Suppose you're directory structure looks
|
||||
|
||||
```
|
||||
application_files/
|
||||
__init__.py
|
||||
example.py
|
||||
```
|
||||
|
||||
And suppose `example.py` defines the following functions.
|
||||
|
||||
```python
|
||||
import ray
|
||||
|
||||
def example_helper(x):
|
||||
return x
|
||||
|
||||
@ray.remote
|
||||
def example_function(x):
|
||||
return example_helper(x)
|
||||
```
|
||||
|
||||
If you simply run
|
||||
|
||||
```python
|
||||
from application_files import example
|
||||
```
|
||||
|
||||
An error message will be printed like the following. This indicates that one of
|
||||
the workers was unable to register the remote function.
|
||||
|
||||
```
|
||||
Traceback (most recent call last):
|
||||
File "/home/ubuntu/ray/lib/python/ray/worker.py", line 813, in fetch_and_register_remote_function
|
||||
function = pickling.loads(serialized_function)
|
||||
ImportError: No module named 'application_files'
|
||||
```
|
||||
|
||||
To make this work, you need to copy your application files to all of the nodes.
|
||||
The following command will do that through Ray, and will add the files to Python
|
||||
path of each worker. This functionality is experimental. You may be able to do
|
||||
something like the following.
|
||||
|
||||
```python
|
||||
import ray
|
||||
|
||||
ray.init(redis_address="<redis-address>")
|
||||
|
||||
ray.experimental.copy_directory("application_files/")
|
||||
|
||||
# Now the import should work.
|
||||
from application_files import example
|
||||
```
|
||||
|
||||
Now you should be able to run the following command.
|
||||
|
||||
```python
|
||||
ray.get([example.example_function.remote(0) for _ in range(1000)])
|
||||
```
|
||||
|
|
1067
scripts/ec2.py
1067
scripts/ec2.py
File diff suppressed because it is too large
Load diff
Loading…
Add table
Reference in a new issue