Remove old files and remove old documentation for copying files around cluster. (#274)

This commit is contained in:
Robert Nishihara 2017-02-13 11:20:04 -08:00 committed by Philipp Moritz
parent cb7f6ca9b5
commit 3934d5f6eb
3 changed files with 0 additions and 1131 deletions

View file

@ -96,67 +96,3 @@ If the web UI doesn't work, it's possible that the web UI processes were never
started (check `ps aux | grep polymer` and `ps aux | grep ray_ui.py`). If they
were started, see if you can fetch the web UI from within the head node (try
`wget http://localhost:8080` on the head node).
### Copying Application Files to Other Nodes (Experimental)
If you're running an application that imports Python files that are present
locally but not on the other machines in the cluster, you may first want to copy
those files to the other machines. One way to do that is through Ray (this is
experimental). Suppose you're directory structure looks
```
application_files/
__init__.py
example.py
```
And suppose `example.py` defines the following functions.
```python
import ray
def example_helper(x):
return x
@ray.remote
def example_function(x):
return example_helper(x)
```
If you simply run
```python
from application_files import example
```
An error message will be printed like the following. This indicates that one of
the workers was unable to register the remote function.
```
Traceback (most recent call last):
File "/home/ubuntu/ray/lib/python/ray/worker.py", line 813, in fetch_and_register_remote_function
function = pickling.loads(serialized_function)
ImportError: No module named 'application_files'
```
To make this work, you need to copy your application files to all of the nodes.
The following command will do that through Ray, and will add the files to Python
path of each worker. This functionality is experimental. You may be able to do
something like the following.
```python
import ray
ray.init(redis_address="<redis-address>")
ray.experimental.copy_directory("application_files/")
# Now the import should work.
from application_files import example
```
Now you should be able to run the following command.
```python
ray.get([example.example_function.remote(0) for _ in range(1000)])
```

View file

File diff suppressed because it is too large Load diff