{ "cells": [ { "cell_type": "markdown", "id": "5a1d28f3", "metadata": {}, "source": [ "# Running Tune experiments with Nevergrad\n", "\n", "In this tutorial we introduce Nevergrad, while running a simple Ray Tune experiment. Tune’s Search Algorithms integrate with Nevergrad and, as a result, allow you to seamlessly scale up a Nevergrad optimization process - without sacrificing performance.\n", "\n", "Nevergrad provides gradient/derivative-free optimization able to handle noise over the objective landscape, including evolutionary, bandit, and Bayesian optimization algorithms. Nevergrad internally supports search spaces which are continuous, discrete or a mixture of thereof. It also provides a library of functions on which to test the optimization algorithms and compare with other benchmarks.\n", "\n", "In this example we minimize a simple objective to briefly demonstrate the usage of Nevergrad with Ray Tune via `NevergradSearch`. It's useful to keep in mind that despite the emphasis on machine learning experiments, Ray Tune optimizes any implicit or explicit objective. Here we assume `nevergrad==0.4.3.post7` library is installed. To learn more, please refer to [Nevergrad website](https://github.com/facebookresearch/nevergrad)." ] }, { "cell_type": "code", "execution_count": 1, "id": "5ab54f85", "metadata": { "tags": [ "remove-cell" ] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Collecting nevergrad==0.4.3.post7\n", " Using cached nevergrad-0.4.3.post7-py3-none-any.whl (400 kB)\n", "Collecting cma>=2.6.0\n", " Downloading cma-3.2.2-py2.py3-none-any.whl (249 kB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m249.1/249.1 kB\u001b[0m \u001b[31m3.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n", "\u001b[?25hRequirement already satisfied: bayesian-optimization>=1.2.0 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from nevergrad==0.4.3.post7) (1.2.0)\n", "Requirement already satisfied: numpy>=1.15.0 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from nevergrad==0.4.3.post7) (1.21.6)\n", "Requirement already satisfied: typing-extensions>=3.6.6 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from nevergrad==0.4.3.post7) (4.1.1)\n", "Requirement already satisfied: scipy>=0.14.0 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from bayesian-optimization>=1.2.0->nevergrad==0.4.3.post7) (1.4.1)\n", "Requirement already satisfied: scikit-learn>=0.18.0 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from bayesian-optimization>=1.2.0->nevergrad==0.4.3.post7) (0.24.2)\n", "Requirement already satisfied: threadpoolctl>=2.0.0 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from scikit-learn>=0.18.0->bayesian-optimization>=1.2.0->nevergrad==0.4.3.post7) (3.0.0)\n", "Requirement already satisfied: joblib>=0.11 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from scikit-learn>=0.18.0->bayesian-optimization>=1.2.0->nevergrad==0.4.3.post7) (1.1.0)\n", "Installing collected packages: cma, nevergrad\n", "Successfully installed cma-3.2.2 nevergrad-0.4.3.post7\n", "\u001b[33mWARNING: There was an error checking the latest version of pip.\u001b[0m\u001b[33m\n", "\u001b[0m" ] } ], "source": [ "# !pip install ray[tune]\n", "!pip install nevergrad==0.4.3.post7 " ] }, { "cell_type": "markdown", "id": "66cb8206", "metadata": {}, "source": [ "Click below to see all the imports we need for this example.\n", "You can also launch directly into a Binder instance to run this notebook yourself.\n", "Just click on the rocket symbol at the top of the navigation." ] }, { "cell_type": "code", "execution_count": 2, "id": "1f6d7a31", "metadata": { "tags": [ "hide-input" ] }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/nevergrad/optimization/differentialevolution.py:107: InefficientSettingsWarning: DE algorithms are inefficient with budget < 60\n", " \"DE algorithms are inefficient with budget < 60\", base.errors.InefficientSettingsWarning\n" ] } ], "source": [ "import time\n", "\n", "import ray\n", "import nevergrad as ng\n", "from ray import tune\n", "from ray.air import session\n", "from ray.tune.search import ConcurrencyLimiter\n", "from ray.tune.search.nevergrad import NevergradSearch" ] }, { "cell_type": "markdown", "id": "41f2c881", "metadata": {}, "source": [ "Let's start by defining a simple evaluation function.\n", "We artificially sleep for a bit (`0.1` seconds) to simulate a long-running ML experiment.\n", "This setup assumes that we're running multiple `step`s of an experiment and try to tune two hyperparameters,\n", "namely `width` and `height`, and `activation`." ] }, { "cell_type": "code", "execution_count": 3, "id": "271bd5c5", "metadata": {}, "outputs": [], "source": [ "def evaluate(step, width, height, activation):\n", " time.sleep(0.1)\n", " activation_boost = 10 if activation==\"relu\" else 1\n", " return (0.1 + width * step / 100) ** (-1) + height * 0.1 + activation_boost" ] }, { "cell_type": "markdown", "id": "f060ea83", "metadata": {}, "source": [ "Next, our `objective` function takes a Tune `config`, evaluates the `score` of your experiment in a training loop,\n", "and uses `session.report` to report the `score` back to Tune." ] }, { "cell_type": "code", "execution_count": 4, "id": "c71fc423", "metadata": {}, "outputs": [], "source": [ "def objective(config):\n", " for step in range(config[\"steps\"]):\n", " score = evaluate(step, config[\"width\"], config[\"height\"], config[\"activation\"])\n", " session.report({\"iterations\": step, \"mean_loss\": score})" ] }, { "cell_type": "code", "execution_count": 5, "id": "619263ee", "metadata": { "lines_to_next_cell": 0, "tags": [ "remove-cell" ] }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "INFO:ray._private.services:View the Ray dashboard at \u001b[1m\u001b[32mhttp://127.0.0.1:8266\u001b[39m\u001b[22m\n" ] }, { "data": { "text/html": [ "
\n", "
\n", "

Ray

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", "\n", "
Python version:3.7.7
Ray version: 2.0.0rc0
Dashboard:http://127.0.0.1:8266
\n", "
\n", "
\n" ], "text/plain": [ "RayContext(dashboard_url='127.0.0.1:8266', python_version='3.7.7', ray_version='2.0.0rc0', ray_commit='{{RAY_COMMIT_SHA}}', address_info={'node_ip_address': '127.0.0.1', 'raylet_ip_address': '127.0.0.1', 'redis_address': None, 'object_store_address': '/tmp/ray/session_2022-07-22_15-23-54_250484_46348/sockets/plasma_store', 'raylet_socket_name': '/tmp/ray/session_2022-07-22_15-23-54_250484_46348/sockets/raylet', 'webui_url': '127.0.0.1:8266', 'session_dir': '/tmp/ray/session_2022-07-22_15-23-54_250484_46348', 'metrics_export_port': 60625, 'gcs_address': '127.0.0.1:63562', 'address': '127.0.0.1:63562', 'dashboard_agent_listen_port': 52365, 'node_id': '833f68610f3299cf23cacacc02fe1c7dce7e9ea49bb948b17af93bf0'})" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "ray.init(configure_logging=False)" ] }, { "cell_type": "markdown", "id": "5b7a4b94", "metadata": {}, "source": [ "Now we construct the hyperparameter search space using `ConfigSpace`" ] }, { "cell_type": "markdown", "id": "e2405373", "metadata": {}, "source": [ "Next we define the search algorithm built from `NevergradSearch`, constrained to a maximum of `4` concurrent trials with a `ConcurrencyLimiter`. Here we use `ng.optimizers.OnePlusOne`, a simple evolutionary algorithm." ] }, { "cell_type": "code", "execution_count": 6, "id": "f099b674", "metadata": {}, "outputs": [], "source": [ "algo = NevergradSearch(\n", " optimizer=ng.optimizers.OnePlusOne,\n", ")\n", "algo = tune.search.ConcurrencyLimiter(algo, max_concurrent=4)" ] }, { "cell_type": "markdown", "id": "4bddc4e5", "metadata": {}, "source": [ "The number of samples is the number of hyperparameter combinations that will be tried out. This Tune run is set to `1000` samples.\n", "(you can decrease this if it takes too long on your machine)." ] }, { "cell_type": "code", "execution_count": 7, "id": "adb807bc", "metadata": {}, "outputs": [], "source": [ "num_samples = 1000" ] }, { "cell_type": "code", "execution_count": 8, "id": "191c7f89", "metadata": { "tags": [ "remove-cell" ] }, "outputs": [], "source": [ "# If 1000 samples take too long, you can reduce this number.\n", "# We override this number here for our smoke tests.\n", "num_samples = 10" ] }, { "cell_type": "markdown", "id": "a3956381", "metadata": {}, "source": [ "Finally, all that's left is to define a search space." ] }, { "cell_type": "code", "execution_count": 9, "id": "8829ffc5", "metadata": {}, "outputs": [], "source": [ "search_config = {\n", " \"steps\": 100,\n", " \"width\": tune.uniform(0, 20),\n", " \"height\": tune.uniform(-100, 100),\n", " \"activation\": tune.choice([\"relu, tanh\"])\n", "}" ] }, { "cell_type": "markdown", "id": "0b9d051f", "metadata": {}, "source": [ "Finally, we run the experiment to `\"min\"`imize the \"mean_loss\" of the `objective` by searching `search_space` via `algo`, `num_samples` times. This previous sentence is fully characterizes the search problem we aim to solve. With this in mind, observe how efficient it is to execute `tuner.fit()`." ] }, { "cell_type": "code", "execution_count": 10, "id": "769f4368", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "WARNING:ray.tune.trainable.function_trainable:Function checkpointing is disabled. This may result in unexpected behavior when using checkpointing features or certain schedulers. To enable, set the train function arguments to be `func(config, checkpoint_dir=None)`.\n" ] }, { "data": { "text/html": [ "== Status ==
Current time: 2022-07-22 15:24:44 (running for 00:00:43.69)
Memory usage on this node: 10.4/16.0 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/4.61 GiB heap, 0.0/2.0 GiB objects
Current best trial: 004f499a with mean_loss=-7.595329711238255 and parameters={'steps': 100, 'width': 2.1174116156230918, 'height': -90.50653873694615, 'activation': 'relu, tanh'}
Result logdir: /Users/kai/ray_results/objective_2022-07-22_15-23-59
Number of trials: 10/10 (10 TERMINATED)
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
Trial name status loc activation height width loss iter total time (s) iterations neg_mean_loss
objective_ee2ca136TERMINATED127.0.0.1:46434relu, tanh 0 10 1.1 100 10.942 99 -1.1
objective_efe1626eTERMINATED127.0.0.1:46441relu, tanh -31.0013 9.28761 -1.99254 100 11.5354 99 1.99254
objective_efe34e4eTERMINATED127.0.0.1:46442relu, tanh 5.21403 9.48974 1.62672 100 11.6606 99 -1.62672
objective_efe55c2aTERMINATED127.0.0.1:46443relu, tanh -20.8721 11.3958 -0.99935 100 11.6083 99 0.99935
objective_f6688086TERMINATED127.0.0.1:46467relu, tanh 57.2829 17.7296 6.78493 100 10.716 99 -6.78493
objective_f85ed926TERMINATED127.0.0.1:46478relu, tanh 40.5543 19.0813 5.10809 100 10.7158 99 -5.10809
objective_f86ee276TERMINATED127.0.0.1:46481relu, tanh 93.8686 1.60757 10.9781 100 10.7415 99 -10.9781
objective_f880a02eTERMINATED127.0.0.1:46484relu, tanh -80.5769 5.84852 -6.88791 100 10.7335 99 6.88791
objective_fe4e7a44TERMINATED127.0.0.1:46499relu, tanh 9.62911 0.622909 3.35823 100 13.1428 99 -3.35823
objective_004f499aTERMINATED127.0.0.1:46504relu, tanh -90.5065 2.11741 -7.59533 100 10.7688 99 7.59533


" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stderr", "output_type": "stream", "text": [ "INFO:ray._private.runtime_env.plugin_schema_manager:Loading the default runtime env schemas: ['/Users/kai/coding/ray/python/ray/_private/runtime_env/../../runtime_env/schemas/working_dir_schema.json', '/Users/kai/coding/ray/python/ray/_private/runtime_env/../../runtime_env/schemas/pip_schema.json'].\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Result for objective_ee2ca136:\n", " date: 2022-07-22_15-24-03\n", " done: false\n", " experiment_id: c0ad5ddb78cc4cc88e8195f5bde34e20\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 11.0\n", " neg_mean_loss: -11.0\n", " node_ip: 127.0.0.1\n", " pid: 46434\n", " time_since_restore: 0.10390329360961914\n", " time_this_iter_s: 0.10390329360961914\n", " time_total_s: 0.10390329360961914\n", " timestamp: 1658499843\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: ee2ca136\n", " warmup_time: 0.003058910369873047\n", " \n", "Result for objective_efe1626e:\n", " date: 2022-07-22_15-24-06\n", " done: false\n", " experiment_id: c6d33a9a30c040c0929b657f1d3e1557\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 7.899868000941147\n", " neg_mean_loss: -7.899868000941147\n", " node_ip: 127.0.0.1\n", " pid: 46441\n", " time_since_restore: 0.10202908515930176\n", " time_this_iter_s: 0.10202908515930176\n", " time_total_s: 0.10202908515930176\n", " timestamp: 1658499846\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: efe1626e\n", " warmup_time: 0.003210783004760742\n", " \n", "Result for objective_efe55c2a:\n", " date: 2022-07-22_15-24-06\n", " done: false\n", " experiment_id: 903e9605ba894aa0bc55297229b8a77b\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 8.91279068144465\n", " neg_mean_loss: -8.91279068144465\n", " node_ip: 127.0.0.1\n", " pid: 46443\n", " time_since_restore: 0.1029670238494873\n", " time_this_iter_s: 0.1029670238494873\n", " time_total_s: 0.1029670238494873\n", " timestamp: 1658499846\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: efe55c2a\n", " warmup_time: 0.0031630992889404297\n", " \n", "Result for objective_efe34e4e:\n", " date: 2022-07-22_15-24-06\n", " done: false\n", " experiment_id: 1efbb9c1becb436e8f304b61e9edd61b\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 11.521403250268252\n", " neg_mean_loss: -11.521403250268252\n", " node_ip: 127.0.0.1\n", " pid: 46442\n", " time_since_restore: 0.10443997383117676\n", " time_this_iter_s: 0.10443997383117676\n", " time_total_s: 0.10443997383117676\n", " timestamp: 1658499846\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: efe34e4e\n", " warmup_time: 0.002650022506713867\n", " \n", "Result for objective_ee2ca136:\n", " date: 2022-07-22_15-24-08\n", " done: false\n", " experiment_id: c0ad5ddb78cc4cc88e8195f5bde34e20\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 45\n", " iterations_since_restore: 46\n", " mean_loss: 1.2173913043478262\n", " neg_mean_loss: -1.2173913043478262\n", " node_ip: 127.0.0.1\n", " pid: 46434\n", " time_since_restore: 5.145823955535889\n", " time_this_iter_s: 0.10791492462158203\n", " time_total_s: 5.145823955535889\n", " timestamp: 1658499848\n", " timesteps_since_restore: 0\n", " training_iteration: 46\n", " trial_id: ee2ca136\n", " warmup_time: 0.003058910369873047\n", " \n", "Result for objective_efe1626e:\n", " date: 2022-07-22_15-24-11\n", " done: false\n", " experiment_id: c6d33a9a30c040c0929b657f1d3e1557\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: -1.8761766831351419\n", " neg_mean_loss: 1.8761766831351419\n", " node_ip: 127.0.0.1\n", " pid: 46441\n", " time_since_restore: 5.138395071029663\n", " time_this_iter_s: 0.10561490058898926\n", " time_total_s: 5.138395071029663\n", " timestamp: 1658499851\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: efe1626e\n", " warmup_time: 0.003210783004760742\n", " \n", "Result for objective_efe55c2a:\n", " date: 2022-07-22_15-24-11\n", " done: false\n", " experiment_id: 903e9605ba894aa0bc55297229b8a77b\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: -0.9039251143764186\n", " neg_mean_loss: 0.9039251143764186\n", " node_ip: 127.0.0.1\n", " pid: 46443\n", " time_since_restore: 5.145689249038696\n", " time_this_iter_s: 0.10677504539489746\n", " time_total_s: 5.145689249038696\n", " timestamp: 1658499851\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: efe55c2a\n", " warmup_time: 0.0031630992889404297\n", " \n", "Result for objective_efe34e4e:\n", " date: 2022-07-22_15-24-11\n", " done: false\n", " experiment_id: 1efbb9c1becb436e8f304b61e9edd61b\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: 1.7406930109818743\n", " neg_mean_loss: -1.7406930109818743\n", " node_ip: 127.0.0.1\n", " pid: 46442\n", " time_since_restore: 5.151263952255249\n", " time_this_iter_s: 0.10529589653015137\n", " time_total_s: 5.151263952255249\n", " timestamp: 1658499851\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: efe34e4e\n", " warmup_time: 0.002650022506713867\n", " \n", "Result for objective_ee2ca136:\n", " date: 2022-07-22_15-24-13\n", " done: false\n", " experiment_id: c0ad5ddb78cc4cc88e8195f5bde34e20\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 92\n", " iterations_since_restore: 93\n", " mean_loss: 1.10752688172043\n", " neg_mean_loss: -1.10752688172043\n", " node_ip: 127.0.0.1\n", " pid: 46434\n", " time_since_restore: 10.185918092727661\n", " time_this_iter_s: 0.10853385925292969\n", " time_total_s: 10.185918092727661\n", " timestamp: 1658499853\n", " timesteps_since_restore: 0\n", " training_iteration: 93\n", " trial_id: ee2ca136\n", " warmup_time: 0.003058910369873047\n", " \n", "Result for objective_ee2ca136:\n", " date: 2022-07-22_15-24-14\n", " done: true\n", " experiment_id: c0ad5ddb78cc4cc88e8195f5bde34e20\n", " experiment_tag: 1_activation=relu_tanh,height=0.0000,steps=100,width=10.0000\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 1.1\n", " neg_mean_loss: -1.1\n", " node_ip: 127.0.0.1\n", " pid: 46434\n", " time_since_restore: 10.941971063613892\n", " time_this_iter_s: 0.10557413101196289\n", " time_total_s: 10.941971063613892\n", " timestamp: 1658499854\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: ee2ca136\n", " warmup_time: 0.003058910369873047\n", " \n", "Result for objective_efe34e4e:\n", " date: 2022-07-22_15-24-16\n", " done: false\n", " experiment_id: 1efbb9c1becb436e8f304b61e9edd61b\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 90\n", " iterations_since_restore: 91\n", " mean_loss: 1.63713376556457\n", " neg_mean_loss: -1.63713376556457\n", " node_ip: 127.0.0.1\n", " pid: 46442\n", " time_since_restore: 9.770322799682617\n", " time_this_iter_s: 0.1040806770324707\n", " time_total_s: 9.770322799682617\n", " timestamp: 1658499856\n", " timesteps_since_restore: 0\n", " training_iteration: 91\n", " trial_id: efe34e4e\n", " warmup_time: 0.002650022506713867\n", " \n", "Result for objective_efe1626e:\n", " date: 2022-07-22_15-24-16\n", " done: false\n", " experiment_id: c6d33a9a30c040c0929b657f1d3e1557\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 92\n", " iterations_since_restore: 93\n", " mean_loss: -1.9844528559710652\n", " neg_mean_loss: 1.9844528559710652\n", " node_ip: 127.0.0.1\n", " pid: 46441\n", " time_since_restore: 9.955276012420654\n", " time_this_iter_s: 0.10721087455749512\n", " time_total_s: 9.955276012420654\n", " timestamp: 1658499856\n", " timesteps_since_restore: 0\n", " training_iteration: 93\n", " trial_id: efe1626e\n", " warmup_time: 0.003210783004760742\n", " \n", "Result for objective_efe55c2a:\n", " date: 2022-07-22_15-24-16\n", " done: false\n", " experiment_id: 903e9605ba894aa0bc55297229b8a77b\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 91\n", " iterations_since_restore: 92\n", " mean_loss: -0.991699632507838\n", " neg_mean_loss: 0.991699632507838\n", " node_ip: 127.0.0.1\n", " pid: 46443\n", " time_since_restore: 9.83866286277771\n", " time_this_iter_s: 0.10593676567077637\n", " time_total_s: 9.83866286277771\n", " timestamp: 1658499856\n", " timesteps_since_restore: 0\n", " training_iteration: 92\n", " trial_id: efe55c2a\n", " warmup_time: 0.0031630992889404297\n", " \n", "Result for objective_f6688086:\n", " date: 2022-07-22_15-24-17\n", " done: false\n", " experiment_id: aa090556819b41c5b1e93d761dc9311a\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 16.728285222315503\n", " neg_mean_loss: -16.728285222315503\n", " node_ip: 127.0.0.1\n", " pid: 46467\n", " time_since_restore: 0.10288572311401367\n", " time_this_iter_s: 0.10288572311401367\n", " time_total_s: 0.10288572311401367\n", " timestamp: 1658499857\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: f6688086\n", " warmup_time: 0.002875089645385742\n", " \n", "Result for objective_efe1626e:\n", " date: 2022-07-22_15-24-17\n", " done: true\n", " experiment_id: c6d33a9a30c040c0929b657f1d3e1557\n", " experiment_tag: 2_activation=relu_tanh,height=-31.0013,steps=100,width=9.2876\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: -1.9925441896649678\n", " neg_mean_loss: 1.9925441896649678\n", " node_ip: 127.0.0.1\n", " pid: 46441\n", " time_since_restore: 11.535439252853394\n", " time_this_iter_s: 0.10611414909362793\n", " time_total_s: 11.535439252853394\n", " timestamp: 1658499857\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: efe1626e\n", " warmup_time: 0.003210783004760742\n", " \n", "Result for objective_efe55c2a:\n", " date: 2022-07-22_15-24-17\n", " done: true\n", " experiment_id: 903e9605ba894aa0bc55297229b8a77b\n", " experiment_tag: 4_activation=relu_tanh,height=-20.8721,steps=100,width=11.3958\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: -0.9993497773424045\n", " neg_mean_loss: 0.9993497773424045\n", " node_ip: 127.0.0.1\n", " pid: 46443\n", " time_since_restore: 11.608299970626831\n", " time_this_iter_s: 0.10889291763305664\n", " time_total_s: 11.608299970626831\n", " timestamp: 1658499857\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: efe55c2a\n", " warmup_time: 0.0031630992889404297\n", " \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Result for objective_efe34e4e:\n", " date: 2022-07-22_15-24-18\n", " done: true\n", " experiment_id: 1efbb9c1becb436e8f304b61e9edd61b\n", " experiment_tag: 3_activation=relu_tanh,height=5.2140,steps=100,width=9.4897\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 1.6267236167220034\n", " neg_mean_loss: -1.6267236167220034\n", " node_ip: 127.0.0.1\n", " pid: 46442\n", " time_since_restore: 11.660571098327637\n", " time_this_iter_s: 0.1082301139831543\n", " time_total_s: 11.660571098327637\n", " timestamp: 1658499858\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: efe34e4e\n", " warmup_time: 0.002650022506713867\n", " \n", "Result for objective_f85ed926:\n", " date: 2022-07-22_15-24-20\n", " done: false\n", " experiment_id: 402684968ef24377ae7787c9af50d3ae\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 15.055429995999226\n", " neg_mean_loss: -15.055429995999226\n", " node_ip: 127.0.0.1\n", " pid: 46478\n", " time_since_restore: 0.10394287109375\n", " time_this_iter_s: 0.10394287109375\n", " time_total_s: 0.10394287109375\n", " timestamp: 1658499860\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: f85ed926\n", " warmup_time: 0.0031621456146240234\n", " \n", "Result for objective_f86ee276:\n", " date: 2022-07-22_15-24-20\n", " done: false\n", " experiment_id: cbdc4d94c3d64e59aa53e4465349dbe1\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 20.386860838650087\n", " neg_mean_loss: -20.386860838650087\n", " node_ip: 127.0.0.1\n", " pid: 46481\n", " time_since_restore: 0.10312676429748535\n", " time_this_iter_s: 0.10312676429748535\n", " time_total_s: 0.10312676429748535\n", " timestamp: 1658499860\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: f86ee276\n", " warmup_time: 0.002810955047607422\n", " \n", "Result for objective_f880a02e:\n", " date: 2022-07-22_15-24-20\n", " done: false\n", " experiment_id: fb5af0209b2b45eda67ab40960a48a99\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 2.94231225430282\n", " neg_mean_loss: -2.94231225430282\n", " node_ip: 127.0.0.1\n", " pid: 46484\n", " time_since_restore: 0.10325503349304199\n", " time_this_iter_s: 0.10325503349304199\n", " time_total_s: 0.10325503349304199\n", " timestamp: 1658499860\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: f880a02e\n", " warmup_time: 0.0027141571044921875\n", " \n", "Result for objective_f6688086:\n", " date: 2022-07-22_15-24-22\n", " done: false\n", " experiment_id: aa090556819b41c5b1e93d761dc9311a\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: 6.846867893893624\n", " neg_mean_loss: -6.846867893893624\n", " node_ip: 127.0.0.1\n", " pid: 46467\n", " time_since_restore: 5.122231721878052\n", " time_this_iter_s: 0.10641169548034668\n", " time_total_s: 5.122231721878052\n", " timestamp: 1658499862\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: f6688086\n", " warmup_time: 0.002875089645385742\n", " \n", "Result for objective_f85ed926:\n", " date: 2022-07-22_15-24-25\n", " done: false\n", " experiment_id: 402684968ef24377ae7787c9af50d3ae\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: 5.1657053480548045\n", " neg_mean_loss: -5.1657053480548045\n", " node_ip: 127.0.0.1\n", " pid: 46478\n", " time_since_restore: 5.151860952377319\n", " time_this_iter_s: 0.10704493522644043\n", " time_total_s: 5.151860952377319\n", " timestamp: 1658499865\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: f85ed926\n", " warmup_time: 0.0031621456146240234\n", " \n", "Result for objective_f86ee276:\n", " date: 2022-07-22_15-24-25\n", " done: false\n", " experiment_id: cbdc4d94c3d64e59aa53e4465349dbe1\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: 11.55568804118157\n", " neg_mean_loss: -11.55568804118157\n", " node_ip: 127.0.0.1\n", " pid: 46481\n", " time_since_restore: 5.140729665756226\n", " time_this_iter_s: 0.1076810359954834\n", " time_total_s: 5.140729665756226\n", " timestamp: 1658499865\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: f86ee276\n", " warmup_time: 0.002810955047607422\n", " \n", "Result for objective_f880a02e:\n", " date: 2022-07-22_15-24-25\n", " done: false\n", " experiment_id: fb5af0209b2b45eda67ab40960a48a99\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: -6.706663109525936\n", " neg_mean_loss: 6.706663109525936\n", " node_ip: 127.0.0.1\n", " pid: 46484\n", " time_since_restore: 5.157264947891235\n", " time_this_iter_s: 0.1107029914855957\n", " time_total_s: 5.157264947891235\n", " timestamp: 1658499865\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: f880a02e\n", " warmup_time: 0.0027141571044921875\n", " \n", "Result for objective_f6688086:\n", " date: 2022-07-22_15-24-27\n", " done: false\n", " experiment_id: aa090556819b41c5b1e93d761dc9311a\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " mean_loss: 6.787930201151392\n", " neg_mean_loss: -6.787930201151392\n", " node_ip: 127.0.0.1\n", " pid: 46467\n", " time_since_restore: 10.178911924362183\n", " time_this_iter_s: 0.10588788986206055\n", " time_total_s: 10.178911924362183\n", " timestamp: 1658499867\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: f6688086\n", " warmup_time: 0.002875089645385742\n", " \n", "Result for objective_f6688086:\n", " date: 2022-07-22_15-24-27\n", " done: true\n", " experiment_id: aa090556819b41c5b1e93d761dc9311a\n", " experiment_tag: 5_activation=relu_tanh,height=57.2829,steps=100,width=17.7296\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 6.784934893475021\n", " neg_mean_loss: -6.784934893475021\n", " node_ip: 127.0.0.1\n", " pid: 46467\n", " time_since_restore: 10.71599006652832\n", " time_this_iter_s: 0.10760498046875\n", " time_total_s: 10.71599006652832\n", " timestamp: 1658499867\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: f6688086\n", " warmup_time: 0.002875089645385742\n", " \n", "Result for objective_fe4e7a44:\n", " date: 2022-07-22_15-24-30\n", " done: false\n", " experiment_id: 95711fcb897a4f9fae3ebd811619fba9\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 11.962911259228687\n", " neg_mean_loss: -11.962911259228687\n", " node_ip: 127.0.0.1\n", " pid: 46499\n", " time_since_restore: 0.10338783264160156\n", " time_this_iter_s: 0.10338783264160156\n", " time_total_s: 0.10338783264160156\n", " timestamp: 1658499870\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: fe4e7a44\n", " warmup_time: 0.002811908721923828\n", " \n", "Result for objective_f85ed926:\n", " date: 2022-07-22_15-24-30\n", " done: false\n", " experiment_id: 402684968ef24377ae7787c9af50d3ae\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " mean_loss: 5.110873373928027\n", " neg_mean_loss: -5.110873373928027\n", " node_ip: 127.0.0.1\n", " pid: 46478\n", " time_since_restore: 10.178997039794922\n", " time_this_iter_s: 0.10618400573730469\n", " time_total_s: 10.178997039794922\n", " timestamp: 1658499870\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: f85ed926\n", " warmup_time: 0.0031621456146240234\n", " \n", "Result for objective_f86ee276:\n", " date: 2022-07-22_15-24-30\n", " done: false\n", " experiment_id: cbdc4d94c3d64e59aa53e4465349dbe1\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " mean_loss: 11.007548256848812\n", " neg_mean_loss: -11.007548256848812\n", " node_ip: 127.0.0.1\n", " pid: 46481\n", " time_since_restore: 10.165759801864624\n", " time_this_iter_s: 0.10771489143371582\n", " time_total_s: 10.165759801864624\n", " timestamp: 1658499870\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: f86ee276\n", " warmup_time: 0.002810955047607422\n", " \n", "Result for objective_f880a02e:\n", " date: 2022-07-22_15-24-30\n", " done: false\n", " experiment_id: fb5af0209b2b45eda67ab40960a48a99\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " mean_loss: -6.87903993853598\n", " neg_mean_loss: 6.87903993853598\n", " node_ip: 127.0.0.1\n", " pid: 46484\n", " time_since_restore: 10.188904047012329\n", " time_this_iter_s: 0.10939908027648926\n", " time_total_s: 10.188904047012329\n", " timestamp: 1658499870\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: f880a02e\n", " warmup_time: 0.0027141571044921875\n", " \n", "Result for objective_f85ed926:\n", " date: 2022-07-22_15-24-31\n", " done: true\n", " experiment_id: 402684968ef24377ae7787c9af50d3ae\n", " experiment_tag: 6_activation=relu_tanh,height=40.5543,steps=100,width=19.0813\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 5.108087948450606\n", " neg_mean_loss: -5.108087948450606\n", " node_ip: 127.0.0.1\n", " pid: 46478\n", " time_since_restore: 10.71581220626831\n", " time_this_iter_s: 0.10749602317810059\n", " time_total_s: 10.71581220626831\n", " timestamp: 1658499871\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: f85ed926\n", " warmup_time: 0.0031621456146240234\n", " \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Result for objective_f86ee276:\n", " date: 2022-07-22_15-24-31\n", " done: true\n", " experiment_id: cbdc4d94c3d64e59aa53e4465349dbe1\n", " experiment_tag: 7_activation=relu_tanh,height=93.8686,steps=100,width=1.6076\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 10.978053669828887\n", " neg_mean_loss: -10.978053669828887\n", " node_ip: 127.0.0.1\n", " pid: 46481\n", " time_since_restore: 10.741472005844116\n", " time_this_iter_s: 0.1456291675567627\n", " time_total_s: 10.741472005844116\n", " timestamp: 1658499871\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: f86ee276\n", " warmup_time: 0.002810955047607422\n", " \n", "Result for objective_f880a02e:\n", " date: 2022-07-22_15-24-31\n", " done: true\n", " experiment_id: fb5af0209b2b45eda67ab40960a48a99\n", " experiment_tag: 8_activation=relu_tanh,height=-80.5769,steps=100,width=5.8485\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: -6.887909370541976\n", " neg_mean_loss: 6.887909370541976\n", " node_ip: 127.0.0.1\n", " pid: 46484\n", " time_since_restore: 10.733515977859497\n", " time_this_iter_s: 0.1060791015625\n", " time_total_s: 10.733515977859497\n", " timestamp: 1658499871\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: f880a02e\n", " warmup_time: 0.0027141571044921875\n", " \n", "Result for objective_004f499a:\n", " date: 2022-07-22_15-24-33\n", " done: false\n", " experiment_id: c3015ffd206444cd9c2eb1152aae5dd0\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 1.9493461263053842\n", " neg_mean_loss: -1.9493461263053842\n", " node_ip: 127.0.0.1\n", " pid: 46504\n", " time_since_restore: 0.1044008731842041\n", " time_this_iter_s: 0.1044008731842041\n", " time_total_s: 0.1044008731842041\n", " timestamp: 1658499873\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 004f499a\n", " warmup_time: 0.0026237964630126953\n", " \n", "Result for objective_fe4e7a44:\n", " date: 2022-07-22_15-24-35\n", " done: false\n", " experiment_id: 95711fcb897a4f9fae3ebd811619fba9\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 25\n", " iterations_since_restore: 26\n", " mean_loss: 5.873327389463485\n", " neg_mean_loss: -5.873327389463485\n", " node_ip: 127.0.0.1\n", " pid: 46499\n", " time_since_restore: 5.199802875518799\n", " time_this_iter_s: 0.1075749397277832\n", " time_total_s: 5.199802875518799\n", " timestamp: 1658499875\n", " timesteps_since_restore: 0\n", " training_iteration: 26\n", " trial_id: fe4e7a44\n", " warmup_time: 0.002811908721923828\n", " \n", "Result for objective_004f499a:\n", " date: 2022-07-22_15-24-38\n", " done: false\n", " experiment_id: c3015ffd206444cd9c2eb1152aae5dd0\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: -7.137564846035238\n", " neg_mean_loss: 7.137564846035238\n", " node_ip: 127.0.0.1\n", " pid: 46504\n", " time_since_restore: 5.15981912612915\n", " time_this_iter_s: 0.10737729072570801\n", " time_total_s: 5.15981912612915\n", " timestamp: 1658499878\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: 004f499a\n", " warmup_time: 0.0026237964630126953\n", " \n", "Result for objective_fe4e7a44:\n", " date: 2022-07-22_15-24-40\n", " done: false\n", " experiment_id: 95711fcb897a4f9fae3ebd811619fba9\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 72\n", " iterations_since_restore: 73\n", " mean_loss: 3.7860835740628094\n", " neg_mean_loss: -3.7860835740628094\n", " node_ip: 127.0.0.1\n", " pid: 46499\n", " time_since_restore: 10.234857082366943\n", " time_this_iter_s: 0.10819101333618164\n", " time_total_s: 10.234857082366943\n", " timestamp: 1658499880\n", " timesteps_since_restore: 0\n", " training_iteration: 73\n", " trial_id: fe4e7a44\n", " warmup_time: 0.002811908721923828\n", " \n", "Result for objective_fe4e7a44:\n", " date: 2022-07-22_15-24-43\n", " done: true\n", " experiment_id: 95711fcb897a4f9fae3ebd811619fba9\n", " experiment_tag: 9_activation=relu_tanh,height=9.6291,steps=100,width=0.6229\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 3.3582342398877607\n", " neg_mean_loss: -3.3582342398877607\n", " node_ip: 127.0.0.1\n", " pid: 46499\n", " time_since_restore: 13.142819166183472\n", " time_this_iter_s: 0.10675215721130371\n", " time_total_s: 13.142819166183472\n", " timestamp: 1658499883\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: fe4e7a44\n", " warmup_time: 0.002811908721923828\n", " \n", "Result for objective_004f499a:\n", " date: 2022-07-22_15-24-43\n", " done: false\n", " experiment_id: c3015ffd206444cd9c2eb1152aae5dd0\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " mean_loss: -7.572268959036313\n", " neg_mean_loss: 7.572268959036313\n", " node_ip: 127.0.0.1\n", " pid: 46504\n", " time_since_restore: 10.228418827056885\n", " time_this_iter_s: 0.10715484619140625\n", " time_total_s: 10.228418827056885\n", " timestamp: 1658499883\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: 004f499a\n", " warmup_time: 0.0026237964630126953\n", " \n", "Result for objective_004f499a:\n", " date: 2022-07-22_15-24-44\n", " done: true\n", " experiment_id: c3015ffd206444cd9c2eb1152aae5dd0\n", " experiment_tag: 10_activation=relu_tanh,height=-90.5065,steps=100,width=2.1174\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: -7.595329711238255\n", " neg_mean_loss: 7.595329711238255\n", " node_ip: 127.0.0.1\n", " pid: 46504\n", " time_since_restore: 10.768790006637573\n", " time_this_iter_s: 0.10774111747741699\n", " time_total_s: 10.768790006637573\n", " timestamp: 1658499884\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 004f499a\n", " warmup_time: 0.0026237964630126953\n", " \n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "INFO:ray.tune.tune:Total run time: 44.70 seconds (43.68 seconds for the tuning loop).\n" ] } ], "source": [ "tuner = tune.Tuner(\n", " objective,\n", " tune_config=tune.TuneConfig(\n", " metric=\"mean_loss\",\n", " mode=\"min\",\n", " search_alg=algo,\n", " num_samples=num_samples,\n", " ),\n", " param_space=search_config,\n", ")\n", "results = tuner.fit()" ] }, { "cell_type": "markdown", "id": "950003be", "metadata": {}, "source": [ "Here are the hyperparamters found to minimize the mean loss of the defined objective." ] }, { "cell_type": "code", "execution_count": 11, "id": "0f021674", "metadata": { "lines_to_next_cell": 0 }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Best hyperparameters found were: {'steps': 100, 'width': 2.1174116156230918, 'height': -90.50653873694615, 'activation': 'relu, tanh'}\n" ] } ], "source": [ "print(\"Best hyperparameters found were: \", results.get_best_result().config)" ] }, { "cell_type": "markdown", "id": "0d3824ae", "metadata": {}, "source": [ "## Optional: passing the (hyper)parameter space into the search algorithm\n", "\n", "We can also pass the search space into `NevergradSearch` using their designed format." ] }, { "cell_type": "code", "execution_count": 12, "id": "89ae7455", "metadata": {}, "outputs": [], "source": [ "space = ng.p.Dict(\n", " width=ng.p.Scalar(lower=0, upper=20),\n", " height=ng.p.Scalar(lower=-100, upper=100),\n", " activation=ng.p.Choice(choices=[\"relu\", \"tanh\"])\n", ")" ] }, { "cell_type": "code", "execution_count": 13, "id": "e52eeab1", "metadata": {}, "outputs": [], "source": [ "algo = NevergradSearch(\n", " optimizer=ng.optimizers.OnePlusOne,\n", " space=space,\n", " metric=\"mean_loss\",\n", " mode=\"min\"\n", ")\n", "algo = tune.search.ConcurrencyLimiter(algo, max_concurrent=4)" ] }, { "cell_type": "markdown", "id": "f8926177", "metadata": {}, "source": [ "Again we run the experiment, this time with a less passed via the `config` and instead passed through `search_alg`." ] }, { "cell_type": "code", "execution_count": 14, "id": "64f39800", "metadata": {}, "outputs": [ { "data": { "text/html": [ "== Status ==
Current time: 2022-07-22 15:25:27 (running for 00:00:43.22)
Memory usage on this node: 10.8/16.0 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/4.61 GiB heap, 0.0/2.0 GiB objects
Result logdir: /Users/kai/ray_results/objective_2022-07-22_15-24-44
Number of trials: 10/10 (10 TERMINATED)
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
Trial name status loc activation height width loss iter total time (s) iterations neg_mean_loss
objective_085274beTERMINATED127.0.0.1:46516tanh 0 10 1.1 100 10.7324 99 -1.1
objective_09dee4f2TERMINATED127.0.0.1:46524tanh -44.4216 12.9653 -3.36485 100 11.3476 99 3.36485
objective_09e0846aTERMINATED127.0.0.1:46525relu -38.0638 11.1574 6.28334 100 11.3103 99 -6.28334
objective_09e21122TERMINATED127.0.0.1:46526relu 41.2509 9.7558514.2276 100 11.3512 99 -14.2276
objective_1045958eTERMINATED127.0.0.1:46544relu -73.2818 5.78832 2.84334 100 10.7372 99 -2.84334
objective_12309db2TERMINATED127.0.0.1:46549relu -94.9666 16.9764 0.562486 100 10.7329 99 -0.562486
objective_12342770TERMINATED127.0.0.1:46550tanh -98.0775 17.2252 -8.74945 100 10.7455 99 8.74945
objective_12374d7eTERMINATED127.0.0.1:46551relu -1.6075918.0841 9.89479 100 10.7348 99 -9.89479
objective_18344524TERMINATED127.0.0.1:46569tanh -41.1284 12.2952 -3.03135 100 12.622 99 3.03135
objective_1a1e29b8TERMINATED127.0.0.1:46576tanh 64.0289 10.0482 7.50242 100 10.8237 99 -7.50242


" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "Result for objective_085274be:\n", " date: 2022-07-22_15-24-47\n", " done: false\n", " experiment_id: be25590fb790400ca28b548b8359a120\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 11.0\n", " neg_mean_loss: -11.0\n", " node_ip: 127.0.0.1\n", " pid: 46516\n", " time_since_restore: 0.10224103927612305\n", " time_this_iter_s: 0.10224103927612305\n", " time_total_s: 0.10224103927612305\n", " timestamp: 1658499887\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 085274be\n", " warmup_time: 0.0029327869415283203\n", " \n", "Result for objective_09dee4f2:\n", " date: 2022-07-22_15-24-49\n", " done: false\n", " experiment_id: 9469ffc8df11476db00e329acd2ae8b6\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 6.557843114006006\n", " neg_mean_loss: -6.557843114006006\n", " node_ip: 127.0.0.1\n", " pid: 46524\n", " time_since_restore: 0.10509586334228516\n", " time_this_iter_s: 0.10509586334228516\n", " time_total_s: 0.10509586334228516\n", " timestamp: 1658499889\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 09dee4f2\n", " warmup_time: 0.002938985824584961\n", " \n", "Result for objective_09e21122:\n", " date: 2022-07-22_15-24-49\n", " done: false\n", " experiment_id: 0a825198482b4bd4aded6c318dff0dcf\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 24.125094234750687\n", " neg_mean_loss: -24.125094234750687\n", " node_ip: 127.0.0.1\n", " pid: 46526\n", " time_since_restore: 0.10476112365722656\n", " time_this_iter_s: 0.10476112365722656\n", " time_total_s: 0.10476112365722656\n", " timestamp: 1658499889\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 09e21122\n", " warmup_time: 0.003228902816772461\n", " \n", "Result for objective_09e0846a:\n", " date: 2022-07-22_15-24-49\n", " done: false\n", " experiment_id: 260410fc1fb24f78af507cd38a8361a9\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 16.193619847512025\n", " neg_mean_loss: -16.193619847512025\n", " node_ip: 127.0.0.1\n", " pid: 46525\n", " time_since_restore: 0.10422873497009277\n", " time_this_iter_s: 0.10422873497009277\n", " time_total_s: 0.10422873497009277\n", " timestamp: 1658499889\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 09e0846a\n", " warmup_time: 0.002719879150390625\n", " \n", "Result for objective_085274be:\n", " date: 2022-07-22_15-24-52\n", " done: false\n", " experiment_id: be25590fb790400ca28b548b8359a120\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: 1.2083333333333333\n", " neg_mean_loss: -1.2083333333333333\n", " node_ip: 127.0.0.1\n", " pid: 46516\n", " time_since_restore: 5.132040977478027\n", " time_this_iter_s: 0.1084139347076416\n", " time_total_s: 5.132040977478027\n", " timestamp: 1658499892\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: 085274be\n", " warmup_time: 0.0029327869415283203\n", " \n", "Result for objective_09dee4f2:\n", " date: 2022-07-22_15-24-54\n", " done: false\n", " experiment_id: 9469ffc8df11476db00e329acd2ae8b6\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: -3.280701838145472\n", " neg_mean_loss: 3.280701838145472\n", " node_ip: 127.0.0.1\n", " pid: 46524\n", " time_since_restore: 5.134023189544678\n", " time_this_iter_s: 0.10428118705749512\n", " time_total_s: 5.134023189544678\n", " timestamp: 1658499894\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: 09dee4f2\n", " warmup_time: 0.002938985824584961\n", " \n", "Result for objective_09e21122:\n", " date: 2022-07-22_15-24-54\n", " done: false\n", " experiment_id: 0a825198482b4bd4aded6c318dff0dcf\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: 14.33852997141215\n", " neg_mean_loss: -14.33852997141215\n", " node_ip: 127.0.0.1\n", " pid: 46526\n", " time_since_restore: 5.139041185379028\n", " time_this_iter_s: 0.10721492767333984\n", " time_total_s: 5.139041185379028\n", " timestamp: 1658499894\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: 09e21122\n", " warmup_time: 0.003228902816772461\n", " \n", "Result for objective_09e0846a:\n", " date: 2022-07-22_15-24-54\n", " done: false\n", " experiment_id: 260410fc1fb24f78af507cd38a8361a9\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: 6.3807469650477024\n", " neg_mean_loss: -6.3807469650477024\n", " node_ip: 127.0.0.1\n", " pid: 46525\n", " time_since_restore: 5.149268865585327\n", " time_this_iter_s: 0.10705304145812988\n", " time_total_s: 5.149268865585327\n", " timestamp: 1658499894\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: 09e0846a\n", " warmup_time: 0.002719879150390625\n", " \n", "Result for objective_085274be:\n", " date: 2022-07-22_15-24-57\n", " done: false\n", " experiment_id: be25590fb790400ca28b548b8359a120\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " mean_loss: 1.1052631578947367\n", " neg_mean_loss: -1.1052631578947367\n", " node_ip: 127.0.0.1\n", " pid: 46516\n", " time_since_restore: 10.194181203842163\n", " time_this_iter_s: 0.10754203796386719\n", " time_total_s: 10.194181203842163\n", " timestamp: 1658499897\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: 085274be\n", " warmup_time: 0.0029327869415283203\n", " \n", "Result for objective_085274be:\n", " date: 2022-07-22_15-24-57\n", " done: true\n", " experiment_id: be25590fb790400ca28b548b8359a120\n", " experiment_tag: 1_activation=tanh,height=0.0000,steps=100,width=10.0000\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 1.1\n", " neg_mean_loss: -1.1\n", " node_ip: 127.0.0.1\n", " pid: 46516\n", " time_since_restore: 10.732417106628418\n", " time_this_iter_s: 0.10759997367858887\n", " time_total_s: 10.732417106628418\n", " timestamp: 1658499897\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 085274be\n", " warmup_time: 0.0029327869415283203\n", " \n", "Result for objective_09dee4f2:\n", " date: 2022-07-22_15-24-59\n", " done: false\n", " experiment_id: 9469ffc8df11476db00e329acd2ae8b6\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 93\n", " iterations_since_restore: 94\n", " mean_loss: -3.3599044605358728\n", " neg_mean_loss: 3.3599044605358728\n", " node_ip: 127.0.0.1\n", " pid: 46524\n", " time_since_restore: 10.055138111114502\n", " time_this_iter_s: 0.10467100143432617\n", " time_total_s: 10.055138111114502\n", " timestamp: 1658499899\n", " timesteps_since_restore: 0\n", " training_iteration: 94\n", " trial_id: 09dee4f2\n", " warmup_time: 0.002938985824584961\n", " \n", "Result for objective_09e21122:\n", " date: 2022-07-22_15-24-59\n", " done: false\n", " experiment_id: 0a825198482b4bd4aded6c318dff0dcf\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 93\n", " iterations_since_restore: 94\n", " mean_loss: 14.23411049568083\n", " neg_mean_loss: -14.23411049568083\n", " node_ip: 127.0.0.1\n", " pid: 46526\n", " time_since_restore: 10.066343307495117\n", " time_this_iter_s: 0.1061861515045166\n", " time_total_s: 10.066343307495117\n", " timestamp: 1658499899\n", " timesteps_since_restore: 0\n", " training_iteration: 94\n", " trial_id: 09e21122\n", " warmup_time: 0.003228902816772461\n", " \n", "Result for objective_09e0846a:\n", " date: 2022-07-22_15-24-59\n", " done: false\n", " experiment_id: 260410fc1fb24f78af507cd38a8361a9\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 93\n", " iterations_since_restore: 94\n", " mean_loss: 6.2890729561522765\n", " neg_mean_loss: -6.2890729561522765\n", " node_ip: 127.0.0.1\n", " pid: 46525\n", " time_since_restore: 10.100499868392944\n", " time_this_iter_s: 0.10625505447387695\n", " time_total_s: 10.100499868392944\n", " timestamp: 1658499899\n", " timesteps_since_restore: 0\n", " training_iteration: 94\n", " trial_id: 09e0846a\n", " warmup_time: 0.002719879150390625\n", " \n", "Result for objective_1045958e:\n", " date: 2022-07-22_15-25-00\n", " done: false\n", " experiment_id: 3579bfc2b346424b833f82f23d459807\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 12.671822501738296\n", " neg_mean_loss: -12.671822501738296\n", " node_ip: 127.0.0.1\n", " pid: 46544\n", " time_since_restore: 0.10491013526916504\n", " time_this_iter_s: 0.10491013526916504\n", " time_total_s: 0.10491013526916504\n", " timestamp: 1658499900\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 1045958e\n", " warmup_time: 0.002847909927368164\n", " \n", "Result for objective_09dee4f2:\n", " date: 2022-07-22_15-25-01\n", " done: true\n", " experiment_id: 9469ffc8df11476db00e329acd2ae8b6\n", " experiment_tag: 2_activation=tanh,height=-44.4216,steps=100,width=12.9653\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: -3.3648509190285747\n", " neg_mean_loss: 3.3648509190285747\n", " node_ip: 127.0.0.1\n", " pid: 46524\n", " time_since_restore: 11.347625017166138\n", " time_this_iter_s: 0.10793185234069824\n", " time_total_s: 11.347625017166138\n", " timestamp: 1658499901\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 09dee4f2\n", " warmup_time: 0.002938985824584961\n", " \n", "Result for objective_09e0846a:\n", " date: 2022-07-22_15-25-01\n", " done: true\n", " experiment_id: 260410fc1fb24f78af507cd38a8361a9\n", " experiment_tag: 3_activation=relu,height=-38.0638,steps=100,width=11.1574\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 6.28333982260153\n", " neg_mean_loss: -6.28333982260153\n", " node_ip: 127.0.0.1\n", " pid: 46525\n", " time_since_restore: 11.310342788696289\n", " time_this_iter_s: 0.10942316055297852\n", " time_total_s: 11.310342788696289\n", " timestamp: 1658499901\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 09e0846a\n", " warmup_time: 0.002719879150390625\n", " \n", "Result for objective_09e21122:\n", " date: 2022-07-22_15-25-01\n", " done: true\n", " experiment_id: 0a825198482b4bd4aded6c318dff0dcf\n", " experiment_tag: 4_activation=relu,height=41.2509,steps=100,width=9.7559\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 14.22757115653867\n", " neg_mean_loss: -14.22757115653867\n", " node_ip: 127.0.0.1\n", " pid: 46526\n", " time_since_restore: 11.351194143295288\n", " time_this_iter_s: 0.10791015625\n", " time_total_s: 11.351194143295288\n", " timestamp: 1658499901\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 09e21122\n", " warmup_time: 0.003228902816772461\n", " \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Result for objective_12309db2:\n", " date: 2022-07-22_15-25-03\n", " done: false\n", " experiment_id: 3915d19b775c4ee1843b5dcf79560a93\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 10.503337773185708\n", " neg_mean_loss: -10.503337773185708\n", " node_ip: 127.0.0.1\n", " pid: 46549\n", " time_since_restore: 0.10407328605651855\n", " time_this_iter_s: 0.10407328605651855\n", " time_total_s: 0.10407328605651855\n", " timestamp: 1658499903\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 12309db2\n", " warmup_time: 0.0030128955841064453\n", " \n", "Result for objective_12342770:\n", " date: 2022-07-22_15-25-03\n", " done: false\n", " experiment_id: 1da16f0c20d7438b93927280dc40e5a1\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 1.1922491879354844\n", " neg_mean_loss: -1.1922491879354844\n", " node_ip: 127.0.0.1\n", " pid: 46550\n", " time_since_restore: 0.1050269603729248\n", " time_this_iter_s: 0.1050269603729248\n", " time_total_s: 0.1050269603729248\n", " timestamp: 1658499903\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: '12342770'\n", " warmup_time: 0.0032460689544677734\n", " \n", "Result for objective_12374d7e:\n", " date: 2022-07-22_15-25-03\n", " done: false\n", " experiment_id: 5788d010ee194eeeabfc3592d37fb2cc\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 19.839240921278268\n", " neg_mean_loss: -19.839240921278268\n", " node_ip: 127.0.0.1\n", " pid: 46551\n", " time_since_restore: 0.10313534736633301\n", " time_this_iter_s: 0.10313534736633301\n", " time_total_s: 0.10313534736633301\n", " timestamp: 1658499903\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 12374d7e\n", " warmup_time: 0.002891063690185547\n", " \n", "Result for objective_1045958e:\n", " date: 2022-07-22_15-25-05\n", " done: false\n", " experiment_id: 3579bfc2b346424b833f82f23d459807\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: 3.0263684682219036\n", " neg_mean_loss: -3.0263684682219036\n", " node_ip: 127.0.0.1\n", " pid: 46544\n", " time_since_restore: 5.180821180343628\n", " time_this_iter_s: 0.10701394081115723\n", " time_total_s: 5.180821180343628\n", " timestamp: 1658499905\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: 1045958e\n", " warmup_time: 0.002847909927368164\n", " \n", "Result for objective_12309db2:\n", " date: 2022-07-22_15-25-08\n", " done: false\n", " experiment_id: 3915d19b775c4ee1843b5dcf79560a93\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: 0.6271166903395393\n", " neg_mean_loss: -0.6271166903395393\n", " node_ip: 127.0.0.1\n", " pid: 46549\n", " time_since_restore: 5.172047138214111\n", " time_this_iter_s: 0.11187624931335449\n", " time_total_s: 5.172047138214111\n", " timestamp: 1658499908\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: 12309db2\n", " warmup_time: 0.0030128955841064453\n", " \n", "Result for objective_12342770:\n", " date: 2022-07-22_15-25-08\n", " done: false\n", " experiment_id: 1da16f0c20d7438b93927280dc40e5a1\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: -8.685737519988487\n", " neg_mean_loss: 8.685737519988487\n", " node_ip: 127.0.0.1\n", " pid: 46550\n", " time_since_restore: 5.172597885131836\n", " time_this_iter_s: 0.10711097717285156\n", " time_total_s: 5.172597885131836\n", " timestamp: 1658499908\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: '12342770'\n", " warmup_time: 0.0032460689544677734\n", " \n", "Result for objective_12374d7e:\n", " date: 2022-07-22_15-25-08\n", " done: false\n", " experiment_id: 5788d010ee194eeeabfc3592d37fb2cc\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: 9.955526689542863\n", " neg_mean_loss: -9.955526689542863\n", " node_ip: 127.0.0.1\n", " pid: 46551\n", " time_since_restore: 5.162422180175781\n", " time_this_iter_s: 0.10872411727905273\n", " time_total_s: 5.162422180175781\n", " timestamp: 1658499908\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: 12374d7e\n", " warmup_time: 0.002891063690185547\n", " \n", "Result for objective_1045958e:\n", " date: 2022-07-22_15-25-10\n", " done: false\n", " experiment_id: 3579bfc2b346424b833f82f23d459807\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " mean_loss: 2.852294770731789\n", " neg_mean_loss: -2.852294770731789\n", " node_ip: 127.0.0.1\n", " pid: 46544\n", " time_since_restore: 10.196197271347046\n", " time_this_iter_s: 0.10780715942382812\n", " time_total_s: 10.196197271347046\n", " timestamp: 1658499910\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: 1045958e\n", " warmup_time: 0.002847909927368164\n", " \n", "Result for objective_1045958e:\n", " date: 2022-07-22_15-25-11\n", " done: true\n", " experiment_id: 3579bfc2b346424b833f82f23d459807\n", " experiment_tag: 5_activation=relu,height=-73.2818,steps=100,width=5.7883\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 2.8433363404373386\n", " neg_mean_loss: -2.8433363404373386\n", " node_ip: 127.0.0.1\n", " pid: 46544\n", " time_since_restore: 10.737220287322998\n", " time_this_iter_s: 0.10835909843444824\n", " time_total_s: 10.737220287322998\n", " timestamp: 1658499911\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 1045958e\n", " warmup_time: 0.002847909927368164\n", " \n", "Result for objective_18344524:\n", " date: 2022-07-22_15-25-13\n", " done: false\n", " experiment_id: 10740ef080664baaa8cca7ba5cb899ac\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 6.887162536491742\n", " neg_mean_loss: -6.887162536491742\n", " node_ip: 127.0.0.1\n", " pid: 46569\n", " time_since_restore: 0.10450887680053711\n", " time_this_iter_s: 0.10450887680053711\n", " time_total_s: 0.10450887680053711\n", " timestamp: 1658499913\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: '18344524'\n", " warmup_time: 0.002858877182006836\n", " \n", "Result for objective_12309db2:\n", " date: 2022-07-22_15-25-13\n", " done: false\n", " experiment_id: 3915d19b775c4ee1843b5dcf79560a93\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " mean_loss: 0.5656126475885976\n", " neg_mean_loss: -0.5656126475885976\n", " node_ip: 127.0.0.1\n", " pid: 46549\n", " time_since_restore: 10.193128108978271\n", " time_this_iter_s: 0.10602211952209473\n", " time_total_s: 10.193128108978271\n", " timestamp: 1658499913\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: 12309db2\n", " warmup_time: 0.0030128955841064453\n", " \n", "Result for objective_12342770:\n", " date: 2022-07-22_15-25-13\n", " done: false\n", " experiment_id: 1da16f0c20d7438b93927280dc40e5a1\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " mean_loss: -8.746369700451542\n", " neg_mean_loss: 8.746369700451542\n", " node_ip: 127.0.0.1\n", " pid: 46550\n", " time_since_restore: 10.210996866226196\n", " time_this_iter_s: 0.10719585418701172\n", " time_total_s: 10.210996866226196\n", " timestamp: 1658499913\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: '12342770'\n", " warmup_time: 0.0032460689544677734\n", " \n", "Result for objective_12374d7e:\n", " date: 2022-07-22_15-25-13\n", " done: false\n", " experiment_id: 5788d010ee194eeeabfc3592d37fb2cc\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " mean_loss: 9.897723841978765\n", " neg_mean_loss: -9.897723841978765\n", " node_ip: 127.0.0.1\n", " pid: 46551\n", " time_since_restore: 10.19912338256836\n", " time_this_iter_s: 0.10725522041320801\n", " time_total_s: 10.19912338256836\n", " timestamp: 1658499913\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: 12374d7e\n", " warmup_time: 0.002891063690185547\n", " \n", "Result for objective_12309db2:\n", " date: 2022-07-22_15-25-14\n", " done: true\n", " experiment_id: 3915d19b775c4ee1843b5dcf79560a93\n", " experiment_tag: 6_activation=relu,height=-94.9666,steps=100,width=16.9764\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 0.5624860552037738\n", " neg_mean_loss: -0.5624860552037738\n", " node_ip: 127.0.0.1\n", " pid: 46549\n", " time_since_restore: 10.73293399810791\n", " time_this_iter_s: 0.1079857349395752\n", " time_total_s: 10.73293399810791\n", " timestamp: 1658499914\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 12309db2\n", " warmup_time: 0.0030128955841064453\n", " \n", "Result for objective_12342770:\n", " date: 2022-07-22_15-25-14\n", " done: true\n", " experiment_id: 1da16f0c20d7438b93927280dc40e5a1\n", " experiment_tag: 7_activation=tanh,height=-98.0775,steps=100,width=17.2252\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: -8.749451683536464\n", " neg_mean_loss: 8.749451683536464\n", " node_ip: 127.0.0.1\n", " pid: 46550\n", " time_since_restore: 10.745534181594849\n", " time_this_iter_s: 0.10716795921325684\n", " time_total_s: 10.745534181594849\n", " timestamp: 1658499914\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: '12342770'\n", " warmup_time: 0.0032460689544677734\n", " \n", "Result for objective_12374d7e:\n", " date: 2022-07-22_15-25-14\n", " done: true\n", " experiment_id: 5788d010ee194eeeabfc3592d37fb2cc\n", " experiment_tag: 8_activation=relu,height=-1.6076,steps=100,width=18.0841\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 9.894786565536863\n", " neg_mean_loss: -9.894786565536863\n", " node_ip: 127.0.0.1\n", " pid: 46551\n", " time_since_restore: 10.734816074371338\n", " time_this_iter_s: 0.10638594627380371\n", " time_total_s: 10.734816074371338\n", " timestamp: 1658499914\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 12374d7e\n", " warmup_time: 0.002891063690185547\n", " \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Result for objective_1a1e29b8:\n", " date: 2022-07-22_15-25-17\n", " done: false\n", " experiment_id: 26fc8c5f05a6407690dc8cada561576e\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 17.40289288319415\n", " neg_mean_loss: -17.40289288319415\n", " node_ip: 127.0.0.1\n", " pid: 46576\n", " time_since_restore: 0.10475707054138184\n", " time_this_iter_s: 0.10475707054138184\n", " time_total_s: 0.10475707054138184\n", " timestamp: 1658499917\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 1a1e29b8\n", " warmup_time: 0.0028810501098632812\n", " \n", "Result for objective_18344524:\n", " date: 2022-07-22_15-25-19\n", " done: false\n", " experiment_id: 10740ef080664baaa8cca7ba5cb899ac\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 30\n", " iterations_since_restore: 31\n", " mean_loss: -2.8488858516331668\n", " neg_mean_loss: 2.8488858516331668\n", " node_ip: 127.0.0.1\n", " pid: 46569\n", " time_since_restore: 5.212157964706421\n", " time_this_iter_s: 0.10646200180053711\n", " time_total_s: 5.212157964706421\n", " timestamp: 1658499919\n", " timesteps_since_restore: 0\n", " training_iteration: 31\n", " trial_id: '18344524'\n", " warmup_time: 0.002858877182006836\n", " \n", "Result for objective_1a1e29b8:\n", " date: 2022-07-22_15-25-22\n", " done: false\n", " experiment_id: 26fc8c5f05a6407690dc8cada561576e\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: 7.610248309424407\n", " neg_mean_loss: -7.610248309424407\n", " node_ip: 127.0.0.1\n", " pid: 46576\n", " time_since_restore: 5.170850038528442\n", " time_this_iter_s: 0.1072230339050293\n", " time_total_s: 5.170850038528442\n", " timestamp: 1658499922\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: 1a1e29b8\n", " warmup_time: 0.0028810501098632812\n", " \n", "Result for objective_18344524:\n", " date: 2022-07-22_15-25-24\n", " done: false\n", " experiment_id: 10740ef080664baaa8cca7ba5cb899ac\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 77\n", " iterations_since_restore: 78\n", " mean_loss: -3.0083151799393004\n", " neg_mean_loss: 3.0083151799393004\n", " node_ip: 127.0.0.1\n", " pid: 46569\n", " time_since_restore: 10.263540983200073\n", " time_this_iter_s: 0.10780215263366699\n", " time_total_s: 10.263540983200073\n", " timestamp: 1658499924\n", " timesteps_since_restore: 0\n", " training_iteration: 78\n", " trial_id: '18344524'\n", " warmup_time: 0.002858877182006836\n", " \n", "Result for objective_18344524:\n", " date: 2022-07-22_15-25-26\n", " done: true\n", " experiment_id: 10740ef080664baaa8cca7ba5cb899ac\n", " experiment_tag: 9_activation=tanh,height=-41.1284,steps=100,width=12.2952\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: -3.0313530888899516\n", " neg_mean_loss: 3.0313530888899516\n", " node_ip: 127.0.0.1\n", " pid: 46569\n", " time_since_restore: 12.62198805809021\n", " time_this_iter_s: 0.1080331802368164\n", " time_total_s: 12.62198805809021\n", " timestamp: 1658499926\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: '18344524'\n", " warmup_time: 0.002858877182006836\n", " \n", "Result for objective_1a1e29b8:\n", " date: 2022-07-22_15-25-27\n", " done: false\n", " experiment_id: 26fc8c5f05a6407690dc8cada561576e\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 93\n", " iterations_since_restore: 94\n", " mean_loss: 7.5087713304782575\n", " neg_mean_loss: -7.5087713304782575\n", " node_ip: 127.0.0.1\n", " pid: 46576\n", " time_since_restore: 10.177540063858032\n", " time_this_iter_s: 0.1076350212097168\n", " time_total_s: 10.177540063858032\n", " timestamp: 1658499927\n", " timesteps_since_restore: 0\n", " training_iteration: 94\n", " trial_id: 1a1e29b8\n", " warmup_time: 0.0028810501098632812\n", " \n", "Result for objective_1a1e29b8:\n", " date: 2022-07-22_15-25-27\n", " done: true\n", " experiment_id: 26fc8c5f05a6407690dc8cada561576e\n", " experiment_tag: 10_activation=tanh,height=64.0289,steps=100,width=10.0482\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 7.502418319119348\n", " neg_mean_loss: -7.502418319119348\n", " node_ip: 127.0.0.1\n", " pid: 46576\n", " time_since_restore: 10.823745012283325\n", " time_this_iter_s: 0.10851097106933594\n", " time_total_s: 10.823745012283325\n", " timestamp: 1658499927\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 1a1e29b8\n", " warmup_time: 0.0028810501098632812\n", " \n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "INFO:ray.tune.tune:Total run time: 43.33 seconds (43.21 seconds for the tuning loop).\n" ] } ], "source": [ "tuner = tune.Tuner(\n", " objective,\n", " tune_config=tune.TuneConfig(\n", "# metric=\"mean_loss\",\n", "# mode=\"min\",\n", " search_alg=algo,\n", " num_samples=num_samples,\n", " ),\n", " param_space={\"steps\": 100},\n", ")\n", "results = tuner.fit()" ] }, { "cell_type": "markdown", "id": "64a17648", "metadata": {}, "source": [ "Here are the hyperparamters found to minimize the mean loss of the defined objective. Note that we have to pass the metric and mode here because we don't set it in the TuneConfig." ] }, { "cell_type": "code", "execution_count": 17, "id": "aac3e88b", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Best hyperparameters found were: {'steps': 100, 'width': 17.225166732233465, 'height': -98.07750812064515, 'activation': 'tanh'}\n" ] } ], "source": [ "print(\"Best hyperparameters found were: \", results.get_best_result(\"mean_loss\", \"min\").config)" ] }, { "cell_type": "code", "execution_count": 15, "id": "0478c1ea", "metadata": { "tags": [ "remove-cell" ] }, "outputs": [], "source": [ "ray.shutdown()" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.7" }, "orphan": true }, "nbformat": 4, "nbformat_minor": 5 }