{ "cells": [ { "cell_type": "markdown", "id": "61966d26", "metadata": {}, "source": [ "# Running Tune experiments with Skopt\n", "\n", "In this tutorial we introduce Skopt, while running a simple Ray Tune experiment. Tune’s Search Algorithms integrate with Skopt and, as a result, allow you to seamlessly scale up a Skopt optimization process - without sacrificing performance.\n", "\n", "Scikit-Optimize, or skopt, is a simple and efficient library to optimize expensive and noisy black-box functions, e.g. large-scale ML experiments. It implements several methods for sequential model-based optimization. Noteably, skopt does not perform gradient-based optimization, and instead uses computationally cheap surrogate models to\n", "approximate the expensive function. In this example we minimize a simple objective to briefly demonstrate the usage of Skopt with Ray Tune via `SkOptSearch`. It's useful to keep in mind that despite the emphasis on machine learning experiments, Ray Tune optimizes any implicit or explicit objective. Here we assume `scikit-opitmize==0.8.1` library is installed. To learn more, please refer to the [Scikit-Optimize website](https://scikit-optimize.github.io).\n" ] }, { "cell_type": "code", "execution_count": 1, "id": "7d3b7ff9", "metadata": { "tags": [ "remove-cell" ] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Collecting scikit-optimize==0.8.1\n", " Using cached scikit_optimize-0.8.1-py2.py3-none-any.whl (101 kB)\n", "Requirement already satisfied: joblib>=0.11 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from scikit-optimize==0.8.1) (1.1.0)\n", "Collecting pyaml>=16.9\n", " Downloading pyaml-21.10.1-py2.py3-none-any.whl (24 kB)\n", "Requirement already satisfied: scikit-learn>=0.20.0 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from scikit-optimize==0.8.1) (0.24.2)\n", "Requirement already satisfied: numpy>=1.13.3 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from scikit-optimize==0.8.1) (1.21.6)\n", "Requirement already satisfied: scipy>=0.19.1 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from scikit-optimize==0.8.1) (1.4.1)\n", "Requirement already satisfied: PyYAML in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from pyaml>=16.9->scikit-optimize==0.8.1) (5.4.1)\n", "Requirement already satisfied: threadpoolctl>=2.0.0 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from scikit-learn>=0.20.0->scikit-optimize==0.8.1) (3.0.0)\n", "Installing collected packages: pyaml, scikit-optimize\n", "Successfully installed pyaml-21.10.1 scikit-optimize-0.8.1\n", "\u001b[33mWARNING: There was an error checking the latest version of pip.\u001b[0m\u001b[33m\n", "\u001b[0m\u001b[31mERROR: Could not find a version that satisfies the requirement sklearn==0.18.2 (from versions: 0.0)\u001b[0m\u001b[31m\n", "\u001b[0m\u001b[31mERROR: No matching distribution found for sklearn==0.18.2\u001b[0m\u001b[31m\n", "\u001b[0m\u001b[33mWARNING: There was an error checking the latest version of pip.\u001b[0m\u001b[33m\n", "\u001b[0m" ] } ], "source": [ "# !pip install ray[tune]\n", "!pip install scikit-optimize==0.8.1\n", "!pip install sklearn==0.18.2" ] }, { "cell_type": "markdown", "id": "a730341c", "metadata": {}, "source": [ "Click below to see all the imports we need for this example.\n", "You can also launch directly into a Binder instance to run this notebook yourself.\n", "Just click on the rocket symbol at the top of the navigation." ] }, { "cell_type": "code", "execution_count": 2, "id": "1591bb7e", "metadata": { "tags": [ "hide-input" ] }, "outputs": [], "source": [ "import time\n", "from typing import Dict, Optional, Any\n", "\n", "import ray\n", "\n", "import skopt\n", "from ray import tune\n", "from ray.air import session\n", "from ray.tune.search import ConcurrencyLimiter\n", "from ray.tune.search.skopt import SkOptSearch" ] }, { "cell_type": "code", "execution_count": 3, "id": "f538eecb", "metadata": { "tags": [ "remove-cell" ] }, "outputs": [ { "data": { "text/html": [ "
\n", "
\n", "

Ray

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", "\n", "
Python version:3.7.7
Ray version: 3.0.0.dev0
Dashboard:http://127.0.0.1:8268
\n", "
\n", "
\n" ], "text/plain": [ "RayContext(dashboard_url='127.0.0.1:8268', python_version='3.7.7', ray_version='3.0.0.dev0', ray_commit='{{RAY_COMMIT_SHA}}', address_info={'node_ip_address': '127.0.0.1', 'raylet_ip_address': '127.0.0.1', 'redis_address': None, 'object_store_address': '/tmp/ray/session_2022-07-22_15-46-17_712380_48353/sockets/plasma_store', 'raylet_socket_name': '/tmp/ray/session_2022-07-22_15-46-17_712380_48353/sockets/raylet', 'webui_url': '127.0.0.1:8268', 'session_dir': '/tmp/ray/session_2022-07-22_15-46-17_712380_48353', 'metrics_export_port': 64916, 'gcs_address': '127.0.0.1:64371', 'address': '127.0.0.1:64371', 'dashboard_agent_listen_port': 52365, 'node_id': 'bf206482b99028fd8c8cfaea9d94a1bd982869a6c7d6f7531ac6bcb8'})" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "ray.init(configure_logging=False)" ] }, { "cell_type": "markdown", "id": "c18346e5", "metadata": {}, "source": [ "Let's start by defining a simple evaluation function. Again, an explicit math formula is queried here for demonstration, yet in practice this is typically a black-box function-- e.g. the performance results after training an ML model. We artificially sleep for a bit (`0.1` seconds) to simulate a long-running ML experiment. This setup assumes that we're running multiple `step`s of an experiment while tuning three hyperparameters, namely `width`, `height`, and `activation`." ] }, { "cell_type": "code", "execution_count": 4, "id": "3cb44451", "metadata": {}, "outputs": [], "source": [ "def evaluate(step, width, height, activation):\n", " time.sleep(0.1)\n", " activation_boost = 10 if activation==\"relu\" else 0\n", " return (0.1 + width * step / 100) ** (-1) + height * 0.1 + activation_boost" ] }, { "cell_type": "markdown", "id": "9ac14837", "metadata": {}, "source": [ "Next, our `objective` function to be optimized takes a Tune `config`, evaluates the `score` of your experiment in a training loop,\n", "and uses `session.report` to report the `score` back to Tune." ] }, { "cell_type": "code", "execution_count": 5, "id": "f4b27c17", "metadata": {}, "outputs": [], "source": [ "def objective(config):\n", " for step in range(config[\"steps\"]):\n", " score = evaluate(step, config[\"width\"], config[\"height\"], config[\"activation\"])\n", " session.report({\"iterations\": step, \"mean_loss\": score})" ] }, { "cell_type": "markdown", "id": "2b48209c", "metadata": {}, "source": [ "Next we define a search space. The critical assumption is that the optimal hyperparamters live within this space. Yet, if the space is very large, then those hyperparameters may be difficult to find in a short amount of time." ] }, { "cell_type": "code", "execution_count": 6, "id": "e1cf2e16", "metadata": {}, "outputs": [], "source": [ "search_space = {\n", " \"steps\": 100,\n", " \"width\": tune.uniform(0, 20),\n", " \"height\": tune.uniform(-100, 100),\n", " \"activation\": tune.choice([\"relu\", \"tanh\"]),\n", "}" ] }, { "cell_type": "markdown", "id": "2892b243", "metadata": {}, "source": [ "The search algorithm is instantiated from the `SkOptSearch` class. We also constrain the the number of concurrent trials to `4` with a `ConcurrencyLimiter`." ] }, { "cell_type": "code", "execution_count": 7, "id": "5b560697", "metadata": {}, "outputs": [], "source": [ "algo = SkOptSearch()\n", "algo = ConcurrencyLimiter(algo, max_concurrent=4)\n" ] }, { "cell_type": "markdown", "id": "573a6c97", "metadata": {}, "source": [ "The number of samples is the number of hyperparameter combinations that will be tried out. This Tune run is set to `1000` samples.\n", "(you can decrease this if it takes too long on your machine)." ] }, { "cell_type": "code", "execution_count": 8, "id": "bebc40db", "metadata": {}, "outputs": [], "source": [ "num_samples = 1000" ] }, { "cell_type": "code", "execution_count": 9, "id": "a86c5a19", "metadata": { "tags": [ "remove-cell" ] }, "outputs": [], "source": [ "# We override here for our smoke tests.\n", "num_samples = 10" ] }, { "cell_type": "markdown", "id": "35c5823b", "metadata": {}, "source": [ "Finally, we run the experiment to `\"min\"`imize the \"mean_loss\" of the `objective` by searching `search_config` via `algo`, `num_samples` times. This previous sentence is fully characterizes the search problem we aim to solve. With this in mind, notice how efficient it is to execute `tuner.fit()`." ] }, { "cell_type": "code", "execution_count": 10, "id": "c86c6919", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Function checkpointing is disabled. This may result in unexpected behavior when using checkpointing features or certain schedulers. To enable, set the train function arguments to be `func(config, checkpoint_dir=None)`.\n" ] }, { "data": { "text/html": [ "== Status ==
Current time: 2022-07-22 15:47:07 (running for 00:00:44.11)
Memory usage on this node: 9.9/16.0 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/5.42 GiB heap, 0.0/2.0 GiB objects
Current best trial: 20e93604 with mean_loss=-8.869507755259557 and parameters={'steps': 100, 'width': 8.246480084328711, 'height': -89.90514307848873, 'activation': 'tanh'}
Result logdir: /Users/kai/ray_results/objective_2022-07-22_15-46-22
Number of trials: 10/10 (10 TERMINATED)
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
Trial name status loc activation height width loss iter total time (s) iterations neg_mean_loss
objective_0ec1ec00TERMINATED127.0.0.1:48435tanh 10.9976 8.60092 1.21584 100 10.941 99 -1.21584
objective_107c17faTERMINATED127.0.0.1:48446relu 49.545511.8085 15.0394 100 11.5877 99 -15.0394
objective_107d75f0TERMINATED127.0.0.1:48447tanh -7.6131 3.10991-0.446727 100 11.7109 99 0.446727
objective_107ef722TERMINATED127.0.0.1:48448relu -75.8366 5.30975 2.60302 100 11.6501 99 -2.60302
objective_17033b12TERMINATED127.0.0.1:48470tanh -89.189112.1463 -8.83643 100 10.7255 99 8.83643
objective_18fbb20aTERMINATED127.0.0.1:48475tanh 33.5568 9.75437 3.45818 100 10.7563 99 -3.45818
objective_190ab4d0TERMINATED127.0.0.1:48478relu -65.148 4.19546 3.7203 100 10.7123 99 -3.7203
objective_191ca3deTERMINATED127.0.0.1:48481tanh -13.4892 6.74078-1.20128 100 13.219 99 1.20128
objective_1eec012eTERMINATED127.0.0.1:48494tanh -50.281914.8287 -4.96053 100 13.2232 99 4.96053
objective_20e93604TERMINATED127.0.0.1:48502tanh -89.9051 8.24648-8.86951 100 10.7925 99 8.86951


" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "Result for objective_0ec1ec00:\n", " date: 2022-07-22_15-46-26\n", " done: false\n", " experiment_id: c3e1e5ac1d6c4d67878bf793d020f99e\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 11.099760178357887\n", " neg_mean_loss: -11.099760178357887\n", " node_ip: 127.0.0.1\n", " pid: 48435\n", " time_since_restore: 0.10384106636047363\n", " time_this_iter_s: 0.10384106636047363\n", " time_total_s: 0.10384106636047363\n", " timestamp: 1658501186\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 0ec1ec00\n", " warmup_time: 0.003099679946899414\n", " \n", "Result for objective_107c17fa:\n", " date: 2022-07-22_15-46-29\n", " done: false\n", " experiment_id: f2a0e8ad4ea64c489424406b75b5c051\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 24.95455186350225\n", " neg_mean_loss: -24.95455186350225\n", " node_ip: 127.0.0.1\n", " pid: 48446\n", " time_since_restore: 0.10309791564941406\n", " time_this_iter_s: 0.10309791564941406\n", " time_total_s: 0.10309791564941406\n", " timestamp: 1658501189\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 107c17fa\n", " warmup_time: 0.0029959678649902344\n", " \n", "Result for objective_107ef722:\n", " date: 2022-07-22_15-46-29\n", " done: false\n", " experiment_id: 286beb82d0f0493dacbafb0ceb9bfe2d\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 12.416336029599446\n", " neg_mean_loss: -12.416336029599446\n", " node_ip: 127.0.0.1\n", " pid: 48448\n", " time_since_restore: 0.10424971580505371\n", " time_this_iter_s: 0.10424971580505371\n", " time_total_s: 0.10424971580505371\n", " timestamp: 1658501189\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 107ef722\n", " warmup_time: 0.0032057762145996094\n", " \n", "Result for objective_107d75f0:\n", " date: 2022-07-22_15-46-29\n", " done: false\n", " experiment_id: a44e7e5deb2e4bc68b31ca0466db50f5\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 9.238689666819806\n", " neg_mean_loss: -9.238689666819806\n", " node_ip: 127.0.0.1\n", " pid: 48447\n", " time_since_restore: 0.10475015640258789\n", " time_this_iter_s: 0.10475015640258789\n", " time_total_s: 0.10475015640258789\n", " timestamp: 1658501189\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 107d75f0\n", " warmup_time: 0.0029137134552001953\n", " \n", "Result for objective_0ec1ec00:\n", " date: 2022-07-22_15-46-31\n", " done: false\n", " experiment_id: c3e1e5ac1d6c4d67878bf793d020f99e\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 45\n", " iterations_since_restore: 46\n", " mean_loss: 1.3516231012646176\n", " neg_mean_loss: -1.3516231012646176\n", " node_ip: 127.0.0.1\n", " pid: 48435\n", " time_since_restore: 5.127159833908081\n", " time_this_iter_s: 0.10582566261291504\n", " time_total_s: 5.127159833908081\n", " timestamp: 1658501191\n", " timesteps_since_restore: 0\n", " training_iteration: 46\n", " trial_id: 0ec1ec00\n", " warmup_time: 0.003099679946899414\n", " \n", "Result for objective_107c17fa:\n", " date: 2022-07-22_15-46-34\n", " done: false\n", " experiment_id: f2a0e8ad4ea64c489424406b75b5c051\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: 15.131543197018463\n", " neg_mean_loss: -15.131543197018463\n", " node_ip: 127.0.0.1\n", " pid: 48446\n", " time_since_restore: 5.161302089691162\n", " time_this_iter_s: 0.10862898826599121\n", " time_total_s: 5.161302089691162\n", " timestamp: 1658501194\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: 107c17fa\n", " warmup_time: 0.0029959678649902344\n", " \n", "Result for objective_107ef722:\n", " date: 2022-07-22_15-46-34\n", " done: false\n", " experiment_id: 286beb82d0f0493dacbafb0ceb9bfe2d\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: 2.8016060611446587\n", " neg_mean_loss: -2.8016060611446587\n", " node_ip: 127.0.0.1\n", " pid: 48448\n", " time_since_restore: 5.157445907592773\n", " time_this_iter_s: 0.10624003410339355\n", " time_total_s: 5.157445907592773\n", " timestamp: 1658501194\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: 107ef722\n", " warmup_time: 0.0032057762145996094\n", " \n", "Result for objective_107d75f0:\n", " date: 2022-07-22_15-46-34\n", " done: false\n", " experiment_id: a44e7e5deb2e4bc68b31ca0466db50f5\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: -0.12096483365591593\n", " neg_mean_loss: 0.12096483365591593\n", " node_ip: 127.0.0.1\n", " pid: 48447\n", " time_since_restore: 5.159098863601685\n", " time_this_iter_s: 0.10865378379821777\n", " time_total_s: 5.159098863601685\n", " timestamp: 1658501194\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: 107d75f0\n", " warmup_time: 0.0029137134552001953\n", " \n", "Result for objective_0ec1ec00:\n", " date: 2022-07-22_15-46-36\n", " done: false\n", " experiment_id: c3e1e5ac1d6c4d67878bf793d020f99e\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 92\n", " iterations_since_restore: 93\n", " mean_loss: 1.2245597846713034\n", " neg_mean_loss: -1.2245597846713034\n", " node_ip: 127.0.0.1\n", " pid: 48435\n", " time_since_restore: 10.187216997146606\n", " time_this_iter_s: 0.10774493217468262\n", " time_total_s: 10.187216997146606\n", " timestamp: 1658501196\n", " timesteps_since_restore: 0\n", " training_iteration: 93\n", " trial_id: 0ec1ec00\n", " warmup_time: 0.003099679946899414\n", " \n", "Result for objective_0ec1ec00:\n", " date: 2022-07-22_15-46-37\n", " done: true\n", " experiment_id: c3e1e5ac1d6c4d67878bf793d020f99e\n", " experiment_tag: 1_activation=tanh,height=10.9976,steps=100,width=8.6009\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 1.2158379996438775\n", " neg_mean_loss: -1.2158379996438775\n", " node_ip: 127.0.0.1\n", " pid: 48435\n", " time_since_restore: 10.9409658908844\n", " time_this_iter_s: 0.10762286186218262\n", " time_total_s: 10.9409658908844\n", " timestamp: 1658501197\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 0ec1ec00\n", " warmup_time: 0.003099679946899414\n", " \n", "Result for objective_107ef722:\n", " date: 2022-07-22_15-46-39\n", " done: false\n", " experiment_id: 286beb82d0f0493dacbafb0ceb9bfe2d\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 91\n", " iterations_since_restore: 92\n", " mean_loss: 2.619098804697928\n", " neg_mean_loss: -2.619098804697928\n", " node_ip: 127.0.0.1\n", " pid: 48448\n", " time_since_restore: 9.851147890090942\n", " time_this_iter_s: 0.1064901351928711\n", " time_total_s: 9.851147890090942\n", " timestamp: 1658501199\n", " timesteps_since_restore: 0\n", " training_iteration: 92\n", " trial_id: 107ef722\n", " warmup_time: 0.0032057762145996094\n", " \n", "Result for objective_107c17fa:\n", " date: 2022-07-22_15-46-39\n", " done: false\n", " experiment_id: f2a0e8ad4ea64c489424406b75b5c051\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 92\n", " iterations_since_restore: 93\n", " mean_loss: 15.0457609631331\n", " neg_mean_loss: -15.0457609631331\n", " node_ip: 127.0.0.1\n", " pid: 48446\n", " time_since_restore: 9.976460695266724\n", " time_this_iter_s: 0.10694289207458496\n", " time_total_s: 9.976460695266724\n", " timestamp: 1658501199\n", " timesteps_since_restore: 0\n", " training_iteration: 93\n", " trial_id: 107c17fa\n", " warmup_time: 0.0029959678649902344\n", " \n", "Result for objective_107d75f0:\n", " date: 2022-07-22_15-46-39\n", " done: false\n", " experiment_id: a44e7e5deb2e4bc68b31ca0466db50f5\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 90\n", " iterations_since_restore: 91\n", " mean_loss: -0.41635399078466045\n", " neg_mean_loss: 0.41635399078466045\n", " node_ip: 127.0.0.1\n", " pid: 48447\n", " time_since_restore: 9.776082992553711\n", " time_this_iter_s: 0.10715007781982422\n", " time_total_s: 9.776082992553711\n", " timestamp: 1658501199\n", " timesteps_since_restore: 0\n", " training_iteration: 91\n", " trial_id: 107d75f0\n", " warmup_time: 0.0029137134552001953\n", " \n", "Result for objective_17033b12:\n", " date: 2022-07-22_15-46-40\n", " done: false\n", " experiment_id: b5f76922e6424055b6aa255677e2f4b8\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 1.0810921171053653\n", " neg_mean_loss: -1.0810921171053653\n", " node_ip: 127.0.0.1\n", " pid: 48470\n", " time_since_restore: 0.10409784317016602\n", " time_this_iter_s: 0.10409784317016602\n", " time_total_s: 0.10409784317016602\n", " timestamp: 1658501200\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 17033b12\n", " warmup_time: 0.0029649734497070312\n", " \n", "Result for objective_107c17fa:\n", " date: 2022-07-22_15-46-41\n", " done: true\n", " experiment_id: f2a0e8ad4ea64c489424406b75b5c051\n", " experiment_tag: 2_activation=relu,height=49.5455,steps=100,width=11.8085\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 15.039366532944502\n", " neg_mean_loss: -15.039366532944502\n", " node_ip: 127.0.0.1\n", " pid: 48446\n", " time_since_restore: 11.587661743164062\n", " time_this_iter_s: 0.10614180564880371\n", " time_total_s: 11.587661743164062\n", " timestamp: 1658501201\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 107c17fa\n", " warmup_time: 0.0029959678649902344\n", " \n", "Result for objective_107ef722:\n", " date: 2022-07-22_15-46-41\n", " done: true\n", " experiment_id: 286beb82d0f0493dacbafb0ceb9bfe2d\n", " experiment_tag: 4_activation=relu,height=-75.8366,steps=100,width=5.3097\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 2.603019812948376\n", " neg_mean_loss: -2.603019812948376\n", " node_ip: 127.0.0.1\n", " pid: 48448\n", " time_since_restore: 11.65014386177063\n", " time_this_iter_s: 0.10755181312561035\n", " time_total_s: 11.65014386177063\n", " timestamp: 1658501201\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 107ef722\n", " warmup_time: 0.0032057762145996094\n", " \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Result for objective_107d75f0:\n", " date: 2022-07-22_15-46-41\n", " done: true\n", " experiment_id: a44e7e5deb2e4bc68b31ca0466db50f5\n", " experiment_tag: 3_activation=tanh,height=-7.6131,steps=100,width=3.1099\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: -0.44672713582845847\n", " neg_mean_loss: 0.44672713582845847\n", " node_ip: 127.0.0.1\n", " pid: 48447\n", " time_since_restore: 11.710935115814209\n", " time_this_iter_s: 0.1116342544555664\n", " time_total_s: 11.710935115814209\n", " timestamp: 1658501201\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 107d75f0\n", " warmup_time: 0.0029137134552001953\n", " \n", "Result for objective_18fbb20a:\n", " date: 2022-07-22_15-46-43\n", " done: false\n", " experiment_id: 5e399bc125a548739dd433f4153019fe\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 13.355684066332913\n", " neg_mean_loss: -13.355684066332913\n", " node_ip: 127.0.0.1\n", " pid: 48475\n", " time_since_restore: 0.10402607917785645\n", " time_this_iter_s: 0.10402607917785645\n", " time_total_s: 0.10402607917785645\n", " timestamp: 1658501203\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 18fbb20a\n", " warmup_time: 0.003000974655151367\n", " \n", "Result for objective_190ab4d0:\n", " date: 2022-07-22_15-46-43\n", " done: false\n", " experiment_id: 1b007a40cf6d425085379a4314739912\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 13.48520025532623\n", " neg_mean_loss: -13.48520025532623\n", " node_ip: 127.0.0.1\n", " pid: 48478\n", " time_since_restore: 0.10344505310058594\n", " time_this_iter_s: 0.10344505310058594\n", " time_total_s: 0.10344505310058594\n", " timestamp: 1658501203\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 190ab4d0\n", " warmup_time: 0.0028760433197021484\n", " \n", "Result for objective_191ca3de:\n", " date: 2022-07-22_15-46-43\n", " done: false\n", " experiment_id: e3e72bbeb8ae46f0a4f1624db62bc730\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 8.651082223836086\n", " neg_mean_loss: -8.651082223836086\n", " node_ip: 127.0.0.1\n", " pid: 48481\n", " time_since_restore: 0.1036369800567627\n", " time_this_iter_s: 0.1036369800567627\n", " time_total_s: 0.1036369800567627\n", " timestamp: 1658501203\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 191ca3de\n", " warmup_time: 0.0029621124267578125\n", " \n", "Result for objective_17033b12:\n", " date: 2022-07-22_15-46-45\n", " done: false\n", " experiment_id: b5f76922e6424055b6aa255677e2f4b8\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: -8.746754140047871\n", " neg_mean_loss: 8.746754140047871\n", " node_ip: 127.0.0.1\n", " pid: 48470\n", " time_since_restore: 5.144078969955444\n", " time_this_iter_s: 0.10620617866516113\n", " time_total_s: 5.144078969955444\n", " timestamp: 1658501205\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: 17033b12\n", " warmup_time: 0.0029649734497070312\n", " \n", "Result for objective_18fbb20a:\n", " date: 2022-07-22_15-46-48\n", " done: false\n", " experiment_id: 5e399bc125a548739dd433f4153019fe\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: 3.5691516291476226\n", " neg_mean_loss: -3.5691516291476226\n", " node_ip: 127.0.0.1\n", " pid: 48475\n", " time_since_restore: 5.178721189498901\n", " time_this_iter_s: 0.10647201538085938\n", " time_total_s: 5.178721189498901\n", " timestamp: 1658501208\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: 18fbb20a\n", " warmup_time: 0.003000974655151367\n", " \n", "Result for objective_190ab4d0:\n", " date: 2022-07-22_15-46-48\n", " done: false\n", " experiment_id: 1b007a40cf6d425085379a4314739912\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: 3.9678564240341583\n", " neg_mean_loss: -3.9678564240341583\n", " node_ip: 127.0.0.1\n", " pid: 48478\n", " time_since_restore: 5.156033754348755\n", " time_this_iter_s: 0.10652899742126465\n", " time_total_s: 5.156033754348755\n", " timestamp: 1658501208\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: 190ab4d0\n", " warmup_time: 0.0028760433197021484\n", " \n", "Result for objective_191ca3de:\n", " date: 2022-07-22_15-46-48\n", " done: false\n", " experiment_id: e3e72bbeb8ae46f0a4f1624db62bc730\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: -1.04293599865772\n", " neg_mean_loss: 1.04293599865772\n", " node_ip: 127.0.0.1\n", " pid: 48481\n", " time_since_restore: 5.1839048862457275\n", " time_this_iter_s: 0.10760998725891113\n", " time_total_s: 5.1839048862457275\n", " timestamp: 1658501208\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: 191ca3de\n", " warmup_time: 0.0029621124267578125\n", " \n", "Result for objective_17033b12:\n", " date: 2022-07-22_15-46-50\n", " done: false\n", " experiment_id: b5f76922e6424055b6aa255677e2f4b8\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " mean_loss: -8.832083655687075\n", " neg_mean_loss: 8.832083655687075\n", " node_ip: 127.0.0.1\n", " pid: 48470\n", " time_since_restore: 10.193100929260254\n", " time_this_iter_s: 0.10694503784179688\n", " time_total_s: 10.193100929260254\n", " timestamp: 1658501210\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: 17033b12\n", " warmup_time: 0.0029649734497070312\n", " \n", "Result for objective_17033b12:\n", " date: 2022-07-22_15-46-50\n", " done: true\n", " experiment_id: b5f76922e6424055b6aa255677e2f4b8\n", " experiment_tag: 5_activation=tanh,height=-89.1891,steps=100,width=12.1463\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: -8.836432551723352\n", " neg_mean_loss: 8.836432551723352\n", " node_ip: 127.0.0.1\n", " pid: 48470\n", " time_since_restore: 10.725535869598389\n", " time_this_iter_s: 0.10705089569091797\n", " time_total_s: 10.725535869598389\n", " timestamp: 1658501210\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 17033b12\n", " warmup_time: 0.0029649734497070312\n", " \n", "Result for objective_1eec012e:\n", " date: 2022-07-22_15-46-53\n", " done: false\n", " experiment_id: a3cab04eb23543ddbf9b1daa9e915851\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 4.9718091604083385\n", " neg_mean_loss: -4.9718091604083385\n", " node_ip: 127.0.0.1\n", " pid: 48494\n", " time_since_restore: 0.10102462768554688\n", " time_this_iter_s: 0.10102462768554688\n", " time_total_s: 0.10102462768554688\n", " timestamp: 1658501213\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 1eec012e\n", " warmup_time: 0.002748250961303711\n", " \n", "Result for objective_18fbb20a:\n", " date: 2022-07-22_15-46-53\n", " done: false\n", " experiment_id: 5e399bc125a548739dd433f4153019fe\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " mean_loss: 3.4635693481489094\n", " neg_mean_loss: -3.4635693481489094\n", " node_ip: 127.0.0.1\n", " pid: 48475\n", " time_since_restore: 10.22235107421875\n", " time_this_iter_s: 0.10835385322570801\n", " time_total_s: 10.22235107421875\n", " timestamp: 1658501213\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: 18fbb20a\n", " warmup_time: 0.003000974655151367\n", " \n", "Result for objective_190ab4d0:\n", " date: 2022-07-22_15-46-53\n", " done: false\n", " experiment_id: 1b007a40cf6d425085379a4314739912\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " mean_loss: 3.732496287450277\n", " neg_mean_loss: -3.732496287450277\n", " node_ip: 127.0.0.1\n", " pid: 48478\n", " time_since_restore: 10.17775583267212\n", " time_this_iter_s: 0.10712075233459473\n", " time_total_s: 10.17775583267212\n", " timestamp: 1658501213\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: 190ab4d0\n", " warmup_time: 0.0028760433197021484\n", " \n", "Result for objective_191ca3de:\n", " date: 2022-07-22_15-46-54\n", " done: false\n", " experiment_id: e3e72bbeb8ae46f0a4f1624db62bc730\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " mean_loss: -1.193549900472242\n", " neg_mean_loss: 1.193549900472242\n", " node_ip: 127.0.0.1\n", " pid: 48481\n", " time_since_restore: 10.230173826217651\n", " time_this_iter_s: 0.10641312599182129\n", " time_total_s: 10.230173826217651\n", " timestamp: 1658501214\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: 191ca3de\n", " warmup_time: 0.0029621124267578125\n", " \n", "Result for objective_18fbb20a:\n", " date: 2022-07-22_15-46-54\n", " done: true\n", " experiment_id: 5e399bc125a548739dd433f4153019fe\n", " experiment_tag: 6_activation=tanh,height=33.5568,steps=100,width=9.7544\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 3.4581764420917516\n", " neg_mean_loss: -3.4581764420917516\n", " node_ip: 127.0.0.1\n", " pid: 48475\n", " time_since_restore: 10.756349325180054\n", " time_this_iter_s: 0.10465312004089355\n", " time_total_s: 10.756349325180054\n", " timestamp: 1658501214\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 18fbb20a\n", " warmup_time: 0.003000974655151367\n", " \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Result for objective_190ab4d0:\n", " date: 2022-07-22_15-46-54\n", " done: true\n", " experiment_id: 1b007a40cf6d425085379a4314739912\n", " experiment_tag: 7_activation=relu,height=-65.1480,steps=100,width=4.1955\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 3.7203002216299694\n", " neg_mean_loss: -3.7203002216299694\n", " node_ip: 127.0.0.1\n", " pid: 48478\n", " time_since_restore: 10.712332725524902\n", " time_this_iter_s: 0.10630083084106445\n", " time_total_s: 10.712332725524902\n", " timestamp: 1658501214\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 190ab4d0\n", " warmup_time: 0.0028760433197021484\n", " \n", "Result for objective_20e93604:\n", " date: 2022-07-22_15-46-56\n", " done: false\n", " experiment_id: e2c12d5518de42acb5f8413171242ef8\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 1.0094856921511255\n", " neg_mean_loss: -1.0094856921511255\n", " node_ip: 127.0.0.1\n", " pid: 48502\n", " time_since_restore: 0.10361671447753906\n", " time_this_iter_s: 0.10361671447753906\n", " time_total_s: 0.10361671447753906\n", " timestamp: 1658501216\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 20e93604\n", " warmup_time: 0.0035440921783447266\n", " \n", "Result for objective_191ca3de:\n", " date: 2022-07-22_15-46-57\n", " done: true\n", " experiment_id: e3e72bbeb8ae46f0a4f1624db62bc730\n", " experiment_tag: 8_activation=tanh,height=-13.4892,steps=100,width=6.7408\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: -1.2012809142667369\n", " neg_mean_loss: 1.2012809142667369\n", " node_ip: 127.0.0.1\n", " pid: 48481\n", " time_since_restore: 13.218971729278564\n", " time_this_iter_s: 2.560670852661133\n", " time_total_s: 13.218971729278564\n", " timestamp: 1658501217\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 191ca3de\n", " warmup_time: 0.0029621124267578125\n", " \n", "Result for objective_1eec012e:\n", " date: 2022-07-22_15-46-58\n", " done: false\n", " experiment_id: a3cab04eb23543ddbf9b1daa9e915851\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 24\n", " iterations_since_restore: 25\n", " mean_loss: -4.754884423931921\n", " neg_mean_loss: 4.754884423931921\n", " node_ip: 127.0.0.1\n", " pid: 48494\n", " time_since_restore: 5.151007890701294\n", " time_this_iter_s: 0.1075141429901123\n", " time_total_s: 5.151007890701294\n", " timestamp: 1658501218\n", " timesteps_since_restore: 0\n", " training_iteration: 25\n", " trial_id: 1eec012e\n", " warmup_time: 0.002748250961303711\n", " \n", "Result for objective_20e93604:\n", " date: 2022-07-22_15-47-02\n", " done: false\n", " experiment_id: e2c12d5518de42acb5f8413171242ef8\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: -8.738995488800374\n", " neg_mean_loss: 8.738995488800374\n", " node_ip: 127.0.0.1\n", " pid: 48502\n", " time_since_restore: 5.183563709259033\n", " time_this_iter_s: 0.10645675659179688\n", " time_total_s: 5.183563709259033\n", " timestamp: 1658501222\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: 20e93604\n", " warmup_time: 0.0035440921783447266\n", " \n", "Result for objective_1eec012e:\n", " date: 2022-07-22_15-47-03\n", " done: false\n", " experiment_id: a3cab04eb23543ddbf9b1daa9e915851\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 71\n", " iterations_since_restore: 72\n", " mean_loss: -4.9341033303033965\n", " neg_mean_loss: 4.9341033303033965\n", " node_ip: 127.0.0.1\n", " pid: 48494\n", " time_since_restore: 10.200856685638428\n", " time_this_iter_s: 0.10717487335205078\n", " time_total_s: 10.200856685638428\n", " timestamp: 1658501223\n", " timesteps_since_restore: 0\n", " training_iteration: 72\n", " trial_id: 1eec012e\n", " warmup_time: 0.002748250961303711\n", " \n", "Result for objective_1eec012e:\n", " date: 2022-07-22_15-47-06\n", " done: true\n", " experiment_id: a3cab04eb23543ddbf9b1daa9e915851\n", " experiment_tag: 9_activation=tanh,height=-50.2819,steps=100,width=14.8287\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: -4.960533899755889\n", " neg_mean_loss: 4.960533899755889\n", " node_ip: 127.0.0.1\n", " pid: 48494\n", " time_since_restore: 13.22315764427185\n", " time_this_iter_s: 0.10771775245666504\n", " time_total_s: 13.22315764427185\n", " timestamp: 1658501226\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 1eec012e\n", " warmup_time: 0.002748250961303711\n", " \n", "Result for objective_20e93604:\n", " date: 2022-07-22_15-47-07\n", " done: false\n", " experiment_id: e2c12d5518de42acb5f8413171242ef8\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " mean_loss: -8.863153212712554\n", " neg_mean_loss: 8.863153212712554\n", " node_ip: 127.0.0.1\n", " pid: 48502\n", " time_since_restore: 10.251462936401367\n", " time_this_iter_s: 0.1052391529083252\n", " time_total_s: 10.251462936401367\n", " timestamp: 1658501227\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: 20e93604\n", " warmup_time: 0.0035440921783447266\n", " \n", "Result for objective_20e93604:\n", " date: 2022-07-22_15-47-07\n", " done: true\n", " experiment_id: e2c12d5518de42acb5f8413171242ef8\n", " experiment_tag: 10_activation=tanh,height=-89.9051,steps=100,width=8.2465\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: -8.869507755259557\n", " neg_mean_loss: 8.869507755259557\n", " node_ip: 127.0.0.1\n", " pid: 48502\n", " time_since_restore: 10.792518854141235\n", " time_this_iter_s: 0.10802984237670898\n", " time_total_s: 10.792518854141235\n", " timestamp: 1658501227\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 20e93604\n", " warmup_time: 0.0035440921783447266\n", " \n" ] } ], "source": [ "tuner = tune.Tuner(\n", " objective,\n", " tune_config=tune.TuneConfig(\n", " metric=\"mean_loss\",\n", " mode=\"min\",\n", " search_alg=algo,\n", " num_samples=num_samples,\n", " ),\n", " param_space=search_space,\n", ")\n", "results = tuner.fit()" ] }, { "cell_type": "markdown", "id": "5f387c95", "metadata": {}, "source": [ "We now have hyperparameters found to minimize the mean loss." ] }, { "cell_type": "code", "execution_count": 11, "id": "19fff170", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Best hyperparameters found were: {'steps': 100, 'width': 8.246480084328711, 'height': -89.90514307848873, 'activation': 'tanh'}\n" ] } ], "source": [ "print(\"Best hyperparameters found were: \", results.get_best_result().config)" ] }, { "cell_type": "markdown", "id": "7e7f43bb", "metadata": {}, "source": [ "## Providing an initial set of hyperparameters\n", "\n", "While defining the search algorithm, we may choose to provide an initial set of hyperparameters that we believe are especially promising or informative, and\n", "pass this information as a helpful starting point for the `SkOptSearch` object. We also can pass the known rewards for these initial params to save on unnecessary computation." ] }, { "cell_type": "code", "execution_count": 12, "id": "c44a3ad2", "metadata": {}, "outputs": [], "source": [ "initial_params = [\n", " {\"width\": 10, \"height\": 0, \"activation\": \"relu\"},\n", " {\"width\": 15, \"height\": -20, \"activation\": \"tanh\"}\n", "]\n", "known_rewards = [-189, -1144]" ] }, { "cell_type": "markdown", "id": "30e19a4b", "metadata": {}, "source": [ "Now the `search_alg` built using `SkOptSearch` takes `points_to_evaluate`." ] }, { "cell_type": "code", "execution_count": 13, "id": "19308593", "metadata": { "lines_to_next_cell": 0 }, "outputs": [], "source": [ "algo = SkOptSearch(points_to_evaluate=initial_params)\n", "algo = ConcurrencyLimiter(algo, max_concurrent=4)" ] }, { "cell_type": "markdown", "id": "bd022018", "metadata": {}, "source": [ "And again run the experiment, this time with initial hyperparameter evaluations:" ] }, { "cell_type": "code", "execution_count": 15, "id": "d9c7c8d4", "metadata": {}, "outputs": [ { "data": { "text/html": [ "== Status ==
Current time: 2022-07-22 15:49:08 (running for 00:00:43.84)
Memory usage on this node: 10.5/16.0 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/5.42 GiB heap, 0.0/2.0 GiB objects
Current best trial: 60e6657e with mean_loss=-6.564839315448637 and parameters={'steps': 100, 'width': 11.45060059898019, 'height': -66.52281754112352, 'activation': 'tanh'}
Result logdir: /Users/kai/ray_results/objective_2022-07-22_15-48-24
Number of trials: 10/10 (10 TERMINATED)
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
Trial name status loc activation height width loss iter total time (s) iterations neg_mean_loss
objective_56a79fceTERMINATED127.0.0.1:48590relu 0 10 10.1 100 10.9393 99 -10.1
objective_583ee2caTERMINATED127.0.0.1:48605tanh -20 15 -1.93311 100 11.6366 99 1.93311
objective_58405952TERMINATED127.0.0.1:48606relu -87.802519.608 1.271 100 11.752 99 -1.271
objective_5841ebe6TERMINATED127.0.0.1:48607relu 49.51 15.0894 15.0175 100 11.6761 99 -15.0175
objective_5ec4da6eTERMINATED127.0.0.1:48629relu 76.083713.7995 17.681 100 10.7667 99 -17.681
objective_60d7fb56TERMINATED127.0.0.1:48634relu -50.453416.3133 5.0162 100 10.75 99 -5.0162
objective_60e6657eTERMINATED127.0.0.1:48637tanh -66.522811.4506 -6.56484 100 10.759 99 6.56484
objective_60f84e42TERMINATED127.0.0.1:48640tanh 99.5965 9.0442510.0701 100 10.71 99 -10.0701
objective_66bdb9e8TERMINATED127.0.0.1:48653tanh -61.954712.3263 -6.11419 100 10.724 99 6.11419
objective_68c346b8TERMINATED127.0.0.1:48660tanh 23.7916 2.108 2.83643 100 10.7375 99 -2.83643


" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "Result for objective_56a79fce:\n", " date: 2022-07-22_15-48-27\n", " done: false\n", " experiment_id: 7046e554c1db465bbed80331b034b40f\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 20.0\n", " neg_mean_loss: -20.0\n", " node_ip: 127.0.0.1\n", " pid: 48590\n", " time_since_restore: 0.10438704490661621\n", " time_this_iter_s: 0.10438704490661621\n", " time_total_s: 0.10438704490661621\n", " timestamp: 1658501307\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 56a79fce\n", " warmup_time: 0.002905130386352539\n", " \n", "Result for objective_583ee2ca:\n", " date: 2022-07-22_15-48-30\n", " done: false\n", " experiment_id: 59c76129c7074e41ab39b330d898ba03\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 8.0\n", " neg_mean_loss: -8.0\n", " node_ip: 127.0.0.1\n", " pid: 48605\n", " time_since_restore: 0.10483098030090332\n", " time_this_iter_s: 0.10483098030090332\n", " time_total_s: 0.10483098030090332\n", " timestamp: 1658501310\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 583ee2ca\n", " warmup_time: 0.0031731128692626953\n", " \n", "Result for objective_5841ebe6:\n", " date: 2022-07-22_15-48-30\n", " done: false\n", " experiment_id: fde2877ded13416195ab1af070eda91a\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 24.951000429599397\n", " neg_mean_loss: -24.951000429599397\n", " node_ip: 127.0.0.1\n", " pid: 48607\n", " time_since_restore: 0.10440397262573242\n", " time_this_iter_s: 0.10440397262573242\n", " time_total_s: 0.10440397262573242\n", " timestamp: 1658501310\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 5841ebe6\n", " warmup_time: 0.0033330917358398438\n", " \n", "Result for objective_58405952:\n", " date: 2022-07-22_15-48-30\n", " done: false\n", " experiment_id: 5902a2d532b3437e8b94d02919e1604c\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 11.21975075417543\n", " neg_mean_loss: -11.21975075417543\n", " node_ip: 127.0.0.1\n", " pid: 48606\n", " time_since_restore: 0.10400986671447754\n", " time_this_iter_s: 0.10400986671447754\n", " time_total_s: 0.10400986671447754\n", " timestamp: 1658501310\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: '58405952'\n", " warmup_time: 0.0027670860290527344\n", " \n", "Result for objective_56a79fce:\n", " date: 2022-07-22_15-48-32\n", " done: false\n", " experiment_id: 7046e554c1db465bbed80331b034b40f\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 45\n", " iterations_since_restore: 46\n", " mean_loss: 10.217391304347826\n", " neg_mean_loss: -10.217391304347826\n", " node_ip: 127.0.0.1\n", " pid: 48590\n", " time_since_restore: 5.135921239852905\n", " time_this_iter_s: 0.10727715492248535\n", " time_total_s: 5.135921239852905\n", " timestamp: 1658501312\n", " timesteps_since_restore: 0\n", " training_iteration: 46\n", " trial_id: 56a79fce\n", " warmup_time: 0.002905130386352539\n", " \n", "Result for objective_583ee2ca:\n", " date: 2022-07-22_15-48-35\n", " done: false\n", " experiment_id: 59c76129c7074e41ab39b330d898ba03\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: -1.8601398601398602\n", " neg_mean_loss: 1.8601398601398602\n", " node_ip: 127.0.0.1\n", " pid: 48605\n", " time_since_restore: 5.154622793197632\n", " time_this_iter_s: 0.10759091377258301\n", " time_total_s: 5.154622793197632\n", " timestamp: 1658501315\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: 583ee2ca\n", " warmup_time: 0.0031731128692626953\n", " \n", "Result for objective_5841ebe6:\n", " date: 2022-07-22_15-48-35\n", " done: false\n", " experiment_id: fde2877ded13416195ab1af070eda91a\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: 15.090043780907216\n", " neg_mean_loss: -15.090043780907216\n", " node_ip: 127.0.0.1\n", " pid: 48607\n", " time_since_restore: 5.153692960739136\n", " time_this_iter_s: 0.10715007781982422\n", " time_total_s: 5.153692960739136\n", " timestamp: 1658501315\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: 5841ebe6\n", " warmup_time: 0.0033330917358398438\n", " \n", "Result for objective_58405952:\n", " date: 2022-07-22_15-48-35\n", " done: false\n", " experiment_id: 5902a2d532b3437e8b94d02919e1604c\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: 1.3270955262548618\n", " neg_mean_loss: -1.3270955262548618\n", " node_ip: 127.0.0.1\n", " pid: 48606\n", " time_since_restore: 5.145922660827637\n", " time_this_iter_s: 0.10549187660217285\n", " time_total_s: 5.145922660827637\n", " timestamp: 1658501315\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: '58405952'\n", " warmup_time: 0.0027670860290527344\n", " \n", "Result for objective_56a79fce:\n", " date: 2022-07-22_15-48-37\n", " done: false\n", " experiment_id: 7046e554c1db465bbed80331b034b40f\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 92\n", " iterations_since_restore: 93\n", " mean_loss: 10.10752688172043\n", " neg_mean_loss: -10.10752688172043\n", " node_ip: 127.0.0.1\n", " pid: 48590\n", " time_since_restore: 10.195838212966919\n", " time_this_iter_s: 0.10654616355895996\n", " time_total_s: 10.195838212966919\n", " timestamp: 1658501317\n", " timesteps_since_restore: 0\n", " training_iteration: 93\n", " trial_id: 56a79fce\n", " warmup_time: 0.002905130386352539\n", " \n", "Result for objective_56a79fce:\n", " date: 2022-07-22_15-48-38\n", " done: true\n", " experiment_id: 7046e554c1db465bbed80331b034b40f\n", " experiment_tag: 1_activation=relu,height=0,steps=100,width=10\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 10.1\n", " neg_mean_loss: -10.1\n", " node_ip: 127.0.0.1\n", " pid: 48590\n", " time_since_restore: 10.939304113388062\n", " time_this_iter_s: 0.10537099838256836\n", " time_total_s: 10.939304113388062\n", " timestamp: 1658501318\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 56a79fce\n", " warmup_time: 0.002905130386352539\n", " \n", "Result for objective_5841ebe6:\n", " date: 2022-07-22_15-48-39\n", " done: false\n", " experiment_id: fde2877ded13416195ab1af070eda91a\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 90\n", " iterations_since_restore: 91\n", " mean_loss: 15.024097555037425\n", " neg_mean_loss: -15.024097555037425\n", " node_ip: 127.0.0.1\n", " pid: 48607\n", " time_since_restore: 9.817665100097656\n", " time_this_iter_s: 0.10596418380737305\n", " time_total_s: 9.817665100097656\n", " timestamp: 1658501319\n", " timesteps_since_restore: 0\n", " training_iteration: 91\n", " trial_id: 5841ebe6\n", " warmup_time: 0.0033330917358398438\n", " \n", "Result for objective_58405952:\n", " date: 2022-07-22_15-48-39\n", " done: false\n", " experiment_id: 5902a2d532b3437e8b94d02919e1604c\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 89\n", " iterations_since_restore: 90\n", " mean_loss: 1.2767270681910414\n", " neg_mean_loss: -1.2767270681910414\n", " node_ip: 127.0.0.1\n", " pid: 48606\n", " time_since_restore: 9.72554087638855\n", " time_this_iter_s: 0.10558199882507324\n", " time_total_s: 9.72554087638855\n", " timestamp: 1658501319\n", " timesteps_since_restore: 0\n", " training_iteration: 90\n", " trial_id: '58405952'\n", " warmup_time: 0.0027670860290527344\n", " \n", "Result for objective_583ee2ca:\n", " date: 2022-07-22_15-48-39\n", " done: false\n", " experiment_id: 59c76129c7074e41ab39b330d898ba03\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 91\n", " iterations_since_restore: 92\n", " mean_loss: -1.9272727272727272\n", " neg_mean_loss: 1.9272727272727272\n", " node_ip: 127.0.0.1\n", " pid: 48605\n", " time_since_restore: 9.925616025924683\n", " time_this_iter_s: 0.1071469783782959\n", " time_total_s: 9.925616025924683\n", " timestamp: 1658501319\n", " timesteps_since_restore: 0\n", " training_iteration: 92\n", " trial_id: 583ee2ca\n", " warmup_time: 0.0031731128692626953\n", " \n", "Result for objective_5ec4da6e:\n", " date: 2022-07-22_15-48-40\n", " done: false\n", " experiment_id: 24cf29dbb0e44fa7a2dbffe353a56c35\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 27.60836685236658\n", " neg_mean_loss: -27.60836685236658\n", " node_ip: 127.0.0.1\n", " pid: 48629\n", " time_since_restore: 0.10513186454772949\n", " time_this_iter_s: 0.10513186454772949\n", " time_total_s: 0.10513186454772949\n", " timestamp: 1658501320\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 5ec4da6e\n", " warmup_time: 0.002888917922973633\n", " \n", "Result for objective_583ee2ca:\n", " date: 2022-07-22_15-48-41\n", " done: true\n", " experiment_id: 59c76129c7074e41ab39b330d898ba03\n", " experiment_tag: 2_activation=tanh,height=-20,steps=100,width=15\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: -1.9331103678929766\n", " neg_mean_loss: 1.9331103678929766\n", " node_ip: 127.0.0.1\n", " pid: 48605\n", " time_since_restore: 11.636637926101685\n", " time_this_iter_s: 0.10811305046081543\n", " time_total_s: 11.636637926101685\n", " timestamp: 1658501321\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 583ee2ca\n", " warmup_time: 0.0031731128692626953\n", " \n", "Result for objective_5841ebe6:\n", " date: 2022-07-22_15-48-41\n", " done: true\n", " experiment_id: fde2877ded13416195ab1af070eda91a\n", " experiment_tag: 4_activation=relu,height=49.5100,steps=100,width=15.0894\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 15.017496549772488\n", " neg_mean_loss: -15.017496549772488\n", " node_ip: 127.0.0.1\n", " pid: 48607\n", " time_since_restore: 11.676102876663208\n", " time_this_iter_s: 0.1074519157409668\n", " time_total_s: 11.676102876663208\n", " timestamp: 1658501321\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 5841ebe6\n", " warmup_time: 0.0033330917358398438\n", " \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Result for objective_58405952:\n", " date: 2022-07-22_15-48-41\n", " done: true\n", " experiment_id: 5902a2d532b3437e8b94d02919e1604c\n", " experiment_tag: 3_activation=relu,height=-87.8025,steps=100,width=19.6080\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 1.271001380631752\n", " neg_mean_loss: -1.271001380631752\n", " node_ip: 127.0.0.1\n", " pid: 48606\n", " time_since_restore: 11.751991033554077\n", " time_this_iter_s: 0.1101231575012207\n", " time_total_s: 11.751991033554077\n", " timestamp: 1658501321\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: '58405952'\n", " warmup_time: 0.0027670860290527344\n", " \n", "Result for objective_60d7fb56:\n", " date: 2022-07-22_15-48-44\n", " done: false\n", " experiment_id: 26d42c9dcc6e4354b3fa7355fc806522\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 14.954664969255782\n", " neg_mean_loss: -14.954664969255782\n", " node_ip: 127.0.0.1\n", " pid: 48634\n", " time_since_restore: 0.10394525527954102\n", " time_this_iter_s: 0.10394525527954102\n", " time_total_s: 0.10394525527954102\n", " timestamp: 1658501324\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 60d7fb56\n", " warmup_time: 0.003077268600463867\n", " \n", "Result for objective_60e6657e:\n", " date: 2022-07-22_15-48-44\n", " done: false\n", " experiment_id: 3f2014eb8b41451bb277cabed9951d90\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 3.3477182458876484\n", " neg_mean_loss: -3.3477182458876484\n", " node_ip: 127.0.0.1\n", " pid: 48637\n", " time_since_restore: 0.10410618782043457\n", " time_this_iter_s: 0.10410618782043457\n", " time_total_s: 0.10410618782043457\n", " timestamp: 1658501324\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 60e6657e\n", " warmup_time: 0.0029239654541015625\n", " \n", "Result for objective_60f84e42:\n", " date: 2022-07-22_15-48-44\n", " done: false\n", " experiment_id: 481882ef37194e6f995c4294a3453fd2\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 19.959649161689548\n", " neg_mean_loss: -19.959649161689548\n", " node_ip: 127.0.0.1\n", " pid: 48640\n", " time_since_restore: 0.10413289070129395\n", " time_this_iter_s: 0.10413289070129395\n", " time_total_s: 0.10413289070129395\n", " timestamp: 1658501324\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 60f84e42\n", " warmup_time: 0.0028841495513916016\n", " \n", "Result for objective_5ec4da6e:\n", " date: 2022-07-22_15-48-45\n", " done: false\n", " experiment_id: 24cf29dbb0e44fa7a2dbffe353a56c35\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: 17.760209862707853\n", " neg_mean_loss: -17.760209862707853\n", " node_ip: 127.0.0.1\n", " pid: 48629\n", " time_since_restore: 5.151746988296509\n", " time_this_iter_s: 0.10759496688842773\n", " time_total_s: 5.151746988296509\n", " timestamp: 1658501325\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: 5ec4da6e\n", " warmup_time: 0.002888917922973633\n", " \n", "Result for objective_60d7fb56:\n", " date: 2022-07-22_15-48-49\n", " done: false\n", " experiment_id: 26d42c9dcc6e4354b3fa7355fc806522\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: 5.08341058857216\n", " neg_mean_loss: -5.08341058857216\n", " node_ip: 127.0.0.1\n", " pid: 48634\n", " time_since_restore: 5.146520137786865\n", " time_this_iter_s: 0.10820508003234863\n", " time_total_s: 5.146520137786865\n", " timestamp: 1658501329\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: 60d7fb56\n", " warmup_time: 0.003077268600463867\n", " \n", "Result for objective_60e6657e:\n", " date: 2022-07-22_15-48-49\n", " done: false\n", " experiment_id: 3f2014eb8b41451bb277cabed9951d90\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: -6.469859332234883\n", " neg_mean_loss: 6.469859332234883\n", " node_ip: 127.0.0.1\n", " pid: 48637\n", " time_since_restore: 5.153999090194702\n", " time_this_iter_s: 0.10596871376037598\n", " time_total_s: 5.153999090194702\n", " timestamp: 1658501329\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: 60e6657e\n", " warmup_time: 0.0029239654541015625\n", " \n", "Result for objective_60f84e42:\n", " date: 2022-07-22_15-48-49\n", " done: false\n", " experiment_id: 481882ef37194e6f995c4294a3453fd2\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: 10.189492101039557\n", " neg_mean_loss: -10.189492101039557\n", " node_ip: 127.0.0.1\n", " pid: 48640\n", " time_since_restore: 5.137938976287842\n", " time_this_iter_s: 0.10651087760925293\n", " time_total_s: 5.137938976287842\n", " timestamp: 1658501329\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: 60f84e42\n", " warmup_time: 0.0028841495513916016\n", " \n", "Result for objective_5ec4da6e:\n", " date: 2022-07-22_15-48-50\n", " done: false\n", " experiment_id: 24cf29dbb0e44fa7a2dbffe353a56c35\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " mean_loss: 17.684869174683147\n", " neg_mean_loss: -17.684869174683147\n", " node_ip: 127.0.0.1\n", " pid: 48629\n", " time_since_restore: 10.229070901870728\n", " time_this_iter_s: 0.11342406272888184\n", " time_total_s: 10.229070901870728\n", " timestamp: 1658501330\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: 5ec4da6e\n", " warmup_time: 0.002888917922973633\n", " \n", "Result for objective_5ec4da6e:\n", " date: 2022-07-22_15-48-51\n", " done: true\n", " experiment_id: 24cf29dbb0e44fa7a2dbffe353a56c35\n", " experiment_tag: 5_activation=relu,height=76.0837,steps=100,width=13.7995\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 17.6810334976324\n", " neg_mean_loss: -17.6810334976324\n", " node_ip: 127.0.0.1\n", " pid: 48629\n", " time_since_restore: 10.766729831695557\n", " time_this_iter_s: 0.10697484016418457\n", " time_total_s: 10.766729831695557\n", " timestamp: 1658501331\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 5ec4da6e\n", " warmup_time: 0.002888917922973633\n", " \n", "Result for objective_66bdb9e8:\n", " date: 2022-07-22_15-48-54\n", " done: false\n", " experiment_id: fae00fc97db044e9965698562b20039b\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 3.804528837217097\n", " neg_mean_loss: -3.804528837217097\n", " node_ip: 127.0.0.1\n", " pid: 48653\n", " time_since_restore: 0.10318303108215332\n", " time_this_iter_s: 0.10318303108215332\n", " time_total_s: 0.10318303108215332\n", " timestamp: 1658501334\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 66bdb9e8\n", " warmup_time: 0.004212856292724609\n", " \n", "Result for objective_60d7fb56:\n", " date: 2022-07-22_15-48-54\n", " done: false\n", " experiment_id: 26d42c9dcc6e4354b3fa7355fc806522\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " mean_loss: 5.019454849577342\n", " neg_mean_loss: -5.019454849577342\n", " node_ip: 127.0.0.1\n", " pid: 48634\n", " time_since_restore: 10.212354183197021\n", " time_this_iter_s: 0.10782504081726074\n", " time_total_s: 10.212354183197021\n", " timestamp: 1658501334\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: 60d7fb56\n", " warmup_time: 0.003077268600463867\n", " \n", "Result for objective_60e6657e:\n", " date: 2022-07-22_15-48-54\n", " done: false\n", " experiment_id: 3f2014eb8b41451bb277cabed9951d90\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " mean_loss: -6.560230936519563\n", " neg_mean_loss: 6.560230936519563\n", " node_ip: 127.0.0.1\n", " pid: 48637\n", " time_since_restore: 10.219763278961182\n", " time_this_iter_s: 0.10767722129821777\n", " time_total_s: 10.219763278961182\n", " timestamp: 1658501334\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: 60e6657e\n", " warmup_time: 0.0029239654541015625\n", " \n", "Result for objective_60f84e42:\n", " date: 2022-07-22_15-48-54\n", " done: false\n", " experiment_id: 481882ef37194e6f995c4294a3453fd2\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " mean_loss: 10.075906679849322\n", " neg_mean_loss: -10.075906679849322\n", " node_ip: 127.0.0.1\n", " pid: 48640\n", " time_since_restore: 10.16911792755127\n", " time_this_iter_s: 0.10426807403564453\n", " time_total_s: 10.16911792755127\n", " timestamp: 1658501334\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: 60f84e42\n", " warmup_time: 0.0028841495513916016\n", " \n", "Result for objective_60d7fb56:\n", " date: 2022-07-22_15-48-54\n", " done: true\n", " experiment_id: 26d42c9dcc6e4354b3fa7355fc806522\n", " experiment_tag: 6_activation=relu,height=-50.4534,steps=100,width=16.3133\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 5.016202769898167\n", " neg_mean_loss: -5.016202769898167\n", " node_ip: 127.0.0.1\n", " pid: 48634\n", " time_since_restore: 10.75002908706665\n", " time_this_iter_s: 0.10717272758483887\n", " time_total_s: 10.75002908706665\n", " timestamp: 1658501334\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 60d7fb56\n", " warmup_time: 0.003077268600463867\n", " \n", "Result for objective_60e6657e:\n", " date: 2022-07-22_15-48-54\n", " done: true\n", " experiment_id: 3f2014eb8b41451bb277cabed9951d90\n", " experiment_tag: 7_activation=tanh,height=-66.5228,steps=100,width=11.4506\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: -6.564839315448637\n", " neg_mean_loss: 6.564839315448637\n", " node_ip: 127.0.0.1\n", " pid: 48637\n", " time_since_restore: 10.759015321731567\n", " time_this_iter_s: 0.10623526573181152\n", " time_total_s: 10.759015321731567\n", " timestamp: 1658501334\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 60e6657e\n", " warmup_time: 0.0029239654541015625\n", " \n", "Result for objective_60f84e42:\n", " date: 2022-07-22_15-48-55\n", " done: true\n", " experiment_id: 481882ef37194e6f995c4294a3453fd2\n", " experiment_tag: 8_activation=tanh,height=99.5965,steps=100,width=9.0442\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 10.070099940211907\n", " neg_mean_loss: -10.070099940211907\n", " node_ip: 127.0.0.1\n", " pid: 48640\n", " time_since_restore: 10.710038900375366\n", " time_this_iter_s: 0.1064748764038086\n", " time_total_s: 10.710038900375366\n", " timestamp: 1658501335\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 60f84e42\n", " warmup_time: 0.0028841495513916016\n", " \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Result for objective_68c346b8:\n", " date: 2022-07-22_15-48-57\n", " done: false\n", " experiment_id: fe241f21de6a4466bcdf713da6d4cd92\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 12.37916148081603\n", " neg_mean_loss: -12.37916148081603\n", " node_ip: 127.0.0.1\n", " pid: 48660\n", " time_since_restore: 0.10358977317810059\n", " time_this_iter_s: 0.10358977317810059\n", " time_total_s: 0.10358977317810059\n", " timestamp: 1658501337\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 68c346b8\n", " warmup_time: 0.0028688907623291016\n", " \n", "Result for objective_66bdb9e8:\n", " date: 2022-07-22_15-48-59\n", " done: false\n", " experiment_id: fae00fc97db044e9965698562b20039b\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: -6.025789125550826\n", " neg_mean_loss: 6.025789125550826\n", " node_ip: 127.0.0.1\n", " pid: 48653\n", " time_since_restore: 5.1440629959106445\n", " time_this_iter_s: 0.10767483711242676\n", " time_total_s: 5.1440629959106445\n", " timestamp: 1658501339\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: 66bdb9e8\n", " warmup_time: 0.004212856292724609\n", " \n", "Result for objective_68c346b8:\n", " date: 2022-07-22_15-49-02\n", " done: false\n", " experiment_id: fe241f21de6a4466bcdf713da6d4cd92\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: 3.295952835154158\n", " neg_mean_loss: -3.295952835154158\n", " node_ip: 127.0.0.1\n", " pid: 48660\n", " time_since_restore: 5.15015983581543\n", " time_this_iter_s: 0.10423922538757324\n", " time_total_s: 5.15015983581543\n", " timestamp: 1658501342\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: 68c346b8\n", " warmup_time: 0.0028688907623291016\n", " \n", "Result for objective_66bdb9e8:\n", " date: 2022-07-22_15-49-04\n", " done: false\n", " experiment_id: fae00fc97db044e9965698562b20039b\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " mean_loss: -6.109904185213174\n", " neg_mean_loss: 6.109904185213174\n", " node_ip: 127.0.0.1\n", " pid: 48653\n", " time_since_restore: 10.1879301071167\n", " time_this_iter_s: 0.10626506805419922\n", " time_total_s: 10.1879301071167\n", " timestamp: 1658501344\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: 66bdb9e8\n", " warmup_time: 0.004212856292724609\n", " \n", "Result for objective_66bdb9e8:\n", " date: 2022-07-22_15-49-04\n", " done: true\n", " experiment_id: fae00fc97db044e9965698562b20039b\n", " experiment_tag: 9_activation=tanh,height=-61.9547,steps=100,width=12.3263\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: -6.114190623827535\n", " neg_mean_loss: 6.114190623827535\n", " node_ip: 127.0.0.1\n", " pid: 48653\n", " time_since_restore: 10.724038124084473\n", " time_this_iter_s: 0.1088099479675293\n", " time_total_s: 10.724038124084473\n", " timestamp: 1658501344\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 66bdb9e8\n", " warmup_time: 0.004212856292724609\n", " \n", "Result for objective_68c346b8:\n", " date: 2022-07-22_15-49-07\n", " done: false\n", " experiment_id: fe241f21de6a4466bcdf713da6d4cd92\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " mean_loss: 2.859579303290837\n", " neg_mean_loss: -2.859579303290837\n", " node_ip: 127.0.0.1\n", " pid: 48660\n", " time_since_restore: 10.199071884155273\n", " time_this_iter_s: 0.10729789733886719\n", " time_total_s: 10.199071884155273\n", " timestamp: 1658501347\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: 68c346b8\n", " warmup_time: 0.0028688907623291016\n", " \n", "Result for objective_68c346b8:\n", " date: 2022-07-22_15-49-08\n", " done: true\n", " experiment_id: fe241f21de6a4466bcdf713da6d4cd92\n", " experiment_tag: 10_activation=tanh,height=23.7916,steps=100,width=2.1080\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 2.836425260080649\n", " neg_mean_loss: -2.836425260080649\n", " node_ip: 127.0.0.1\n", " pid: 48660\n", " time_since_restore: 10.7375328540802\n", " time_this_iter_s: 0.1081850528717041\n", " time_total_s: 10.7375328540802\n", " timestamp: 1658501348\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 68c346b8\n", " warmup_time: 0.0028688907623291016\n", " \n" ] } ], "source": [ "tuner = tune.Tuner(\n", " objective,\n", " tune_config=tune.TuneConfig(\n", " metric=\"mean_loss\",\n", " mode=\"min\",\n", " search_alg=algo,\n", " num_samples=num_samples,\n", " ),\n", " param_space=search_space,\n", ")\n", "results = tuner.fit()" ] }, { "cell_type": "markdown", "id": "04c9d550", "metadata": {}, "source": [ "And we again show the ideal hyperparameters." ] }, { "cell_type": "code", "execution_count": 16, "id": "d7267095", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Best hyperparameters found were: {'steps': 100, 'width': 11.45060059898019, 'height': -66.52281754112352, 'activation': 'tanh'}\n" ] } ], "source": [ "print(\"Best hyperparameters found were: \", results.get_best_result().config)" ] }, { "cell_type": "code", "execution_count": 17, "id": "23739237", "metadata": { "tags": [ "remove-cell" ] }, "outputs": [], "source": [ "ray.shutdown()" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.7" }, "orphan": true }, "nbformat": 4, "nbformat_minor": 5 }