2022-04-25 11:10:58 -07:00
{
"cells": [
{
"cell_type": "markdown",
"id": "61966d26",
"metadata": {},
"source": [
"# Running Tune experiments with Skopt\n",
"\n",
"In this tutorial we introduce Skopt, while running a simple Ray Tune experiment. Tune’
"\n",
"Scikit-Optimize, or skopt, is a simple and efficient library to optimize expensive and noisy black-box functions, e.g. large-scale ML experiments. It implements several methods for sequential model-based optimization. Noteably, skopt does not perform gradient-based optimization, and instead uses computationally cheap surrogate models to\n",
"approximate the expensive function. In this example we minimize a simple objective to briefly demonstrate the usage of Skopt with Ray Tune via `SkOptSearch`. It's useful to keep in mind that despite the emphasis on machine learning experiments, Ray Tune optimizes any implicit or explicit objective. Here we assume `scikit-opitmize==0.8.1` library is installed. To learn more, please refer to the [Scikit-Optimize website](https://scikit-optimize.github.io).\n"
]
},
{
"cell_type": "code",
2022-07-24 18:53:57 +01:00
"execution_count": 1,
2022-04-25 11:10:58 -07:00
"id": "7d3b7ff9",
"metadata": {
"tags": [
"remove-cell"
]
},
2022-07-24 18:53:57 +01:00
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Collecting scikit-optimize==0.8.1\n",
" Using cached scikit_optimize-0.8.1-py2.py3-none-any.whl (101 kB)\n",
"Requirement already satisfied: joblib>=0.11 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from scikit-optimize==0.8.1) (1.1.0)\n",
"Collecting pyaml>=16.9\n",
" Downloading pyaml-21.10.1-py2.py3-none-any.whl (24 kB)\n",
"Requirement already satisfied: scikit-learn>=0.20.0 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from scikit-optimize==0.8.1) (0.24.2)\n",
"Requirement already satisfied: numpy>=1.13.3 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from scikit-optimize==0.8.1) (1.21.6)\n",
"Requirement already satisfied: scipy>=0.19.1 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from scikit-optimize==0.8.1) (1.4.1)\n",
"Requirement already satisfied: PyYAML in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from pyaml>=16.9->scikit-optimize==0.8.1) (5.4.1)\n",
"Requirement already satisfied: threadpoolctl>=2.0.0 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from scikit-learn>=0.20.0->scikit-optimize==0.8.1) (3.0.0)\n",
"Installing collected packages: pyaml, scikit-optimize\n",
"Successfully installed pyaml-21.10.1 scikit-optimize-0.8.1\n",
"\u001b[33mWARNING: There was an error checking the latest version of pip.\u001b[0m\u001b[33m\n",
"\u001b[0m\u001b[31mERROR: Could not find a version that satisfies the requirement sklearn==0.18.2 (from versions: 0.0)\u001b[0m\u001b[31m\n",
"\u001b[0m\u001b[31mERROR: No matching distribution found for sklearn==0.18.2\u001b[0m\u001b[31m\n",
"\u001b[0m\u001b[33mWARNING: There was an error checking the latest version of pip.\u001b[0m\u001b[33m\n",
"\u001b[0m"
]
}
],
2022-04-25 11:10:58 -07:00
"source": [
"# !pip install ray[tune]\n",
"!pip install scikit-optimize==0.8.1\n",
"!pip install sklearn==0.18.2"
]
},
{
"cell_type": "markdown",
"id": "a730341c",
"metadata": {},
"source": [
"Click below to see all the imports we need for this example.\n",
"You can also launch directly into a Binder instance to run this notebook yourself.\n",
"Just click on the rocket symbol at the top of the navigation."
]
},
{
"cell_type": "code",
2022-07-24 18:53:57 +01:00
"execution_count": 2,
2022-04-25 11:10:58 -07:00
"id": "1591bb7e",
"metadata": {
"tags": [
"hide-input"
]
},
"outputs": [],
"source": [
"import time\n",
"from typing import Dict, Optional, Any\n",
"\n",
"import ray\n",
"\n",
"import skopt\n",
"from ray import tune\n",
2022-06-30 10:37:31 -07:00
"from ray.air import session\n",
2022-06-25 14:55:30 +01:00
"from ray.tune.search import ConcurrencyLimiter\n",
"from ray.tune.search.skopt import SkOptSearch"
2022-04-25 11:10:58 -07:00
]
},
{
"cell_type": "code",
2022-07-24 18:53:57 +01:00
"execution_count": 3,
2022-04-25 11:10:58 -07:00
"id": "f538eecb",
"metadata": {
"tags": [
"remove-cell"
]
},
2022-07-24 18:53:57 +01:00
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
" <div style=\"margin-left: 50px;display: flex;flex-direction: row;align-items: center\">\n",
" <h3 style=\"color: var(--jp-ui-font-color0)\">Ray</h3>\n",
" <svg version=\"1.1\" id=\"ray\" width=\"3em\" viewBox=\"0 0 144.5 144.6\" style=\"margin-left: 3em;margin-right: 3em\">\n",
" <g id=\"layer-1\">\n",
" <path fill=\"#00a2e9\" class=\"st0\" d=\"M97.3,77.2c-3.8-1.1-6.2,0.9-8.3,5.1c-3.5,6.8-9.9,9.9-17.4,9.6S58,88.1,54.8,81.2c-1.4-3-3-4-6.3-4.1\n",
" c-5.6-0.1-9.9,0.1-13.1,6.4c-3.8,7.6-13.6,10.2-21.8,7.6C5.2,88.4-0.4,80.5,0,71.7c0.1-8.4,5.7-15.8,13.8-18.2\n",
" c8.4-2.6,17.5,0.7,22.3,8c1.3,1.9,1.3,5.2,3.6,5.6c3.9,0.6,8,0.2,12,0.2c1.8,0,1.9-1.6,2.4-2.8c3.5-7.8,9.7-11.8,18-11.9\n",
" c8.2-0.1,14.4,3.9,17.8,11.4c1.3,2.8,2.9,3.6,5.7,3.3c1-0.1,2,0.1,3,0c2.8-0.5,6.4,1.7,8.1-2.7s-2.3-5.5-4.1-7.5\n",
" c-5.1-5.7-10.9-10.8-16.1-16.3C84,38,81.9,37.1,78,38.3C66.7,42,56.2,35.7,53,24.1C50.3,14,57.3,2.8,67.7,0.5\n",
" C78.4-2,89,4.7,91.5,15.3c0.1,0.3,0.1,0.5,0.2,0.8c0.7,3.4,0.7,6.9-0.8,9.8c-1.7,3.2-0.8,5,1.5,7.2c6.7,6.5,13.3,13,19.8,19.7\n",
" c1.8,1.8,3,2.1,5.5,1.2c9.1-3.4,17.9-0.6,23.4,7c4.8,6.9,4.6,16.1-0.4,22.9c-5.4,7.2-14.2,9.9-23.1,6.5c-2.3-0.9-3.5-0.6-5.1,1.1\n",
" c-6.7,6.9-13.6,13.7-20.5,20.4c-1.8,1.8-2.5,3.2-1.4,5.9c3.5,8.7,0.3,18.6-7.7,23.6c-7.9,5-18.2,3.8-24.8-2.9\n",
" c-6.4-6.4-7.4-16.2-2.5-24.3c4.9-7.8,14.5-11,23.1-7.8c3,1.1,4.7,0.5,6.9-1.7C91.7,98.4,98,92.3,104.2,86c1.6-1.6,4.1-2.7,2.6-6.2\n",
" c-1.4-3.3-3.8-2.5-6.2-2.6C99.8,77.2,98.9,77.2,97.3,77.2z M72.1,29.7c5.5,0.1,9.9-4.3,10-9.8c0-0.1,0-0.2,0-0.3\n",
" C81.8,14,77,9.8,71.5,10.2c-5,0.3-9,4.2-9.3,9.2c-0.2,5.5,4,10.1,9.5,10.3C71.8,29.7,72,29.7,72.1,29.7z M72.3,62.3\n",
" c-5.4-0.1-9.9,4.2-10.1,9.7c0,0.2,0,0.3,0,0.5c0.2,5.4,4.5,9.7,9.9,10c5.1,0.1,9.9-4.7,10.1-9.8c0.2-5.5-4-10-9.5-10.3\n",
" C72.6,62.3,72.4,62.3,72.3,62.3z M115,72.5c0.1,5.4,4.5,9.7,9.8,9.9c5.6-0.2,10-4.8,10-10.4c-0.2-5.4-4.6-9.7-10-9.7\n",
" c-5.3-0.1-9.8,4.2-9.9,9.5C115,72.1,115,72.3,115,72.5z M19.5,62.3c-5.4,0.1-9.8,4.4-10,9.8c-0.1,5.1,5.2,10.4,10.2,10.3\n",
" c5.6-0.2,10-4.9,9.8-10.5c-0.1-5.4-4.5-9.7-9.9-9.6C19.6,62.3,19.5,62.3,19.5,62.3z M71.8,134.6c5.9,0.2,10.3-3.9,10.4-9.6\n",
" c0.5-5.5-3.6-10.4-9.1-10.8c-5.5-0.5-10.4,3.6-10.8,9.1c0,0.5,0,0.9,0,1.4c-0.2,5.3,4,9.8,9.3,10\n",
" C71.6,134.6,71.7,134.6,71.8,134.6z\"/>\n",
" </g>\n",
" </svg>\n",
" <table>\n",
" <tr>\n",
" <td style=\"text-align: left\"><b>Python version:</b></td>\n",
" <td style=\"text-align: left\"><b>3.7.7</b></td>\n",
" </tr>\n",
" <tr>\n",
" <td style=\"text-align: left\"><b>Ray version:</b></td>\n",
2022-07-27 18:43:27 +00:00
" <td style=\"text-align: left\"><b> 2.0.0rc0</b></td>\n",
2022-07-24 18:53:57 +01:00
" </tr>\n",
" <tr>\n",
" <td style=\"text-align: left\"><b>Dashboard:</b></td>\n",
" <td style=\"text-align: left\"><b><a href=\"http://127.0.0.1:8268\" target=\"_blank\">http://127.0.0.1:8268</a></b></td>\n",
"</tr>\n",
"\n",
" </table>\n",
" </div>\n",
"</div>\n"
],
"text/plain": [
2022-07-27 18:43:27 +00:00
"RayContext(dashboard_url='127.0.0.1:8268', python_version='3.7.7', ray_version='2.0.0rc0', ray_commit='{{RAY_COMMIT_SHA}}', address_info={'node_ip_address': '127.0.0.1', 'raylet_ip_address': '127.0.0.1', 'redis_address': None, 'object_store_address': '/tmp/ray/session_2022-07-22_15-46-17_712380_48353/sockets/plasma_store', 'raylet_socket_name': '/tmp/ray/session_2022-07-22_15-46-17_712380_48353/sockets/raylet', 'webui_url': '127.0.0.1:8268', 'session_dir': '/tmp/ray/session_2022-07-22_15-46-17_712380_48353', 'metrics_export_port': 64916, 'gcs_address': '127.0.0.1:64371', 'address': '127.0.0.1:64371', 'dashboard_agent_listen_port': 52365, 'node_id': 'bf206482b99028fd8c8cfaea9d94a1bd982869a6c7d6f7531ac6bcb8'})"
2022-07-24 18:53:57 +01:00
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
2022-04-25 11:10:58 -07:00
"source": [
"ray.init(configure_logging=False)"
]
},
{
"cell_type": "markdown",
"id": "c18346e5",
"metadata": {},
"source": [
"Let's start by defining a simple evaluation function. Again, an explicit math formula is queried here for demonstration, yet in practice this is typically a black-box function-- e.g. the performance results after training an ML model. We artificially sleep for a bit (`0.1` seconds) to simulate a long-running ML experiment. This setup assumes that we're running multiple `step`s of an experiment while tuning three hyperparameters, namely `width`, `height`, and `activation`."
]
},
{
"cell_type": "code",
2022-07-24 18:53:57 +01:00
"execution_count": 4,
2022-04-25 11:10:58 -07:00
"id": "3cb44451",
"metadata": {},
"outputs": [],
"source": [
"def evaluate(step, width, height, activation):\n",
" time.sleep(0.1)\n",
" activation_boost = 10 if activation==\"relu\" else 0\n",
" return (0.1 + width * step / 100) ** (-1) + height * 0.1 + activation_boost"
]
},
{
"cell_type": "markdown",
"id": "9ac14837",
"metadata": {},
"source": [
"Next, our `objective` function to be optimized takes a Tune `config`, evaluates the `score` of your experiment in a training loop,\n",
2022-06-30 10:37:31 -07:00
"and uses `session.report` to report the `score` back to Tune."
2022-04-25 11:10:58 -07:00
]
},
{
"cell_type": "code",
2022-07-24 18:53:57 +01:00
"execution_count": 5,
2022-04-25 11:10:58 -07:00
"id": "f4b27c17",
"metadata": {},
"outputs": [],
"source": [
"def objective(config):\n",
" for step in range(config[\"steps\"]):\n",
" score = evaluate(step, config[\"width\"], config[\"height\"], config[\"activation\"])\n",
2022-06-30 10:37:31 -07:00
" session.report({\"iterations\": step, \"mean_loss\": score})"
2022-04-25 11:10:58 -07:00
]
},
{
"cell_type": "markdown",
"id": "2b48209c",
"metadata": {},
"source": [
"Next we define a search space. The critical assumption is that the optimal hyperparamters live within this space. Yet, if the space is very large, then those hyperparameters may be difficult to find in a short amount of time."
]
},
{
"cell_type": "code",
2022-07-24 18:53:57 +01:00
"execution_count": 6,
2022-04-25 11:10:58 -07:00
"id": "e1cf2e16",
"metadata": {},
"outputs": [],
"source": [
"search_space = {\n",
" \"steps\": 100,\n",
" \"width\": tune.uniform(0, 20),\n",
" \"height\": tune.uniform(-100, 100),\n",
" \"activation\": tune.choice([\"relu\", \"tanh\"]),\n",
"}"
]
},
{
"cell_type": "markdown",
"id": "2892b243",
"metadata": {},
"source": [
"The search algorithm is instantiated from the `SkOptSearch` class. We also constrain the the number of concurrent trials to `4` with a `ConcurrencyLimiter`."
]
},
{
"cell_type": "code",
2022-07-24 18:53:57 +01:00
"execution_count": 7,
2022-04-25 11:10:58 -07:00
"id": "5b560697",
"metadata": {},
"outputs": [],
"source": [
"algo = SkOptSearch()\n",
"algo = ConcurrencyLimiter(algo, max_concurrent=4)\n"
]
},
{
"cell_type": "markdown",
"id": "573a6c97",
"metadata": {},
"source": [
"The number of samples is the number of hyperparameter combinations that will be tried out. This Tune run is set to `1000` samples.\n",
"(you can decrease this if it takes too long on your machine)."
]
},
{
"cell_type": "code",
2022-07-24 18:53:57 +01:00
"execution_count": 8,
2022-04-25 11:10:58 -07:00
"id": "bebc40db",
"metadata": {},
"outputs": [],
"source": [
"num_samples = 1000"
]
},
{
"cell_type": "code",
2022-07-24 18:53:57 +01:00
"execution_count": 9,
2022-04-25 11:10:58 -07:00
"id": "a86c5a19",
"metadata": {
"tags": [
"remove-cell"
]
},
"outputs": [],
"source": [
"# We override here for our smoke tests.\n",
"num_samples = 10"
]
},
{
"cell_type": "markdown",
"id": "35c5823b",
"metadata": {},
"source": [
2022-07-24 18:53:57 +01:00
"Finally, we run the experiment to `\"min\"`imize the \"mean_loss\" of the `objective` by searching `search_config` via `algo`, `num_samples` times. This previous sentence is fully characterizes the search problem we aim to solve. With this in mind, notice how efficient it is to execute `tuner.fit()`."
2022-04-25 11:10:58 -07:00
]
},
{
"cell_type": "code",
2022-07-24 18:53:57 +01:00
"execution_count": 10,
2022-04-25 11:10:58 -07:00
"id": "c86c6919",
"metadata": {},
2022-07-24 18:53:57 +01:00
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Function checkpointing is disabled. This may result in unexpected behavior when using checkpointing features or certain schedulers. To enable, set the train function arguments to be `func(config, checkpoint_dir=None)`.\n"
]
},
{
"data": {
"text/html": [
"== Status ==<br>Current time: 2022-07-22 15:47:07 (running for 00:00:44.11)<br>Memory usage on this node: 9.9/16.0 GiB<br>Using FIFO scheduling algorithm.<br>Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/5.42 GiB heap, 0.0/2.0 GiB objects<br>Current best trial: 20e93604 with mean_loss=-8.869507755259557 and parameters={'steps': 100, 'width': 8.246480084328711, 'height': -89.90514307848873, 'activation': 'tanh'}<br>Result logdir: /Users/kai/ray_results/objective_2022-07-22_15-46-22<br>Number of trials: 10/10 (10 TERMINATED)<br><table>\n",
"<thead>\n",
"<tr><th>Trial name </th><th>status </th><th>loc </th><th>activation </th><th style=\"text-align: right;\"> height</th><th style=\"text-align: right;\"> width</th><th style=\"text-align: right;\"> loss</th><th style=\"text-align: right;\"> iter</th><th style=\"text-align: right;\"> total time (s)</th><th style=\"text-align: right;\"> iterations</th><th style=\"text-align: right;\"> neg_mean_loss</th></tr>\n",
"</thead>\n",
"<tbody>\n",
"<tr><td>objective_0ec1ec00</td><td>TERMINATED</td><td>127.0.0.1:48435</td><td>tanh </td><td style=\"text-align: right;\"> 10.9976</td><td style=\"text-align: right;\"> 8.60092</td><td style=\"text-align: right;\"> 1.21584 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 10.941 </td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\"> -1.21584 </td></tr>\n",
"<tr><td>objective_107c17fa</td><td>TERMINATED</td><td>127.0.0.1:48446</td><td>relu </td><td style=\"text-align: right;\"> 49.5455</td><td style=\"text-align: right;\">11.8085 </td><td style=\"text-align: right;\">15.0394 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 11.5877</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\"> -15.0394 </td></tr>\n",
"<tr><td>objective_107d75f0</td><td>TERMINATED</td><td>127.0.0.1:48447</td><td>tanh </td><td style=\"text-align: right;\"> -7.6131</td><td style=\"text-align: right;\"> 3.10991</td><td style=\"text-align: right;\">-0.446727</td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 11.7109</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\"> 0.446727</td></tr>\n",
"<tr><td>objective_107ef722</td><td>TERMINATED</td><td>127.0.0.1:48448</td><td>relu </td><td style=\"text-align: right;\">-75.8366</td><td style=\"text-align: right;\"> 5.30975</td><td style=\"text-align: right;\"> 2.60302 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 11.6501</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\"> -2.60302 </td></tr>\n",
"<tr><td>objective_17033b12</td><td>TERMINATED</td><td>127.0.0.1:48470</td><td>tanh </td><td style=\"text-align: right;\">-89.1891</td><td style=\"text-align: right;\">12.1463 </td><td style=\"text-align: right;\">-8.83643 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 10.7255</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\"> 8.83643 </td></tr>\n",
"<tr><td>objective_18fbb20a</td><td>TERMINATED</td><td>127.0.0.1:48475</td><td>tanh </td><td style=\"text-align: right;\"> 33.5568</td><td style=\"text-align: right;\"> 9.75437</td><td style=\"text-align: right;\"> 3.45818 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 10.7563</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\"> -3.45818 </td></tr>\n",
"<tr><td>objective_190ab4d0</td><td>TERMINATED</td><td>127.0.0.1:48478</td><td>relu </td><td style=\"text-align: right;\">-65.148 </td><td style=\"text-align: right;\"> 4.19546</td><td style=\"text-align: right;\"> 3.7203 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 10.7123</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\"> -3.7203 </td></tr>\n",
"<tr><td>objective_191ca3de</td><td>TERMINATED</td><td>127.0.0.1:48481</td><td>tanh </td><td style=\"text-align: right;\">-13.4892</td><td style=\"text-align: right;\"> 6.74078</td><td style=\"text-align: right;\">-1.20128 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 13.219 </td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\"> 1.20128 </td></tr>\n",
"<tr><td>objective_1eec012e</td><td>TERMINATED</td><td>127.0.0.1:48494</td><td>tanh </td><td style=\"text-align: right;\">-50.2819</td><td style=\"text-align: right;\">14.8287 </td><td style=\"text-align: right;\">-4.96053 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 13.2232</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\"> 4.96053 </td></tr>\n",
"<tr><td>objective_20e93604</td><td>TERMINATED</td><td>127.0.0.1:48502</td><td>tanh </td><td style=\"text-align: right;\">-89.9051</td><td style=\"text-align: right;\"> 8.24648</td><td style=\"text-align: right;\">-8.86951 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 10.7925</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\"> 8.86951 </td></tr>\n",
"</tbody>\n",
"</table><br><br>"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Result for objective_0ec1ec00:\n",
" date: 2022-07-22_15-46-26\n",
" done: false\n",
" experiment_id: c3e1e5ac1d6c4d67878bf793d020f99e\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 11.099760178357887\n",
" neg_mean_loss: -11.099760178357887\n",
" node_ip: 127.0.0.1\n",
" pid: 48435\n",
" time_since_restore: 0.10384106636047363\n",
" time_this_iter_s: 0.10384106636047363\n",
" time_total_s: 0.10384106636047363\n",
" timestamp: 1658501186\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: 0ec1ec00\n",
" warmup_time: 0.003099679946899414\n",
" \n",
"Result for objective_107c17fa:\n",
" date: 2022-07-22_15-46-29\n",
" done: false\n",
" experiment_id: f2a0e8ad4ea64c489424406b75b5c051\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 24.95455186350225\n",
" neg_mean_loss: -24.95455186350225\n",
" node_ip: 127.0.0.1\n",
" pid: 48446\n",
" time_since_restore: 0.10309791564941406\n",
" time_this_iter_s: 0.10309791564941406\n",
" time_total_s: 0.10309791564941406\n",
" timestamp: 1658501189\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: 107c17fa\n",
" warmup_time: 0.0029959678649902344\n",
" \n",
"Result for objective_107ef722:\n",
" date: 2022-07-22_15-46-29\n",
" done: false\n",
" experiment_id: 286beb82d0f0493dacbafb0ceb9bfe2d\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 12.416336029599446\n",
" neg_mean_loss: -12.416336029599446\n",
" node_ip: 127.0.0.1\n",
" pid: 48448\n",
" time_since_restore: 0.10424971580505371\n",
" time_this_iter_s: 0.10424971580505371\n",
" time_total_s: 0.10424971580505371\n",
" timestamp: 1658501189\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: 107ef722\n",
" warmup_time: 0.0032057762145996094\n",
" \n",
"Result for objective_107d75f0:\n",
" date: 2022-07-22_15-46-29\n",
" done: false\n",
" experiment_id: a44e7e5deb2e4bc68b31ca0466db50f5\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 9.238689666819806\n",
" neg_mean_loss: -9.238689666819806\n",
" node_ip: 127.0.0.1\n",
" pid: 48447\n",
" time_since_restore: 0.10475015640258789\n",
" time_this_iter_s: 0.10475015640258789\n",
" time_total_s: 0.10475015640258789\n",
" timestamp: 1658501189\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: 107d75f0\n",
" warmup_time: 0.0029137134552001953\n",
" \n",
"Result for objective_0ec1ec00:\n",
" date: 2022-07-22_15-46-31\n",
" done: false\n",
" experiment_id: c3e1e5ac1d6c4d67878bf793d020f99e\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 45\n",
" iterations_since_restore: 46\n",
" mean_loss: 1.3516231012646176\n",
" neg_mean_loss: -1.3516231012646176\n",
" node_ip: 127.0.0.1\n",
" pid: 48435\n",
" time_since_restore: 5.127159833908081\n",
" time_this_iter_s: 0.10582566261291504\n",
" time_total_s: 5.127159833908081\n",
" timestamp: 1658501191\n",
" timesteps_since_restore: 0\n",
" training_iteration: 46\n",
" trial_id: 0ec1ec00\n",
" warmup_time: 0.003099679946899414\n",
" \n",
"Result for objective_107c17fa:\n",
" date: 2022-07-22_15-46-34\n",
" done: false\n",
" experiment_id: f2a0e8ad4ea64c489424406b75b5c051\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" mean_loss: 15.131543197018463\n",
" neg_mean_loss: -15.131543197018463\n",
" node_ip: 127.0.0.1\n",
" pid: 48446\n",
" time_since_restore: 5.161302089691162\n",
" time_this_iter_s: 0.10862898826599121\n",
" time_total_s: 5.161302089691162\n",
" timestamp: 1658501194\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: 107c17fa\n",
" warmup_time: 0.0029959678649902344\n",
" \n",
"Result for objective_107ef722:\n",
" date: 2022-07-22_15-46-34\n",
" done: false\n",
" experiment_id: 286beb82d0f0493dacbafb0ceb9bfe2d\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" mean_loss: 2.8016060611446587\n",
" neg_mean_loss: -2.8016060611446587\n",
" node_ip: 127.0.0.1\n",
" pid: 48448\n",
" time_since_restore: 5.157445907592773\n",
" time_this_iter_s: 0.10624003410339355\n",
" time_total_s: 5.157445907592773\n",
" timestamp: 1658501194\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: 107ef722\n",
" warmup_time: 0.0032057762145996094\n",
" \n",
"Result for objective_107d75f0:\n",
" date: 2022-07-22_15-46-34\n",
" done: false\n",
" experiment_id: a44e7e5deb2e4bc68b31ca0466db50f5\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" mean_loss: -0.12096483365591593\n",
" neg_mean_loss: 0.12096483365591593\n",
" node_ip: 127.0.0.1\n",
" pid: 48447\n",
" time_since_restore: 5.159098863601685\n",
" time_this_iter_s: 0.10865378379821777\n",
" time_total_s: 5.159098863601685\n",
" timestamp: 1658501194\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: 107d75f0\n",
" warmup_time: 0.0029137134552001953\n",
" \n",
"Result for objective_0ec1ec00:\n",
" date: 2022-07-22_15-46-36\n",
" done: false\n",
" experiment_id: c3e1e5ac1d6c4d67878bf793d020f99e\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 92\n",
" iterations_since_restore: 93\n",
" mean_loss: 1.2245597846713034\n",
" neg_mean_loss: -1.2245597846713034\n",
" node_ip: 127.0.0.1\n",
" pid: 48435\n",
" time_since_restore: 10.187216997146606\n",
" time_this_iter_s: 0.10774493217468262\n",
" time_total_s: 10.187216997146606\n",
" timestamp: 1658501196\n",
" timesteps_since_restore: 0\n",
" training_iteration: 93\n",
" trial_id: 0ec1ec00\n",
" warmup_time: 0.003099679946899414\n",
" \n",
"Result for objective_0ec1ec00:\n",
" date: 2022-07-22_15-46-37\n",
" done: true\n",
" experiment_id: c3e1e5ac1d6c4d67878bf793d020f99e\n",
" experiment_tag: 1_activation=tanh,height=10.9976,steps=100,width=8.6009\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: 1.2158379996438775\n",
" neg_mean_loss: -1.2158379996438775\n",
" node_ip: 127.0.0.1\n",
" pid: 48435\n",
" time_since_restore: 10.9409658908844\n",
" time_this_iter_s: 0.10762286186218262\n",
" time_total_s: 10.9409658908844\n",
" timestamp: 1658501197\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: 0ec1ec00\n",
" warmup_time: 0.003099679946899414\n",
" \n",
"Result for objective_107ef722:\n",
" date: 2022-07-22_15-46-39\n",
" done: false\n",
" experiment_id: 286beb82d0f0493dacbafb0ceb9bfe2d\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 91\n",
" iterations_since_restore: 92\n",
" mean_loss: 2.619098804697928\n",
" neg_mean_loss: -2.619098804697928\n",
" node_ip: 127.0.0.1\n",
" pid: 48448\n",
" time_since_restore: 9.851147890090942\n",
" time_this_iter_s: 0.1064901351928711\n",
" time_total_s: 9.851147890090942\n",
" timestamp: 1658501199\n",
" timesteps_since_restore: 0\n",
" training_iteration: 92\n",
" trial_id: 107ef722\n",
" warmup_time: 0.0032057762145996094\n",
" \n",
"Result for objective_107c17fa:\n",
" date: 2022-07-22_15-46-39\n",
" done: false\n",
" experiment_id: f2a0e8ad4ea64c489424406b75b5c051\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 92\n",
" iterations_since_restore: 93\n",
" mean_loss: 15.0457609631331\n",
" neg_mean_loss: -15.0457609631331\n",
" node_ip: 127.0.0.1\n",
" pid: 48446\n",
" time_since_restore: 9.976460695266724\n",
" time_this_iter_s: 0.10694289207458496\n",
" time_total_s: 9.976460695266724\n",
" timestamp: 1658501199\n",
" timesteps_since_restore: 0\n",
" training_iteration: 93\n",
" trial_id: 107c17fa\n",
" warmup_time: 0.0029959678649902344\n",
" \n",
"Result for objective_107d75f0:\n",
" date: 2022-07-22_15-46-39\n",
" done: false\n",
" experiment_id: a44e7e5deb2e4bc68b31ca0466db50f5\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 90\n",
" iterations_since_restore: 91\n",
" mean_loss: -0.41635399078466045\n",
" neg_mean_loss: 0.41635399078466045\n",
" node_ip: 127.0.0.1\n",
" pid: 48447\n",
" time_since_restore: 9.776082992553711\n",
" time_this_iter_s: 0.10715007781982422\n",
" time_total_s: 9.776082992553711\n",
" timestamp: 1658501199\n",
" timesteps_since_restore: 0\n",
" training_iteration: 91\n",
" trial_id: 107d75f0\n",
" warmup_time: 0.0029137134552001953\n",
" \n",
"Result for objective_17033b12:\n",
" date: 2022-07-22_15-46-40\n",
" done: false\n",
" experiment_id: b5f76922e6424055b6aa255677e2f4b8\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 1.0810921171053653\n",
" neg_mean_loss: -1.0810921171053653\n",
" node_ip: 127.0.0.1\n",
" pid: 48470\n",
" time_since_restore: 0.10409784317016602\n",
" time_this_iter_s: 0.10409784317016602\n",
" time_total_s: 0.10409784317016602\n",
" timestamp: 1658501200\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: 17033b12\n",
" warmup_time: 0.0029649734497070312\n",
" \n",
"Result for objective_107c17fa:\n",
" date: 2022-07-22_15-46-41\n",
" done: true\n",
" experiment_id: f2a0e8ad4ea64c489424406b75b5c051\n",
" experiment_tag: 2_activation=relu,height=49.5455,steps=100,width=11.8085\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: 15.039366532944502\n",
" neg_mean_loss: -15.039366532944502\n",
" node_ip: 127.0.0.1\n",
" pid: 48446\n",
" time_since_restore: 11.587661743164062\n",
" time_this_iter_s: 0.10614180564880371\n",
" time_total_s: 11.587661743164062\n",
" timestamp: 1658501201\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: 107c17fa\n",
" warmup_time: 0.0029959678649902344\n",
" \n",
"Result for objective_107ef722:\n",
" date: 2022-07-22_15-46-41\n",
" done: true\n",
" experiment_id: 286beb82d0f0493dacbafb0ceb9bfe2d\n",
" experiment_tag: 4_activation=relu,height=-75.8366,steps=100,width=5.3097\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: 2.603019812948376\n",
" neg_mean_loss: -2.603019812948376\n",
" node_ip: 127.0.0.1\n",
" pid: 48448\n",
" time_since_restore: 11.65014386177063\n",
" time_this_iter_s: 0.10755181312561035\n",
" time_total_s: 11.65014386177063\n",
" timestamp: 1658501201\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: 107ef722\n",
" warmup_time: 0.0032057762145996094\n",
" \n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Result for objective_107d75f0:\n",
" date: 2022-07-22_15-46-41\n",
" done: true\n",
" experiment_id: a44e7e5deb2e4bc68b31ca0466db50f5\n",
" experiment_tag: 3_activation=tanh,height=-7.6131,steps=100,width=3.1099\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: -0.44672713582845847\n",
" neg_mean_loss: 0.44672713582845847\n",
" node_ip: 127.0.0.1\n",
" pid: 48447\n",
" time_since_restore: 11.710935115814209\n",
" time_this_iter_s: 0.1116342544555664\n",
" time_total_s: 11.710935115814209\n",
" timestamp: 1658501201\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: 107d75f0\n",
" warmup_time: 0.0029137134552001953\n",
" \n",
"Result for objective_18fbb20a:\n",
" date: 2022-07-22_15-46-43\n",
" done: false\n",
" experiment_id: 5e399bc125a548739dd433f4153019fe\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 13.355684066332913\n",
" neg_mean_loss: -13.355684066332913\n",
" node_ip: 127.0.0.1\n",
" pid: 48475\n",
" time_since_restore: 0.10402607917785645\n",
" time_this_iter_s: 0.10402607917785645\n",
" time_total_s: 0.10402607917785645\n",
" timestamp: 1658501203\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: 18fbb20a\n",
" warmup_time: 0.003000974655151367\n",
" \n",
"Result for objective_190ab4d0:\n",
" date: 2022-07-22_15-46-43\n",
" done: false\n",
" experiment_id: 1b007a40cf6d425085379a4314739912\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 13.48520025532623\n",
" neg_mean_loss: -13.48520025532623\n",
" node_ip: 127.0.0.1\n",
" pid: 48478\n",
" time_since_restore: 0.10344505310058594\n",
" time_this_iter_s: 0.10344505310058594\n",
" time_total_s: 0.10344505310058594\n",
" timestamp: 1658501203\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: 190ab4d0\n",
" warmup_time: 0.0028760433197021484\n",
" \n",
"Result for objective_191ca3de:\n",
" date: 2022-07-22_15-46-43\n",
" done: false\n",
" experiment_id: e3e72bbeb8ae46f0a4f1624db62bc730\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 8.651082223836086\n",
" neg_mean_loss: -8.651082223836086\n",
" node_ip: 127.0.0.1\n",
" pid: 48481\n",
" time_since_restore: 0.1036369800567627\n",
" time_this_iter_s: 0.1036369800567627\n",
" time_total_s: 0.1036369800567627\n",
" timestamp: 1658501203\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: 191ca3de\n",
" warmup_time: 0.0029621124267578125\n",
" \n",
"Result for objective_17033b12:\n",
" date: 2022-07-22_15-46-45\n",
" done: false\n",
" experiment_id: b5f76922e6424055b6aa255677e2f4b8\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" mean_loss: -8.746754140047871\n",
" neg_mean_loss: 8.746754140047871\n",
" node_ip: 127.0.0.1\n",
" pid: 48470\n",
" time_since_restore: 5.144078969955444\n",
" time_this_iter_s: 0.10620617866516113\n",
" time_total_s: 5.144078969955444\n",
" timestamp: 1658501205\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: 17033b12\n",
" warmup_time: 0.0029649734497070312\n",
" \n",
"Result for objective_18fbb20a:\n",
" date: 2022-07-22_15-46-48\n",
" done: false\n",
" experiment_id: 5e399bc125a548739dd433f4153019fe\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" mean_loss: 3.5691516291476226\n",
" neg_mean_loss: -3.5691516291476226\n",
" node_ip: 127.0.0.1\n",
" pid: 48475\n",
" time_since_restore: 5.178721189498901\n",
" time_this_iter_s: 0.10647201538085938\n",
" time_total_s: 5.178721189498901\n",
" timestamp: 1658501208\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: 18fbb20a\n",
" warmup_time: 0.003000974655151367\n",
" \n",
"Result for objective_190ab4d0:\n",
" date: 2022-07-22_15-46-48\n",
" done: false\n",
" experiment_id: 1b007a40cf6d425085379a4314739912\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" mean_loss: 3.9678564240341583\n",
" neg_mean_loss: -3.9678564240341583\n",
" node_ip: 127.0.0.1\n",
" pid: 48478\n",
" time_since_restore: 5.156033754348755\n",
" time_this_iter_s: 0.10652899742126465\n",
" time_total_s: 5.156033754348755\n",
" timestamp: 1658501208\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: 190ab4d0\n",
" warmup_time: 0.0028760433197021484\n",
" \n",
"Result for objective_191ca3de:\n",
" date: 2022-07-22_15-46-48\n",
" done: false\n",
" experiment_id: e3e72bbeb8ae46f0a4f1624db62bc730\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" mean_loss: -1.04293599865772\n",
" neg_mean_loss: 1.04293599865772\n",
" node_ip: 127.0.0.1\n",
" pid: 48481\n",
" time_since_restore: 5.1839048862457275\n",
" time_this_iter_s: 0.10760998725891113\n",
" time_total_s: 5.1839048862457275\n",
" timestamp: 1658501208\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: 191ca3de\n",
" warmup_time: 0.0029621124267578125\n",
" \n",
"Result for objective_17033b12:\n",
" date: 2022-07-22_15-46-50\n",
" done: false\n",
" experiment_id: b5f76922e6424055b6aa255677e2f4b8\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 94\n",
" iterations_since_restore: 95\n",
" mean_loss: -8.832083655687075\n",
" neg_mean_loss: 8.832083655687075\n",
" node_ip: 127.0.0.1\n",
" pid: 48470\n",
" time_since_restore: 10.193100929260254\n",
" time_this_iter_s: 0.10694503784179688\n",
" time_total_s: 10.193100929260254\n",
" timestamp: 1658501210\n",
" timesteps_since_restore: 0\n",
" training_iteration: 95\n",
" trial_id: 17033b12\n",
" warmup_time: 0.0029649734497070312\n",
" \n",
"Result for objective_17033b12:\n",
" date: 2022-07-22_15-46-50\n",
" done: true\n",
" experiment_id: b5f76922e6424055b6aa255677e2f4b8\n",
" experiment_tag: 5_activation=tanh,height=-89.1891,steps=100,width=12.1463\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: -8.836432551723352\n",
" neg_mean_loss: 8.836432551723352\n",
" node_ip: 127.0.0.1\n",
" pid: 48470\n",
" time_since_restore: 10.725535869598389\n",
" time_this_iter_s: 0.10705089569091797\n",
" time_total_s: 10.725535869598389\n",
" timestamp: 1658501210\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: 17033b12\n",
" warmup_time: 0.0029649734497070312\n",
" \n",
"Result for objective_1eec012e:\n",
" date: 2022-07-22_15-46-53\n",
" done: false\n",
" experiment_id: a3cab04eb23543ddbf9b1daa9e915851\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 4.9718091604083385\n",
" neg_mean_loss: -4.9718091604083385\n",
" node_ip: 127.0.0.1\n",
" pid: 48494\n",
" time_since_restore: 0.10102462768554688\n",
" time_this_iter_s: 0.10102462768554688\n",
" time_total_s: 0.10102462768554688\n",
" timestamp: 1658501213\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: 1eec012e\n",
" warmup_time: 0.002748250961303711\n",
" \n",
"Result for objective_18fbb20a:\n",
" date: 2022-07-22_15-46-53\n",
" done: false\n",
" experiment_id: 5e399bc125a548739dd433f4153019fe\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 94\n",
" iterations_since_restore: 95\n",
" mean_loss: 3.4635693481489094\n",
" neg_mean_loss: -3.4635693481489094\n",
" node_ip: 127.0.0.1\n",
" pid: 48475\n",
" time_since_restore: 10.22235107421875\n",
" time_this_iter_s: 0.10835385322570801\n",
" time_total_s: 10.22235107421875\n",
" timestamp: 1658501213\n",
" timesteps_since_restore: 0\n",
" training_iteration: 95\n",
" trial_id: 18fbb20a\n",
" warmup_time: 0.003000974655151367\n",
" \n",
"Result for objective_190ab4d0:\n",
" date: 2022-07-22_15-46-53\n",
" done: false\n",
" experiment_id: 1b007a40cf6d425085379a4314739912\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 94\n",
" iterations_since_restore: 95\n",
" mean_loss: 3.732496287450277\n",
" neg_mean_loss: -3.732496287450277\n",
" node_ip: 127.0.0.1\n",
" pid: 48478\n",
" time_since_restore: 10.17775583267212\n",
" time_this_iter_s: 0.10712075233459473\n",
" time_total_s: 10.17775583267212\n",
" timestamp: 1658501213\n",
" timesteps_since_restore: 0\n",
" training_iteration: 95\n",
" trial_id: 190ab4d0\n",
" warmup_time: 0.0028760433197021484\n",
" \n",
"Result for objective_191ca3de:\n",
" date: 2022-07-22_15-46-54\n",
" done: false\n",
" experiment_id: e3e72bbeb8ae46f0a4f1624db62bc730\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 94\n",
" iterations_since_restore: 95\n",
" mean_loss: -1.193549900472242\n",
" neg_mean_loss: 1.193549900472242\n",
" node_ip: 127.0.0.1\n",
" pid: 48481\n",
" time_since_restore: 10.230173826217651\n",
" time_this_iter_s: 0.10641312599182129\n",
" time_total_s: 10.230173826217651\n",
" timestamp: 1658501214\n",
" timesteps_since_restore: 0\n",
" training_iteration: 95\n",
" trial_id: 191ca3de\n",
" warmup_time: 0.0029621124267578125\n",
" \n",
"Result for objective_18fbb20a:\n",
" date: 2022-07-22_15-46-54\n",
" done: true\n",
" experiment_id: 5e399bc125a548739dd433f4153019fe\n",
" experiment_tag: 6_activation=tanh,height=33.5568,steps=100,width=9.7544\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: 3.4581764420917516\n",
" neg_mean_loss: -3.4581764420917516\n",
" node_ip: 127.0.0.1\n",
" pid: 48475\n",
" time_since_restore: 10.756349325180054\n",
" time_this_iter_s: 0.10465312004089355\n",
" time_total_s: 10.756349325180054\n",
" timestamp: 1658501214\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: 18fbb20a\n",
" warmup_time: 0.003000974655151367\n",
" \n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Result for objective_190ab4d0:\n",
" date: 2022-07-22_15-46-54\n",
" done: true\n",
" experiment_id: 1b007a40cf6d425085379a4314739912\n",
" experiment_tag: 7_activation=relu,height=-65.1480,steps=100,width=4.1955\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: 3.7203002216299694\n",
" neg_mean_loss: -3.7203002216299694\n",
" node_ip: 127.0.0.1\n",
" pid: 48478\n",
" time_since_restore: 10.712332725524902\n",
" time_this_iter_s: 0.10630083084106445\n",
" time_total_s: 10.712332725524902\n",
" timestamp: 1658501214\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: 190ab4d0\n",
" warmup_time: 0.0028760433197021484\n",
" \n",
"Result for objective_20e93604:\n",
" date: 2022-07-22_15-46-56\n",
" done: false\n",
" experiment_id: e2c12d5518de42acb5f8413171242ef8\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 1.0094856921511255\n",
" neg_mean_loss: -1.0094856921511255\n",
" node_ip: 127.0.0.1\n",
" pid: 48502\n",
" time_since_restore: 0.10361671447753906\n",
" time_this_iter_s: 0.10361671447753906\n",
" time_total_s: 0.10361671447753906\n",
" timestamp: 1658501216\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: 20e93604\n",
" warmup_time: 0.0035440921783447266\n",
" \n",
"Result for objective_191ca3de:\n",
" date: 2022-07-22_15-46-57\n",
" done: true\n",
" experiment_id: e3e72bbeb8ae46f0a4f1624db62bc730\n",
" experiment_tag: 8_activation=tanh,height=-13.4892,steps=100,width=6.7408\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: -1.2012809142667369\n",
" neg_mean_loss: 1.2012809142667369\n",
" node_ip: 127.0.0.1\n",
" pid: 48481\n",
" time_since_restore: 13.218971729278564\n",
" time_this_iter_s: 2.560670852661133\n",
" time_total_s: 13.218971729278564\n",
" timestamp: 1658501217\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: 191ca3de\n",
" warmup_time: 0.0029621124267578125\n",
" \n",
"Result for objective_1eec012e:\n",
" date: 2022-07-22_15-46-58\n",
" done: false\n",
" experiment_id: a3cab04eb23543ddbf9b1daa9e915851\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 24\n",
" iterations_since_restore: 25\n",
" mean_loss: -4.754884423931921\n",
" neg_mean_loss: 4.754884423931921\n",
" node_ip: 127.0.0.1\n",
" pid: 48494\n",
" time_since_restore: 5.151007890701294\n",
" time_this_iter_s: 0.1075141429901123\n",
" time_total_s: 5.151007890701294\n",
" timestamp: 1658501218\n",
" timesteps_since_restore: 0\n",
" training_iteration: 25\n",
" trial_id: 1eec012e\n",
" warmup_time: 0.002748250961303711\n",
" \n",
"Result for objective_20e93604:\n",
" date: 2022-07-22_15-47-02\n",
" done: false\n",
" experiment_id: e2c12d5518de42acb5f8413171242ef8\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" mean_loss: -8.738995488800374\n",
" neg_mean_loss: 8.738995488800374\n",
" node_ip: 127.0.0.1\n",
" pid: 48502\n",
" time_since_restore: 5.183563709259033\n",
" time_this_iter_s: 0.10645675659179688\n",
" time_total_s: 5.183563709259033\n",
" timestamp: 1658501222\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: 20e93604\n",
" warmup_time: 0.0035440921783447266\n",
" \n",
"Result for objective_1eec012e:\n",
" date: 2022-07-22_15-47-03\n",
" done: false\n",
" experiment_id: a3cab04eb23543ddbf9b1daa9e915851\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 71\n",
" iterations_since_restore: 72\n",
" mean_loss: -4.9341033303033965\n",
" neg_mean_loss: 4.9341033303033965\n",
" node_ip: 127.0.0.1\n",
" pid: 48494\n",
" time_since_restore: 10.200856685638428\n",
" time_this_iter_s: 0.10717487335205078\n",
" time_total_s: 10.200856685638428\n",
" timestamp: 1658501223\n",
" timesteps_since_restore: 0\n",
" training_iteration: 72\n",
" trial_id: 1eec012e\n",
" warmup_time: 0.002748250961303711\n",
" \n",
"Result for objective_1eec012e:\n",
" date: 2022-07-22_15-47-06\n",
" done: true\n",
" experiment_id: a3cab04eb23543ddbf9b1daa9e915851\n",
" experiment_tag: 9_activation=tanh,height=-50.2819,steps=100,width=14.8287\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: -4.960533899755889\n",
" neg_mean_loss: 4.960533899755889\n",
" node_ip: 127.0.0.1\n",
" pid: 48494\n",
" time_since_restore: 13.22315764427185\n",
" time_this_iter_s: 0.10771775245666504\n",
" time_total_s: 13.22315764427185\n",
" timestamp: 1658501226\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: 1eec012e\n",
" warmup_time: 0.002748250961303711\n",
" \n",
"Result for objective_20e93604:\n",
" date: 2022-07-22_15-47-07\n",
" done: false\n",
" experiment_id: e2c12d5518de42acb5f8413171242ef8\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 94\n",
" iterations_since_restore: 95\n",
" mean_loss: -8.863153212712554\n",
" neg_mean_loss: 8.863153212712554\n",
" node_ip: 127.0.0.1\n",
" pid: 48502\n",
" time_since_restore: 10.251462936401367\n",
" time_this_iter_s: 0.1052391529083252\n",
" time_total_s: 10.251462936401367\n",
" timestamp: 1658501227\n",
" timesteps_since_restore: 0\n",
" training_iteration: 95\n",
" trial_id: 20e93604\n",
" warmup_time: 0.0035440921783447266\n",
" \n",
"Result for objective_20e93604:\n",
" date: 2022-07-22_15-47-07\n",
" done: true\n",
" experiment_id: e2c12d5518de42acb5f8413171242ef8\n",
" experiment_tag: 10_activation=tanh,height=-89.9051,steps=100,width=8.2465\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: -8.869507755259557\n",
" neg_mean_loss: 8.869507755259557\n",
" node_ip: 127.0.0.1\n",
" pid: 48502\n",
" time_since_restore: 10.792518854141235\n",
" time_this_iter_s: 0.10802984237670898\n",
" time_total_s: 10.792518854141235\n",
" timestamp: 1658501227\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: 20e93604\n",
" warmup_time: 0.0035440921783447266\n",
" \n"
]
}
],
2022-04-25 11:10:58 -07:00
"source": [
2022-07-24 18:53:57 +01:00
"tuner = tune.Tuner(\n",
2022-04-25 11:10:58 -07:00
" objective,\n",
2022-07-24 18:53:57 +01:00
" tune_config=tune.TuneConfig(\n",
" metric=\"mean_loss\",\n",
" mode=\"min\",\n",
" search_alg=algo,\n",
" num_samples=num_samples,\n",
" ),\n",
" param_space=search_space,\n",
")\n",
"results = tuner.fit()"
2022-04-25 11:10:58 -07:00
]
},
{
"cell_type": "markdown",
"id": "5f387c95",
"metadata": {},
"source": [
"We now have hyperparameters found to minimize the mean loss."
]
},
{
"cell_type": "code",
2022-07-24 18:53:57 +01:00
"execution_count": 11,
2022-04-25 11:10:58 -07:00
"id": "19fff170",
"metadata": {},
2022-07-24 18:53:57 +01:00
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Best hyperparameters found were: {'steps': 100, 'width': 8.246480084328711, 'height': -89.90514307848873, 'activation': 'tanh'}\n"
]
}
],
2022-04-25 11:10:58 -07:00
"source": [
2022-07-24 18:53:57 +01:00
"print(\"Best hyperparameters found were: \", results.get_best_result().config)"
2022-04-25 11:10:58 -07:00
]
},
{
"cell_type": "markdown",
"id": "7e7f43bb",
"metadata": {},
"source": [
"## Providing an initial set of hyperparameters\n",
"\n",
"While defining the search algorithm, we may choose to provide an initial set of hyperparameters that we believe are especially promising or informative, and\n",
"pass this information as a helpful starting point for the `SkOptSearch` object. We also can pass the known rewards for these initial params to save on unnecessary computation."
]
},
{
"cell_type": "code",
2022-07-24 18:53:57 +01:00
"execution_count": 12,
2022-04-25 11:10:58 -07:00
"id": "c44a3ad2",
"metadata": {},
"outputs": [],
"source": [
"initial_params = [\n",
" {\"width\": 10, \"height\": 0, \"activation\": \"relu\"},\n",
" {\"width\": 15, \"height\": -20, \"activation\": \"tanh\"}\n",
"]\n",
"known_rewards = [-189, -1144]"
]
},
{
"cell_type": "markdown",
"id": "30e19a4b",
"metadata": {},
"source": [
"Now the `search_alg` built using `SkOptSearch` takes `points_to_evaluate`."
]
},
{
"cell_type": "code",
2022-07-24 18:53:57 +01:00
"execution_count": 13,
2022-04-25 11:10:58 -07:00
"id": "19308593",
"metadata": {
"lines_to_next_cell": 0
},
"outputs": [],
"source": [
"algo = SkOptSearch(points_to_evaluate=initial_params)\n",
"algo = ConcurrencyLimiter(algo, max_concurrent=4)"
]
},
{
"cell_type": "markdown",
"id": "bd022018",
"metadata": {},
"source": [
"And again run the experiment, this time with initial hyperparameter evaluations:"
]
},
{
"cell_type": "code",
2022-07-24 18:53:57 +01:00
"execution_count": 15,
2022-04-25 11:10:58 -07:00
"id": "d9c7c8d4",
"metadata": {},
2022-07-24 18:53:57 +01:00
"outputs": [
{
"data": {
"text/html": [
"== Status ==<br>Current time: 2022-07-22 15:49:08 (running for 00:00:43.84)<br>Memory usage on this node: 10.5/16.0 GiB<br>Using FIFO scheduling algorithm.<br>Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/5.42 GiB heap, 0.0/2.0 GiB objects<br>Current best trial: 60e6657e with mean_loss=-6.564839315448637 and parameters={'steps': 100, 'width': 11.45060059898019, 'height': -66.52281754112352, 'activation': 'tanh'}<br>Result logdir: /Users/kai/ray_results/objective_2022-07-22_15-48-24<br>Number of trials: 10/10 (10 TERMINATED)<br><table>\n",
"<thead>\n",
"<tr><th>Trial name </th><th>status </th><th>loc </th><th>activation </th><th style=\"text-align: right;\"> height</th><th style=\"text-align: right;\"> width</th><th style=\"text-align: right;\"> loss</th><th style=\"text-align: right;\"> iter</th><th style=\"text-align: right;\"> total time (s)</th><th style=\"text-align: right;\"> iterations</th><th style=\"text-align: right;\"> neg_mean_loss</th></tr>\n",
"</thead>\n",
"<tbody>\n",
"<tr><td>objective_56a79fce</td><td>TERMINATED</td><td>127.0.0.1:48590</td><td>relu </td><td style=\"text-align: right;\"> 0 </td><td style=\"text-align: right;\">10 </td><td style=\"text-align: right;\">10.1 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 10.9393</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\"> -10.1 </td></tr>\n",
"<tr><td>objective_583ee2ca</td><td>TERMINATED</td><td>127.0.0.1:48605</td><td>tanh </td><td style=\"text-align: right;\">-20 </td><td style=\"text-align: right;\">15 </td><td style=\"text-align: right;\">-1.93311</td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 11.6366</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\"> 1.93311</td></tr>\n",
"<tr><td>objective_58405952</td><td>TERMINATED</td><td>127.0.0.1:48606</td><td>relu </td><td style=\"text-align: right;\">-87.8025</td><td style=\"text-align: right;\">19.608 </td><td style=\"text-align: right;\"> 1.271 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 11.752 </td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\"> -1.271 </td></tr>\n",
"<tr><td>objective_5841ebe6</td><td>TERMINATED</td><td>127.0.0.1:48607</td><td>relu </td><td style=\"text-align: right;\"> 49.51 </td><td style=\"text-align: right;\">15.0894 </td><td style=\"text-align: right;\">15.0175 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 11.6761</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\"> -15.0175 </td></tr>\n",
"<tr><td>objective_5ec4da6e</td><td>TERMINATED</td><td>127.0.0.1:48629</td><td>relu </td><td style=\"text-align: right;\"> 76.0837</td><td style=\"text-align: right;\">13.7995 </td><td style=\"text-align: right;\">17.681 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 10.7667</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\"> -17.681 </td></tr>\n",
"<tr><td>objective_60d7fb56</td><td>TERMINATED</td><td>127.0.0.1:48634</td><td>relu </td><td style=\"text-align: right;\">-50.4534</td><td style=\"text-align: right;\">16.3133 </td><td style=\"text-align: right;\"> 5.0162 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 10.75 </td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\"> -5.0162 </td></tr>\n",
"<tr><td>objective_60e6657e</td><td>TERMINATED</td><td>127.0.0.1:48637</td><td>tanh </td><td style=\"text-align: right;\">-66.5228</td><td style=\"text-align: right;\">11.4506 </td><td style=\"text-align: right;\">-6.56484</td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 10.759 </td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\"> 6.56484</td></tr>\n",
"<tr><td>objective_60f84e42</td><td>TERMINATED</td><td>127.0.0.1:48640</td><td>tanh </td><td style=\"text-align: right;\"> 99.5965</td><td style=\"text-align: right;\"> 9.04425</td><td style=\"text-align: right;\">10.0701 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 10.71 </td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\"> -10.0701 </td></tr>\n",
"<tr><td>objective_66bdb9e8</td><td>TERMINATED</td><td>127.0.0.1:48653</td><td>tanh </td><td style=\"text-align: right;\">-61.9547</td><td style=\"text-align: right;\">12.3263 </td><td style=\"text-align: right;\">-6.11419</td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 10.724 </td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\"> 6.11419</td></tr>\n",
"<tr><td>objective_68c346b8</td><td>TERMINATED</td><td>127.0.0.1:48660</td><td>tanh </td><td style=\"text-align: right;\"> 23.7916</td><td style=\"text-align: right;\"> 2.108 </td><td style=\"text-align: right;\"> 2.83643</td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 10.7375</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\"> -2.83643</td></tr>\n",
"</tbody>\n",
"</table><br><br>"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Result for objective_56a79fce:\n",
" date: 2022-07-22_15-48-27\n",
" done: false\n",
" experiment_id: 7046e554c1db465bbed80331b034b40f\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 20.0\n",
" neg_mean_loss: -20.0\n",
" node_ip: 127.0.0.1\n",
" pid: 48590\n",
" time_since_restore: 0.10438704490661621\n",
" time_this_iter_s: 0.10438704490661621\n",
" time_total_s: 0.10438704490661621\n",
" timestamp: 1658501307\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: 56a79fce\n",
" warmup_time: 0.002905130386352539\n",
" \n",
"Result for objective_583ee2ca:\n",
" date: 2022-07-22_15-48-30\n",
" done: false\n",
" experiment_id: 59c76129c7074e41ab39b330d898ba03\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 8.0\n",
" neg_mean_loss: -8.0\n",
" node_ip: 127.0.0.1\n",
" pid: 48605\n",
" time_since_restore: 0.10483098030090332\n",
" time_this_iter_s: 0.10483098030090332\n",
" time_total_s: 0.10483098030090332\n",
" timestamp: 1658501310\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: 583ee2ca\n",
" warmup_time: 0.0031731128692626953\n",
" \n",
"Result for objective_5841ebe6:\n",
" date: 2022-07-22_15-48-30\n",
" done: false\n",
" experiment_id: fde2877ded13416195ab1af070eda91a\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 24.951000429599397\n",
" neg_mean_loss: -24.951000429599397\n",
" node_ip: 127.0.0.1\n",
" pid: 48607\n",
" time_since_restore: 0.10440397262573242\n",
" time_this_iter_s: 0.10440397262573242\n",
" time_total_s: 0.10440397262573242\n",
" timestamp: 1658501310\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: 5841ebe6\n",
" warmup_time: 0.0033330917358398438\n",
" \n",
"Result for objective_58405952:\n",
" date: 2022-07-22_15-48-30\n",
" done: false\n",
" experiment_id: 5902a2d532b3437e8b94d02919e1604c\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 11.21975075417543\n",
" neg_mean_loss: -11.21975075417543\n",
" node_ip: 127.0.0.1\n",
" pid: 48606\n",
" time_since_restore: 0.10400986671447754\n",
" time_this_iter_s: 0.10400986671447754\n",
" time_total_s: 0.10400986671447754\n",
" timestamp: 1658501310\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: '58405952'\n",
" warmup_time: 0.0027670860290527344\n",
" \n",
"Result for objective_56a79fce:\n",
" date: 2022-07-22_15-48-32\n",
" done: false\n",
" experiment_id: 7046e554c1db465bbed80331b034b40f\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 45\n",
" iterations_since_restore: 46\n",
" mean_loss: 10.217391304347826\n",
" neg_mean_loss: -10.217391304347826\n",
" node_ip: 127.0.0.1\n",
" pid: 48590\n",
" time_since_restore: 5.135921239852905\n",
" time_this_iter_s: 0.10727715492248535\n",
" time_total_s: 5.135921239852905\n",
" timestamp: 1658501312\n",
" timesteps_since_restore: 0\n",
" training_iteration: 46\n",
" trial_id: 56a79fce\n",
" warmup_time: 0.002905130386352539\n",
" \n",
"Result for objective_583ee2ca:\n",
" date: 2022-07-22_15-48-35\n",
" done: false\n",
" experiment_id: 59c76129c7074e41ab39b330d898ba03\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" mean_loss: -1.8601398601398602\n",
" neg_mean_loss: 1.8601398601398602\n",
" node_ip: 127.0.0.1\n",
" pid: 48605\n",
" time_since_restore: 5.154622793197632\n",
" time_this_iter_s: 0.10759091377258301\n",
" time_total_s: 5.154622793197632\n",
" timestamp: 1658501315\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: 583ee2ca\n",
" warmup_time: 0.0031731128692626953\n",
" \n",
"Result for objective_5841ebe6:\n",
" date: 2022-07-22_15-48-35\n",
" done: false\n",
" experiment_id: fde2877ded13416195ab1af070eda91a\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" mean_loss: 15.090043780907216\n",
" neg_mean_loss: -15.090043780907216\n",
" node_ip: 127.0.0.1\n",
" pid: 48607\n",
" time_since_restore: 5.153692960739136\n",
" time_this_iter_s: 0.10715007781982422\n",
" time_total_s: 5.153692960739136\n",
" timestamp: 1658501315\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: 5841ebe6\n",
" warmup_time: 0.0033330917358398438\n",
" \n",
"Result for objective_58405952:\n",
" date: 2022-07-22_15-48-35\n",
" done: false\n",
" experiment_id: 5902a2d532b3437e8b94d02919e1604c\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" mean_loss: 1.3270955262548618\n",
" neg_mean_loss: -1.3270955262548618\n",
" node_ip: 127.0.0.1\n",
" pid: 48606\n",
" time_since_restore: 5.145922660827637\n",
" time_this_iter_s: 0.10549187660217285\n",
" time_total_s: 5.145922660827637\n",
" timestamp: 1658501315\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: '58405952'\n",
" warmup_time: 0.0027670860290527344\n",
" \n",
"Result for objective_56a79fce:\n",
" date: 2022-07-22_15-48-37\n",
" done: false\n",
" experiment_id: 7046e554c1db465bbed80331b034b40f\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 92\n",
" iterations_since_restore: 93\n",
" mean_loss: 10.10752688172043\n",
" neg_mean_loss: -10.10752688172043\n",
" node_ip: 127.0.0.1\n",
" pid: 48590\n",
" time_since_restore: 10.195838212966919\n",
" time_this_iter_s: 0.10654616355895996\n",
" time_total_s: 10.195838212966919\n",
" timestamp: 1658501317\n",
" timesteps_since_restore: 0\n",
" training_iteration: 93\n",
" trial_id: 56a79fce\n",
" warmup_time: 0.002905130386352539\n",
" \n",
"Result for objective_56a79fce:\n",
" date: 2022-07-22_15-48-38\n",
" done: true\n",
" experiment_id: 7046e554c1db465bbed80331b034b40f\n",
" experiment_tag: 1_activation=relu,height=0,steps=100,width=10\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: 10.1\n",
" neg_mean_loss: -10.1\n",
" node_ip: 127.0.0.1\n",
" pid: 48590\n",
" time_since_restore: 10.939304113388062\n",
" time_this_iter_s: 0.10537099838256836\n",
" time_total_s: 10.939304113388062\n",
" timestamp: 1658501318\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: 56a79fce\n",
" warmup_time: 0.002905130386352539\n",
" \n",
"Result for objective_5841ebe6:\n",
" date: 2022-07-22_15-48-39\n",
" done: false\n",
" experiment_id: fde2877ded13416195ab1af070eda91a\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 90\n",
" iterations_since_restore: 91\n",
" mean_loss: 15.024097555037425\n",
" neg_mean_loss: -15.024097555037425\n",
" node_ip: 127.0.0.1\n",
" pid: 48607\n",
" time_since_restore: 9.817665100097656\n",
" time_this_iter_s: 0.10596418380737305\n",
" time_total_s: 9.817665100097656\n",
" timestamp: 1658501319\n",
" timesteps_since_restore: 0\n",
" training_iteration: 91\n",
" trial_id: 5841ebe6\n",
" warmup_time: 0.0033330917358398438\n",
" \n",
"Result for objective_58405952:\n",
" date: 2022-07-22_15-48-39\n",
" done: false\n",
" experiment_id: 5902a2d532b3437e8b94d02919e1604c\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 89\n",
" iterations_since_restore: 90\n",
" mean_loss: 1.2767270681910414\n",
" neg_mean_loss: -1.2767270681910414\n",
" node_ip: 127.0.0.1\n",
" pid: 48606\n",
" time_since_restore: 9.72554087638855\n",
" time_this_iter_s: 0.10558199882507324\n",
" time_total_s: 9.72554087638855\n",
" timestamp: 1658501319\n",
" timesteps_since_restore: 0\n",
" training_iteration: 90\n",
" trial_id: '58405952'\n",
" warmup_time: 0.0027670860290527344\n",
" \n",
"Result for objective_583ee2ca:\n",
" date: 2022-07-22_15-48-39\n",
" done: false\n",
" experiment_id: 59c76129c7074e41ab39b330d898ba03\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 91\n",
" iterations_since_restore: 92\n",
" mean_loss: -1.9272727272727272\n",
" neg_mean_loss: 1.9272727272727272\n",
" node_ip: 127.0.0.1\n",
" pid: 48605\n",
" time_since_restore: 9.925616025924683\n",
" time_this_iter_s: 0.1071469783782959\n",
" time_total_s: 9.925616025924683\n",
" timestamp: 1658501319\n",
" timesteps_since_restore: 0\n",
" training_iteration: 92\n",
" trial_id: 583ee2ca\n",
" warmup_time: 0.0031731128692626953\n",
" \n",
"Result for objective_5ec4da6e:\n",
" date: 2022-07-22_15-48-40\n",
" done: false\n",
" experiment_id: 24cf29dbb0e44fa7a2dbffe353a56c35\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 27.60836685236658\n",
" neg_mean_loss: -27.60836685236658\n",
" node_ip: 127.0.0.1\n",
" pid: 48629\n",
" time_since_restore: 0.10513186454772949\n",
" time_this_iter_s: 0.10513186454772949\n",
" time_total_s: 0.10513186454772949\n",
" timestamp: 1658501320\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: 5ec4da6e\n",
" warmup_time: 0.002888917922973633\n",
" \n",
"Result for objective_583ee2ca:\n",
" date: 2022-07-22_15-48-41\n",
" done: true\n",
" experiment_id: 59c76129c7074e41ab39b330d898ba03\n",
" experiment_tag: 2_activation=tanh,height=-20,steps=100,width=15\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: -1.9331103678929766\n",
" neg_mean_loss: 1.9331103678929766\n",
" node_ip: 127.0.0.1\n",
" pid: 48605\n",
" time_since_restore: 11.636637926101685\n",
" time_this_iter_s: 0.10811305046081543\n",
" time_total_s: 11.636637926101685\n",
" timestamp: 1658501321\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: 583ee2ca\n",
" warmup_time: 0.0031731128692626953\n",
" \n",
"Result for objective_5841ebe6:\n",
" date: 2022-07-22_15-48-41\n",
" done: true\n",
" experiment_id: fde2877ded13416195ab1af070eda91a\n",
" experiment_tag: 4_activation=relu,height=49.5100,steps=100,width=15.0894\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: 15.017496549772488\n",
" neg_mean_loss: -15.017496549772488\n",
" node_ip: 127.0.0.1\n",
" pid: 48607\n",
" time_since_restore: 11.676102876663208\n",
" time_this_iter_s: 0.1074519157409668\n",
" time_total_s: 11.676102876663208\n",
" timestamp: 1658501321\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: 5841ebe6\n",
" warmup_time: 0.0033330917358398438\n",
" \n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Result for objective_58405952:\n",
" date: 2022-07-22_15-48-41\n",
" done: true\n",
" experiment_id: 5902a2d532b3437e8b94d02919e1604c\n",
" experiment_tag: 3_activation=relu,height=-87.8025,steps=100,width=19.6080\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: 1.271001380631752\n",
" neg_mean_loss: -1.271001380631752\n",
" node_ip: 127.0.0.1\n",
" pid: 48606\n",
" time_since_restore: 11.751991033554077\n",
" time_this_iter_s: 0.1101231575012207\n",
" time_total_s: 11.751991033554077\n",
" timestamp: 1658501321\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: '58405952'\n",
" warmup_time: 0.0027670860290527344\n",
" \n",
"Result for objective_60d7fb56:\n",
" date: 2022-07-22_15-48-44\n",
" done: false\n",
" experiment_id: 26d42c9dcc6e4354b3fa7355fc806522\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 14.954664969255782\n",
" neg_mean_loss: -14.954664969255782\n",
" node_ip: 127.0.0.1\n",
" pid: 48634\n",
" time_since_restore: 0.10394525527954102\n",
" time_this_iter_s: 0.10394525527954102\n",
" time_total_s: 0.10394525527954102\n",
" timestamp: 1658501324\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: 60d7fb56\n",
" warmup_time: 0.003077268600463867\n",
" \n",
"Result for objective_60e6657e:\n",
" date: 2022-07-22_15-48-44\n",
" done: false\n",
" experiment_id: 3f2014eb8b41451bb277cabed9951d90\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 3.3477182458876484\n",
" neg_mean_loss: -3.3477182458876484\n",
" node_ip: 127.0.0.1\n",
" pid: 48637\n",
" time_since_restore: 0.10410618782043457\n",
" time_this_iter_s: 0.10410618782043457\n",
" time_total_s: 0.10410618782043457\n",
" timestamp: 1658501324\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: 60e6657e\n",
" warmup_time: 0.0029239654541015625\n",
" \n",
"Result for objective_60f84e42:\n",
" date: 2022-07-22_15-48-44\n",
" done: false\n",
" experiment_id: 481882ef37194e6f995c4294a3453fd2\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 19.959649161689548\n",
" neg_mean_loss: -19.959649161689548\n",
" node_ip: 127.0.0.1\n",
" pid: 48640\n",
" time_since_restore: 0.10413289070129395\n",
" time_this_iter_s: 0.10413289070129395\n",
" time_total_s: 0.10413289070129395\n",
" timestamp: 1658501324\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: 60f84e42\n",
" warmup_time: 0.0028841495513916016\n",
" \n",
"Result for objective_5ec4da6e:\n",
" date: 2022-07-22_15-48-45\n",
" done: false\n",
" experiment_id: 24cf29dbb0e44fa7a2dbffe353a56c35\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" mean_loss: 17.760209862707853\n",
" neg_mean_loss: -17.760209862707853\n",
" node_ip: 127.0.0.1\n",
" pid: 48629\n",
" time_since_restore: 5.151746988296509\n",
" time_this_iter_s: 0.10759496688842773\n",
" time_total_s: 5.151746988296509\n",
" timestamp: 1658501325\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: 5ec4da6e\n",
" warmup_time: 0.002888917922973633\n",
" \n",
"Result for objective_60d7fb56:\n",
" date: 2022-07-22_15-48-49\n",
" done: false\n",
" experiment_id: 26d42c9dcc6e4354b3fa7355fc806522\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" mean_loss: 5.08341058857216\n",
" neg_mean_loss: -5.08341058857216\n",
" node_ip: 127.0.0.1\n",
" pid: 48634\n",
" time_since_restore: 5.146520137786865\n",
" time_this_iter_s: 0.10820508003234863\n",
" time_total_s: 5.146520137786865\n",
" timestamp: 1658501329\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: 60d7fb56\n",
" warmup_time: 0.003077268600463867\n",
" \n",
"Result for objective_60e6657e:\n",
" date: 2022-07-22_15-48-49\n",
" done: false\n",
" experiment_id: 3f2014eb8b41451bb277cabed9951d90\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" mean_loss: -6.469859332234883\n",
" neg_mean_loss: 6.469859332234883\n",
" node_ip: 127.0.0.1\n",
" pid: 48637\n",
" time_since_restore: 5.153999090194702\n",
" time_this_iter_s: 0.10596871376037598\n",
" time_total_s: 5.153999090194702\n",
" timestamp: 1658501329\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: 60e6657e\n",
" warmup_time: 0.0029239654541015625\n",
" \n",
"Result for objective_60f84e42:\n",
" date: 2022-07-22_15-48-49\n",
" done: false\n",
" experiment_id: 481882ef37194e6f995c4294a3453fd2\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" mean_loss: 10.189492101039557\n",
" neg_mean_loss: -10.189492101039557\n",
" node_ip: 127.0.0.1\n",
" pid: 48640\n",
" time_since_restore: 5.137938976287842\n",
" time_this_iter_s: 0.10651087760925293\n",
" time_total_s: 5.137938976287842\n",
" timestamp: 1658501329\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: 60f84e42\n",
" warmup_time: 0.0028841495513916016\n",
" \n",
"Result for objective_5ec4da6e:\n",
" date: 2022-07-22_15-48-50\n",
" done: false\n",
" experiment_id: 24cf29dbb0e44fa7a2dbffe353a56c35\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 94\n",
" iterations_since_restore: 95\n",
" mean_loss: 17.684869174683147\n",
" neg_mean_loss: -17.684869174683147\n",
" node_ip: 127.0.0.1\n",
" pid: 48629\n",
" time_since_restore: 10.229070901870728\n",
" time_this_iter_s: 0.11342406272888184\n",
" time_total_s: 10.229070901870728\n",
" timestamp: 1658501330\n",
" timesteps_since_restore: 0\n",
" training_iteration: 95\n",
" trial_id: 5ec4da6e\n",
" warmup_time: 0.002888917922973633\n",
" \n",
"Result for objective_5ec4da6e:\n",
" date: 2022-07-22_15-48-51\n",
" done: true\n",
" experiment_id: 24cf29dbb0e44fa7a2dbffe353a56c35\n",
" experiment_tag: 5_activation=relu,height=76.0837,steps=100,width=13.7995\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: 17.6810334976324\n",
" neg_mean_loss: -17.6810334976324\n",
" node_ip: 127.0.0.1\n",
" pid: 48629\n",
" time_since_restore: 10.766729831695557\n",
" time_this_iter_s: 0.10697484016418457\n",
" time_total_s: 10.766729831695557\n",
" timestamp: 1658501331\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: 5ec4da6e\n",
" warmup_time: 0.002888917922973633\n",
" \n",
"Result for objective_66bdb9e8:\n",
" date: 2022-07-22_15-48-54\n",
" done: false\n",
" experiment_id: fae00fc97db044e9965698562b20039b\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 3.804528837217097\n",
" neg_mean_loss: -3.804528837217097\n",
" node_ip: 127.0.0.1\n",
" pid: 48653\n",
" time_since_restore: 0.10318303108215332\n",
" time_this_iter_s: 0.10318303108215332\n",
" time_total_s: 0.10318303108215332\n",
" timestamp: 1658501334\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: 66bdb9e8\n",
" warmup_time: 0.004212856292724609\n",
" \n",
"Result for objective_60d7fb56:\n",
" date: 2022-07-22_15-48-54\n",
" done: false\n",
" experiment_id: 26d42c9dcc6e4354b3fa7355fc806522\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 94\n",
" iterations_since_restore: 95\n",
" mean_loss: 5.019454849577342\n",
" neg_mean_loss: -5.019454849577342\n",
" node_ip: 127.0.0.1\n",
" pid: 48634\n",
" time_since_restore: 10.212354183197021\n",
" time_this_iter_s: 0.10782504081726074\n",
" time_total_s: 10.212354183197021\n",
" timestamp: 1658501334\n",
" timesteps_since_restore: 0\n",
" training_iteration: 95\n",
" trial_id: 60d7fb56\n",
" warmup_time: 0.003077268600463867\n",
" \n",
"Result for objective_60e6657e:\n",
" date: 2022-07-22_15-48-54\n",
" done: false\n",
" experiment_id: 3f2014eb8b41451bb277cabed9951d90\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 94\n",
" iterations_since_restore: 95\n",
" mean_loss: -6.560230936519563\n",
" neg_mean_loss: 6.560230936519563\n",
" node_ip: 127.0.0.1\n",
" pid: 48637\n",
" time_since_restore: 10.219763278961182\n",
" time_this_iter_s: 0.10767722129821777\n",
" time_total_s: 10.219763278961182\n",
" timestamp: 1658501334\n",
" timesteps_since_restore: 0\n",
" training_iteration: 95\n",
" trial_id: 60e6657e\n",
" warmup_time: 0.0029239654541015625\n",
" \n",
"Result for objective_60f84e42:\n",
" date: 2022-07-22_15-48-54\n",
" done: false\n",
" experiment_id: 481882ef37194e6f995c4294a3453fd2\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 94\n",
" iterations_since_restore: 95\n",
" mean_loss: 10.075906679849322\n",
" neg_mean_loss: -10.075906679849322\n",
" node_ip: 127.0.0.1\n",
" pid: 48640\n",
" time_since_restore: 10.16911792755127\n",
" time_this_iter_s: 0.10426807403564453\n",
" time_total_s: 10.16911792755127\n",
" timestamp: 1658501334\n",
" timesteps_since_restore: 0\n",
" training_iteration: 95\n",
" trial_id: 60f84e42\n",
" warmup_time: 0.0028841495513916016\n",
" \n",
"Result for objective_60d7fb56:\n",
" date: 2022-07-22_15-48-54\n",
" done: true\n",
" experiment_id: 26d42c9dcc6e4354b3fa7355fc806522\n",
" experiment_tag: 6_activation=relu,height=-50.4534,steps=100,width=16.3133\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: 5.016202769898167\n",
" neg_mean_loss: -5.016202769898167\n",
" node_ip: 127.0.0.1\n",
" pid: 48634\n",
" time_since_restore: 10.75002908706665\n",
" time_this_iter_s: 0.10717272758483887\n",
" time_total_s: 10.75002908706665\n",
" timestamp: 1658501334\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: 60d7fb56\n",
" warmup_time: 0.003077268600463867\n",
" \n",
"Result for objective_60e6657e:\n",
" date: 2022-07-22_15-48-54\n",
" done: true\n",
" experiment_id: 3f2014eb8b41451bb277cabed9951d90\n",
" experiment_tag: 7_activation=tanh,height=-66.5228,steps=100,width=11.4506\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: -6.564839315448637\n",
" neg_mean_loss: 6.564839315448637\n",
" node_ip: 127.0.0.1\n",
" pid: 48637\n",
" time_since_restore: 10.759015321731567\n",
" time_this_iter_s: 0.10623526573181152\n",
" time_total_s: 10.759015321731567\n",
" timestamp: 1658501334\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: 60e6657e\n",
" warmup_time: 0.0029239654541015625\n",
" \n",
"Result for objective_60f84e42:\n",
" date: 2022-07-22_15-48-55\n",
" done: true\n",
" experiment_id: 481882ef37194e6f995c4294a3453fd2\n",
" experiment_tag: 8_activation=tanh,height=99.5965,steps=100,width=9.0442\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: 10.070099940211907\n",
" neg_mean_loss: -10.070099940211907\n",
" node_ip: 127.0.0.1\n",
" pid: 48640\n",
" time_since_restore: 10.710038900375366\n",
" time_this_iter_s: 0.1064748764038086\n",
" time_total_s: 10.710038900375366\n",
" timestamp: 1658501335\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: 60f84e42\n",
" warmup_time: 0.0028841495513916016\n",
" \n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Result for objective_68c346b8:\n",
" date: 2022-07-22_15-48-57\n",
" done: false\n",
" experiment_id: fe241f21de6a4466bcdf713da6d4cd92\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 12.37916148081603\n",
" neg_mean_loss: -12.37916148081603\n",
" node_ip: 127.0.0.1\n",
" pid: 48660\n",
" time_since_restore: 0.10358977317810059\n",
" time_this_iter_s: 0.10358977317810059\n",
" time_total_s: 0.10358977317810059\n",
" timestamp: 1658501337\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: 68c346b8\n",
" warmup_time: 0.0028688907623291016\n",
" \n",
"Result for objective_66bdb9e8:\n",
" date: 2022-07-22_15-48-59\n",
" done: false\n",
" experiment_id: fae00fc97db044e9965698562b20039b\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" mean_loss: -6.025789125550826\n",
" neg_mean_loss: 6.025789125550826\n",
" node_ip: 127.0.0.1\n",
" pid: 48653\n",
" time_since_restore: 5.1440629959106445\n",
" time_this_iter_s: 0.10767483711242676\n",
" time_total_s: 5.1440629959106445\n",
" timestamp: 1658501339\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: 66bdb9e8\n",
" warmup_time: 0.004212856292724609\n",
" \n",
"Result for objective_68c346b8:\n",
" date: 2022-07-22_15-49-02\n",
" done: false\n",
" experiment_id: fe241f21de6a4466bcdf713da6d4cd92\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" mean_loss: 3.295952835154158\n",
" neg_mean_loss: -3.295952835154158\n",
" node_ip: 127.0.0.1\n",
" pid: 48660\n",
" time_since_restore: 5.15015983581543\n",
" time_this_iter_s: 0.10423922538757324\n",
" time_total_s: 5.15015983581543\n",
" timestamp: 1658501342\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: 68c346b8\n",
" warmup_time: 0.0028688907623291016\n",
" \n",
"Result for objective_66bdb9e8:\n",
" date: 2022-07-22_15-49-04\n",
" done: false\n",
" experiment_id: fae00fc97db044e9965698562b20039b\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 94\n",
" iterations_since_restore: 95\n",
" mean_loss: -6.109904185213174\n",
" neg_mean_loss: 6.109904185213174\n",
" node_ip: 127.0.0.1\n",
" pid: 48653\n",
" time_since_restore: 10.1879301071167\n",
" time_this_iter_s: 0.10626506805419922\n",
" time_total_s: 10.1879301071167\n",
" timestamp: 1658501344\n",
" timesteps_since_restore: 0\n",
" training_iteration: 95\n",
" trial_id: 66bdb9e8\n",
" warmup_time: 0.004212856292724609\n",
" \n",
"Result for objective_66bdb9e8:\n",
" date: 2022-07-22_15-49-04\n",
" done: true\n",
" experiment_id: fae00fc97db044e9965698562b20039b\n",
" experiment_tag: 9_activation=tanh,height=-61.9547,steps=100,width=12.3263\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: -6.114190623827535\n",
" neg_mean_loss: 6.114190623827535\n",
" node_ip: 127.0.0.1\n",
" pid: 48653\n",
" time_since_restore: 10.724038124084473\n",
" time_this_iter_s: 0.1088099479675293\n",
" time_total_s: 10.724038124084473\n",
" timestamp: 1658501344\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: 66bdb9e8\n",
" warmup_time: 0.004212856292724609\n",
" \n",
"Result for objective_68c346b8:\n",
" date: 2022-07-22_15-49-07\n",
" done: false\n",
" experiment_id: fe241f21de6a4466bcdf713da6d4cd92\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 94\n",
" iterations_since_restore: 95\n",
" mean_loss: 2.859579303290837\n",
" neg_mean_loss: -2.859579303290837\n",
" node_ip: 127.0.0.1\n",
" pid: 48660\n",
" time_since_restore: 10.199071884155273\n",
" time_this_iter_s: 0.10729789733886719\n",
" time_total_s: 10.199071884155273\n",
" timestamp: 1658501347\n",
" timesteps_since_restore: 0\n",
" training_iteration: 95\n",
" trial_id: 68c346b8\n",
" warmup_time: 0.0028688907623291016\n",
" \n",
"Result for objective_68c346b8:\n",
" date: 2022-07-22_15-49-08\n",
" done: true\n",
" experiment_id: fe241f21de6a4466bcdf713da6d4cd92\n",
" experiment_tag: 10_activation=tanh,height=23.7916,steps=100,width=2.1080\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: 2.836425260080649\n",
" neg_mean_loss: -2.836425260080649\n",
" node_ip: 127.0.0.1\n",
" pid: 48660\n",
" time_since_restore: 10.7375328540802\n",
" time_this_iter_s: 0.1081850528717041\n",
" time_total_s: 10.7375328540802\n",
" timestamp: 1658501348\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: 68c346b8\n",
" warmup_time: 0.0028688907623291016\n",
" \n"
]
}
],
2022-04-25 11:10:58 -07:00
"source": [
2022-07-24 18:53:57 +01:00
"tuner = tune.Tuner(\n",
2022-04-25 11:10:58 -07:00
" objective,\n",
2022-07-24 18:53:57 +01:00
" tune_config=tune.TuneConfig(\n",
" metric=\"mean_loss\",\n",
" mode=\"min\",\n",
" search_alg=algo,\n",
" num_samples=num_samples,\n",
" ),\n",
" param_space=search_space,\n",
")\n",
"results = tuner.fit()"
2022-04-25 11:10:58 -07:00
]
},
{
"cell_type": "markdown",
"id": "04c9d550",
"metadata": {},
"source": [
"And we again show the ideal hyperparameters."
]
},
{
"cell_type": "code",
2022-07-24 18:53:57 +01:00
"execution_count": 16,
2022-04-25 11:10:58 -07:00
"id": "d7267095",
"metadata": {},
2022-07-24 18:53:57 +01:00
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Best hyperparameters found were: {'steps': 100, 'width': 11.45060059898019, 'height': -66.52281754112352, 'activation': 'tanh'}\n"
]
}
],
2022-04-25 11:10:58 -07:00
"source": [
2022-07-24 18:53:57 +01:00
"print(\"Best hyperparameters found were: \", results.get_best_result().config)"
2022-04-25 11:10:58 -07:00
]
},
{
"cell_type": "code",
2022-07-24 18:53:57 +01:00
"execution_count": 17,
2022-04-25 11:10:58 -07:00
"id": "23739237",
"metadata": {
"tags": [
"remove-cell"
]
},
"outputs": [],
"source": [
"ray.shutdown()"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
2022-07-24 18:53:57 +01:00
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.7"
},
2022-04-25 11:10:58 -07:00
"orphan": true
},
"nbformat": 4,
"nbformat_minor": 5
}
