2022-03-25 01:04:53 -07:00
{
"cells": [
{
"cell_type": "markdown",
2022-04-25 11:10:58 -07:00
"id": "a587ce4e",
2022-03-25 01:04:53 -07:00
"metadata": {},
"source": [
"# Running Tune experiments with Optuna\n",
"\n",
2022-04-25 11:10:58 -07:00
"In this tutorial we introduce Optuna, while running a simple Ray Tune experiment. Tune’
2022-03-25 01:04:53 -07:00
"\n",
2022-04-25 11:10:58 -07:00
"Similar to Ray Tune, Optuna is an automatic hyperparameter optimization software framework, particularly designed for machine learning. It features an imperative (\"how\" over \"what\" emphasis), define-by-run style user API. With Optuna, a user has the ability to dynamically construct the search spaces for the hyperparameters. Optuna falls in the domain of \"derivative-free optimization\" and \"black-box optimization\".\n",
2022-03-25 01:04:53 -07:00
"\n",
2022-04-25 11:10:58 -07:00
"In this example we minimize a simple objective to briefly demonstrate the usage of Optuna with Ray Tune via `OptunaSearch`, including examples of conditional search spaces (string together relationships between hyperparameters), and the multi-objective problem (measure trade-offs among all important metrics). It's useful to keep in mind that despite the emphasis on machine learning experiments, Ray Tune optimizes any implicit or explicit objective. Here we assume `optuna==2.9.1` library is installed. To learn more, please refer to [Optuna website](https://optuna.org/).\n",
"\n",
"Please note that sophisticated schedulers, such as `AsyncHyperBandScheduler`, may not work correctly with multi-objective optimization, since they typically expect a scalar score to compare fitness among trials.\n"
]
},
{
"cell_type": "code",
2022-07-24 18:53:57 +01:00
"execution_count": 1,
2022-04-25 11:10:58 -07:00
"id": "aeaf9ff0",
"metadata": {
"tags": [
"remove-cell"
]
},
2022-07-24 18:53:57 +01:00
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: optuna==2.9.1 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (2.9.1)\n",
"Requirement already satisfied: alembic in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from optuna==2.9.1) (1.7.6)\n",
"Requirement already satisfied: colorlog in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from optuna==2.9.1) (6.6.0)\n",
"Requirement already satisfied: PyYAML in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from optuna==2.9.1) (5.4.1)\n",
"Requirement already satisfied: scipy!=1.4.0 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from optuna==2.9.1) (1.4.1)\n",
"Requirement already satisfied: tqdm in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from optuna==2.9.1) (4.64.0)\n",
"Requirement already satisfied: numpy in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from optuna==2.9.1) (1.21.6)\n",
"Requirement already satisfied: cliff in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from optuna==2.9.1) (3.10.1)\n",
"Requirement already satisfied: sqlalchemy>=1.1.0 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from optuna==2.9.1) (1.4.32)\n",
"Requirement already satisfied: packaging>=20.0 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from optuna==2.9.1) (21.3)\n",
"Requirement already satisfied: cmaes>=0.8.2 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from optuna==2.9.1) (0.8.2)\n",
"Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from packaging>=20.0->optuna==2.9.1) (2.4.7)\n",
"Requirement already satisfied: importlib-metadata in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from sqlalchemy>=1.1.0->optuna==2.9.1) (4.10.1)\n",
"Requirement already satisfied: greenlet!=0.4.17 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from sqlalchemy>=1.1.0->optuna==2.9.1) (1.1.2)\n",
"Requirement already satisfied: importlib-resources in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from alembic->optuna==2.9.1) (5.4.0)\n",
"Requirement already satisfied: Mako in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from alembic->optuna==2.9.1) (1.1.6)\n",
"Requirement already satisfied: autopage>=0.4.0 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from cliff->optuna==2.9.1) (0.5.0)\n",
"Requirement already satisfied: cmd2>=1.0.0 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from cliff->optuna==2.9.1) (2.4.0)\n",
"Requirement already satisfied: PrettyTable>=0.7.2 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from cliff->optuna==2.9.1) (3.2.0)\n",
"Requirement already satisfied: stevedore>=2.0.1 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from cliff->optuna==2.9.1) (3.5.0)\n",
"Requirement already satisfied: pbr!=2.1.0,>=2.0.0 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from cliff->optuna==2.9.1) (5.8.1)\n",
"Requirement already satisfied: typing-extensions in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from cmd2>=1.0.0->cliff->optuna==2.9.1) (4.1.1)\n",
"Requirement already satisfied: attrs>=16.3.0 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from cmd2>=1.0.0->cliff->optuna==2.9.1) (21.2.0)\n",
"Requirement already satisfied: pyperclip>=1.6 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from cmd2>=1.0.0->cliff->optuna==2.9.1) (1.8.2)\n",
"Requirement already satisfied: wcwidth>=0.1.7 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from cmd2>=1.0.0->cliff->optuna==2.9.1) (0.2.5)\n",
"Requirement already satisfied: zipp>=0.5 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from importlib-metadata->sqlalchemy>=1.1.0->optuna==2.9.1) (3.7.0)\n",
"Requirement already satisfied: MarkupSafe>=0.9.2 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from Mako->alembic->optuna==2.9.1) (2.0.1)\n",
"\u001b[33mWARNING: There was an error checking the latest version of pip.\u001b[0m\u001b[33m\n",
"\u001b[0m"
]
}
],
2022-04-25 11:10:58 -07:00
"source": [
"# !pip install ray[tune]\n",
"!pip install optuna==2.9.1"
]
},
{
"cell_type": "markdown",
"id": "467466a3",
"metadata": {},
"source": [
2022-03-25 01:04:53 -07:00
"Click below to see all the imports we need for this example.\n",
"You can also launch directly into a Binder instance to run this notebook yourself.\n",
"Just click on the rocket symbol at the top of the navigation."
]
},
{
"cell_type": "code",
2022-07-24 18:53:57 +01:00
"execution_count": 2,
2022-04-25 11:10:58 -07:00
"id": "fb1ad624",
2022-03-25 01:04:53 -07:00
"metadata": {
"tags": [
"hide-input"
]
},
"outputs": [],
"source": [
"import time\n",
"from typing import Dict, Optional, Any\n",
"\n",
"import ray\n",
"from ray import tune\n",
2022-06-30 10:37:31 -07:00
"from ray.air import session\n",
2022-06-25 14:55:30 +01:00
"from ray.tune.search import ConcurrencyLimiter\n",
"from ray.tune.search.optuna import OptunaSearch"
2022-03-25 01:04:53 -07:00
]
},
{
"cell_type": "code",
2022-07-24 18:53:57 +01:00
"execution_count": 3,
2022-04-25 11:10:58 -07:00
"id": "e64f0b44",
2022-03-25 01:04:53 -07:00
"metadata": {
"tags": [
"remove-cell"
]
},
2022-07-24 18:53:57 +01:00
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
" <div style=\"margin-left: 50px;display: flex;flex-direction: row;align-items: center\">\n",
" <h3 style=\"color: var(--jp-ui-font-color0)\">Ray</h3>\n",
" <svg version=\"1.1\" id=\"ray\" width=\"3em\" viewBox=\"0 0 144.5 144.6\" style=\"margin-left: 3em;margin-right: 3em\">\n",
" <g id=\"layer-1\">\n",
" <path fill=\"#00a2e9\" class=\"st0\" d=\"M97.3,77.2c-3.8-1.1-6.2,0.9-8.3,5.1c-3.5,6.8-9.9,9.9-17.4,9.6S58,88.1,54.8,81.2c-1.4-3-3-4-6.3-4.1\n",
" c-5.6-0.1-9.9,0.1-13.1,6.4c-3.8,7.6-13.6,10.2-21.8,7.6C5.2,88.4-0.4,80.5,0,71.7c0.1-8.4,5.7-15.8,13.8-18.2\n",
" c8.4-2.6,17.5,0.7,22.3,8c1.3,1.9,1.3,5.2,3.6,5.6c3.9,0.6,8,0.2,12,0.2c1.8,0,1.9-1.6,2.4-2.8c3.5-7.8,9.7-11.8,18-11.9\n",
" c8.2-0.1,14.4,3.9,17.8,11.4c1.3,2.8,2.9,3.6,5.7,3.3c1-0.1,2,0.1,3,0c2.8-0.5,6.4,1.7,8.1-2.7s-2.3-5.5-4.1-7.5\n",
" c-5.1-5.7-10.9-10.8-16.1-16.3C84,38,81.9,37.1,78,38.3C66.7,42,56.2,35.7,53,24.1C50.3,14,57.3,2.8,67.7,0.5\n",
" C78.4-2,89,4.7,91.5,15.3c0.1,0.3,0.1,0.5,0.2,0.8c0.7,3.4,0.7,6.9-0.8,9.8c-1.7,3.2-0.8,5,1.5,7.2c6.7,6.5,13.3,13,19.8,19.7\n",
" c1.8,1.8,3,2.1,5.5,1.2c9.1-3.4,17.9-0.6,23.4,7c4.8,6.9,4.6,16.1-0.4,22.9c-5.4,7.2-14.2,9.9-23.1,6.5c-2.3-0.9-3.5-0.6-5.1,1.1\n",
" c-6.7,6.9-13.6,13.7-20.5,20.4c-1.8,1.8-2.5,3.2-1.4,5.9c3.5,8.7,0.3,18.6-7.7,23.6c-7.9,5-18.2,3.8-24.8-2.9\n",
" c-6.4-6.4-7.4-16.2-2.5-24.3c4.9-7.8,14.5-11,23.1-7.8c3,1.1,4.7,0.5,6.9-1.7C91.7,98.4,98,92.3,104.2,86c1.6-1.6,4.1-2.7,2.6-6.2\n",
" c-1.4-3.3-3.8-2.5-6.2-2.6C99.8,77.2,98.9,77.2,97.3,77.2z M72.1,29.7c5.5,0.1,9.9-4.3,10-9.8c0-0.1,0-0.2,0-0.3\n",
" C81.8,14,77,9.8,71.5,10.2c-5,0.3-9,4.2-9.3,9.2c-0.2,5.5,4,10.1,9.5,10.3C71.8,29.7,72,29.7,72.1,29.7z M72.3,62.3\n",
" c-5.4-0.1-9.9,4.2-10.1,9.7c0,0.2,0,0.3,0,0.5c0.2,5.4,4.5,9.7,9.9,10c5.1,0.1,9.9-4.7,10.1-9.8c0.2-5.5-4-10-9.5-10.3\n",
" C72.6,62.3,72.4,62.3,72.3,62.3z M115,72.5c0.1,5.4,4.5,9.7,9.8,9.9c5.6-0.2,10-4.8,10-10.4c-0.2-5.4-4.6-9.7-10-9.7\n",
" c-5.3-0.1-9.8,4.2-9.9,9.5C115,72.1,115,72.3,115,72.5z M19.5,62.3c-5.4,0.1-9.8,4.4-10,9.8c-0.1,5.1,5.2,10.4,10.2,10.3\n",
" c5.6-0.2,10-4.9,9.8-10.5c-0.1-5.4-4.5-9.7-9.9-9.6C19.6,62.3,19.5,62.3,19.5,62.3z M71.8,134.6c5.9,0.2,10.3-3.9,10.4-9.6\n",
" c0.5-5.5-3.6-10.4-9.1-10.8c-5.5-0.5-10.4,3.6-10.8,9.1c0,0.5,0,0.9,0,1.4c-0.2,5.3,4,9.8,9.3,10\n",
" C71.6,134.6,71.7,134.6,71.8,134.6z\"/>\n",
" </g>\n",
" </svg>\n",
" <table>\n",
" <tr>\n",
" <td style=\"text-align: left\"><b>Python version:</b></td>\n",
" <td style=\"text-align: left\"><b>3.7.7</b></td>\n",
" </tr>\n",
" <tr>\n",
" <td style=\"text-align: left\"><b>Ray version:</b></td>\n",
" <td style=\"text-align: left\"><b> 3.0.0.dev0</b></td>\n",
" </tr>\n",
" <tr>\n",
" <td style=\"text-align: left\"><b>Dashboard:</b></td>\n",
" <td style=\"text-align: left\"><b><a href=\"http://127.0.0.1:8265\" target=\"_blank\">http://127.0.0.1:8265</a></b></td>\n",
"</tr>\n",
"\n",
" </table>\n",
" </div>\n",
"</div>\n"
],
"text/plain": [
"RayContext(dashboard_url='127.0.0.1:8265', python_version='3.7.7', ray_version='3.0.0.dev0', ray_commit='{{RAY_COMMIT_SHA}}', address_info={'node_ip_address': '127.0.0.1', 'raylet_ip_address': '127.0.0.1', 'redis_address': None, 'object_store_address': '/tmp/ray/session_2022-07-22_15-21-42_643152_46061/sockets/plasma_store', 'raylet_socket_name': '/tmp/ray/session_2022-07-22_15-21-42_643152_46061/sockets/raylet', 'webui_url': '127.0.0.1:8265', 'session_dir': '/tmp/ray/session_2022-07-22_15-21-42_643152_46061', 'metrics_export_port': 60351, 'gcs_address': '127.0.0.1:65028', 'address': '127.0.0.1:65028', 'dashboard_agent_listen_port': 52365, 'node_id': 'b7abfa8f9f137b294759a2e68a71edb9920c011a97e390fd1e0b71bf'})"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
2022-03-25 01:04:53 -07:00
"source": [
"ray.init(configure_logging=False)"
]
},
{
"cell_type": "markdown",
2022-04-25 11:10:58 -07:00
"id": "56b0c685",
2022-03-25 01:04:53 -07:00
"metadata": {},
"source": [
"Let's start by defining a simple evaluation function.\n",
"An explicit math formula is queried here for demonstration, yet in practice this is typically a black-box function-- e.g. the performance results after training an ML model.\n",
"We artificially sleep for a bit (`0.1` seconds) to simulate a long-running ML experiment.\n",
"This setup assumes that we're running multiple `step`s of an experiment while tuning three hyperparameters,\n",
"namely `width`, `height`, and `activation`."
]
},
{
"cell_type": "code",
2022-07-24 18:53:57 +01:00
"execution_count": 4,
2022-04-25 11:10:58 -07:00
"id": "90a11f98",
2022-03-25 01:04:53 -07:00
"metadata": {},
"outputs": [],
"source": [
"def evaluate(step, width, height, activation):\n",
" time.sleep(0.1)\n",
" activation_boost = 10 if activation==\"relu\" else 0\n",
" return (0.1 + width * step / 100) ** (-1) + height * 0.1 + activation_boost"
]
},
{
"cell_type": "markdown",
2022-04-25 11:10:58 -07:00
"id": "bc579b83",
2022-03-25 01:04:53 -07:00
"metadata": {},
"source": [
2022-04-25 11:10:58 -07:00
"Next, our `objective` function to be optimized takes a Tune `config`, evaluates the `score` of your experiment in a training loop,\n",
2022-06-30 10:37:31 -07:00
"and uses `session.report` to report the `score` back to Tune."
2022-03-25 01:04:53 -07:00
]
},
{
"cell_type": "code",
2022-07-24 18:53:57 +01:00
"execution_count": 5,
2022-04-25 11:10:58 -07:00
"id": "3a11d0e0",
2022-03-25 01:04:53 -07:00
"metadata": {
"lines_to_next_cell": 0
},
"outputs": [],
"source": [
"def objective(config):\n",
" for step in range(config[\"steps\"]):\n",
" score = evaluate(step, config[\"width\"], config[\"height\"], config[\"activation\"])\n",
2022-06-30 10:37:31 -07:00
" session.report({\"iterations\": step, \"mean_loss\": score})\n",
" "
2022-03-25 01:04:53 -07:00
]
},
{
"cell_type": "markdown",
2022-04-25 11:10:58 -07:00
"id": "c58bd20b",
2022-03-25 01:04:53 -07:00
"metadata": {},
"source": [
"Next we define a search space. The critical assumption is that the optimal hyperparamters live within this space. Yet, if the space is very large, then those hyperparamters may be difficult to find in a short amount of time.\n",
"\n",
"The simplest case is a search space with independent dimensions. In this case, a config dictionary will suffice."
]
},
{
"cell_type": "code",
2022-07-24 18:53:57 +01:00
"execution_count": 6,
2022-04-25 11:10:58 -07:00
"id": "c3e4eecb",
2022-03-25 01:04:53 -07:00
"metadata": {},
"outputs": [],
"source": [
"search_space = {\n",
" \"steps\": 100,\n",
" \"width\": tune.uniform(0, 20),\n",
" \"height\": tune.uniform(-100, 100),\n",
" \"activation\": tune.choice([\"relu\", \"tanh\"]),\n",
"}"
]
},
{
"cell_type": "markdown",
2022-04-25 11:10:58 -07:00
"id": "ef0c666d",
2022-03-25 01:04:53 -07:00
"metadata": {},
"source": [
"Here we define the Optuna search algorithm:"
]
},
{
"cell_type": "code",
2022-07-24 18:53:57 +01:00
"execution_count": 7,
2022-04-25 11:10:58 -07:00
"id": "f23cadc8",
2022-03-25 01:04:53 -07:00
"metadata": {},
"outputs": [],
"source": [
2022-04-25 11:10:58 -07:00
"algo = OptunaSearch()"
2022-03-25 01:04:53 -07:00
]
},
{
"cell_type": "markdown",
2022-04-25 11:10:58 -07:00
"id": "4287fa79",
2022-03-25 01:04:53 -07:00
"metadata": {},
"source": [
"We also constrain the the number of concurrent trials to `4` with a `ConcurrencyLimiter`."
]
},
{
"cell_type": "code",
2022-07-24 18:53:57 +01:00
"execution_count": 8,
2022-04-25 11:10:58 -07:00
"id": "68022ea4",
2022-03-25 01:04:53 -07:00
"metadata": {},
"outputs": [],
"source": [
2022-04-25 11:10:58 -07:00
"algo = ConcurrencyLimiter(algo, max_concurrent=4)\n"
2022-03-25 01:04:53 -07:00
]
},
{
"cell_type": "markdown",
2022-04-25 11:10:58 -07:00
"id": "4c250f74",
2022-03-25 01:04:53 -07:00
"metadata": {},
"source": [
2022-04-25 11:10:58 -07:00
"The number of samples is the number of hyperparameter combinations that will be tried out. This Tune run is set to `1000` samples.\n",
2022-03-25 01:04:53 -07:00
"(you can decrease this if it takes too long on your machine)."
]
},
{
"cell_type": "code",
2022-07-24 18:53:57 +01:00
"execution_count": 9,
2022-04-25 11:10:58 -07:00
"id": "f6c21314",
2022-03-25 01:04:53 -07:00
"metadata": {},
"outputs": [],
"source": [
"num_samples = 1000"
]
},
{
"cell_type": "code",
2022-07-24 18:53:57 +01:00
"execution_count": 10,
2022-04-25 11:10:58 -07:00
"id": "9533aabf",
2022-03-25 01:04:53 -07:00
"metadata": {
"tags": [
"remove-cell"
]
},
"outputs": [],
"source": [
"# We override here for our smoke tests.\n",
"num_samples = 10"
]
},
{
"cell_type": "markdown",
2022-04-25 11:10:58 -07:00
"id": "92942b88",
2022-03-25 01:04:53 -07:00
"metadata": {},
"source": [
2022-07-24 18:53:57 +01:00
"Finally, we run the experiment to `\"min\"`imize the \"mean_loss\" of the `objective` by searching `search_space` via `algo`, `num_samples` times. This previous sentence is fully characterizes the search problem we aim to solve. With this in mind, notice how efficient it is to execute `tuner.fit()`."
2022-03-25 01:04:53 -07:00
]
},
{
"cell_type": "code",
2022-07-24 18:53:57 +01:00
"execution_count": 11,
2022-04-25 11:10:58 -07:00
"id": "4e224bb2",
2022-03-25 01:04:53 -07:00
"metadata": {},
2022-07-24 18:53:57 +01:00
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Function checkpointing is disabled. This may result in unexpected behavior when using checkpointing features or certain schedulers. To enable, set the train function arguments to be `func(config, checkpoint_dir=None)`.\n",
"\u001b[32m[I 2022-07-22 15:21:47,769]\u001b[0m A new study created in memory with name: optuna\u001b[0m\n"
]
},
{
"data": {
"text/html": [
"== Status ==<br>Current time: 2022-07-22 15:22:32 (running for 00:00:43.89)<br>Memory usage on this node: 10.1/16.0 GiB<br>Using FIFO scheduling algorithm.<br>Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/5.07 GiB heap, 0.0/2.0 GiB objects<br>Current best trial: b1bad2fe with mean_loss=-1.5166144695235813 and parameters={'steps': 100, 'width': 7.688465886501621, 'height': -16.46289560854555, 'activation': 'tanh'}<br>Result logdir: /Users/kai/ray_results/optuna_example<br>Number of trials: 10/10 (10 TERMINATED)<br><table>\n",
"<thead>\n",
"<tr><th>Trial name </th><th>status </th><th>loc </th><th>activation </th><th style=\"text-align: right;\"> height</th><th style=\"text-align: right;\"> width</th><th style=\"text-align: right;\"> loss</th><th style=\"text-align: right;\"> iter</th><th style=\"text-align: right;\"> total time (s)</th><th style=\"text-align: right;\"> iterations</th><th style=\"text-align: right;\"> neg_mean_loss</th></tr>\n",
"</thead>\n",
"<tbody>\n",
"<tr><td>objective_9f689668</td><td>TERMINATED</td><td>127.0.0.1:46141</td><td>tanh </td><td style=\"text-align: right;\"> 32.9806 </td><td style=\"text-align: right;\">16.0406 </td><td style=\"text-align: right;\"> 3.36064 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 11.0409</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\"> -3.36064 </td></tr>\n",
"<tr><td>objective_a11d2104</td><td>TERMINATED</td><td>127.0.0.1:46149</td><td>relu </td><td style=\"text-align: right;\"> 72.2627 </td><td style=\"text-align: right;\"> 0.763881</td><td style=\"text-align: right;\">18.3942 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 11.7018</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\"> -18.3942 </td></tr>\n",
"<tr><td>objective_a11eef8e</td><td>TERMINATED</td><td>127.0.0.1:46150</td><td>relu </td><td style=\"text-align: right;\">-32.8474 </td><td style=\"text-align: right;\">13.1144 </td><td style=\"text-align: right;\"> 6.79169 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 11.8071</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\"> -6.79169 </td></tr>\n",
"<tr><td>objective_a1209e56</td><td>TERMINATED</td><td>127.0.0.1:46151</td><td>tanh </td><td style=\"text-align: right;\"> 75.6408 </td><td style=\"text-align: right;\">10.4415 </td><td style=\"text-align: right;\"> 7.65989 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 11.6987</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\"> -7.65989 </td></tr>\n",
"<tr><td>objective_a7b383e6</td><td>TERMINATED</td><td>127.0.0.1:46174</td><td>relu </td><td style=\"text-align: right;\"> 50.1501 </td><td style=\"text-align: right;\"> 3.4612 </td><td style=\"text-align: right;\">15.2986 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 10.7015</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\"> -15.2986 </td></tr>\n",
"<tr><td>objective_a9c844c8</td><td>TERMINATED</td><td>127.0.0.1:46183</td><td>relu </td><td style=\"text-align: right;\">-40.3931 </td><td style=\"text-align: right;\">14.0525 </td><td style=\"text-align: right;\"> 6.03205 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 10.6687</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\"> -6.03205 </td></tr>\n",
"<tr><td>objective_a9cb308e</td><td>TERMINATED</td><td>127.0.0.1:46184</td><td>tanh </td><td style=\"text-align: right;\"> 48.4802 </td><td style=\"text-align: right;\">12.7746 </td><td style=\"text-align: right;\"> 4.92647 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 10.7122</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\"> -4.92647 </td></tr>\n",
"<tr><td>objective_a9d8332e</td><td>TERMINATED</td><td>127.0.0.1:46189</td><td>tanh </td><td style=\"text-align: right;\"> -1.26682</td><td style=\"text-align: right;\">10.9788 </td><td style=\"text-align: right;\">-0.0355157</td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 10.7089</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\"> 0.0355157</td></tr>\n",
"<tr><td>objective_af948a1a</td><td>TERMINATED</td><td>127.0.0.1:46204</td><td>tanh </td><td style=\"text-align: right;\"> 61.6381 </td><td style=\"text-align: right;\">11.0105 </td><td style=\"text-align: right;\"> 6.25472 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 10.744 </td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\"> -6.25472 </td></tr>\n",
"<tr><td>objective_b1bad2fe</td><td>TERMINATED</td><td>127.0.0.1:46209</td><td>tanh </td><td style=\"text-align: right;\">-16.4629 </td><td style=\"text-align: right;\"> 7.68847 </td><td style=\"text-align: right;\">-1.51661 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 10.7274</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\"> 1.51661 </td></tr>\n",
"</tbody>\n",
"</table><br><br>"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Result for objective_9f689668:\n",
" date: 2022-07-22_15-21-51\n",
" done: false\n",
" experiment_id: 7d29e5c95aa44d77becc24b23a326b9b\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 13.298063134872326\n",
" neg_mean_loss: -13.298063134872326\n",
" node_ip: 127.0.0.1\n",
" pid: 46141\n",
" time_since_restore: 0.10468196868896484\n",
" time_this_iter_s: 0.10468196868896484\n",
" time_total_s: 0.10468196868896484\n",
" timestamp: 1658499711\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: 9f689668\n",
" warmup_time: 0.003930091857910156\n",
" \n",
"Result for objective_a1209e56:\n",
" date: 2022-07-22_15-21-54\n",
" done: false\n",
" experiment_id: 70d9788c327c43fb94450e5084416b81\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 17.564083106485064\n",
" neg_mean_loss: -17.564083106485064\n",
" node_ip: 127.0.0.1\n",
" pid: 46151\n",
" time_since_restore: 0.10416483879089355\n",
" time_this_iter_s: 0.10416483879089355\n",
" time_total_s: 0.10416483879089355\n",
" timestamp: 1658499714\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: a1209e56\n",
" warmup_time: 0.0026938915252685547\n",
" \n",
"Result for objective_a11d2104:\n",
" date: 2022-07-22_15-21-54\n",
" done: false\n",
" experiment_id: af2598364a004cd28f37ede19afd1ffb\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 27.2262748487915\n",
" neg_mean_loss: -27.2262748487915\n",
" node_ip: 127.0.0.1\n",
" pid: 46149\n",
" time_since_restore: 0.10507392883300781\n",
" time_this_iter_s: 0.10507392883300781\n",
" time_total_s: 0.10507392883300781\n",
" timestamp: 1658499714\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: a11d2104\n",
" warmup_time: 0.0031232833862304688\n",
" \n",
"Result for objective_a11eef8e:\n",
" date: 2022-07-22_15-21-54\n",
" done: false\n",
" experiment_id: e4a5922603d44b459c23607a42d5e574\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 16.71525618736072\n",
" neg_mean_loss: -16.71525618736072\n",
" node_ip: 127.0.0.1\n",
" pid: 46150\n",
" time_since_restore: 0.10363006591796875\n",
" time_this_iter_s: 0.10363006591796875\n",
" time_total_s: 0.10363006591796875\n",
" timestamp: 1658499714\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: a11eef8e\n",
" warmup_time: 0.0028891563415527344\n",
" \n",
"Result for objective_9f689668:\n",
" date: 2022-07-22_15-21-56\n",
" done: false\n",
" experiment_id: 7d29e5c95aa44d77becc24b23a326b9b\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 45\n",
" iterations_since_restore: 46\n",
" mean_loss: 3.434707064844201\n",
" neg_mean_loss: -3.434707064844201\n",
" node_ip: 127.0.0.1\n",
" pid: 46141\n",
" time_since_restore: 5.202661752700806\n",
" time_this_iter_s: 0.10527586936950684\n",
" time_total_s: 5.202661752700806\n",
" timestamp: 1658499716\n",
" timesteps_since_restore: 0\n",
" training_iteration: 46\n",
" trial_id: 9f689668\n",
" warmup_time: 0.003930091857910156\n",
" \n",
"Result for objective_a1209e56:\n",
" date: 2022-07-22_15-21-59\n",
" done: false\n",
" experiment_id: 70d9788c327c43fb94450e5084416b81\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" mean_loss: 7.763782433167051\n",
" neg_mean_loss: -7.763782433167051\n",
" node_ip: 127.0.0.1\n",
" pid: 46151\n",
" time_since_restore: 5.146381139755249\n",
" time_this_iter_s: 0.10691022872924805\n",
" time_total_s: 5.146381139755249\n",
" timestamp: 1658499719\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: a1209e56\n",
" warmup_time: 0.0026938915252685547\n",
" \n",
"Result for objective_a11eef8e:\n",
" date: 2022-07-22_15-21-59\n",
" done: false\n",
" experiment_id: e4a5922603d44b459c23607a42d5e574\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" mean_loss: 6.874905099790281\n",
" neg_mean_loss: -6.874905099790281\n",
" node_ip: 127.0.0.1\n",
" pid: 46150\n",
" time_since_restore: 5.1618921756744385\n",
" time_this_iter_s: 0.10776615142822266\n",
" time_total_s: 5.1618921756744385\n",
" timestamp: 1658499719\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: a11eef8e\n",
" warmup_time: 0.0028891563415527344\n",
" \n",
"Result for objective_a11d2104:\n",
" date: 2022-07-22_15-21-59\n",
" done: false\n",
" experiment_id: af2598364a004cd28f37ede19afd1ffb\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" mean_loss: 19.4048102011043\n",
" neg_mean_loss: -19.4048102011043\n",
" node_ip: 127.0.0.1\n",
" pid: 46149\n",
" time_since_restore: 5.170748233795166\n",
" time_this_iter_s: 0.11556220054626465\n",
" time_total_s: 5.170748233795166\n",
" timestamp: 1658499719\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: a11d2104\n",
" warmup_time: 0.0031232833862304688\n",
" \n",
"Result for objective_9f689668:\n",
" date: 2022-07-22_15-22-01\n",
" done: false\n",
" experiment_id: 7d29e5c95aa44d77becc24b23a326b9b\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 92\n",
" iterations_since_restore: 93\n",
" mean_loss: 3.3653696881211697\n",
" neg_mean_loss: -3.3653696881211697\n",
" node_ip: 127.0.0.1\n",
" pid: 46141\n",
" time_since_restore: 10.248206853866577\n",
" time_this_iter_s: 0.10745501518249512\n",
" time_total_s: 10.248206853866577\n",
" timestamp: 1658499721\n",
" timesteps_since_restore: 0\n",
" training_iteration: 93\n",
" trial_id: 9f689668\n",
" warmup_time: 0.003930091857910156\n",
" \n",
"Result for objective_9f689668:\n",
" date: 2022-07-22_15-22-02\n",
" done: true\n",
" experiment_id: 7d29e5c95aa44d77becc24b23a326b9b\n",
" experiment_tag: 1_activation=tanh,height=32.9806,steps=100,width=16.0406\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: 3.3606404197394255\n",
" neg_mean_loss: -3.3606404197394255\n",
" node_ip: 127.0.0.1\n",
" pid: 46141\n",
" time_since_restore: 11.04089903831482\n",
" time_this_iter_s: 0.10824704170227051\n",
" time_total_s: 11.04089903831482\n",
" timestamp: 1658499722\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: 9f689668\n",
" warmup_time: 0.003930091857910156\n",
" \n",
"Result for objective_a1209e56:\n",
" date: 2022-07-22_15-22-04\n",
" done: false\n",
" experiment_id: 70d9788c327c43fb94450e5084416b81\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 90\n",
" iterations_since_restore: 91\n",
" mean_loss: 7.669375143981996\n",
" neg_mean_loss: -7.669375143981996\n",
" node_ip: 127.0.0.1\n",
" pid: 46151\n",
" time_since_restore: 9.724387168884277\n",
" time_this_iter_s: 0.10670638084411621\n",
" time_total_s: 9.724387168884277\n",
" timestamp: 1658499724\n",
" timesteps_since_restore: 0\n",
" training_iteration: 91\n",
" trial_id: a1209e56\n",
" warmup_time: 0.0026938915252685547\n",
" \n",
"Result for objective_a11d2104:\n",
" date: 2022-07-22_15-22-04\n",
" done: false\n",
" experiment_id: af2598364a004cd28f37ede19afd1ffb\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 90\n",
" iterations_since_restore: 91\n",
" mean_loss: 18.49612779999574\n",
" neg_mean_loss: -18.49612779999574\n",
" node_ip: 127.0.0.1\n",
" pid: 46149\n",
" time_since_restore: 9.818917274475098\n",
" time_this_iter_s: 0.10725140571594238\n",
" time_total_s: 9.818917274475098\n",
" timestamp: 1658499724\n",
" timesteps_since_restore: 0\n",
" training_iteration: 91\n",
" trial_id: a11d2104\n",
" warmup_time: 0.0031232833862304688\n",
" \n",
"Result for objective_a11eef8e:\n",
" date: 2022-07-22_15-22-03\n",
" done: false\n",
" experiment_id: e4a5922603d44b459c23607a42d5e574\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 89\n",
" iterations_since_restore: 90\n",
" mean_loss: 6.800205168670805\n",
" neg_mean_loss: -6.800205168670805\n",
" node_ip: 127.0.0.1\n",
" pid: 46150\n",
" time_since_restore: 9.714852094650269\n",
" time_this_iter_s: 0.10286402702331543\n",
" time_total_s: 9.714852094650269\n",
" timestamp: 1658499723\n",
" timesteps_since_restore: 0\n",
" training_iteration: 90\n",
" trial_id: a11eef8e\n",
" warmup_time: 0.0028891563415527344\n",
" \n",
"Result for objective_a7b383e6:\n",
" date: 2022-07-22_15-22-05\n",
" done: false\n",
" experiment_id: d2b2d4c7590142b3bdd660dcf6c947aa\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 25.015010793208496\n",
" neg_mean_loss: -25.015010793208496\n",
" node_ip: 127.0.0.1\n",
" pid: 46174\n",
" time_since_restore: 0.10384607315063477\n",
" time_this_iter_s: 0.10384607315063477\n",
" time_total_s: 0.10384607315063477\n",
" timestamp: 1658499725\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: a7b383e6\n",
" warmup_time: 0.002817869186401367\n",
" \n",
"Result for objective_a1209e56:\n",
" date: 2022-07-22_15-22-05\n",
" done: true\n",
" experiment_id: 70d9788c327c43fb94450e5084416b81\n",
" experiment_tag: 4_activation=tanh,height=75.6408,steps=100,width=10.4415\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: 7.659894851608276\n",
" neg_mean_loss: -7.659894851608276\n",
" node_ip: 127.0.0.1\n",
" pid: 46151\n",
" time_since_restore: 11.698678970336914\n",
" time_this_iter_s: 0.10874581336975098\n",
" time_total_s: 11.698678970336914\n",
" timestamp: 1658499725\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: a1209e56\n",
" warmup_time: 0.0026938915252685547\n",
" \n",
"Result for objective_a11d2104:\n",
" date: 2022-07-22_15-22-05\n",
" done: true\n",
" experiment_id: af2598364a004cd28f37ede19afd1ffb\n",
" experiment_tag: 2_activation=relu,height=72.2627,steps=100,width=0.7639\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: 18.394168929385685\n",
" neg_mean_loss: -18.394168929385685\n",
" node_ip: 127.0.0.1\n",
" pid: 46149\n",
" time_since_restore: 11.701792001724243\n",
" time_this_iter_s: 0.10734391212463379\n",
" time_total_s: 11.701792001724243\n",
" timestamp: 1658499725\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: a11d2104\n",
" warmup_time: 0.0031232833862304688\n",
" \n",
"Result for objective_a11eef8e:\n",
" date: 2022-07-22_15-22-06\n",
" done: true\n",
" experiment_id: e4a5922603d44b459c23607a42d5e574\n",
" experiment_tag: 3_activation=relu,height=-32.8474,steps=100,width=13.1144\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: 6.791690049131352\n",
" neg_mean_loss: -6.791690049131352\n",
" node_ip: 127.0.0.1\n",
" pid: 46150\n",
" time_since_restore: 11.807135105133057\n",
" time_this_iter_s: 0.10672497749328613\n",
" time_total_s: 11.807135105133057\n",
" timestamp: 1658499726\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: a11eef8e\n",
" warmup_time: 0.0028891563415527344\n",
" \n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Result for objective_a9cb308e:\n",
" date: 2022-07-22_15-22-08\n",
" done: false\n",
" experiment_id: fcc8b8f4880c471c8e8545228d7e591a\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 14.84802392036395\n",
" neg_mean_loss: -14.84802392036395\n",
" node_ip: 127.0.0.1\n",
" pid: 46184\n",
" time_since_restore: 0.1039888858795166\n",
" time_this_iter_s: 0.1039888858795166\n",
" time_total_s: 0.1039888858795166\n",
" timestamp: 1658499728\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: a9cb308e\n",
" warmup_time: 0.0027310848236083984\n",
" \n",
"Result for objective_a9c844c8:\n",
" date: 2022-07-22_15-22-08\n",
" done: false\n",
" experiment_id: 3ef3f95b10fb4a65a21c80130f69ed3e\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 15.960686077942514\n",
" neg_mean_loss: -15.960686077942514\n",
" node_ip: 127.0.0.1\n",
" pid: 46183\n",
" time_since_restore: 0.1004641056060791\n",
" time_this_iter_s: 0.1004641056060791\n",
" time_total_s: 0.1004641056060791\n",
" timestamp: 1658499728\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: a9c844c8\n",
" warmup_time: 0.002599954605102539\n",
" \n",
"Result for objective_a9d8332e:\n",
" date: 2022-07-22_15-22-08\n",
" done: false\n",
" experiment_id: 9c2f8aa7e4ed45c8867025673b146195\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 9.873318211243287\n",
" neg_mean_loss: -9.873318211243287\n",
" node_ip: 127.0.0.1\n",
" pid: 46189\n",
" time_since_restore: 0.10463905334472656\n",
" time_this_iter_s: 0.10463905334472656\n",
" time_total_s: 0.10463905334472656\n",
" timestamp: 1658499728\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: a9d8332e\n",
" warmup_time: 0.002753019332885742\n",
" \n",
"Result for objective_a7b383e6:\n",
" date: 2022-07-22_15-22-10\n",
" done: false\n",
" experiment_id: d2b2d4c7590142b3bdd660dcf6c947aa\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" mean_loss: 15.594128340598196\n",
" neg_mean_loss: -15.594128340598196\n",
" node_ip: 127.0.0.1\n",
" pid: 46174\n",
" time_since_restore: 5.12034797668457\n",
" time_this_iter_s: 0.10577988624572754\n",
" time_total_s: 5.12034797668457\n",
" timestamp: 1658499730\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: a7b383e6\n",
" warmup_time: 0.002817869186401367\n",
" \n",
"Result for objective_a9cb308e:\n",
" date: 2022-07-22_15-22-13\n",
" done: false\n",
" experiment_id: fcc8b8f4880c471c8e8545228d7e591a\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" mean_loss: 5.011849709111575\n",
" neg_mean_loss: -5.011849709111575\n",
" node_ip: 127.0.0.1\n",
" pid: 46184\n",
" time_since_restore: 5.150436162948608\n",
" time_this_iter_s: 0.10827016830444336\n",
" time_total_s: 5.150436162948608\n",
" timestamp: 1658499733\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: a9cb308e\n",
" warmup_time: 0.0027310848236083984\n",
" \n",
"Result for objective_a9c844c8:\n",
" date: 2022-07-22_15-22-13\n",
" done: false\n",
" experiment_id: 3ef3f95b10fb4a65a21c80130f69ed3e\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" mean_loss: 6.109835222238134\n",
" neg_mean_loss: -6.109835222238134\n",
" node_ip: 127.0.0.1\n",
" pid: 46183\n",
" time_since_restore: 5.136260032653809\n",
" time_this_iter_s: 0.10730218887329102\n",
" time_total_s: 5.136260032653809\n",
" timestamp: 1658499733\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: a9c844c8\n",
" warmup_time: 0.002599954605102539\n",
" \n",
"Result for objective_a9d8332e:\n",
" date: 2022-07-22_15-22-13\n",
" done: false\n",
" experiment_id: 9c2f8aa7e4ed45c8867025673b146195\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" mean_loss: 0.06343129971586822\n",
" neg_mean_loss: -0.06343129971586822\n",
" node_ip: 127.0.0.1\n",
" pid: 46189\n",
" time_since_restore: 5.141700983047485\n",
" time_this_iter_s: 0.1074991226196289\n",
" time_total_s: 5.141700983047485\n",
" timestamp: 1658499733\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: a9d8332e\n",
" warmup_time: 0.002753019332885742\n",
" \n",
"Result for objective_a7b383e6:\n",
" date: 2022-07-22_15-22-15\n",
" done: false\n",
" experiment_id: d2b2d4c7590142b3bdd660dcf6c947aa\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 94\n",
" iterations_since_restore: 95\n",
" mean_loss: 15.313204013214245\n",
" neg_mean_loss: -15.313204013214245\n",
" node_ip: 127.0.0.1\n",
" pid: 46174\n",
" time_since_restore: 10.163710832595825\n",
" time_this_iter_s: 0.10754895210266113\n",
" time_total_s: 10.163710832595825\n",
" timestamp: 1658499735\n",
" timesteps_since_restore: 0\n",
" training_iteration: 95\n",
" trial_id: a7b383e6\n",
" warmup_time: 0.002817869186401367\n",
" \n",
"Result for objective_a7b383e6:\n",
" date: 2022-07-22_15-22-15\n",
" done: true\n",
" experiment_id: d2b2d4c7590142b3bdd660dcf6c947aa\n",
" experiment_tag: 5_activation=relu,height=50.1501,steps=100,width=3.4612\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: 15.298570798421121\n",
" neg_mean_loss: -15.298570798421121\n",
" node_ip: 127.0.0.1\n",
" pid: 46174\n",
" time_since_restore: 10.701509952545166\n",
" time_this_iter_s: 0.10684585571289062\n",
" time_total_s: 10.701509952545166\n",
" timestamp: 1658499735\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: a7b383e6\n",
" warmup_time: 0.002817869186401367\n",
" \n",
"Result for objective_af948a1a:\n",
" date: 2022-07-22_15-22-18\n",
" done: false\n",
" experiment_id: d27f58d6527a4d47ae4c12496f5275cf\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 16.16381214127525\n",
" neg_mean_loss: -16.16381214127525\n",
" node_ip: 127.0.0.1\n",
" pid: 46204\n",
" time_since_restore: 0.10407209396362305\n",
" time_this_iter_s: 0.10407209396362305\n",
" time_total_s: 0.10407209396362305\n",
" timestamp: 1658499738\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: af948a1a\n",
" warmup_time: 0.002669095993041992\n",
" \n",
"Result for objective_a9cb308e:\n",
" date: 2022-07-22_15-22-18\n",
" done: false\n",
" experiment_id: fcc8b8f4880c471c8e8545228d7e591a\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 94\n",
" iterations_since_restore: 95\n",
" mean_loss: 4.930613328484136\n",
" neg_mean_loss: -4.930613328484136\n",
" node_ip: 127.0.0.1\n",
" pid: 46184\n",
" time_since_restore: 10.175731897354126\n",
" time_this_iter_s: 0.10618400573730469\n",
" time_total_s: 10.175731897354126\n",
" timestamp: 1658499738\n",
" timesteps_since_restore: 0\n",
" training_iteration: 95\n",
" trial_id: a9cb308e\n",
" warmup_time: 0.0027310848236083984\n",
" \n",
"Result for objective_a9d8332e:\n",
" date: 2022-07-22_15-22-18\n",
" done: false\n",
" experiment_id: 9c2f8aa7e4ed45c8867025673b146195\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 94\n",
" iterations_since_restore: 95\n",
" mean_loss: -0.030712998363490673\n",
" neg_mean_loss: 0.030712998363490673\n",
" node_ip: 127.0.0.1\n",
" pid: 46189\n",
" time_since_restore: 10.171820163726807\n",
" time_this_iter_s: 0.10807609558105469\n",
" time_total_s: 10.171820163726807\n",
" timestamp: 1658499738\n",
" timesteps_since_restore: 0\n",
" training_iteration: 95\n",
" trial_id: a9d8332e\n",
" warmup_time: 0.002753019332885742\n",
" \n",
"Result for objective_a9c844c8:\n",
" date: 2022-07-22_15-22-18\n",
" done: false\n",
" experiment_id: 3ef3f95b10fb4a65a21c80130f69ed3e\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 95\n",
" iterations_since_restore: 96\n",
" mean_loss: 6.035035952030598\n",
" neg_mean_loss: -6.035035952030598\n",
" node_ip: 127.0.0.1\n",
" pid: 46183\n",
" time_since_restore: 10.23954701423645\n",
" time_this_iter_s: 0.10863399505615234\n",
" time_total_s: 10.23954701423645\n",
" timestamp: 1658499738\n",
" timesteps_since_restore: 0\n",
" training_iteration: 96\n",
" trial_id: a9c844c8\n",
" warmup_time: 0.002599954605102539\n",
" \n",
"Result for objective_a9cb308e:\n",
" date: 2022-07-22_15-22-19\n",
" done: true\n",
" experiment_id: fcc8b8f4880c471c8e8545228d7e591a\n",
" experiment_tag: 7_activation=tanh,height=48.4802,steps=100,width=12.7746\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: 4.926474869576359\n",
" neg_mean_loss: -4.926474869576359\n",
" node_ip: 127.0.0.1\n",
" pid: 46184\n",
" time_since_restore: 10.712197065353394\n",
" time_this_iter_s: 0.10726809501647949\n",
" time_total_s: 10.712197065353394\n",
" timestamp: 1658499739\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: a9cb308e\n",
" warmup_time: 0.0027310848236083984\n",
" \n",
"Result for objective_a9c844c8:\n",
" date: 2022-07-22_15-22-19\n",
" done: true\n",
" experiment_id: 3ef3f95b10fb4a65a21c80130f69ed3e\n",
" experiment_tag: 6_activation=relu,height=-40.3931,steps=100,width=14.0525\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: 6.03205335569619\n",
" neg_mean_loss: -6.03205335569619\n",
" node_ip: 127.0.0.1\n",
" pid: 46183\n",
" time_since_restore: 10.668684959411621\n",
" time_this_iter_s: 0.1074681282043457\n",
" time_total_s: 10.668684959411621\n",
" timestamp: 1658499739\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: a9c844c8\n",
" warmup_time: 0.002599954605102539\n",
" \n",
"Result for objective_a9d8332e:\n",
" date: 2022-07-22_15-22-19\n",
" done: true\n",
" experiment_id: 9c2f8aa7e4ed45c8867025673b146195\n",
" experiment_tag: 8_activation=tanh,height=-1.2668,steps=100,width=10.9788\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: -0.035515719608698634\n",
" neg_mean_loss: 0.035515719608698634\n",
" node_ip: 127.0.0.1\n",
" pid: 46189\n",
" time_since_restore: 10.708936929702759\n",
" time_this_iter_s: 0.10948991775512695\n",
" time_total_s: 10.708936929702759\n",
" timestamp: 1658499739\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: a9d8332e\n",
" warmup_time: 0.002753019332885742\n",
" \n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Result for objective_b1bad2fe:\n",
" date: 2022-07-22_15-22-21\n",
" done: false\n",
" experiment_id: f58c8e48e2324dd69c28f526bf96497c\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 8.353710439145445\n",
" neg_mean_loss: -8.353710439145445\n",
" node_ip: 127.0.0.1\n",
" pid: 46209\n",
" time_since_restore: 0.10369992256164551\n",
" time_this_iter_s: 0.10369992256164551\n",
" time_total_s: 0.10369992256164551\n",
" timestamp: 1658499741\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: b1bad2fe\n",
" warmup_time: 0.0026597976684570312\n",
" \n",
"Result for objective_af948a1a:\n",
" date: 2022-07-22_15-22-23\n",
" done: false\n",
" experiment_id: d27f58d6527a4d47ae4c12496f5275cf\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" mean_loss: 6.353387644558537\n",
" neg_mean_loss: -6.353387644558537\n",
" node_ip: 127.0.0.1\n",
" pid: 46204\n",
" time_since_restore: 5.161527872085571\n",
" time_this_iter_s: 0.10784792900085449\n",
" time_total_s: 5.161527872085571\n",
" timestamp: 1658499743\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: af948a1a\n",
" warmup_time: 0.002669095993041992\n",
" \n",
"Result for objective_b1bad2fe:\n",
" date: 2022-07-22_15-22-26\n",
" done: false\n",
" experiment_id: f58c8e48e2324dd69c28f526bf96497c\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" mean_loss: -1.3770075531258914\n",
" neg_mean_loss: 1.3770075531258914\n",
" node_ip: 127.0.0.1\n",
" pid: 46209\n",
" time_since_restore: 5.158898115158081\n",
" time_this_iter_s: 0.10701417922973633\n",
" time_total_s: 5.158898115158081\n",
" timestamp: 1658499746\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: b1bad2fe\n",
" warmup_time: 0.0026597976684570312\n",
" \n",
"Result for objective_af948a1a:\n",
" date: 2022-07-22_15-22-28\n",
" done: false\n",
" experiment_id: d27f58d6527a4d47ae4c12496f5275cf\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 94\n",
" iterations_since_restore: 95\n",
" mean_loss: 6.259506962612728\n",
" neg_mean_loss: -6.259506962612728\n",
" node_ip: 127.0.0.1\n",
" pid: 46204\n",
" time_since_restore: 10.20361876487732\n",
" time_this_iter_s: 0.10824179649353027\n",
" time_total_s: 10.20361876487732\n",
" timestamp: 1658499748\n",
" timesteps_since_restore: 0\n",
" training_iteration: 95\n",
" trial_id: af948a1a\n",
" warmup_time: 0.002669095993041992\n",
" \n",
"Result for objective_af948a1a:\n",
" date: 2022-07-22_15-22-28\n",
" done: true\n",
" experiment_id: d27f58d6527a4d47ae4c12496f5275cf\n",
" experiment_tag: 9_activation=tanh,height=61.6381,steps=100,width=11.0105\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: 6.254717826198094\n",
" neg_mean_loss: -6.254717826198094\n",
" node_ip: 127.0.0.1\n",
" pid: 46204\n",
" time_since_restore: 10.744001865386963\n",
" time_this_iter_s: 0.10536718368530273\n",
" time_total_s: 10.744001865386963\n",
" timestamp: 1658499748\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: af948a1a\n",
" warmup_time: 0.002669095993041992\n",
" \n",
"Result for objective_b1bad2fe:\n",
" date: 2022-07-22_15-22-31\n",
" done: false\n",
" experiment_id: f58c8e48e2324dd69c28f526bf96497c\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 94\n",
" iterations_since_restore: 95\n",
" mean_loss: -1.5098109958909571\n",
" neg_mean_loss: 1.5098109958909571\n",
" node_ip: 127.0.0.1\n",
" pid: 46209\n",
" time_since_restore: 10.190011024475098\n",
" time_this_iter_s: 0.10639405250549316\n",
" time_total_s: 10.190011024475098\n",
" timestamp: 1658499751\n",
" timesteps_since_restore: 0\n",
" training_iteration: 95\n",
" trial_id: b1bad2fe\n",
" warmup_time: 0.0026597976684570312\n",
" \n",
"Result for objective_b1bad2fe:\n",
" date: 2022-07-22_15-22-32\n",
" done: true\n",
" experiment_id: f58c8e48e2324dd69c28f526bf96497c\n",
" experiment_tag: 10_activation=tanh,height=-16.4629,steps=100,width=7.6885\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: -1.5166144695235813\n",
" neg_mean_loss: 1.5166144695235813\n",
" node_ip: 127.0.0.1\n",
" pid: 46209\n",
" time_since_restore: 10.727396011352539\n",
" time_this_iter_s: 0.10742807388305664\n",
" time_total_s: 10.727396011352539\n",
" timestamp: 1658499752\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: b1bad2fe\n",
" warmup_time: 0.0026597976684570312\n",
" \n"
]
}
],
2022-03-25 01:04:53 -07:00
"source": [
2022-07-24 18:53:57 +01:00
"tuner = tune.Tuner(\n",
2022-04-25 11:10:58 -07:00
" objective,\n",
2022-07-24 18:53:57 +01:00
" tune_config=tune.TuneConfig(\n",
" metric=\"mean_loss\",\n",
" mode=\"min\",\n",
" search_alg=algo,\n",
" num_samples=num_samples,\n",
" ),\n",
" param_space=search_space,\n",
")\n",
"results = tuner.fit()"
2022-03-25 01:04:53 -07:00
]
},
{
"cell_type": "markdown",
2022-04-25 11:10:58 -07:00
"id": "b66aab6a",
2022-03-25 01:04:53 -07:00
"metadata": {},
"source": [
2022-04-25 11:10:58 -07:00
"And now we have the hyperparameters found to minimize the mean loss."
2022-03-25 01:04:53 -07:00
]
},
{
"cell_type": "code",
2022-07-24 18:53:57 +01:00
"execution_count": 12,
2022-04-25 11:10:58 -07:00
"id": "e69db02e",
2022-03-25 01:04:53 -07:00
"metadata": {},
2022-07-24 18:53:57 +01:00
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Best hyperparameters found were: {'steps': 100, 'width': 7.688465886501621, 'height': -16.46289560854555, 'activation': 'tanh'}\n"
]
}
],
2022-03-25 01:04:53 -07:00
"source": [
2022-07-24 18:53:57 +01:00
"print(\"Best hyperparameters found were: \", results.get_best_result().config)"
2022-03-25 01:04:53 -07:00
]
},
{
"cell_type": "markdown",
2022-04-25 11:10:58 -07:00
"id": "d545d30b",
2022-03-25 01:04:53 -07:00
"metadata": {},
"source": [
2022-04-25 11:10:58 -07:00
"## Providing an initial set of hyperparameters\n",
"\n",
2022-03-25 01:04:53 -07:00
"While defining the search algorithm, we may choose to provide an initial set of hyperparameters that we believe are especially promising or informative, and\n",
"pass this information as a helpful starting point for the `OptunaSearch` object."
]
},
{
"cell_type": "code",
2022-07-24 18:53:57 +01:00
"execution_count": 13,
2022-04-25 11:10:58 -07:00
"id": "7596b7f4",
2022-03-25 01:04:53 -07:00
"metadata": {},
"outputs": [],
"source": [
"initial_params = [\n",
" {\"width\": 1, \"height\": 2, \"activation\": \"relu\"},\n",
" {\"width\": 4, \"height\": 2, \"activation\": \"relu\"},\n",
"]"
]
},
{
"cell_type": "markdown",
2022-04-25 11:10:58 -07:00
"id": "f84bbff0",
2022-03-25 01:04:53 -07:00
"metadata": {},
"source": [
"Now the `search_alg` built using `OptunaSearch` takes `points_to_evaluate`."
]
},
{
"cell_type": "code",
2022-07-24 18:53:57 +01:00
"execution_count": 14,
2022-04-25 11:10:58 -07:00
"id": "320d1935",
2022-03-25 01:04:53 -07:00
"metadata": {
"lines_to_next_cell": 0
},
"outputs": [],
"source": [
"searcher = OptunaSearch(points_to_evaluate=initial_params)\n",
"algo = ConcurrencyLimiter(searcher, max_concurrent=4)"
]
},
{
"cell_type": "markdown",
2022-04-25 11:10:58 -07:00
"id": "9147d9a2",
2022-03-25 01:04:53 -07:00
"metadata": {},
"source": [
"And run the experiment with initial hyperparameter evaluations:"
]
},
{
"cell_type": "code",
2022-07-24 18:53:57 +01:00
"execution_count": 15,
2022-04-25 11:10:58 -07:00
"id": "ee442efd",
2022-03-25 01:04:53 -07:00
"metadata": {},
2022-07-24 18:53:57 +01:00
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"\u001b[32m[I 2022-07-22 15:22:32,644]\u001b[0m A new study created in memory with name: optuna\u001b[0m\n",
"/Users/kai/coding/ray/python/ray/tune/search/optuna/optuna_search.py:389: ExperimentalWarning: enqueue_trial is experimental (supported from v1.2.0). The interface can change in the future.\n",
" self._ot_study.enqueue_trial(point)\n",
"/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/optuna/study/study.py:857: ExperimentalWarning: create_trial is experimental (supported from v2.0.0). The interface can change in the future.\n",
" create_trial(state=TrialState.WAITING, system_attrs={\"fixed_params\": params})\n",
"/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/optuna/study/study.py:857: ExperimentalWarning: add_trial is experimental (supported from v2.0.0). The interface can change in the future.\n",
" create_trial(state=TrialState.WAITING, system_attrs={\"fixed_params\": params})\n"
]
},
{
"data": {
"text/html": [
"== Status ==<br>Current time: 2022-07-22 15:23:15 (running for 00:00:42.97)<br>Memory usage on this node: 10.1/16.0 GiB<br>Using FIFO scheduling algorithm.<br>Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/5.07 GiB heap, 0.0/2.0 GiB objects<br>Current best trial: c3762426 with mean_loss=-8.621481572769046 and parameters={'steps': 100, 'width': 0.7236705053153614, 'height': -98.46320622400306, 'activation': 'tanh'}<br>Result logdir: /Users/kai/ray_results/optuna_example_2<br>Number of trials: 10/10 (10 TERMINATED)<br><table>\n",
"<thead>\n",
"<tr><th>Trial name </th><th>status </th><th>loc </th><th>activation </th><th style=\"text-align: right;\"> height</th><th style=\"text-align: right;\"> width</th><th style=\"text-align: right;\"> loss</th><th style=\"text-align: right;\"> iter</th><th style=\"text-align: right;\"> total time (s)</th><th style=\"text-align: right;\"> iterations</th><th style=\"text-align: right;\"> neg_mean_loss</th></tr>\n",
"</thead>\n",
"<tbody>\n",
"<tr><td>objective_b9acff32</td><td>TERMINATED</td><td>127.0.0.1:46220</td><td>relu </td><td style=\"text-align: right;\"> 2 </td><td style=\"text-align: right;\"> 1 </td><td style=\"text-align: right;\">11.1174 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 10.6837</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\"> -11.1174 </td></tr>\n",
"<tr><td>objective_bb3b2702</td><td>TERMINATED</td><td>127.0.0.1:46227</td><td>relu </td><td style=\"text-align: right;\"> 2 </td><td style=\"text-align: right;\"> 4 </td><td style=\"text-align: right;\">10.4463 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 11.2816</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\"> -10.4463 </td></tr>\n",
"<tr><td>objective_bb3c8714</td><td>TERMINATED</td><td>127.0.0.1:46228</td><td>relu </td><td style=\"text-align: right;\">-90.7016 </td><td style=\"text-align: right;\"> 0.566597</td><td style=\"text-align: right;\"> 2.44285 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 11.2766</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\"> -2.44285 </td></tr>\n",
"<tr><td>objective_bb3e0210</td><td>TERMINATED</td><td>127.0.0.1:46229</td><td>tanh </td><td style=\"text-align: right;\">-81.4501 </td><td style=\"text-align: right;\">14.783 </td><td style=\"text-align: right;\">-8.07715 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 11.2196</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\"> 8.07715 </td></tr>\n",
"<tr><td>objective_c19b9dde</td><td>TERMINATED</td><td>127.0.0.1:46248</td><td>relu </td><td style=\"text-align: right;\"> 86.466 </td><td style=\"text-align: right;\"> 0.308659</td><td style=\"text-align: right;\">21.1122 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 10.7226</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\"> -21.1122 </td></tr>\n",
"<tr><td>objective_c3762426</td><td>TERMINATED</td><td>127.0.0.1:46253</td><td>tanh </td><td style=\"text-align: right;\">-98.4632 </td><td style=\"text-align: right;\"> 0.723671</td><td style=\"text-align: right;\">-8.62148 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 10.7271</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\"> 8.62148 </td></tr>\n",
"<tr><td>objective_c385528e</td><td>TERMINATED</td><td>127.0.0.1:46256</td><td>relu </td><td style=\"text-align: right;\">-54.8368 </td><td style=\"text-align: right;\">11.0765 </td><td style=\"text-align: right;\"> 4.60669 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 10.7498</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\"> -4.60669 </td></tr>\n",
"<tr><td>objective_c387a7c8</td><td>TERMINATED</td><td>127.0.0.1:46257</td><td>tanh </td><td style=\"text-align: right;\"> 42.9466 </td><td style=\"text-align: right;\"> 8.01708 </td><td style=\"text-align: right;\"> 4.41909 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 10.7214</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\"> -4.41909 </td></tr>\n",
"<tr><td>objective_c97678f8</td><td>TERMINATED</td><td>127.0.0.1:46272</td><td>tanh </td><td style=\"text-align: right;\"> 9.25214</td><td style=\"text-align: right;\">15.5288 </td><td style=\"text-align: right;\"> 0.989841</td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 12.4591</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\"> -0.989841</td></tr>\n",
"<tr><td>objective_cb6180fe</td><td>TERMINATED</td><td>127.0.0.1:46277</td><td>relu </td><td style=\"text-align: right;\"> 41.6647 </td><td style=\"text-align: right;\"> 8.0585 </td><td style=\"text-align: right;\">14.2903 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 10.752 </td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\"> -14.2903 </td></tr>\n",
"</tbody>\n",
"</table><br><br>"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Result for objective_b9acff32:\n",
" date: 2022-07-22_15-22-35\n",
" done: false\n",
" experiment_id: ec3c260f0ad348fa9b188f9c93c01eba\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 20.2\n",
" neg_mean_loss: -20.2\n",
" node_ip: 127.0.0.1\n",
" pid: 46220\n",
" time_since_restore: 0.10049605369567871\n",
" time_this_iter_s: 0.10049605369567871\n",
" time_total_s: 0.10049605369567871\n",
" timestamp: 1658499755\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: b9acff32\n",
" warmup_time: 0.0029239654541015625\n",
" \n",
"Result for objective_bb3e0210:\n",
" date: 2022-07-22_15-22-37\n",
" done: false\n",
" experiment_id: 08a7bc47c140405296e58a049cd2f6b4\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 1.8549886153795079\n",
" neg_mean_loss: -1.8549886153795079\n",
" node_ip: 127.0.0.1\n",
" pid: 46229\n",
" time_since_restore: 0.10361814498901367\n",
" time_this_iter_s: 0.10361814498901367\n",
" time_total_s: 0.10361814498901367\n",
" timestamp: 1658499757\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: bb3e0210\n",
" warmup_time: 0.002833843231201172\n",
" \n",
"Result for objective_bb3b2702:\n",
" date: 2022-07-22_15-22-37\n",
" done: false\n",
" experiment_id: 2f984120624b4e9aa3392fcc1b16b811\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 20.2\n",
" neg_mean_loss: -20.2\n",
" node_ip: 127.0.0.1\n",
" pid: 46227\n",
" time_since_restore: 0.10420107841491699\n",
" time_this_iter_s: 0.10420107841491699\n",
" time_total_s: 0.10420107841491699\n",
" timestamp: 1658499757\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: bb3b2702\n",
" warmup_time: 0.002672910690307617\n",
" \n",
"Result for objective_bb3c8714:\n",
" date: 2022-07-22_15-22-37\n",
" done: false\n",
" experiment_id: 5fdcb12c4e234f1d9904a09b8705c0bf\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 10.929835533284928\n",
" neg_mean_loss: -10.929835533284928\n",
" node_ip: 127.0.0.1\n",
" pid: 46228\n",
" time_since_restore: 0.1048588752746582\n",
" time_this_iter_s: 0.1048588752746582\n",
" time_total_s: 0.1048588752746582\n",
" timestamp: 1658499757\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: bb3c8714\n",
" warmup_time: 0.0027511119842529297\n",
" \n",
"Result for objective_b9acff32:\n",
" date: 2022-07-22_15-22-40\n",
" done: false\n",
" experiment_id: ec3c260f0ad348fa9b188f9c93c01eba\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 48\n",
" iterations_since_restore: 49\n",
" mean_loss: 11.924137931034483\n",
" neg_mean_loss: -11.924137931034483\n",
" node_ip: 127.0.0.1\n",
" pid: 46220\n",
" time_since_restore: 5.207761287689209\n",
" time_this_iter_s: 0.10753417015075684\n",
" time_total_s: 5.207761287689209\n",
" timestamp: 1658499760\n",
" timesteps_since_restore: 0\n",
" training_iteration: 49\n",
" trial_id: b9acff32\n",
" warmup_time: 0.0029239654541015625\n",
" \n",
"Result for objective_bb3e0210:\n",
" date: 2022-07-22_15-22-43\n",
" done: false\n",
" experiment_id: 08a7bc47c140405296e58a049cd2f6b4\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" mean_loss: -8.003127819283387\n",
" neg_mean_loss: 8.003127819283387\n",
" node_ip: 127.0.0.1\n",
" pid: 46229\n",
" time_since_restore: 5.188760042190552\n",
" time_this_iter_s: 0.10853385925292969\n",
" time_total_s: 5.188760042190552\n",
" timestamp: 1658499763\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: bb3e0210\n",
" warmup_time: 0.002833843231201172\n",
" \n",
"Result for objective_bb3b2702:\n",
" date: 2022-07-22_15-22-43\n",
" done: false\n",
" experiment_id: 2f984120624b4e9aa3392fcc1b16b811\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" mean_loss: 10.705050505050504\n",
" neg_mean_loss: -10.705050505050504\n",
" node_ip: 127.0.0.1\n",
" pid: 46227\n",
" time_since_restore: 5.1574530601501465\n",
" time_this_iter_s: 0.10719585418701172\n",
" time_total_s: 5.1574530601501465\n",
" timestamp: 1658499763\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: bb3b2702\n",
" warmup_time: 0.002672910690307617\n",
" \n",
"Result for objective_bb3c8714:\n",
" date: 2022-07-22_15-22-43\n",
" done: false\n",
" experiment_id: 5fdcb12c4e234f1d9904a09b8705c0bf\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" mean_loss: 3.6598337799021596\n",
" neg_mean_loss: -3.6598337799021596\n",
" node_ip: 127.0.0.1\n",
" pid: 46228\n",
" time_since_restore: 5.163677930831909\n",
" time_this_iter_s: 0.10699272155761719\n",
" time_total_s: 5.163677930831909\n",
" timestamp: 1658499763\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: bb3c8714\n",
" warmup_time: 0.0027511119842529297\n",
" \n",
"Result for objective_b9acff32:\n",
" date: 2022-07-22_15-22-45\n",
" done: false\n",
" experiment_id: ec3c260f0ad348fa9b188f9c93c01eba\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 95\n",
" iterations_since_restore: 96\n",
" mean_loss: 11.152380952380952\n",
" neg_mean_loss: -11.152380952380952\n",
" node_ip: 127.0.0.1\n",
" pid: 46220\n",
" time_since_restore: 10.250982284545898\n",
" time_this_iter_s: 0.1073312759399414\n",
" time_total_s: 10.250982284545898\n",
" timestamp: 1658499765\n",
" timesteps_since_restore: 0\n",
" training_iteration: 96\n",
" trial_id: b9acff32\n",
" warmup_time: 0.0029239654541015625\n",
" \n",
"Result for objective_b9acff32:\n",
" date: 2022-07-22_15-22-45\n",
" done: true\n",
" experiment_id: ec3c260f0ad348fa9b188f9c93c01eba\n",
" experiment_tag: 1_activation=relu,height=2,steps=100,width=1\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: 11.11743119266055\n",
" neg_mean_loss: -11.11743119266055\n",
" node_ip: 127.0.0.1\n",
" pid: 46220\n",
" time_since_restore: 10.68371295928955\n",
" time_this_iter_s: 0.10785365104675293\n",
" time_total_s: 10.68371295928955\n",
" timestamp: 1658499765\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: b9acff32\n",
" warmup_time: 0.0029239654541015625\n",
" \n",
"Result for objective_bb3e0210:\n",
" date: 2022-07-22_15-22-47\n",
" done: false\n",
" experiment_id: 08a7bc47c140405296e58a049cd2f6b4\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 93\n",
" iterations_since_restore: 94\n",
" mean_loss: -8.072800015147221\n",
" neg_mean_loss: 8.072800015147221\n",
" node_ip: 127.0.0.1\n",
" pid: 46229\n",
" time_since_restore: 10.099772214889526\n",
" time_this_iter_s: 0.10250687599182129\n",
" time_total_s: 10.099772214889526\n",
" timestamp: 1658499767\n",
" timesteps_since_restore: 0\n",
" training_iteration: 94\n",
" trial_id: bb3e0210\n",
" warmup_time: 0.002833843231201172\n",
" \n",
"Result for objective_bb3c8714:\n",
" date: 2022-07-22_15-22-47\n",
" done: false\n",
" experiment_id: 5fdcb12c4e234f1d9904a09b8705c0bf\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 92\n",
" iterations_since_restore: 93\n",
" mean_loss: 2.53944357306219\n",
" neg_mean_loss: -2.53944357306219\n",
" node_ip: 127.0.0.1\n",
" pid: 46228\n",
" time_since_restore: 9.980368852615356\n",
" time_this_iter_s: 0.10658979415893555\n",
" time_total_s: 9.980368852615356\n",
" timestamp: 1658499767\n",
" timesteps_since_restore: 0\n",
" training_iteration: 93\n",
" trial_id: bb3c8714\n",
" warmup_time: 0.0027511119842529297\n",
" \n",
"Result for objective_bb3b2702:\n",
" date: 2022-07-22_15-22-47\n",
" done: false\n",
" experiment_id: 2f984120624b4e9aa3392fcc1b16b811\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 92\n",
" iterations_since_restore: 93\n",
" mean_loss: 10.464550264550265\n",
" neg_mean_loss: -10.464550264550265\n",
" node_ip: 127.0.0.1\n",
" pid: 46227\n",
" time_since_restore: 9.964121103286743\n",
" time_this_iter_s: 0.10547995567321777\n",
" time_total_s: 9.964121103286743\n",
" timestamp: 1658499767\n",
" timesteps_since_restore: 0\n",
" training_iteration: 93\n",
" trial_id: bb3b2702\n",
" warmup_time: 0.002672910690307617\n",
" \n",
"Result for objective_c19b9dde:\n",
" date: 2022-07-22_15-22-48\n",
" done: false\n",
" experiment_id: fab35a2f68c347c4bb815c268e9500a6\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 28.646596578422436\n",
" neg_mean_loss: -28.646596578422436\n",
" node_ip: 127.0.0.1\n",
" pid: 46248\n",
" time_since_restore: 0.10230708122253418\n",
" time_this_iter_s: 0.10230708122253418\n",
" time_total_s: 0.10230708122253418\n",
" timestamp: 1658499768\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: c19b9dde\n",
" warmup_time: 0.0026540756225585938\n",
" \n",
"Result for objective_bb3e0210:\n",
" date: 2022-07-22_15-22-49\n",
" done: true\n",
" experiment_id: 08a7bc47c140405296e58a049cd2f6b4\n",
" experiment_tag: 4_activation=tanh,height=-81.4501,steps=100,width=14.7830\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: -8.077146761191683\n",
" neg_mean_loss: 8.077146761191683\n",
" node_ip: 127.0.0.1\n",
" pid: 46229\n",
" time_since_restore: 11.219618082046509\n",
" time_this_iter_s: 0.1108238697052002\n",
" time_total_s: 11.219618082046509\n",
" timestamp: 1658499769\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: bb3e0210\n",
" warmup_time: 0.002833843231201172\n",
" \n",
"Result for objective_bb3c8714:\n",
" date: 2022-07-22_15-22-49\n",
" done: true\n",
" experiment_id: 5fdcb12c4e234f1d9904a09b8705c0bf\n",
" experiment_tag: 3_activation=relu,height=-90.7016,steps=100,width=0.5666\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: 2.442852652712544\n",
" neg_mean_loss: -2.442852652712544\n",
" node_ip: 127.0.0.1\n",
" pid: 46228\n",
" time_since_restore: 11.276630163192749\n",
" time_this_iter_s: 0.10824012756347656\n",
" time_total_s: 11.276630163192749\n",
" timestamp: 1658499769\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: bb3c8714\n",
" warmup_time: 0.0027511119842529297\n",
" \n",
"Result for objective_bb3b2702:\n",
" date: 2022-07-22_15-22-49\n",
" done: true\n",
" experiment_id: 2f984120624b4e9aa3392fcc1b16b811\n",
" experiment_tag: 2_activation=relu,height=2,steps=100,width=4\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: 10.446305418719211\n",
" neg_mean_loss: -10.446305418719211\n",
" node_ip: 127.0.0.1\n",
" pid: 46227\n",
" time_since_restore: 11.281576871871948\n",
" time_this_iter_s: 0.1068716049194336\n",
" time_total_s: 11.281576871871948\n",
" timestamp: 1658499769\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: bb3b2702\n",
" warmup_time: 0.002672910690307617\n",
" \n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Result for objective_c3762426:\n",
" date: 2022-07-22_15-22-51\n",
" done: false\n",
" experiment_id: 034596724b2a4c9f86726e9a15fb386b\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 0.15367937759969408\n",
" neg_mean_loss: -0.15367937759969408\n",
" node_ip: 127.0.0.1\n",
" pid: 46253\n",
" time_since_restore: 0.1050560474395752\n",
" time_this_iter_s: 0.1050560474395752\n",
" time_total_s: 0.1050560474395752\n",
" timestamp: 1658499771\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: c3762426\n",
" warmup_time: 0.0028579235076904297\n",
" \n",
"Result for objective_c385528e:\n",
" date: 2022-07-22_15-22-51\n",
" done: false\n",
" experiment_id: 3044e8f1500f4b16aba04b72105c67be\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 14.516317633001044\n",
" neg_mean_loss: -14.516317633001044\n",
" node_ip: 127.0.0.1\n",
" pid: 46256\n",
" time_since_restore: 0.10402488708496094\n",
" time_this_iter_s: 0.10402488708496094\n",
" time_total_s: 0.10402488708496094\n",
" timestamp: 1658499771\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: c385528e\n",
" warmup_time: 0.0027010440826416016\n",
" \n",
"Result for objective_c387a7c8:\n",
" date: 2022-07-22_15-22-51\n",
" done: false\n",
" experiment_id: fc7f3d46b136437cab526c2fa6ea2944\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 14.294664350689382\n",
" neg_mean_loss: -14.294664350689382\n",
" node_ip: 127.0.0.1\n",
" pid: 46257\n",
" time_since_restore: 0.10461091995239258\n",
" time_this_iter_s: 0.10461091995239258\n",
" time_total_s: 0.10461091995239258\n",
" timestamp: 1658499771\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: c387a7c8\n",
" warmup_time: 0.0025899410247802734\n",
" \n",
"Result for objective_c19b9dde:\n",
" date: 2022-07-22_15-22-53\n",
" done: false\n",
" experiment_id: fab35a2f68c347c4bb815c268e9500a6\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" mean_loss: 22.72706638821166\n",
" neg_mean_loss: -22.72706638821166\n",
" node_ip: 127.0.0.1\n",
" pid: 46248\n",
" time_since_restore: 5.118642330169678\n",
" time_this_iter_s: 0.10826611518859863\n",
" time_total_s: 5.118642330169678\n",
" timestamp: 1658499773\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: c19b9dde\n",
" warmup_time: 0.0026540756225585938\n",
" \n",
"Result for objective_c3762426:\n",
" date: 2022-07-22_15-22-56\n",
" done: false\n",
" experiment_id: 034596724b2a4c9f86726e9a15fb386b\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" mean_loss: -7.5742395372701825\n",
" neg_mean_loss: 7.5742395372701825\n",
" node_ip: 127.0.0.1\n",
" pid: 46253\n",
" time_since_restore: 5.14322304725647\n",
" time_this_iter_s: 0.10359907150268555\n",
" time_total_s: 5.14322304725647\n",
" timestamp: 1658499776\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: c3762426\n",
" warmup_time: 0.0028579235076904297\n",
" \n",
"Result for objective_c385528e:\n",
" date: 2022-07-22_15-22-56\n",
" done: false\n",
" experiment_id: 3044e8f1500f4b16aba04b72105c67be\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" mean_loss: 4.704784733826337\n",
" neg_mean_loss: -4.704784733826337\n",
" node_ip: 127.0.0.1\n",
" pid: 46256\n",
" time_since_restore: 5.16935396194458\n",
" time_this_iter_s: 0.10719513893127441\n",
" time_total_s: 5.16935396194458\n",
" timestamp: 1658499776\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: c385528e\n",
" warmup_time: 0.0027010440826416016\n",
" \n",
"Result for objective_c387a7c8:\n",
" date: 2022-07-22_15-22-56\n",
" done: false\n",
" experiment_id: fc7f3d46b136437cab526c2fa6ea2944\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" mean_loss: 4.553194134463865\n",
" neg_mean_loss: -4.553194134463865\n",
" node_ip: 127.0.0.1\n",
" pid: 46257\n",
" time_since_restore: 5.1458728313446045\n",
" time_this_iter_s: 0.10374999046325684\n",
" time_total_s: 5.1458728313446045\n",
" timestamp: 1658499776\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: c387a7c8\n",
" warmup_time: 0.0025899410247802734\n",
" \n",
"Result for objective_c19b9dde:\n",
" date: 2022-07-22_15-22-58\n",
" done: false\n",
" experiment_id: fab35a2f68c347c4bb815c268e9500a6\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 94\n",
" iterations_since_restore: 95\n",
" mean_loss: 21.20978139671461\n",
" neg_mean_loss: -21.20978139671461\n",
" node_ip: 127.0.0.1\n",
" pid: 46248\n",
" time_since_restore: 10.183105230331421\n",
" time_this_iter_s: 0.10718917846679688\n",
" time_total_s: 10.183105230331421\n",
" timestamp: 1658499778\n",
" timesteps_since_restore: 0\n",
" training_iteration: 95\n",
" trial_id: c19b9dde\n",
" warmup_time: 0.0026540756225585938\n",
" \n",
"Result for objective_c19b9dde:\n",
" date: 2022-07-22_15-22-59\n",
" done: true\n",
" experiment_id: fab35a2f68c347c4bb815c268e9500a6\n",
" experiment_tag: 5_activation=relu,height=86.4660,steps=100,width=0.3087\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: 21.112246386473707\n",
" neg_mean_loss: -21.112246386473707\n",
" node_ip: 127.0.0.1\n",
" pid: 46248\n",
" time_since_restore: 10.722583055496216\n",
" time_this_iter_s: 0.10788774490356445\n",
" time_total_s: 10.722583055496216\n",
" timestamp: 1658499779\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: c19b9dde\n",
" warmup_time: 0.0026540756225585938\n",
" \n",
"Result for objective_c97678f8:\n",
" date: 2022-07-22_15-23-01\n",
" done: false\n",
" experiment_id: 7c4187f99b9246bdbae534ce20f33bfc\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 10.925214276212655\n",
" neg_mean_loss: -10.925214276212655\n",
" node_ip: 127.0.0.1\n",
" pid: 46272\n",
" time_since_restore: 0.1039581298828125\n",
" time_this_iter_s: 0.1039581298828125\n",
" time_total_s: 0.1039581298828125\n",
" timestamp: 1658499781\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: c97678f8\n",
" warmup_time: 0.002711057662963867\n",
" \n",
"Result for objective_c3762426:\n",
" date: 2022-07-22_15-23-01\n",
" done: false\n",
" experiment_id: 034596724b2a4c9f86726e9a15fb386b\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 94\n",
" iterations_since_restore: 95\n",
" mean_loss: -8.564680574270156\n",
" neg_mean_loss: 8.564680574270156\n",
" node_ip: 127.0.0.1\n",
" pid: 46253\n",
" time_since_restore: 10.190189123153687\n",
" time_this_iter_s: 0.10848307609558105\n",
" time_total_s: 10.190189123153687\n",
" timestamp: 1658499781\n",
" timesteps_since_restore: 0\n",
" training_iteration: 95\n",
" trial_id: c3762426\n",
" warmup_time: 0.0028579235076904297\n",
" \n",
"Result for objective_c387a7c8:\n",
" date: 2022-07-22_15-23-01\n",
" done: false\n",
" experiment_id: fc7f3d46b136437cab526c2fa6ea2944\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 94\n",
" iterations_since_restore: 95\n",
" mean_loss: 4.425622066068654\n",
" neg_mean_loss: -4.425622066068654\n",
" node_ip: 127.0.0.1\n",
" pid: 46257\n",
" time_since_restore: 10.181554794311523\n",
" time_this_iter_s: 0.10594582557678223\n",
" time_total_s: 10.181554794311523\n",
" timestamp: 1658499781\n",
" timesteps_since_restore: 0\n",
" training_iteration: 95\n",
" trial_id: c387a7c8\n",
" warmup_time: 0.0025899410247802734\n",
" \n",
"Result for objective_c385528e:\n",
" date: 2022-07-22_15-23-01\n",
" done: false\n",
" experiment_id: 3044e8f1500f4b16aba04b72105c67be\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 94\n",
" iterations_since_restore: 95\n",
" mean_loss: 4.611447627123378\n",
" neg_mean_loss: -4.611447627123378\n",
" node_ip: 127.0.0.1\n",
" pid: 46256\n",
" time_since_restore: 10.214791059494019\n",
" time_this_iter_s: 0.10866808891296387\n",
" time_total_s: 10.214791059494019\n",
" timestamp: 1658499781\n",
" timesteps_since_restore: 0\n",
" training_iteration: 95\n",
" trial_id: c385528e\n",
" warmup_time: 0.0027010440826416016\n",
" \n",
"Result for objective_c3762426:\n",
" date: 2022-07-22_15-23-02\n",
" done: true\n",
" experiment_id: 034596724b2a4c9f86726e9a15fb386b\n",
" experiment_tag: 6_activation=tanh,height=-98.4632,steps=100,width=0.7237\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: -8.621481572769046\n",
" neg_mean_loss: 8.621481572769046\n",
" node_ip: 127.0.0.1\n",
" pid: 46253\n",
" time_since_restore: 10.72708010673523\n",
" time_this_iter_s: 0.10719490051269531\n",
" time_total_s: 10.72708010673523\n",
" timestamp: 1658499782\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: c3762426\n",
" warmup_time: 0.0028579235076904297\n",
" \n",
"Result for objective_c387a7c8:\n",
" date: 2022-07-22_15-23-02\n",
" done: true\n",
" experiment_id: fc7f3d46b136437cab526c2fa6ea2944\n",
" experiment_tag: 8_activation=tanh,height=42.9466,steps=100,width=8.0171\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: 4.419090335709762\n",
" neg_mean_loss: -4.419090335709762\n",
" node_ip: 127.0.0.1\n",
" pid: 46257\n",
" time_since_restore: 10.721372842788696\n",
" time_this_iter_s: 0.10873699188232422\n",
" time_total_s: 10.721372842788696\n",
" timestamp: 1658499782\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: c387a7c8\n",
" warmup_time: 0.0025899410247802734\n",
" \n",
"Result for objective_c385528e:\n",
" date: 2022-07-22_15-23-02\n",
" done: true\n",
" experiment_id: 3044e8f1500f4b16aba04b72105c67be\n",
" experiment_tag: 7_activation=relu,height=-54.8368,steps=100,width=11.0765\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: 4.606686500095965\n",
" neg_mean_loss: -4.606686500095965\n",
" node_ip: 127.0.0.1\n",
" pid: 46256\n",
" time_since_restore: 10.749844074249268\n",
" time_this_iter_s: 0.10710501670837402\n",
" time_total_s: 10.749844074249268\n",
" timestamp: 1658499782\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: c385528e\n",
" warmup_time: 0.0027010440826416016\n",
" \n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Result for objective_cb6180fe:\n",
" date: 2022-07-22_15-23-04\n",
" done: false\n",
" experiment_id: 0cfd73afdca24f8f894b2a3e7ebb8cea\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 24.166473702046588\n",
" neg_mean_loss: -24.166473702046588\n",
" node_ip: 127.0.0.1\n",
" pid: 46277\n",
" time_since_restore: 0.10389995574951172\n",
" time_this_iter_s: 0.10389995574951172\n",
" time_total_s: 0.10389995574951172\n",
" timestamp: 1658499784\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: cb6180fe\n",
" warmup_time: 0.0027260780334472656\n",
" \n",
"Result for objective_c97678f8:\n",
" date: 2022-07-22_15-23-06\n",
" done: false\n",
" experiment_id: 7c4187f99b9246bdbae534ce20f33bfc\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 31\n",
" iterations_since_restore: 32\n",
" mean_loss: 1.1287174007116527\n",
" neg_mean_loss: -1.1287174007116527\n",
" node_ip: 127.0.0.1\n",
" pid: 46272\n",
" time_since_restore: 5.140314340591431\n",
" time_this_iter_s: 0.10663819313049316\n",
" time_total_s: 5.140314340591431\n",
" timestamp: 1658499786\n",
" timesteps_since_restore: 0\n",
" training_iteration: 32\n",
" trial_id: c97678f8\n",
" warmup_time: 0.002711057662963867\n",
" \n",
"Result for objective_cb6180fe:\n",
" date: 2022-07-22_15-23-09\n",
" done: false\n",
" experiment_id: 0cfd73afdca24f8f894b2a3e7ebb8cea\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" mean_loss: 14.42370878193022\n",
" neg_mean_loss: -14.42370878193022\n",
" node_ip: 127.0.0.1\n",
" pid: 46277\n",
" time_since_restore: 5.139087915420532\n",
" time_this_iter_s: 0.10686802864074707\n",
" time_total_s: 5.139087915420532\n",
" timestamp: 1658499789\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: cb6180fe\n",
" warmup_time: 0.0027260780334472656\n",
" \n",
"Result for objective_c97678f8:\n",
" date: 2022-07-22_15-23-11\n",
" done: false\n",
" experiment_id: 7c4187f99b9246bdbae534ce20f33bfc\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 78\n",
" iterations_since_restore: 79\n",
" mean_loss: 1.0070978091228147\n",
" neg_mean_loss: -1.0070978091228147\n",
" node_ip: 127.0.0.1\n",
" pid: 46272\n",
" time_since_restore: 10.206932067871094\n",
" time_this_iter_s: 0.10774993896484375\n",
" time_total_s: 10.206932067871094\n",
" timestamp: 1658499791\n",
" timesteps_since_restore: 0\n",
" training_iteration: 79\n",
" trial_id: c97678f8\n",
" warmup_time: 0.002711057662963867\n",
" \n",
"Result for objective_c97678f8:\n",
" date: 2022-07-22_15-23-14\n",
" done: true\n",
" experiment_id: 7c4187f99b9246bdbae534ce20f33bfc\n",
" experiment_tag: 9_activation=tanh,height=9.2521,steps=100,width=15.5288\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: 0.9898408262377837\n",
" neg_mean_loss: -0.9898408262377837\n",
" node_ip: 127.0.0.1\n",
" pid: 46272\n",
" time_since_restore: 12.459095239639282\n",
" time_this_iter_s: 0.10785508155822754\n",
" time_total_s: 12.459095239639282\n",
" timestamp: 1658499794\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: c97678f8\n",
" warmup_time: 0.002711057662963867\n",
" \n",
"Result for objective_cb6180fe:\n",
" date: 2022-07-22_15-23-15\n",
" done: false\n",
" experiment_id: 0cfd73afdca24f8f894b2a3e7ebb8cea\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 94\n",
" iterations_since_restore: 95\n",
" mean_loss: 14.296767042885353\n",
" neg_mean_loss: -14.296767042885353\n",
" node_ip: 127.0.0.1\n",
" pid: 46277\n",
" time_since_restore: 10.220713138580322\n",
" time_this_iter_s: 0.10750126838684082\n",
" time_total_s: 10.220713138580322\n",
" timestamp: 1658499795\n",
" timesteps_since_restore: 0\n",
" training_iteration: 95\n",
" trial_id: cb6180fe\n",
" warmup_time: 0.0027260780334472656\n",
" \n",
"Result for objective_cb6180fe:\n",
" date: 2022-07-22_15-23-15\n",
" done: true\n",
" experiment_id: 0cfd73afdca24f8f894b2a3e7ebb8cea\n",
" experiment_tag: 10_activation=relu,height=41.6647,steps=100,width=8.0585\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: 14.29026803363612\n",
" neg_mean_loss: -14.29026803363612\n",
" node_ip: 127.0.0.1\n",
" pid: 46277\n",
" time_since_restore: 10.752035856246948\n",
" time_this_iter_s: 0.1066899299621582\n",
" time_total_s: 10.752035856246948\n",
" timestamp: 1658499795\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: cb6180fe\n",
" warmup_time: 0.0027260780334472656\n",
" \n"
]
}
],
2022-03-25 01:04:53 -07:00
"source": [
2022-07-24 18:53:57 +01:00
"tuner = tune.Tuner(\n",
2022-03-25 01:04:53 -07:00
" objective,\n",
2022-07-24 18:53:57 +01:00
" tune_config=tune.TuneConfig(\n",
" metric=\"mean_loss\",\n",
" mode=\"min\",\n",
" search_alg=algo,\n",
" num_samples=num_samples,\n",
" ),\n",
" param_space=search_space,\n",
")\n",
"results = tuner.fit()"
2022-04-25 11:10:58 -07:00
]
},
{
"cell_type": "markdown",
"id": "ccfe15e2",
"metadata": {},
"source": [
"We take another look at the optimal hyperparamters."
]
},
{
"cell_type": "code",
2022-07-24 18:53:57 +01:00
"execution_count": 16,
2022-04-25 11:10:58 -07:00
"id": "fcfa0c2e",
"metadata": {},
2022-07-24 18:53:57 +01:00
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Best hyperparameters found were: {'steps': 100, 'width': 0.7236705053153614, 'height': -98.46320622400306, 'activation': 'tanh'}\n"
]
}
],
2022-04-25 11:10:58 -07:00
"source": [
2022-07-24 18:53:57 +01:00
"print(\"Best hyperparameters found were: \", results.get_best_result().config)"
2022-03-25 01:04:53 -07:00
]
},
{
"cell_type": "markdown",
2022-04-25 11:10:58 -07:00
"id": "88080576",
2022-03-25 01:04:53 -07:00
"metadata": {},
"source": [
2022-04-25 11:10:58 -07:00
"## Conditional search spaces \n",
"\n",
2022-03-25 01:04:53 -07:00
"Sometimes we may want to build a more complicated search space that has conditional dependencies on other hyperparameters. In this case, we pass a define-by-run function to the `search_alg` argument in `ray.tune()`."
]
},
{
"cell_type": "code",
2022-07-24 18:53:57 +01:00
"execution_count": 17,
2022-04-25 11:10:58 -07:00
"id": "f0acc2fc",
2022-03-25 01:04:53 -07:00
"metadata": {
"lines_to_next_cell": 0
},
"outputs": [],
"source": [
"def define_by_run_func(trial) -> Optional[Dict[str, Any]]:\n",
" \"\"\"Define-by-run function to create the search space.\n",
"\n",
" Ensure no actual computation takes place here. That should go into\n",
2022-07-24 18:53:57 +01:00
" the trainable passed to ``Tuner()`` (in this example, that's\n",
2022-03-25 01:04:53 -07:00
" ``objective``).\n",
"\n",
" For more information, see https://optuna.readthedocs.io/en/stable\\\n",
" /tutorial/10_key_features/002_configurations.html\n",
"\n",
" This function should either return None or a dict with constant values.\n",
" \"\"\"\n",
"\n",
" activation = trial.suggest_categorical(\"activation\", [\"relu\", \"tanh\"])\n",
"\n",
" # Define-by-run allows for conditional search spaces.\n",
" if activation == \"relu\":\n",
" trial.suggest_float(\"width\", 0, 20)\n",
" trial.suggest_float(\"height\", -100, 100)\n",
" else:\n",
" trial.suggest_float(\"width\", -1, 21)\n",
" trial.suggest_float(\"height\", -101, 101)\n",
" \n",
" # Return all constants in a dictionary.\n",
" return {\"steps\": 100}"
]
},
{
"cell_type": "markdown",
2022-04-25 11:10:58 -07:00
"id": "4c9d0945",
2022-03-25 01:04:53 -07:00
"metadata": {},
"source": [
"As before, we create the `search_alg` from `OptunaSearch` and `ConcurrencyLimiter`, this time we define the scope of search via the `space` argument and provide no initialization. We also must specific metric and mode when using `space`. "
]
},
{
"cell_type": "code",
2022-07-24 18:53:57 +01:00
"execution_count": 18,
2022-04-25 11:10:58 -07:00
"id": "906f9ffc",
2022-03-25 01:04:53 -07:00
"metadata": {},
2022-07-24 18:53:57 +01:00
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"\u001b[32m[I 2022-07-22 15:23:15,784]\u001b[0m A new study created in memory with name: optuna\u001b[0m\n"
]
}
],
2022-03-25 01:04:53 -07:00
"source": [
"searcher = OptunaSearch(space=define_by_run_func, metric=\"mean_loss\", mode=\"min\")\n",
"algo = ConcurrencyLimiter(searcher, max_concurrent=4)"
]
},
{
"cell_type": "markdown",
2022-04-25 11:10:58 -07:00
"id": "fea9399c",
2022-03-25 01:04:53 -07:00
"metadata": {},
"source": [
"Running the experiment with a define-by-run search space:"
]
},
{
"cell_type": "code",
2022-07-24 18:53:57 +01:00
"execution_count": 19,
2022-04-25 11:10:58 -07:00
"id": "bf0ee932",
"metadata": {},
2022-07-24 18:53:57 +01:00
"outputs": [
{
"data": {
"text/html": [
"== Status ==<br>Current time: 2022-07-22 15:23:58 (running for 00:00:43.15)<br>Memory usage on this node: 10.4/16.0 GiB<br>Using FIFO scheduling algorithm.<br>Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/5.07 GiB heap, 0.0/2.0 GiB objects<br>Result logdir: /Users/kai/ray_results/optuna_example_3<br>Number of trials: 10/10 (10 TERMINATED)<br><table>\n",
"<thead>\n",
"<tr><th>Trial name </th><th>status </th><th>loc </th><th>activation </th><th style=\"text-align: right;\"> height</th><th style=\"text-align: right;\"> steps</th><th style=\"text-align: right;\"> width</th><th style=\"text-align: right;\"> loss</th><th style=\"text-align: right;\"> iter</th><th style=\"text-align: right;\"> total time (s)</th><th style=\"text-align: right;\"> iterations</th><th style=\"text-align: right;\"> neg_mean_loss</th></tr>\n",
"</thead>\n",
"<tbody>\n",
"<tr><td>objective_d363eed6</td><td>TERMINATED</td><td>127.0.0.1:46290</td><td>relu </td><td style=\"text-align: right;\"> 96.9804 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 6.47293 </td><td style=\"text-align: right;\">19.8517 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 10.735 </td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\"> -19.8517 </td></tr>\n",
"<tr><td>objective_d4f3a700</td><td>TERMINATED</td><td>127.0.0.1:46298</td><td>tanh </td><td style=\"text-align: right;\">-76.4387 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\">17.7358 </td><td style=\"text-align: right;\">-7.58724 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 11.396 </td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\"> 7.58724 </td></tr>\n",
"<tr><td>objective_d4f50ce4</td><td>TERMINATED</td><td>127.0.0.1:46299</td><td>relu </td><td style=\"text-align: right;\">-89.8609 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 6.28321 </td><td style=\"text-align: right;\"> 1.17212 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 11.4182</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\"> -1.17212 </td></tr>\n",
"<tr><td>objective_d4f67908</td><td>TERMINATED</td><td>127.0.0.1:46300</td><td>relu </td><td style=\"text-align: right;\"> 67.9266 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\">14.0599 </td><td style=\"text-align: right;\">16.864 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 11.341 </td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\"> -16.864 </td></tr>\n",
"<tr><td>objective_db5c0402</td><td>TERMINATED</td><td>127.0.0.1:46317</td><td>tanh </td><td style=\"text-align: right;\">-54.13 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\">11.4712 </td><td style=\"text-align: right;\">-5.32571 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 10.705 </td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\"> 5.32571 </td></tr>\n",
"<tr><td>objective_dd487b56</td><td>TERMINATED</td><td>127.0.0.1:46322</td><td>relu </td><td style=\"text-align: right;\">-25.9818 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 0.758497</td><td style=\"text-align: right;\"> 8.57703 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 10.6916</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\"> -8.57703 </td></tr>\n",
"<tr><td>objective_dd4b4e94</td><td>TERMINATED</td><td>127.0.0.1:46323</td><td>relu </td><td style=\"text-align: right;\"> 54.6085 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 1.2361 </td><td style=\"text-align: right;\">16.2163 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 10.6711</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\"> -16.2163 </td></tr>\n",
"<tr><td>objective_dd5a8bca</td><td>TERMINATED</td><td>127.0.0.1:46328</td><td>tanh </td><td style=\"text-align: right;\"> 5.22131</td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\">17.0952 </td><td style=\"text-align: right;\"> 0.580871</td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 10.7017</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\"> -0.580871</td></tr>\n",
"<tr><td>objective_e3482178</td><td>TERMINATED</td><td>127.0.0.1:46341</td><td>tanh </td><td style=\"text-align: right;\"> 78.844 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\">15.5079 </td><td style=\"text-align: right;\"> 7.94912 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 12.6417</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\"> -7.94912 </td></tr>\n",
"<tr><td>objective_e532d6e0</td><td>TERMINATED</td><td>127.0.0.1:46353</td><td>tanh </td><td style=\"text-align: right;\">-66.9988 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 7.15087 </td><td style=\"text-align: right;\">-6.56059 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 10.7486</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\"> 6.56059 </td></tr>\n",
"</tbody>\n",
"</table><br><br>"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Result for objective_d363eed6:\n",
" date: 2022-07-22_15-23-18\n",
" done: false\n",
" experiment_id: af8f846e60254bc794827e76909df4f0\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 29.698044266420816\n",
" neg_mean_loss: -29.698044266420816\n",
" node_ip: 127.0.0.1\n",
" pid: 46290\n",
" time_since_restore: 0.10497498512268066\n",
" time_this_iter_s: 0.10497498512268066\n",
" time_total_s: 0.10497498512268066\n",
" timestamp: 1658499798\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: d363eed6\n",
" warmup_time: 0.0027971267700195312\n",
" \n",
"Result for objective_d4f3a700:\n",
" date: 2022-07-22_15-23-21\n",
" done: false\n",
" experiment_id: 52e5f2a557cf4466939b6c80bb9ff905\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 2.3561338711040003\n",
" neg_mean_loss: -2.3561338711040003\n",
" node_ip: 127.0.0.1\n",
" pid: 46298\n",
" time_since_restore: 0.10145115852355957\n",
" time_this_iter_s: 0.10145115852355957\n",
" time_total_s: 0.10145115852355957\n",
" timestamp: 1658499801\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: d4f3a700\n",
" warmup_time: 0.0033049583435058594\n",
" \n",
"Result for objective_d4f67908:\n",
" date: 2022-07-22_15-23-21\n",
" done: false\n",
" experiment_id: 230c6da41a834386be865bb19780b3a5\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 26.792658178391687\n",
" neg_mean_loss: -26.792658178391687\n",
" node_ip: 127.0.0.1\n",
" pid: 46300\n",
" time_since_restore: 0.10428690910339355\n",
" time_this_iter_s: 0.10428690910339355\n",
" time_total_s: 0.10428690910339355\n",
" timestamp: 1658499801\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: d4f67908\n",
" warmup_time: 0.0030798912048339844\n",
" \n",
"Result for objective_d4f50ce4:\n",
" date: 2022-07-22_15-23-21\n",
" done: false\n",
" experiment_id: f2396d75f50b417f9fae16d1c9ddcda4\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 11.013906266700227\n",
" neg_mean_loss: -11.013906266700227\n",
" node_ip: 127.0.0.1\n",
" pid: 46299\n",
" time_since_restore: 0.1020050048828125\n",
" time_this_iter_s: 0.1020050048828125\n",
" time_total_s: 0.1020050048828125\n",
" timestamp: 1658499801\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: d4f50ce4\n",
" warmup_time: 0.0027348995208740234\n",
" \n",
"Result for objective_d363eed6:\n",
" date: 2022-07-22_15-23-23\n",
" done: false\n",
" experiment_id: af8f846e60254bc794827e76909df4f0\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" mean_loss: 20.016284953435374\n",
" neg_mean_loss: -20.016284953435374\n",
" node_ip: 127.0.0.1\n",
" pid: 46290\n",
" time_since_restore: 5.138504981994629\n",
" time_this_iter_s: 0.10720086097717285\n",
" time_total_s: 5.138504981994629\n",
" timestamp: 1658499803\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: d363eed6\n",
" warmup_time: 0.0027971267700195312\n",
" \n",
"Result for objective_d4f3a700:\n",
" date: 2022-07-22_15-23-26\n",
" done: false\n",
" experiment_id: 52e5f2a557cf4466939b6c80bb9ff905\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" mean_loss: -7.525324371061028\n",
" neg_mean_loss: 7.525324371061028\n",
" node_ip: 127.0.0.1\n",
" pid: 46298\n",
" time_since_restore: 5.153075218200684\n",
" time_this_iter_s: 0.10515189170837402\n",
" time_total_s: 5.153075218200684\n",
" timestamp: 1658499806\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: d4f3a700\n",
" warmup_time: 0.0033049583435058594\n",
" \n",
"Result for objective_d4f67908:\n",
" date: 2022-07-22_15-23-26\n",
" done: false\n",
" experiment_id: 230c6da41a834386be865bb19780b3a5\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" mean_loss: 16.94173032408916\n",
" neg_mean_loss: -16.94173032408916\n",
" node_ip: 127.0.0.1\n",
" pid: 46300\n",
" time_since_restore: 5.153704881668091\n",
" time_this_iter_s: 0.10537075996398926\n",
" time_total_s: 5.153704881668091\n",
" timestamp: 1658499806\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: d4f67908\n",
" warmup_time: 0.0030798912048339844\n",
" \n",
"Result for objective_d4f50ce4:\n",
" date: 2022-07-22_15-23-26\n",
" done: false\n",
" experiment_id: f2396d75f50b417f9fae16d1c9ddcda4\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" mean_loss: 1.3414410369095506\n",
" neg_mean_loss: -1.3414410369095506\n",
" node_ip: 127.0.0.1\n",
" pid: 46299\n",
" time_since_restore: 5.1622138023376465\n",
" time_this_iter_s: 0.10337996482849121\n",
" time_total_s: 5.1622138023376465\n",
" timestamp: 1658499806\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: d4f50ce4\n",
" warmup_time: 0.0027348995208740234\n",
" \n",
"Result for objective_d363eed6:\n",
" date: 2022-07-22_15-23-28\n",
" done: false\n",
" experiment_id: af8f846e60254bc794827e76909df4f0\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 94\n",
" iterations_since_restore: 95\n",
" mean_loss: 19.85973747786284\n",
" neg_mean_loss: -19.85973747786284\n",
" node_ip: 127.0.0.1\n",
" pid: 46290\n",
" time_since_restore: 10.1943039894104\n",
" time_this_iter_s: 0.12109899520874023\n",
" time_total_s: 10.1943039894104\n",
" timestamp: 1658499808\n",
" timesteps_since_restore: 0\n",
" training_iteration: 95\n",
" trial_id: d363eed6\n",
" warmup_time: 0.0027971267700195312\n",
" \n",
"Result for objective_d363eed6:\n",
" date: 2022-07-22_15-23-29\n",
" done: true\n",
" experiment_id: af8f846e60254bc794827e76909df4f0\n",
" experiment_tag: 1_activation=relu,height=96.9804,steps=100,width=6.4729\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: 19.851696631549423\n",
" neg_mean_loss: -19.851696631549423\n",
" node_ip: 127.0.0.1\n",
" pid: 46290\n",
" time_since_restore: 10.734967947006226\n",
" time_this_iter_s: 0.10534882545471191\n",
" time_total_s: 10.734967947006226\n",
" timestamp: 1658499809\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: d363eed6\n",
" warmup_time: 0.0027971267700195312\n",
" \n",
"Result for objective_d4f67908:\n",
" date: 2022-07-22_15-23-31\n",
" done: false\n",
" experiment_id: 230c6da41a834386be865bb19780b3a5\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 93\n",
" iterations_since_restore: 94\n",
" mean_loss: 16.86855533914186\n",
" neg_mean_loss: -16.86855533914186\n",
" node_ip: 127.0.0.1\n",
" pid: 46300\n",
" time_since_restore: 10.096334218978882\n",
" time_this_iter_s: 0.10534310340881348\n",
" time_total_s: 10.096334218978882\n",
" timestamp: 1658499811\n",
" timesteps_since_restore: 0\n",
" training_iteration: 94\n",
" trial_id: d4f67908\n",
" warmup_time: 0.0030798912048339844\n",
" \n",
"Result for objective_d4f50ce4:\n",
" date: 2022-07-22_15-23-31\n",
" done: false\n",
" experiment_id: f2396d75f50b417f9fae16d1c9ddcda4\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 92\n",
" iterations_since_restore: 93\n",
" mean_loss: 1.183958166642718\n",
" neg_mean_loss: -1.183958166642718\n",
" node_ip: 127.0.0.1\n",
" pid: 46299\n",
" time_since_restore: 9.976859092712402\n",
" time_this_iter_s: 0.1057121753692627\n",
" time_total_s: 9.976859092712402\n",
" timestamp: 1658499811\n",
" timesteps_since_restore: 0\n",
" training_iteration: 93\n",
" trial_id: d4f50ce4\n",
" warmup_time: 0.0027348995208740234\n",
" \n",
"Result for objective_d4f3a700:\n",
" date: 2022-07-22_15-23-31\n",
" done: false\n",
" experiment_id: 52e5f2a557cf4466939b6c80bb9ff905\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 93\n",
" iterations_since_restore: 94\n",
" mean_loss: -7.583604593881161\n",
" neg_mean_loss: 7.583604593881161\n",
" node_ip: 127.0.0.1\n",
" pid: 46298\n",
" time_since_restore: 10.091833114624023\n",
" time_this_iter_s: 0.10643887519836426\n",
" time_total_s: 10.091833114624023\n",
" timestamp: 1658499811\n",
" timesteps_since_restore: 0\n",
" training_iteration: 94\n",
" trial_id: d4f3a700\n",
" warmup_time: 0.0033049583435058594\n",
" \n",
"Result for objective_db5c0402:\n",
" date: 2022-07-22_15-23-31\n",
" done: false\n",
" experiment_id: 69a34f42e51f4b97a5237fa5ae2be8d8\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 4.587004175501654\n",
" neg_mean_loss: -4.587004175501654\n",
" node_ip: 127.0.0.1\n",
" pid: 46317\n",
" time_since_restore: 0.10449004173278809\n",
" time_this_iter_s: 0.10449004173278809\n",
" time_total_s: 0.10449004173278809\n",
" timestamp: 1658499811\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: db5c0402\n",
" warmup_time: 0.003406047821044922\n",
" \n",
"Result for objective_d4f67908:\n",
" date: 2022-07-22_15-23-32\n",
" done: true\n",
" experiment_id: 230c6da41a834386be865bb19780b3a5\n",
" experiment_tag: 4_activation=relu,height=67.9266,steps=100,width=14.0599\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: 16.86398832185618\n",
" neg_mean_loss: -16.86398832185618\n",
" node_ip: 127.0.0.1\n",
" pid: 46300\n",
" time_since_restore: 11.341041088104248\n",
" time_this_iter_s: 0.10513019561767578\n",
" time_total_s: 11.341041088104248\n",
" timestamp: 1658499812\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: d4f67908\n",
" warmup_time: 0.0030798912048339844\n",
" \n",
"Result for objective_d4f3a700:\n",
" date: 2022-07-22_15-23-32\n",
" done: true\n",
" experiment_id: 52e5f2a557cf4466939b6c80bb9ff905\n",
" experiment_tag: 2_activation=tanh,height=-76.4387,steps=100,width=17.7358\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: -7.587236125634034\n",
" neg_mean_loss: 7.587236125634034\n",
" node_ip: 127.0.0.1\n",
" pid: 46298\n",
" time_since_restore: 11.396040916442871\n",
" time_this_iter_s: 0.10666394233703613\n",
" time_total_s: 11.396040916442871\n",
" timestamp: 1658499812\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: d4f3a700\n",
" warmup_time: 0.0033049583435058594\n",
" \n",
"Result for objective_d4f50ce4:\n",
" date: 2022-07-22_15-23-32\n",
" done: true\n",
" experiment_id: f2396d75f50b417f9fae16d1c9ddcda4\n",
" experiment_tag: 3_activation=relu,height=-89.8609,steps=100,width=6.2832\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: 1.1721245345451354\n",
" neg_mean_loss: -1.1721245345451354\n",
" node_ip: 127.0.0.1\n",
" pid: 46299\n",
" time_since_restore: 11.418162822723389\n",
" time_this_iter_s: 0.11828303337097168\n",
" time_total_s: 11.418162822723389\n",
" timestamp: 1658499812\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: d4f50ce4\n",
" warmup_time: 0.0027348995208740234\n",
" \n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Result for objective_dd4b4e94:\n",
" date: 2022-07-22_15-23-35\n",
" done: false\n",
" experiment_id: c6036d4a26674f86946fd7c8c895e85c\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 25.460850093028082\n",
" neg_mean_loss: -25.460850093028082\n",
" node_ip: 127.0.0.1\n",
" pid: 46323\n",
" time_since_restore: 0.10411691665649414\n",
" time_this_iter_s: 0.10411691665649414\n",
" time_total_s: 0.10411691665649414\n",
" timestamp: 1658499815\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: dd4b4e94\n",
" warmup_time: 0.003142118453979492\n",
" \n",
"Result for objective_dd487b56:\n",
" date: 2022-07-22_15-23-35\n",
" done: false\n",
" experiment_id: 75e118a5733e454fab54f4a20c036852\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 17.401819651158622\n",
" neg_mean_loss: -17.401819651158622\n",
" node_ip: 127.0.0.1\n",
" pid: 46322\n",
" time_since_restore: 0.1048891544342041\n",
" time_this_iter_s: 0.1048891544342041\n",
" time_total_s: 0.1048891544342041\n",
" timestamp: 1658499815\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: dd487b56\n",
" warmup_time: 0.0028679370880126953\n",
" \n",
"Result for objective_dd5a8bca:\n",
" date: 2022-07-22_15-23-35\n",
" done: false\n",
" experiment_id: 775683e6d95a4705846a37e809b474d7\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 10.522131438315931\n",
" neg_mean_loss: -10.522131438315931\n",
" node_ip: 127.0.0.1\n",
" pid: 46328\n",
" time_since_restore: 0.10494732856750488\n",
" time_this_iter_s: 0.10494732856750488\n",
" time_total_s: 0.10494732856750488\n",
" timestamp: 1658499815\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: dd5a8bca\n",
" warmup_time: 0.002931833267211914\n",
" \n",
"Result for objective_db5c0402:\n",
" date: 2022-07-22_15-23-36\n",
" done: false\n",
" experiment_id: 69a34f42e51f4b97a5237fa5ae2be8d8\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" mean_loss: -5.230895562358142\n",
" neg_mean_loss: 5.230895562358142\n",
" node_ip: 127.0.0.1\n",
" pid: 46317\n",
" time_since_restore: 5.120880842208862\n",
" time_this_iter_s: 0.1076059341430664\n",
" time_total_s: 5.120880842208862\n",
" timestamp: 1658499816\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: db5c0402\n",
" warmup_time: 0.003406047821044922\n",
" \n",
"Result for objective_dd4b4e94:\n",
" date: 2022-07-22_15-23-40\n",
" done: false\n",
" experiment_id: c6036d4a26674f86946fd7c8c895e85c\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" mean_loss: 16.929346028466988\n",
" neg_mean_loss: -16.929346028466988\n",
" node_ip: 127.0.0.1\n",
" pid: 46323\n",
" time_since_restore: 5.137623071670532\n",
" time_this_iter_s: 0.10670709609985352\n",
" time_total_s: 5.137623071670532\n",
" timestamp: 1658499820\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: dd4b4e94\n",
" warmup_time: 0.003142118453979492\n",
" \n",
"Result for objective_dd487b56:\n",
" date: 2022-07-22_15-23-40\n",
" done: false\n",
" experiment_id: 75e118a5733e454fab54f4a20c036852\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" mean_loss: 9.592430453048937\n",
" neg_mean_loss: -9.592430453048937\n",
" node_ip: 127.0.0.1\n",
" pid: 46322\n",
" time_since_restore: 5.140692949295044\n",
" time_this_iter_s: 0.1086587905883789\n",
" time_total_s: 5.140692949295044\n",
" timestamp: 1658499820\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: dd487b56\n",
" warmup_time: 0.0028679370880126953\n",
" \n",
"Result for objective_dd5a8bca:\n",
" date: 2022-07-22_15-23-40\n",
" done: false\n",
" experiment_id: 775683e6d95a4705846a37e809b474d7\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" mean_loss: 0.6450610360030851\n",
" neg_mean_loss: -0.6450610360030851\n",
" node_ip: 127.0.0.1\n",
" pid: 46328\n",
" time_since_restore: 5.138496160507202\n",
" time_this_iter_s: 0.10809993743896484\n",
" time_total_s: 5.138496160507202\n",
" timestamp: 1658499820\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: dd5a8bca\n",
" warmup_time: 0.002931833267211914\n",
" \n",
"Result for objective_db5c0402:\n",
" date: 2022-07-22_15-23-41\n",
" done: false\n",
" experiment_id: 69a34f42e51f4b97a5237fa5ae2be8d8\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 94\n",
" iterations_since_restore: 95\n",
" mean_loss: -5.321109063262834\n",
" neg_mean_loss: 5.321109063262834\n",
" node_ip: 127.0.0.1\n",
" pid: 46317\n",
" time_since_restore: 10.162945985794067\n",
" time_this_iter_s: 0.10671615600585938\n",
" time_total_s: 10.162945985794067\n",
" timestamp: 1658499821\n",
" timesteps_since_restore: 0\n",
" training_iteration: 95\n",
" trial_id: db5c0402\n",
" warmup_time: 0.003406047821044922\n",
" \n",
"Result for objective_db5c0402:\n",
" date: 2022-07-22_15-23-42\n",
" done: true\n",
" experiment_id: 69a34f42e51f4b97a5237fa5ae2be8d8\n",
" experiment_tag: 5_activation=tanh,height=-54.1300,steps=100,width=11.4712\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: -5.325709301325668\n",
" neg_mean_loss: 5.325709301325668\n",
" node_ip: 127.0.0.1\n",
" pid: 46317\n",
" time_since_restore: 10.704953908920288\n",
" time_this_iter_s: 0.1082158088684082\n",
" time_total_s: 10.704953908920288\n",
" timestamp: 1658499822\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: db5c0402\n",
" warmup_time: 0.003406047821044922\n",
" \n",
"Result for objective_dd4b4e94:\n",
" date: 2022-07-22_15-23-45\n",
" done: false\n",
" experiment_id: c6036d4a26674f86946fd7c8c895e85c\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 94\n",
" iterations_since_restore: 95\n",
" mean_loss: 16.253282229309562\n",
" neg_mean_loss: -16.253282229309562\n",
" node_ip: 127.0.0.1\n",
" pid: 46323\n",
" time_since_restore: 10.140441179275513\n",
" time_this_iter_s: 0.10501408576965332\n",
" time_total_s: 10.140441179275513\n",
" timestamp: 1658499825\n",
" timesteps_since_restore: 0\n",
" training_iteration: 95\n",
" trial_id: dd4b4e94\n",
" warmup_time: 0.003142118453979492\n",
" \n",
"Result for objective_dd487b56:\n",
" date: 2022-07-22_15-23-45\n",
" done: false\n",
" experiment_id: 75e118a5733e454fab54f4a20c036852\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 94\n",
" iterations_since_restore: 95\n",
" mean_loss: 8.631851056235014\n",
" neg_mean_loss: -8.631851056235014\n",
" node_ip: 127.0.0.1\n",
" pid: 46322\n",
" time_since_restore: 10.156053066253662\n",
" time_this_iter_s: 0.10682201385498047\n",
" time_total_s: 10.156053066253662\n",
" timestamp: 1658499825\n",
" timesteps_since_restore: 0\n",
" training_iteration: 95\n",
" trial_id: dd487b56\n",
" warmup_time: 0.0028679370880126953\n",
" \n",
"Result for objective_e3482178:\n",
" date: 2022-07-22_15-23-45\n",
" done: false\n",
" experiment_id: 6709b686be234eb996457210f3efbe08\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 17.884403838898738\n",
" neg_mean_loss: -17.884403838898738\n",
" node_ip: 127.0.0.1\n",
" pid: 46341\n",
" time_since_restore: 0.10499191284179688\n",
" time_this_iter_s: 0.10499191284179688\n",
" time_total_s: 0.10499191284179688\n",
" timestamp: 1658499825\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: e3482178\n",
" warmup_time: 0.002877950668334961\n",
" \n",
"Result for objective_dd5a8bca:\n",
" date: 2022-07-22_15-23-45\n",
" done: false\n",
" experiment_id: 775683e6d95a4705846a37e809b474d7\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 94\n",
" iterations_since_restore: 95\n",
" mean_loss: 0.583976365762005\n",
" neg_mean_loss: -0.583976365762005\n",
" node_ip: 127.0.0.1\n",
" pid: 46328\n",
" time_since_restore: 10.165286302566528\n",
" time_this_iter_s: 0.10989904403686523\n",
" time_total_s: 10.165286302566528\n",
" timestamp: 1658499825\n",
" timesteps_since_restore: 0\n",
" training_iteration: 95\n",
" trial_id: dd5a8bca\n",
" warmup_time: 0.002931833267211914\n",
" \n",
"Result for objective_dd4b4e94:\n",
" date: 2022-07-22_15-23-45\n",
" done: true\n",
" experiment_id: c6036d4a26674f86946fd7c8c895e85c\n",
" experiment_tag: 7_activation=relu,height=54.6085,steps=100,width=1.2361\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: 16.216283787762706\n",
" neg_mean_loss: -16.216283787762706\n",
" node_ip: 127.0.0.1\n",
" pid: 46323\n",
" time_since_restore: 10.67109203338623\n",
" time_this_iter_s: 0.1058969497680664\n",
" time_total_s: 10.67109203338623\n",
" timestamp: 1658499825\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: dd4b4e94\n",
" warmup_time: 0.003142118453979492\n",
" \n",
"Result for objective_dd487b56:\n",
" date: 2022-07-22_15-23-45\n",
" done: true\n",
" experiment_id: 75e118a5733e454fab54f4a20c036852\n",
" experiment_tag: 6_activation=relu,height=-25.9818,steps=100,width=0.7585\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: 8.577028987245935\n",
" neg_mean_loss: -8.577028987245935\n",
" node_ip: 127.0.0.1\n",
" pid: 46322\n",
" time_since_restore: 10.691627979278564\n",
" time_this_iter_s: 0.10821914672851562\n",
" time_total_s: 10.691627979278564\n",
" timestamp: 1658499825\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: dd487b56\n",
" warmup_time: 0.0028679370880126953\n",
" \n",
"Result for objective_dd5a8bca:\n",
" date: 2022-07-22_15-23-45\n",
" done: true\n",
" experiment_id: 775683e6d95a4705846a37e809b474d7\n",
" experiment_tag: 8_activation=tanh,height=5.2213,steps=100,width=17.0952\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: 0.5808712318418436\n",
" neg_mean_loss: -0.5808712318418436\n",
" node_ip: 127.0.0.1\n",
" pid: 46328\n",
" time_since_restore: 10.701741218566895\n",
" time_this_iter_s: 0.11321616172790527\n",
" time_total_s: 10.701741218566895\n",
" timestamp: 1658499825\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: dd5a8bca\n",
" warmup_time: 0.002931833267211914\n",
" \n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Result for objective_e532d6e0:\n",
" date: 2022-07-22_15-23-48\n",
" done: false\n",
" experiment_id: 34d05fd425474b82a450c22c2a063843\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 3.30012197692201\n",
" neg_mean_loss: -3.30012197692201\n",
" node_ip: 127.0.0.1\n",
" pid: 46353\n",
" time_since_restore: 0.10400509834289551\n",
" time_this_iter_s: 0.10400509834289551\n",
" time_total_s: 0.10400509834289551\n",
" timestamp: 1658499828\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: e532d6e0\n",
" warmup_time: 0.0028901100158691406\n",
" \n",
"Result for objective_e3482178:\n",
" date: 2022-07-22_15-23-50\n",
" done: false\n",
" experiment_id: 6709b686be234eb996457210f3efbe08\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 29\n",
" iterations_since_restore: 30\n",
" mean_loss: 8.10192352359083\n",
" neg_mean_loss: -8.10192352359083\n",
" node_ip: 127.0.0.1\n",
" pid: 46341\n",
" time_since_restore: 5.183844089508057\n",
" time_this_iter_s: 0.10574007034301758\n",
" time_total_s: 5.183844089508057\n",
" timestamp: 1658499830\n",
" timesteps_since_restore: 0\n",
" training_iteration: 30\n",
" trial_id: e3482178\n",
" warmup_time: 0.002877950668334961\n",
" \n",
"Result for objective_e532d6e0:\n",
" date: 2022-07-22_15-23-53\n",
" done: false\n",
" experiment_id: 34d05fd425474b82a450c22c2a063843\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 48\n",
" iterations_since_restore: 49\n",
" mean_loss: -6.416785967274913\n",
" neg_mean_loss: 6.416785967274913\n",
" node_ip: 127.0.0.1\n",
" pid: 46353\n",
" time_since_restore: 5.212116003036499\n",
" time_this_iter_s: 0.10714387893676758\n",
" time_total_s: 5.212116003036499\n",
" timestamp: 1658499833\n",
" timesteps_since_restore: 0\n",
" training_iteration: 49\n",
" trial_id: e532d6e0\n",
" warmup_time: 0.0028901100158691406\n",
" \n",
"Result for objective_e3482178:\n",
" date: 2022-07-22_15-23-55\n",
" done: false\n",
" experiment_id: 6709b686be234eb996457210f3efbe08\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 76\n",
" iterations_since_restore: 77\n",
" mean_loss: 7.968536513457016\n",
" neg_mean_loss: -7.968536513457016\n",
" node_ip: 127.0.0.1\n",
" pid: 46341\n",
" time_since_restore: 10.190052032470703\n",
" time_this_iter_s: 0.10843491554260254\n",
" time_total_s: 10.190052032470703\n",
" timestamp: 1658499835\n",
" timesteps_since_restore: 0\n",
" training_iteration: 77\n",
" trial_id: e3482178\n",
" warmup_time: 0.002877950668334961\n",
" \n",
"Result for objective_e3482178:\n",
" date: 2022-07-22_15-23-57\n",
" done: true\n",
" experiment_id: 6709b686be234eb996457210f3efbe08\n",
" experiment_tag: 9_activation=tanh,height=78.8440,steps=100,width=15.5079\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: 7.9491170267942515\n",
" neg_mean_loss: -7.9491170267942515\n",
" node_ip: 127.0.0.1\n",
" pid: 46341\n",
" time_since_restore: 12.64172887802124\n",
" time_this_iter_s: 0.10761594772338867\n",
" time_total_s: 12.64172887802124\n",
" timestamp: 1658499837\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: e3482178\n",
" warmup_time: 0.002877950668334961\n",
" \n",
"Result for objective_e532d6e0:\n",
" date: 2022-07-22_15-23-58\n",
" done: false\n",
" experiment_id: 34d05fd425474b82a450c22c2a063843\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 95\n",
" iterations_since_restore: 96\n",
" mean_loss: -6.554810275125985\n",
" neg_mean_loss: 6.554810275125985\n",
" node_ip: 127.0.0.1\n",
" pid: 46353\n",
" time_since_restore: 10.305040121078491\n",
" time_this_iter_s: 0.10702300071716309\n",
" time_total_s: 10.305040121078491\n",
" timestamp: 1658499838\n",
" timesteps_since_restore: 0\n",
" training_iteration: 96\n",
" trial_id: e532d6e0\n",
" warmup_time: 0.0028901100158691406\n",
" \n",
"Result for objective_e532d6e0:\n",
" date: 2022-07-22_15-23-58\n",
" done: true\n",
" experiment_id: 34d05fd425474b82a450c22c2a063843\n",
" experiment_tag: 10_activation=tanh,height=-66.9988,steps=100,width=7.1509\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: -6.560589957038967\n",
" neg_mean_loss: 6.560589957038967\n",
" node_ip: 127.0.0.1\n",
" pid: 46353\n",
" time_since_restore: 10.74857211112976\n",
" time_this_iter_s: 0.1075899600982666\n",
" time_total_s: 10.74857211112976\n",
" timestamp: 1658499838\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: e532d6e0\n",
" warmup_time: 0.0028901100158691406\n",
" \n"
]
}
],
2022-03-25 01:04:53 -07:00
"source": [
2022-07-24 18:53:57 +01:00
"tuner = tune.Tuner(\n",
2022-03-25 01:04:53 -07:00
" objective,\n",
2022-07-24 18:53:57 +01:00
" tune_config=tune.TuneConfig(\n",
" search_alg=algo,\n",
" num_samples=num_samples,\n",
" ),\n",
")\n",
"results = tuner.fit()"
2022-04-25 11:10:58 -07:00
]
},
{
"cell_type": "markdown",
"id": "11e1ee04",
"metadata": {},
"source": [
"We take a look again at the optimal hyperparameters."
]
},
{
"cell_type": "code",
2022-07-24 18:53:57 +01:00
"execution_count": 21,
2022-04-25 11:10:58 -07:00
"id": "13e4ce18",
"metadata": {
"lines_to_next_cell": 0
},
2022-07-24 18:53:57 +01:00
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Best hyperparameters for loss found were: {'activation': 'tanh', 'width': 17.73584230792165, 'height': -76.43866128895999, 'steps': 100}\n"
]
}
],
2022-04-25 11:10:58 -07:00
"source": [
2022-07-24 18:53:57 +01:00
"print(\"Best hyperparameters for loss found were: \", results.get_best_result(\"mean_loss\", \"min\").config)"
2022-03-25 01:04:53 -07:00
]
},
{
"cell_type": "markdown",
2022-04-25 11:10:58 -07:00
"id": "34bbd066",
2022-03-25 01:04:53 -07:00
"metadata": {},
"source": [
2022-04-25 11:10:58 -07:00
"## Multi-objective optimization\n",
"\n",
2022-03-25 01:04:53 -07:00
"Finally, let's take a look at the multi-objective case."
]
},
{
"cell_type": "code",
2022-07-24 18:53:57 +01:00
"execution_count": 22,
2022-04-25 11:10:58 -07:00
"id": "b233cbea",
2022-03-25 01:04:53 -07:00
"metadata": {},
"outputs": [],
"source": [
"def multi_objective(config):\n",
" # Hyperparameters\n",
" width, height = config[\"width\"], config[\"height\"]\n",
"\n",
" for step in range(config[\"steps\"]):\n",
" # Iterative training function - can be any arbitrary training procedure\n",
" intermediate_score = evaluate(step, config[\"width\"], config[\"height\"], config[\"activation\"])\n",
" # Feed the score back back to Tune.\n",
2022-06-30 10:37:31 -07:00
" session.report({\n",
" \"iterations\": step, \"loss\": intermediate_score, \"gain\": intermediate_score * width\n",
" })"
2022-03-25 01:04:53 -07:00
]
},
{
"cell_type": "markdown",
2022-04-25 11:10:58 -07:00
"id": "338e5108",
2022-03-25 01:04:53 -07:00
"metadata": {},
"source": [
"We define the `OptunaSearch` object this time with metric and mode as list arguments."
]
},
{
"cell_type": "code",
2022-07-24 18:53:57 +01:00
"execution_count": 23,
2022-04-25 11:10:58 -07:00
"id": "624d0bc8",
2022-03-25 01:04:53 -07:00
"metadata": {},
2022-07-24 18:53:57 +01:00
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"\u001b[32m[I 2022-07-22 15:26:50,680]\u001b[0m A new study created in memory with name: optuna\u001b[0m\n"
]
},
{
"data": {
"text/html": [
"== Status ==<br>Current time: 2022-07-22 15:27:34 (running for 00:00:43.55)<br>Memory usage on this node: 8.2/16.0 GiB<br>Using FIFO scheduling algorithm.<br>Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/5.07 GiB heap, 0.0/2.0 GiB objects<br>Result logdir: /Users/kai/ray_results/multi_objective_2022-07-22_15-26-50<br>Number of trials: 10/10 (10 TERMINATED)<br><table>\n",
"<thead>\n",
"<tr><th>Trial name </th><th>status </th><th>loc </th><th>activation </th><th style=\"text-align: right;\"> height</th><th style=\"text-align: right;\"> width</th><th style=\"text-align: right;\"> iter</th><th style=\"text-align: right;\"> total time (s)</th><th style=\"text-align: right;\"> iterations</th><th style=\"text-align: right;\"> loss</th><th style=\"text-align: right;\"> gain</th></tr>\n",
"</thead>\n",
"<tbody>\n",
"<tr><td>multi_objective_5378f1e8</td><td>TERMINATED</td><td>127.0.0.1:46621</td><td>tanh </td><td style=\"text-align: right;\">-47.5236</td><td style=\"text-align: right;\">10.9256 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 10.7064</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\">-4.66076</td><td style=\"text-align: right;\">-50.9216 </td></tr>\n",
"<tr><td>multi_objective_550962f4</td><td>TERMINATED</td><td>127.0.0.1:46631</td><td>relu </td><td style=\"text-align: right;\">-40.6828</td><td style=\"text-align: right;\"> 4.34953</td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 11.3948</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\"> 6.15869</td><td style=\"text-align: right;\"> 26.7874 </td></tr>\n",
"<tr><td>multi_objective_550ad9a4</td><td>TERMINATED</td><td>127.0.0.1:46632</td><td>tanh </td><td style=\"text-align: right;\">-42.0563</td><td style=\"text-align: right;\"> 2.02124</td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 11.2755</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\">-3.72967</td><td style=\"text-align: right;\"> -7.53857</td></tr>\n",
"<tr><td>multi_objective_550c4faa</td><td>TERMINATED</td><td>127.0.0.1:46633</td><td>relu </td><td style=\"text-align: right;\">-56.2838</td><td style=\"text-align: right;\"> 5.65237</td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 11.407 </td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\"> 4.54719</td><td style=\"text-align: right;\"> 25.7024 </td></tr>\n",
"<tr><td>multi_objective_5b6c4ba2</td><td>TERMINATED</td><td>127.0.0.1:46679</td><td>relu </td><td style=\"text-align: right;\"> 54.9133</td><td style=\"text-align: right;\">16.9039 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 10.6698</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\">15.5507 </td><td style=\"text-align: right;\">262.867 </td></tr>\n",
"<tr><td>multi_objective_5d51f2d2</td><td>TERMINATED</td><td>127.0.0.1:46686</td><td>tanh </td><td style=\"text-align: right;\"> 16.5793</td><td style=\"text-align: right;\"> 5.16709</td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 10.7451</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\"> 1.84967</td><td style=\"text-align: right;\"> 9.55744</td></tr>\n",
"<tr><td>multi_objective_5d636760</td><td>TERMINATED</td><td>127.0.0.1:46689</td><td>tanh </td><td style=\"text-align: right;\">-13.8895</td><td style=\"text-align: right;\">13.6043 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 10.6897</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\">-1.31525</td><td style=\"text-align: right;\">-17.8931 </td></tr>\n",
"<tr><td>multi_objective_5d66093e</td><td>TERMINATED</td><td>127.0.0.1:46690</td><td>relu </td><td style=\"text-align: right;\">-51.6714</td><td style=\"text-align: right;\">19.8211 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 10.7246</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\"> 4.88357</td><td style=\"text-align: right;\"> 96.7974 </td></tr>\n",
"<tr><td>multi_objective_634c9444</td><td>TERMINATED</td><td>127.0.0.1:46710</td><td>relu </td><td style=\"text-align: right;\">-12.3384</td><td style=\"text-align: right;\">15.1097 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 13.1986</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\"> 8.83256</td><td style=\"text-align: right;\">133.457 </td></tr>\n",
"<tr><td>multi_objective_65469a2e</td><td>TERMINATED</td><td>127.0.0.1:46719</td><td>tanh </td><td style=\"text-align: right;\"> 19.6774</td><td style=\"text-align: right;\">13.6172 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 10.7279</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\"> 2.04137</td><td style=\"text-align: right;\"> 27.7978 </td></tr>\n",
"</tbody>\n",
"</table><br><br>"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Result for multi_objective_5378f1e8:\n",
" date: 2022-07-22_15-26-53\n",
" done: false\n",
" experiment_id: 4ce03cab420a449a9f4827c3b6b76c38\n",
" gain: 57.33359096413655\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" loss: 5.247638036988612\n",
" node_ip: 127.0.0.1\n",
" pid: 46621\n",
" time_since_restore: 0.10418081283569336\n",
" time_this_iter_s: 0.10418081283569336\n",
" time_total_s: 0.10418081283569336\n",
" timestamp: 1658500013\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: 5378f1e8\n",
" warmup_time: 0.002785921096801758\n",
" \n",
"Result for multi_objective_550ad9a4:\n",
" date: 2022-07-22_15-26-56\n",
" done: false\n",
" experiment_id: 102a4b06449b428b9870a75d02812be6\n",
" gain: 11.711822825618256\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" loss: 5.794371166191576\n",
" node_ip: 127.0.0.1\n",
" pid: 46632\n",
" time_since_restore: 0.10288000106811523\n",
" time_this_iter_s: 0.10288000106811523\n",
" time_total_s: 0.10288000106811523\n",
" timestamp: 1658500016\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: 550ad9a4\n",
" warmup_time: 0.0030989646911621094\n",
" \n",
"Result for multi_objective_550c4faa:\n",
" date: 2022-07-22_15-26-56\n",
" done: false\n",
" experiment_id: 538d9621144b4ecea8c91ec7b70c8a9d\n",
" gain: 81.23378320141829\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" loss: 14.371624912716872\n",
" node_ip: 127.0.0.1\n",
" pid: 46633\n",
" time_since_restore: 0.10215306282043457\n",
" time_this_iter_s: 0.10215306282043457\n",
" time_total_s: 0.10215306282043457\n",
" timestamp: 1658500016\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: 550c4faa\n",
" warmup_time: 0.0036542415618896484\n",
" \n",
"Result for multi_objective_550962f4:\n",
" date: 2022-07-22_15-26-56\n",
" done: false\n",
" experiment_id: b916abdd07bd4e7cb4e9a1451cd76d64\n",
" gain: 69.29543163212045\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" loss: 15.93172358423837\n",
" node_ip: 127.0.0.1\n",
" pid: 46631\n",
" time_since_restore: 0.10244321823120117\n",
" time_this_iter_s: 0.10244321823120117\n",
" time_total_s: 0.10244321823120117\n",
" timestamp: 1658500016\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: 550962f4\n",
" warmup_time: 0.003384828567504883\n",
" \n",
"Result for multi_objective_5378f1e8:\n",
" date: 2022-07-22_15-26-58\n",
" done: false\n",
" experiment_id: 4ce03cab420a449a9f4827c3b6b76c38\n",
" gain: -49.83538614085625\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" loss: -4.561341152769549\n",
" node_ip: 127.0.0.1\n",
" pid: 46621\n",
" time_since_restore: 5.1368408203125\n",
" time_this_iter_s: 0.1180417537689209\n",
" time_total_s: 5.1368408203125\n",
" timestamp: 1658500018\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: 5378f1e8\n",
" warmup_time: 0.002785921096801758\n",
" \n",
"Result for multi_objective_550ad9a4:\n",
" date: 2022-07-22_15-27-01\n",
" done: false\n",
" experiment_id: 102a4b06449b428b9870a75d02812be6\n",
" gain: -6.575568930702184\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" loss: -3.2532328725144097\n",
" node_ip: 127.0.0.1\n",
" pid: 46632\n",
" time_since_restore: 5.146106719970703\n",
" time_this_iter_s: 0.10509181022644043\n",
" time_total_s: 5.146106719970703\n",
" timestamp: 1658500021\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: 550ad9a4\n",
" warmup_time: 0.0030989646911621094\n",
" \n",
"Result for multi_objective_550c4faa:\n",
" date: 2022-07-22_15-27-01\n",
" done: false\n",
" experiment_id: 538d9621144b4ecea8c91ec7b70c8a9d\n",
" gain: 26.760529995257745\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" loss: 4.73438862995905\n",
" node_ip: 127.0.0.1\n",
" pid: 46633\n",
" time_since_restore: 5.1661481857299805\n",
" time_this_iter_s: 0.10656309127807617\n",
" time_total_s: 5.1661481857299805\n",
" timestamp: 1658500021\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: 550c4faa\n",
" warmup_time: 0.0036542415618896484\n",
" \n",
"Result for multi_objective_550962f4:\n",
" date: 2022-07-22_15-27-01\n",
" done: false\n",
" experiment_id: b916abdd07bd4e7cb4e9a1451cd76d64\n",
" gain: 27.828615153405433\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" loss: 6.398081286367129\n",
" node_ip: 127.0.0.1\n",
" pid: 46631\n",
" time_since_restore: 5.169018983840942\n",
" time_this_iter_s: 0.10616683959960938\n",
" time_total_s: 5.169018983840942\n",
" timestamp: 1658500021\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: 550962f4\n",
" warmup_time: 0.003384828567504883\n",
" \n",
"Result for multi_objective_5378f1e8:\n",
" date: 2022-07-22_15-27-03\n",
" done: false\n",
" experiment_id: 4ce03cab420a449a9f4827c3b6b76c38\n",
" gain: -50.86883186480195\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 94\n",
" iterations_since_restore: 95\n",
" loss: -4.65593053743818\n",
" node_ip: 127.0.0.1\n",
" pid: 46621\n",
" time_since_restore: 10.17048692703247\n",
" time_this_iter_s: 0.10630917549133301\n",
" time_total_s: 10.17048692703247\n",
" timestamp: 1658500023\n",
" timesteps_since_restore: 0\n",
" training_iteration: 95\n",
" trial_id: 5378f1e8\n",
" warmup_time: 0.002785921096801758\n",
" \n",
"Result for multi_objective_5378f1e8:\n",
" date: 2022-07-22_15-27-04\n",
" done: true\n",
" experiment_id: 4ce03cab420a449a9f4827c3b6b76c38\n",
" experiment_tag: 1_activation=tanh,height=-47.5236,steps=100,width=10.9256\n",
" gain: -50.92155508732405\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" loss: -4.66075619693878\n",
" node_ip: 127.0.0.1\n",
" pid: 46621\n",
" time_since_restore: 10.706397771835327\n",
" time_this_iter_s: 0.10694766044616699\n",
" time_total_s: 10.706397771835327\n",
" timestamp: 1658500024\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: 5378f1e8\n",
" warmup_time: 0.002785921096801758\n",
" \n",
"Result for multi_objective_550ad9a4:\n",
" date: 2022-07-22_15-27-05\n",
" done: false\n",
" experiment_id: 102a4b06449b428b9870a75d02812be6\n",
" gain: -7.479635489825483\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 93\n",
" iterations_since_restore: 94\n",
" loss: -3.700515697784228\n",
" node_ip: 127.0.0.1\n",
" pid: 46632\n",
" time_since_restore: 10.056212902069092\n",
" time_this_iter_s: 0.10589313507080078\n",
" time_total_s: 10.056212902069092\n",
" timestamp: 1658500025\n",
" timesteps_since_restore: 0\n",
" training_iteration: 94\n",
" trial_id: 550ad9a4\n",
" warmup_time: 0.0030989646911621094\n",
" \n",
"Result for multi_objective_550c4faa:\n",
" date: 2022-07-22_15-27-05\n",
" done: false\n",
" experiment_id: 538d9621144b4ecea8c91ec7b70c8a9d\n",
" gain: 25.776502806907246\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 92\n",
" iterations_since_restore: 93\n",
" loss: 4.5602976409942295\n",
" node_ip: 127.0.0.1\n",
" pid: 46633\n",
" time_since_restore: 9.99274206161499\n",
" time_this_iter_s: 0.10583209991455078\n",
" time_total_s: 9.99274206161499\n",
" timestamp: 1658500025\n",
" timesteps_since_restore: 0\n",
" training_iteration: 93\n",
" trial_id: 550c4faa\n",
" warmup_time: 0.0036542415618896484\n",
" \n",
"Result for multi_objective_550962f4:\n",
" date: 2022-07-22_15-27-05\n",
" done: false\n",
" experiment_id: b916abdd07bd4e7cb4e9a1451cd76d64\n",
" gain: 26.860636112316993\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 92\n",
" iterations_since_restore: 93\n",
" loss: 6.175533072801938\n",
" node_ip: 127.0.0.1\n",
" pid: 46631\n",
" time_since_restore: 9.971143007278442\n",
" time_this_iter_s: 0.1054527759552002\n",
" time_total_s: 9.971143007278442\n",
" timestamp: 1658500025\n",
" timesteps_since_restore: 0\n",
" training_iteration: 93\n",
" trial_id: 550962f4\n",
" warmup_time: 0.003384828567504883\n",
" \n",
"Result for multi_objective_5b6c4ba2:\n",
" date: 2022-07-22_15-27-06\n",
" done: false\n",
" experiment_id: f143c125a0734166ac832270ac08dbb6\n",
" gain: 430.90195437776714\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" loss: 25.491329340230966\n",
" node_ip: 127.0.0.1\n",
" pid: 46679\n",
" time_since_restore: 0.10215902328491211\n",
" time_this_iter_s: 0.10215902328491211\n",
" time_total_s: 0.10215902328491211\n",
" timestamp: 1658500026\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: 5b6c4ba2\n",
" warmup_time: 0.002663135528564453\n",
" \n",
"Result for multi_objective_550ad9a4:\n",
" date: 2022-07-22_15-27-07\n",
" done: true\n",
" experiment_id: 102a4b06449b428b9870a75d02812be6\n",
" experiment_tag: 3_activation=tanh,height=-42.0563,steps=100,width=2.0212\n",
" gain: -7.538566547680232\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" loss: -3.729671571085163\n",
" node_ip: 127.0.0.1\n",
" pid: 46632\n",
" time_since_restore: 11.27550482749939\n",
" time_this_iter_s: 0.10642290115356445\n",
" time_total_s: 11.27550482749939\n",
" timestamp: 1658500027\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: 550ad9a4\n",
" warmup_time: 0.0030989646911621094\n",
" \n",
"Result for multi_objective_550962f4:\n",
" date: 2022-07-22_15-27-07\n",
" done: true\n",
" experiment_id: b916abdd07bd4e7cb4e9a1451cd76d64\n",
" experiment_tag: 2_activation=relu,height=-40.6828,steps=100,width=4.3495\n",
" gain: 26.787356208375382\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" loss: 6.15868527856241\n",
" node_ip: 127.0.0.1\n",
" pid: 46631\n",
" time_since_restore: 11.394789934158325\n",
" time_this_iter_s: 0.10491585731506348\n",
" time_total_s: 11.394789934158325\n",
" timestamp: 1658500027\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: 550962f4\n",
" warmup_time: 0.003384828567504883\n",
" \n",
"Result for multi_objective_550c4faa:\n",
" date: 2022-07-22_15-27-07\n",
" done: true\n",
" experiment_id: 538d9621144b4ecea8c91ec7b70c8a9d\n",
" experiment_tag: 4_activation=relu,height=-56.2838,steps=100,width=5.6524\n",
" gain: 25.702421216002982\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" loss: 4.547191359402326\n",
" node_ip: 127.0.0.1\n",
" pid: 46633\n",
" time_since_restore: 11.406961917877197\n",
" time_this_iter_s: 0.12177181243896484\n",
" time_total_s: 11.406961917877197\n",
" timestamp: 1658500027\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: 550c4faa\n",
" warmup_time: 0.0036542415618896484\n",
" \n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Result for multi_objective_5d51f2d2:\n",
" date: 2022-07-22_15-27-09\n",
" done: false\n",
" experiment_id: 1433194530b14db4b77ce40a69d65407\n",
" gain: 60.23765295445784\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" loss: 11.657934184433229\n",
" node_ip: 127.0.0.1\n",
" pid: 46686\n",
" time_since_restore: 0.10274410247802734\n",
" time_this_iter_s: 0.10274410247802734\n",
" time_total_s: 0.10274410247802734\n",
" timestamp: 1658500029\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: 5d51f2d2\n",
" warmup_time: 0.0028769969940185547\n",
" \n",
"Result for multi_objective_5d636760:\n",
" date: 2022-07-22_15-27-10\n",
" done: false\n",
" experiment_id: b8dccac5d8914569b1c2ca061ad47a56\n",
" gain: 117.14725547845826\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" loss: 8.611046741256526\n",
" node_ip: 127.0.0.1\n",
" pid: 46689\n",
" time_since_restore: 0.10460686683654785\n",
" time_this_iter_s: 0.10460686683654785\n",
" time_total_s: 0.10460686683654785\n",
" timestamp: 1658500030\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: 5d636760\n",
" warmup_time: 0.0028090476989746094\n",
" \n",
"Result for multi_objective_5d66093e:\n",
" date: 2022-07-22_15-27-10\n",
" done: false\n",
" experiment_id: e7259e986278458899b46b485e069ec9\n",
" gain: 294.00300634729234\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" loss: 14.832864636325777\n",
" node_ip: 127.0.0.1\n",
" pid: 46690\n",
" time_since_restore: 0.10294318199157715\n",
" time_this_iter_s: 0.10294318199157715\n",
" time_total_s: 0.10294318199157715\n",
" timestamp: 1658500030\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: 5d66093e\n",
" warmup_time: 0.0027511119842529297\n",
" \n",
"Result for multi_objective_5b6c4ba2:\n",
" date: 2022-07-22_15-27-11\n",
" done: false\n",
" experiment_id: f143c125a0734166ac832270ac08dbb6\n",
" gain: 263.92128797150474\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 48\n",
" iterations_since_restore: 49\n",
" loss: 15.6130748612977\n",
" node_ip: 127.0.0.1\n",
" pid: 46679\n",
" time_since_restore: 5.2078468799591064\n",
" time_this_iter_s: 0.10803008079528809\n",
" time_total_s: 5.2078468799591064\n",
" timestamp: 1658500031\n",
" timesteps_since_restore: 0\n",
" training_iteration: 49\n",
" trial_id: 5b6c4ba2\n",
" warmup_time: 0.002663135528564453\n",
" \n",
"Result for multi_objective_5d51f2d2:\n",
" date: 2022-07-22_15-27-14\n",
" done: false\n",
" experiment_id: 1433194530b14db4b77ce40a69d65407\n",
" gain: 10.610216987384797\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" loss: 2.053420165872076\n",
" node_ip: 127.0.0.1\n",
" pid: 46686\n",
" time_since_restore: 5.178762912750244\n",
" time_this_iter_s: 0.10710978507995605\n",
" time_total_s: 5.178762912750244\n",
" timestamp: 1658500034\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: 5d51f2d2\n",
" warmup_time: 0.0028769969940185547\n",
" \n",
"Result for multi_objective_5d636760:\n",
" date: 2022-07-22_15-27-15\n",
" done: false\n",
" experiment_id: b8dccac5d8914569b1c2ca061ad47a56\n",
" gain: -16.800839364465826\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" loss: -1.234965449842409\n",
" node_ip: 127.0.0.1\n",
" pid: 46689\n",
" time_since_restore: 5.146665811538696\n",
" time_this_iter_s: 0.10526585578918457\n",
" time_total_s: 5.146665811538696\n",
" timestamp: 1658500035\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: 5d636760\n",
" warmup_time: 0.0028090476989746094\n",
" \n",
"Result for multi_objective_5d66093e:\n",
" date: 2022-07-22_15-27-15\n",
" done: false\n",
" experiment_id: e7259e986278458899b46b485e069ec9\n",
" gain: 97.89753254610552\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" loss: 4.939068026982622\n",
" node_ip: 127.0.0.1\n",
" pid: 46690\n",
" time_since_restore: 5.1532862186431885\n",
" time_this_iter_s: 0.10752320289611816\n",
" time_total_s: 5.1532862186431885\n",
" timestamp: 1658500035\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: 5d66093e\n",
" warmup_time: 0.0027511119842529297\n",
" \n",
"Result for multi_objective_5b6c4ba2:\n",
" date: 2022-07-22_15-27-16\n",
" done: false\n",
" experiment_id: f143c125a0734166ac832270ac08dbb6\n",
" gain: 262.90943552209455\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 95\n",
" iterations_since_restore: 96\n",
" loss: 15.553215620072208\n",
" node_ip: 127.0.0.1\n",
" pid: 46679\n",
" time_since_restore: 10.240519762039185\n",
" time_this_iter_s: 0.10443782806396484\n",
" time_total_s: 10.240519762039185\n",
" timestamp: 1658500036\n",
" timesteps_since_restore: 0\n",
" training_iteration: 96\n",
" trial_id: 5b6c4ba2\n",
" warmup_time: 0.002663135528564453\n",
" \n",
"Result for multi_objective_5b6c4ba2:\n",
" date: 2022-07-22_15-27-17\n",
" done: true\n",
" experiment_id: f143c125a0734166ac832270ac08dbb6\n",
" experiment_tag: 5_activation=relu,height=54.9133,steps=100,width=16.9039\n",
" gain: 262.86741923011425\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" loss: 15.550730017197484\n",
" node_ip: 127.0.0.1\n",
" pid: 46679\n",
" time_since_restore: 10.669770956039429\n",
" time_this_iter_s: 0.10598993301391602\n",
" time_total_s: 10.669770956039429\n",
" timestamp: 1658500037\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: 5b6c4ba2\n",
" warmup_time: 0.002663135528564453\n",
" \n",
"Result for multi_objective_634c9444:\n",
" date: 2022-07-22_15-27-19\n",
" done: false\n",
" experiment_id: 1c6a5d9e38394f1ca770da6a92b6a0ee\n",
" gain: 283.55076351521427\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" loss: 18.766157678691147\n",
" node_ip: 127.0.0.1\n",
" pid: 46710\n",
" time_since_restore: 0.10465264320373535\n",
" time_this_iter_s: 0.10465264320373535\n",
" time_total_s: 0.10465264320373535\n",
" timestamp: 1658500039\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: 634c9444\n",
" warmup_time: 0.004335880279541016\n",
" \n",
"Result for multi_objective_5d51f2d2:\n",
" date: 2022-07-22_15-27-20\n",
" done: false\n",
" experiment_id: 1433194530b14db4b77ce40a69d65407\n",
" gain: 9.60907229177291\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 94\n",
" iterations_since_restore: 95\n",
" loss: 1.8596662860626851\n",
" node_ip: 127.0.0.1\n",
" pid: 46686\n",
" time_since_restore: 10.218497037887573\n",
" time_this_iter_s: 0.10553288459777832\n",
" time_total_s: 10.218497037887573\n",
" timestamp: 1658500040\n",
" timesteps_since_restore: 0\n",
" training_iteration: 95\n",
" trial_id: 5d51f2d2\n",
" warmup_time: 0.0028769969940185547\n",
" \n",
"Result for multi_objective_5d636760:\n",
" date: 2022-07-22_15-27-20\n",
" done: false\n",
" experiment_id: b8dccac5d8914569b1c2ca061ad47a56\n",
" gain: -17.840160182810624\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 94\n",
" iterations_since_restore: 95\n",
" loss: -1.311361948500235\n",
" node_ip: 127.0.0.1\n",
" pid: 46689\n",
" time_since_restore: 10.16199278831482\n",
" time_this_iter_s: 0.10700488090515137\n",
" time_total_s: 10.16199278831482\n",
" timestamp: 1658500040\n",
" timesteps_since_restore: 0\n",
" training_iteration: 95\n",
" trial_id: 5d636760\n",
" warmup_time: 0.0028090476989746094\n",
" \n",
"Result for multi_objective_5d66093e:\n",
" date: 2022-07-22_15-27-20\n",
" done: false\n",
" experiment_id: e7259e986278458899b46b485e069ec9\n",
" gain: 96.85061995053432\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 94\n",
" iterations_since_restore: 95\n",
" loss: 4.886249816008858\n",
" node_ip: 127.0.0.1\n",
" pid: 46690\n",
" time_since_restore: 10.197196006774902\n",
" time_this_iter_s: 0.10463690757751465\n",
" time_total_s: 10.197196006774902\n",
" timestamp: 1658500040\n",
" timesteps_since_restore: 0\n",
" training_iteration: 95\n",
" trial_id: 5d66093e\n",
" warmup_time: 0.0027511119842529297\n",
" \n",
"Result for multi_objective_5d51f2d2:\n",
" date: 2022-07-22_15-27-20\n",
" done: true\n",
" experiment_id: 1433194530b14db4b77ce40a69d65407\n",
" experiment_tag: 6_activation=tanh,height=16.5793,steps=100,width=5.1671\n",
" gain: 9.557436804323405\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" loss: 1.8496731491334866\n",
" node_ip: 127.0.0.1\n",
" pid: 46686\n",
" time_since_restore: 10.745142936706543\n",
" time_this_iter_s: 0.10509181022644043\n",
" time_total_s: 10.745142936706543\n",
" timestamp: 1658500040\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: 5d51f2d2\n",
" warmup_time: 0.0028769969940185547\n",
" \n",
"Result for multi_objective_5d636760:\n",
" date: 2022-07-22_15-27-20\n",
" done: true\n",
" experiment_id: b8dccac5d8914569b1c2ca061ad47a56\n",
" experiment_tag: 7_activation=tanh,height=-13.8895,steps=100,width=13.6043\n",
" gain: -17.893079155032332\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" loss: -1.3152518197690004\n",
" node_ip: 127.0.0.1\n",
" pid: 46689\n",
" time_since_restore: 10.689707040786743\n",
" time_this_iter_s: 0.10322403907775879\n",
" time_total_s: 10.689707040786743\n",
" timestamp: 1658500040\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: 5d636760\n",
" warmup_time: 0.0028090476989746094\n",
" \n",
"Result for multi_objective_5d66093e:\n",
" date: 2022-07-22_15-27-20\n",
" done: true\n",
" experiment_id: e7259e986278458899b46b485e069ec9\n",
" experiment_tag: 8_activation=relu,height=-51.6714,steps=100,width=19.8211\n",
" gain: 96.79744897015512\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" loss: 4.8835672653630535\n",
" node_ip: 127.0.0.1\n",
" pid: 46690\n",
" time_since_restore: 10.72464108467102\n",
" time_this_iter_s: 0.10328292846679688\n",
" time_total_s: 10.72464108467102\n",
" timestamp: 1658500040\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: 5d66093e\n",
" warmup_time: 0.0027511119842529297\n",
" \n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Result for multi_objective_65469a2e:\n",
" date: 2022-07-22_15-27-23\n",
" done: false\n",
" experiment_id: 015ee29ddb8b4c239d85a022a23d4093\n",
" gain: 162.96737941207124\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" loss: 11.96774070054979\n",
" node_ip: 127.0.0.1\n",
" pid: 46719\n",
" time_since_restore: 0.10403180122375488\n",
" time_this_iter_s: 0.10403180122375488\n",
" time_total_s: 0.10403180122375488\n",
" timestamp: 1658500043\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: 65469a2e\n",
" warmup_time: 0.0033850669860839844\n",
" \n",
"Result for multi_objective_634c9444:\n",
" date: 2022-07-22_15-27-24\n",
" done: false\n",
" experiment_id: 1c6a5d9e38394f1ca770da6a92b6a0ee\n",
" gain: 136.50874589912112\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 24\n",
" iterations_since_restore: 25\n",
" loss: 9.034518610724309\n",
" node_ip: 127.0.0.1\n",
" pid: 46710\n",
" time_since_restore: 5.125189781188965\n",
" time_this_iter_s: 0.11187601089477539\n",
" time_total_s: 5.125189781188965\n",
" timestamp: 1658500044\n",
" timesteps_since_restore: 0\n",
" training_iteration: 25\n",
" trial_id: 634c9444\n",
" warmup_time: 0.004335880279541016\n",
" \n",
"Result for multi_objective_65469a2e:\n",
" date: 2022-07-22_15-27-28\n",
" done: false\n",
" experiment_id: 015ee29ddb8b4c239d85a022a23d4093\n",
" gain: 28.890088364068774\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" loss: 2.1215846238952016\n",
" node_ip: 127.0.0.1\n",
" pid: 46719\n",
" time_since_restore: 5.174077749252319\n",
" time_this_iter_s: 0.10734677314758301\n",
" time_total_s: 5.174077749252319\n",
" timestamp: 1658500048\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: 65469a2e\n",
" warmup_time: 0.0033850669860839844\n",
" \n",
"Result for multi_objective_634c9444:\n",
" date: 2022-07-22_15-27-30\n",
" done: false\n",
" experiment_id: 1c6a5d9e38394f1ca770da6a92b6a0ee\n",
" gain: 133.84933935751317\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 71\n",
" iterations_since_restore: 72\n",
" loss: 8.85851188137237\n",
" node_ip: 127.0.0.1\n",
" pid: 46710\n",
" time_since_restore: 10.204095840454102\n",
" time_this_iter_s: 0.10942888259887695\n",
" time_total_s: 10.204095840454102\n",
" timestamp: 1658500050\n",
" timesteps_since_restore: 0\n",
" training_iteration: 72\n",
" trial_id: 634c9444\n",
" warmup_time: 0.004335880279541016\n",
" \n",
"Result for multi_objective_634c9444:\n",
" date: 2022-07-22_15-27-33\n",
" done: true\n",
" experiment_id: 1c6a5d9e38394f1ca770da6a92b6a0ee\n",
" experiment_tag: 9_activation=relu,height=-12.3384,steps=100,width=15.1097\n",
" gain: 133.45728949146132\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" loss: 8.83256495915982\n",
" node_ip: 127.0.0.1\n",
" pid: 46710\n",
" time_since_restore: 13.198630809783936\n",
" time_this_iter_s: 0.1067969799041748\n",
" time_total_s: 13.198630809783936\n",
" timestamp: 1658500053\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: 634c9444\n",
" warmup_time: 0.004335880279541016\n",
" \n",
"Result for multi_objective_65469a2e:\n",
" date: 2022-07-22_15-27-33\n",
" done: false\n",
" experiment_id: 015ee29ddb8b4c239d85a022a23d4093\n",
" gain: 27.850744704436202\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 94\n",
" iterations_since_restore: 95\n",
" loss: 2.0452589477867855\n",
" node_ip: 127.0.0.1\n",
" pid: 46719\n",
" time_since_restore: 10.19201397895813\n",
" time_this_iter_s: 0.10735702514648438\n",
" time_total_s: 10.19201397895813\n",
" timestamp: 1658500053\n",
" timesteps_since_restore: 0\n",
" training_iteration: 95\n",
" trial_id: 65469a2e\n",
" warmup_time: 0.0033850669860839844\n",
" \n",
"Result for multi_objective_65469a2e:\n",
" date: 2022-07-22_15-27-34\n",
" done: true\n",
" experiment_id: 015ee29ddb8b4c239d85a022a23d4093\n",
" experiment_tag: 10_activation=tanh,height=19.6774,steps=100,width=13.6172\n",
" gain: 27.797824972416166\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" loss: 2.041372712191397\n",
" node_ip: 127.0.0.1\n",
" pid: 46719\n",
" time_since_restore: 10.727944135665894\n",
" time_this_iter_s: 0.10885024070739746\n",
" time_total_s: 10.727944135665894\n",
" timestamp: 1658500054\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: 65469a2e\n",
" warmup_time: 0.0033850669860839844\n",
" \n"
]
}
],
2022-03-25 01:04:53 -07:00
"source": [
"searcher = OptunaSearch(metric=[\"loss\", \"gain\"], mode=[\"min\", \"max\"])\n",
"algo = ConcurrencyLimiter(searcher, max_concurrent=4)\n",
"\n",
2022-07-24 18:53:57 +01:00
"tuner = tune.Tuner(\n",
2022-03-25 01:04:53 -07:00
" multi_objective,\n",
2022-07-24 18:53:57 +01:00
" tune_config=tune.TuneConfig(\n",
" search_alg=algo,\n",
" num_samples=num_samples,\n",
" ),\n",
" param_space=search_space\n",
")\n",
"results = tuner.fit()"
2022-03-25 01:04:53 -07:00
]
},
{
"cell_type": "markdown",
2022-04-25 11:10:58 -07:00
"id": "df42b8b3",
2022-03-25 01:04:53 -07:00
"metadata": {},
"source": [
2022-07-24 18:53:57 +01:00
"Now there are two hyperparameter sets for the two objectives."
]
},
{
"cell_type": "code",
"execution_count": 24,
"id": "183fef1a",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Best hyperparameters for loss found were: {'steps': 100, 'width': 10.925599395387751, 'height': -47.52361963011387, 'activation': 'tanh'}\n",
"Best hyperparameters for gain found were: {'steps': 100, 'width': 16.90386360893735, 'height': 54.91329340230965, 'activation': 'relu'}\n"
]
}
],
"source": [
"print(\"Best hyperparameters for loss found were: \", results.get_best_result(\"loss\", \"min\").config)\n",
"print(\"Best hyperparameters for gain found were: \", results.get_best_result(\"gain\", \"max\").config)"
]
},
{
"cell_type": "markdown",
"id": "cdf4d49a",
"metadata": {},
"source": [
2022-03-25 01:04:53 -07:00
"We can mix-and-match the use of initial hyperparameter evaluations, conditional search spaces via define-by-run functions, and multi-objective tasks. This is also true of scheduler usage, with the exception of multi-objective optimization-- schedulers typically rely on a single scalar score, rather than the two scores we use here: loss, gain."
]
},
{
"cell_type": "code",
2022-07-24 18:53:57 +01:00
"execution_count": 25,
2022-04-25 11:10:58 -07:00
"id": "a058fdb3",
2022-03-25 01:04:53 -07:00
"metadata": {
"tags": [
"remove-cell"
]
},
"outputs": [],
"source": [
"ray.shutdown()"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
2022-07-24 18:53:57 +01:00
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.7"
},
2022-03-25 01:04:53 -07:00
"orphan": true
},
"nbformat": 4,
"nbformat_minor": 5
}
