mirror of
https://github.com/vale981/ray
synced 2025-03-09 12:56:46 -04:00
803c094534
This PR updates the Ray AIR/Tune ipynb examples to use the Tuner() API instead of tune.run(). Signed-off-by: Kai Fricke <kai@anyscale.com> Signed-off-by: Richard Liaw <rliaw@berkeley.edu> Signed-off-by: Xiaowei Jiang <xwjiang2010@gmail.com> Signed-off-by: Kai Fricke <coding@kaifricke.com> Co-authored-by: Richard Liaw <rliaw@berkeley.edu> Co-authored-by: Xiaowei Jiang <xwjiang2010@gmail.com>
2311 lines
96 KiB
Text
2311 lines
96 KiB
Text
{
|
||
"cells": [
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "58fc50bc",
|
||
"metadata": {},
|
||
"source": [
|
||
"# Running Tune experiments with HyperOpt\n",
|
||
"\n",
|
||
"In this tutorial we introduce HyperOpt, while running a simple Ray Tune experiment. Tune’s Search Algorithms integrate with HyperOpt and, as a result, allow you to seamlessly scale up a Hyperopt optimization process - without sacrificing performance.\n",
|
||
"\n",
|
||
"HyperOpt provides gradient/derivative-free optimization able to handle noise over the objective landscape, including evolutionary, bandit, and Bayesian optimization algorithms. Nevergrad internally supports search spaces which are continuous, discrete or a mixture of thereof. It also provides a library of functions on which to test the optimization algorithms and compare with other benchmarks.\n",
|
||
"\n",
|
||
"In this example we minimize a simple objective to briefly demonstrate the usage of HyperOpt with Ray Tune via `HyperOptSearch`. It's useful to keep in mind that despite the emphasis on machine learning experiments, Ray Tune optimizes any implicit or explicit objective. Here we assume `hyperopt==0.2.5` library is installed. To learn more, please refer to [HyperOpt website](http://hyperopt.github.io/hyperopt).\n",
|
||
"\n",
|
||
"We include a important example on conditional search spaces (stringing together relationships among hyperparameters)."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "e4586d28",
|
||
"metadata": {},
|
||
"source": [
|
||
"Background information:\n",
|
||
"- [HyperOpt website](http://hyperopt.github.io/hyperopt)\n",
|
||
"\n",
|
||
"Necessary requirements:\n",
|
||
"- `pip install ray[tune]`\n",
|
||
"- `pip install hyperopt==0.2.5`"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 1,
|
||
"id": "6567f2dc",
|
||
"metadata": {
|
||
"tags": [
|
||
"remove-cell"
|
||
],
|
||
"vscode": {
|
||
"languageId": "python"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Collecting hyperopt==0.2.5\n",
|
||
" Using cached hyperopt-0.2.5-py2.py3-none-any.whl (965 kB)\n",
|
||
"Requirement already satisfied: numpy in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from hyperopt==0.2.5) (1.21.6)\n",
|
||
"Requirement already satisfied: cloudpickle in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from hyperopt==0.2.5) (2.0.0)\n",
|
||
"Requirement already satisfied: six in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from hyperopt==0.2.5) (1.16.0)\n",
|
||
"Requirement already satisfied: scipy in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from hyperopt==0.2.5) (1.4.1)\n",
|
||
"Requirement already satisfied: future in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from hyperopt==0.2.5) (0.18.2)\n",
|
||
"Requirement already satisfied: networkx>=2.2 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from hyperopt==0.2.5) (2.6.3)\n",
|
||
"Requirement already satisfied: tqdm in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from hyperopt==0.2.5) (4.64.0)\n",
|
||
"Installing collected packages: hyperopt\n",
|
||
" Attempting uninstall: hyperopt\n",
|
||
" Found existing installation: hyperopt 0.2.7\n",
|
||
" Uninstalling hyperopt-0.2.7:\n",
|
||
" Successfully uninstalled hyperopt-0.2.7\n",
|
||
"Successfully installed hyperopt-0.2.5\n",
|
||
"\u001b[33mWARNING: There was an error checking the latest version of pip.\u001b[0m\u001b[33m\n",
|
||
"\u001b[0m"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"# !pip install ray[tune]\n",
|
||
"!pip install hyperopt==0.2.5"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "b8e9e0cd",
|
||
"metadata": {},
|
||
"source": [
|
||
"Click below to see all the imports we need for this example.\n",
|
||
"You can also launch directly into a Binder instance to run this notebook yourself.\n",
|
||
"Just click on the rocket symbol at the top of the navigation."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 2,
|
||
"id": "6592315e",
|
||
"metadata": {
|
||
"tags": [
|
||
"hide-input"
|
||
],
|
||
"vscode": {
|
||
"languageId": "python"
|
||
}
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"import time\n",
|
||
"\n",
|
||
"import ray\n",
|
||
"from ray import tune\n",
|
||
"from ray.air import session\n",
|
||
"from ray.tune.search import ConcurrencyLimiter\n",
|
||
"from ray.tune.search.hyperopt import HyperOptSearch\n",
|
||
"from hyperopt import hp"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "d4b6d1d5",
|
||
"metadata": {},
|
||
"source": [
|
||
"Let's start by defining a simple evaluation function.\n",
|
||
"We artificially sleep for a bit (`0.1` seconds) to simulate a long-running ML experiment.\n",
|
||
"This setup assumes that we're running multiple `step`s of an experiment and try to tune two hyperparameters,\n",
|
||
"namely `width` and `height`."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 3,
|
||
"id": "12d4efc8",
|
||
"metadata": {
|
||
"vscode": {
|
||
"languageId": "python"
|
||
}
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"def evaluate(step, width, height):\n",
|
||
" time.sleep(0.1)\n",
|
||
" return (0.1 + width * step / 100) ** (-1) + height * 0.1"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "4f4f5aa2",
|
||
"metadata": {},
|
||
"source": [
|
||
"Next, our ``objective`` function takes a Tune ``config``, evaluates the `score` of your experiment in a training loop,\n",
|
||
"and uses `session.report` to report the `score` back to Tune."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 4,
|
||
"id": "c9818009",
|
||
"metadata": {
|
||
"vscode": {
|
||
"languageId": "python"
|
||
}
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"def objective(config):\n",
|
||
" for step in range(config[\"steps\"]):\n",
|
||
" score = evaluate(step, config[\"width\"], config[\"height\"])\n",
|
||
" session.report({\"iterations\": step, \"mean_loss\": score})"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 5,
|
||
"id": "33eddcb9",
|
||
"metadata": {
|
||
"tags": [
|
||
"remove-cell"
|
||
],
|
||
"vscode": {
|
||
"languageId": "python"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/html": [
|
||
"<div>\n",
|
||
" <div style=\"margin-left: 50px;display: flex;flex-direction: row;align-items: center\">\n",
|
||
" <h3 style=\"color: var(--jp-ui-font-color0)\">Ray</h3>\n",
|
||
" <svg version=\"1.1\" id=\"ray\" width=\"3em\" viewBox=\"0 0 144.5 144.6\" style=\"margin-left: 3em;margin-right: 3em\">\n",
|
||
" <g id=\"layer-1\">\n",
|
||
" <path fill=\"#00a2e9\" class=\"st0\" d=\"M97.3,77.2c-3.8-1.1-6.2,0.9-8.3,5.1c-3.5,6.8-9.9,9.9-17.4,9.6S58,88.1,54.8,81.2c-1.4-3-3-4-6.3-4.1\n",
|
||
" c-5.6-0.1-9.9,0.1-13.1,6.4c-3.8,7.6-13.6,10.2-21.8,7.6C5.2,88.4-0.4,80.5,0,71.7c0.1-8.4,5.7-15.8,13.8-18.2\n",
|
||
" c8.4-2.6,17.5,0.7,22.3,8c1.3,1.9,1.3,5.2,3.6,5.6c3.9,0.6,8,0.2,12,0.2c1.8,0,1.9-1.6,2.4-2.8c3.5-7.8,9.7-11.8,18-11.9\n",
|
||
" c8.2-0.1,14.4,3.9,17.8,11.4c1.3,2.8,2.9,3.6,5.7,3.3c1-0.1,2,0.1,3,0c2.8-0.5,6.4,1.7,8.1-2.7s-2.3-5.5-4.1-7.5\n",
|
||
" c-5.1-5.7-10.9-10.8-16.1-16.3C84,38,81.9,37.1,78,38.3C66.7,42,56.2,35.7,53,24.1C50.3,14,57.3,2.8,67.7,0.5\n",
|
||
" C78.4-2,89,4.7,91.5,15.3c0.1,0.3,0.1,0.5,0.2,0.8c0.7,3.4,0.7,6.9-0.8,9.8c-1.7,3.2-0.8,5,1.5,7.2c6.7,6.5,13.3,13,19.8,19.7\n",
|
||
" c1.8,1.8,3,2.1,5.5,1.2c9.1-3.4,17.9-0.6,23.4,7c4.8,6.9,4.6,16.1-0.4,22.9c-5.4,7.2-14.2,9.9-23.1,6.5c-2.3-0.9-3.5-0.6-5.1,1.1\n",
|
||
" c-6.7,6.9-13.6,13.7-20.5,20.4c-1.8,1.8-2.5,3.2-1.4,5.9c3.5,8.7,0.3,18.6-7.7,23.6c-7.9,5-18.2,3.8-24.8-2.9\n",
|
||
" c-6.4-6.4-7.4-16.2-2.5-24.3c4.9-7.8,14.5-11,23.1-7.8c3,1.1,4.7,0.5,6.9-1.7C91.7,98.4,98,92.3,104.2,86c1.6-1.6,4.1-2.7,2.6-6.2\n",
|
||
" c-1.4-3.3-3.8-2.5-6.2-2.6C99.8,77.2,98.9,77.2,97.3,77.2z M72.1,29.7c5.5,0.1,9.9-4.3,10-9.8c0-0.1,0-0.2,0-0.3\n",
|
||
" C81.8,14,77,9.8,71.5,10.2c-5,0.3-9,4.2-9.3,9.2c-0.2,5.5,4,10.1,9.5,10.3C71.8,29.7,72,29.7,72.1,29.7z M72.3,62.3\n",
|
||
" c-5.4-0.1-9.9,4.2-10.1,9.7c0,0.2,0,0.3,0,0.5c0.2,5.4,4.5,9.7,9.9,10c5.1,0.1,9.9-4.7,10.1-9.8c0.2-5.5-4-10-9.5-10.3\n",
|
||
" C72.6,62.3,72.4,62.3,72.3,62.3z M115,72.5c0.1,5.4,4.5,9.7,9.8,9.9c5.6-0.2,10-4.8,10-10.4c-0.2-5.4-4.6-9.7-10-9.7\n",
|
||
" c-5.3-0.1-9.8,4.2-9.9,9.5C115,72.1,115,72.3,115,72.5z M19.5,62.3c-5.4,0.1-9.8,4.4-10,9.8c-0.1,5.1,5.2,10.4,10.2,10.3\n",
|
||
" c5.6-0.2,10-4.9,9.8-10.5c-0.1-5.4-4.5-9.7-9.9-9.6C19.6,62.3,19.5,62.3,19.5,62.3z M71.8,134.6c5.9,0.2,10.3-3.9,10.4-9.6\n",
|
||
" c0.5-5.5-3.6-10.4-9.1-10.8c-5.5-0.5-10.4,3.6-10.8,9.1c0,0.5,0,0.9,0,1.4c-0.2,5.3,4,9.8,9.3,10\n",
|
||
" C71.6,134.6,71.7,134.6,71.8,134.6z\"/>\n",
|
||
" </g>\n",
|
||
" </svg>\n",
|
||
" <table>\n",
|
||
" <tr>\n",
|
||
" <td style=\"text-align: left\"><b>Python version:</b></td>\n",
|
||
" <td style=\"text-align: left\"><b>3.7.7</b></td>\n",
|
||
" </tr>\n",
|
||
" <tr>\n",
|
||
" <td style=\"text-align: left\"><b>Ray version:</b></td>\n",
|
||
" <td style=\"text-align: left\"><b> 3.0.0.dev0</b></td>\n",
|
||
" </tr>\n",
|
||
" <tr>\n",
|
||
" <td style=\"text-align: left\"><b>Dashboard:</b></td>\n",
|
||
" <td style=\"text-align: left\"><b><a href=\"http://127.0.0.1:8266\" target=\"_blank\">http://127.0.0.1:8266</a></b></td>\n",
|
||
"</tr>\n",
|
||
"\n",
|
||
" </table>\n",
|
||
" </div>\n",
|
||
"</div>\n"
|
||
],
|
||
"text/plain": [
|
||
"RayContext(dashboard_url='127.0.0.1:8266', python_version='3.7.7', ray_version='3.0.0.dev0', ray_commit='{{RAY_COMMIT_SHA}}', address_info={'node_ip_address': '127.0.0.1', 'raylet_ip_address': '127.0.0.1', 'redis_address': None, 'object_store_address': '/tmp/ray/session_2022-07-22_15-30-59_720199_47076/sockets/plasma_store', 'raylet_socket_name': '/tmp/ray/session_2022-07-22_15-30-59_720199_47076/sockets/raylet', 'webui_url': '127.0.0.1:8266', 'session_dir': '/tmp/ray/session_2022-07-22_15-30-59_720199_47076', 'metrics_export_port': 61594, 'gcs_address': '127.0.0.1:63379', 'address': '127.0.0.1:63379', 'dashboard_agent_listen_port': 52365, 'node_id': '433e2d825b2bbfe76eb6845ceeda5c337fc35c367b60f2b28526b153'})"
|
||
]
|
||
},
|
||
"execution_count": 5,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"ray.init(configure_logging=False)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "5be35d5e",
|
||
"metadata": {},
|
||
"source": [
|
||
"While defining the search algorithm, we may choose to provide an initial set of hyperparameters that we believe are especially promising or informative, and\n",
|
||
"pass this information as a helpful starting point for the `HyperOptSearch` object.\n",
|
||
"\n",
|
||
"We also set the maximum concurrent trials to `4` with a `ConcurrencyLimiter`."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 6,
|
||
"id": "d4615bed",
|
||
"metadata": {
|
||
"lines_to_next_cell": 0,
|
||
"vscode": {
|
||
"languageId": "python"
|
||
}
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"initial_params = [\n",
|
||
" {\"width\": 1, \"height\": 2, \"activation\": \"relu\"},\n",
|
||
" {\"width\": 4, \"height\": 2, \"activation\": \"tanh\"},\n",
|
||
"]\n",
|
||
"algo = HyperOptSearch(points_to_evaluate=initial_params)\n",
|
||
"algo = ConcurrencyLimiter(algo, max_concurrent=4)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "2a51e7c1",
|
||
"metadata": {},
|
||
"source": [
|
||
"The number of samples is the number of hyperparameter combinations that will be tried out. This Tune run is set to `1000` samples.\n",
|
||
"(you can decrease this if it takes too long on your machine)."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 7,
|
||
"id": "2dbb2be0",
|
||
"metadata": {
|
||
"vscode": {
|
||
"languageId": "python"
|
||
}
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"num_samples = 1000"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 8,
|
||
"id": "950558ed",
|
||
"metadata": {
|
||
"tags": [
|
||
"remove-cell"
|
||
],
|
||
"vscode": {
|
||
"languageId": "python"
|
||
}
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"# If 1000 samples take too long, you can reduce this number.\n",
|
||
"# We override this number here for our smoke tests.\n",
|
||
"num_samples = 10"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "6e3629cb",
|
||
"metadata": {},
|
||
"source": [
|
||
"Next we define a search space. The critical assumption is that the optimal hyperparamters live within this space. Yet, if the space is very large, then those hyperparameters may be difficult to find in a short amount of time."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 9,
|
||
"id": "65189946",
|
||
"metadata": {
|
||
"vscode": {
|
||
"languageId": "python"
|
||
}
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"search_config = {\n",
|
||
" \"steps\": 100,\n",
|
||
" \"width\": tune.uniform(0, 20),\n",
|
||
" \"height\": tune.uniform(-100, 100),\n",
|
||
" \"activation\": tune.choice([\"relu\", \"tanh\"])\n",
|
||
"}"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "1b94c93b",
|
||
"metadata": {},
|
||
"source": [
|
||
"Finally, we run the experiment to `\"min\"`imize the \"mean_loss\" of the `objective` by searching `search_config` via `algo`, `num_samples` times. This previous sentence is fully characterizes the search problem we aim to solve. With this in mind, notice how efficient it is to execute `tuner.fit()`."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 10,
|
||
"id": "9a99a3a7",
|
||
"metadata": {
|
||
"vscode": {
|
||
"languageId": "python"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stderr",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Function checkpointing is disabled. This may result in unexpected behavior when using checkpointing features or certain schedulers. To enable, set the train function arguments to be `func(config, checkpoint_dir=None)`.\n"
|
||
]
|
||
},
|
||
{
|
||
"data": {
|
||
"text/html": [
|
||
"== Status ==<br>Current time: 2022-07-22 15:31:49 (running for 00:00:43.71)<br>Memory usage on this node: 10.4/16.0 GiB<br>Using FIFO scheduling algorithm.<br>Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/5.29 GiB heap, 0.0/2.0 GiB objects<br>Current best trial: f59fe9d6 with mean_loss=-2.5719085451008423 and parameters={'steps': 100, 'width': 5.584304853357766, 'height': -27.49576980043919, 'activation': 'tanh'}<br>Result logdir: /Users/kai/ray_results/objective_2022-07-22_15-31-04<br>Number of trials: 10/10 (10 TERMINATED)<br><table>\n",
|
||
"<thead>\n",
|
||
"<tr><th>Trial name </th><th>status </th><th>loc </th><th>activation </th><th style=\"text-align: right;\"> height</th><th style=\"text-align: right;\"> steps</th><th style=\"text-align: right;\"> width</th><th style=\"text-align: right;\"> loss</th><th style=\"text-align: right;\"> iter</th><th style=\"text-align: right;\"> total time (s)</th><th style=\"text-align: right;\"> iterations</th><th style=\"text-align: right;\"> neg_mean_loss</th></tr>\n",
|
||
"</thead>\n",
|
||
"<tbody>\n",
|
||
"<tr><td>objective_eb74f122</td><td>TERMINATED</td><td>127.0.0.1:47156</td><td>relu </td><td style=\"text-align: right;\"> 2 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 1 </td><td style=\"text-align: right;\"> 1.11743 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 10.9008</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\"> -1.11743 </td></tr>\n",
|
||
"<tr><td>objective_ed2010ec</td><td>TERMINATED</td><td>127.0.0.1:47161</td><td>tanh </td><td style=\"text-align: right;\"> 2 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 4 </td><td style=\"text-align: right;\"> 0.446305</td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 11.5098</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\"> -0.446305</td></tr>\n",
|
||
"<tr><td>objective_ed217cf2</td><td>TERMINATED</td><td>127.0.0.1:47162</td><td>tanh </td><td style=\"text-align: right;\">-20.6075 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\">13.061 </td><td style=\"text-align: right;\">-1.98401 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 11.5623</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\"> 1.98401 </td></tr>\n",
|
||
"<tr><td>objective_ed2322be</td><td>TERMINATED</td><td>127.0.0.1:47163</td><td>tanh </td><td style=\"text-align: right;\"> 19.9564 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\">10.6836 </td><td style=\"text-align: right;\"> 2.0893 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 11.6053</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\"> -2.0893 </td></tr>\n",
|
||
"<tr><td>objective_f3a0bef8</td><td>TERMINATED</td><td>127.0.0.1:47180</td><td>tanh </td><td style=\"text-align: right;\"> -7.43915</td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 2.23969</td><td style=\"text-align: right;\">-0.312378</td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 10.7378</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\"> 0.312378</td></tr>\n",
|
||
"<tr><td>objective_f59fe9d6</td><td>TERMINATED</td><td>127.0.0.1:47185</td><td>tanh </td><td style=\"text-align: right;\">-27.4958 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 5.5843 </td><td style=\"text-align: right;\">-2.57191 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 10.7145</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\"> 2.57191 </td></tr>\n",
|
||
"<tr><td>objective_f5aedf9a</td><td>TERMINATED</td><td>127.0.0.1:47188</td><td>tanh </td><td style=\"text-align: right;\"> 48.706 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\">19.7352 </td><td style=\"text-align: right;\"> 4.92153 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 10.7341</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\"> -4.92153 </td></tr>\n",
|
||
"<tr><td>objective_f5b1ec08</td><td>TERMINATED</td><td>127.0.0.1:47189</td><td>tanh </td><td style=\"text-align: right;\"> 4.14098</td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\">16.2739 </td><td style=\"text-align: right;\"> 0.475784</td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 10.7034</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\"> -0.475784</td></tr>\n",
|
||
"<tr><td>objective_fb926d32</td><td>TERMINATED</td><td>127.0.0.1:47205</td><td>tanh </td><td style=\"text-align: right;\"> 44.778 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\">10.0724 </td><td style=\"text-align: right;\"> 4.57708 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 13.1109</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\"> -4.57708 </td></tr>\n",
|
||
"<tr><td>objective_fd91e28e</td><td>TERMINATED</td><td>127.0.0.1:47214</td><td>relu </td><td style=\"text-align: right;\"> -2.9623 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\">11.8215 </td><td style=\"text-align: right;\">-0.211508</td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 10.7934</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\"> 0.211508</td></tr>\n",
|
||
"</tbody>\n",
|
||
"</table><br><br>"
|
||
],
|
||
"text/plain": [
|
||
"<IPython.core.display.HTML object>"
|
||
]
|
||
},
|
||
"metadata": {},
|
||
"output_type": "display_data"
|
||
},
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Result for objective_eb74f122:\n",
|
||
" date: 2022-07-22_15-31-08\n",
|
||
" done: false\n",
|
||
" experiment_id: 0ed1b6ba6d99477dba0632102d1bc531\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 0\n",
|
||
" iterations_since_restore: 1\n",
|
||
" mean_loss: 10.2\n",
|
||
" neg_mean_loss: -10.2\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47156\n",
|
||
" time_since_restore: 0.10458922386169434\n",
|
||
" time_this_iter_s: 0.10458922386169434\n",
|
||
" time_total_s: 0.10458922386169434\n",
|
||
" timestamp: 1658500268\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 1\n",
|
||
" trial_id: eb74f122\n",
|
||
" warmup_time: 0.002730131149291992\n",
|
||
" \n",
|
||
"Result for objective_ed2010ec:\n",
|
||
" date: 2022-07-22_15-31-11\n",
|
||
" done: false\n",
|
||
" experiment_id: 3acf6a8ccf8442a7adcf98cc92362216\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 0\n",
|
||
" iterations_since_restore: 1\n",
|
||
" mean_loss: 10.2\n",
|
||
" neg_mean_loss: -10.2\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47161\n",
|
||
" time_since_restore: 0.1025240421295166\n",
|
||
" time_this_iter_s: 0.1025240421295166\n",
|
||
" time_total_s: 0.1025240421295166\n",
|
||
" timestamp: 1658500271\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 1\n",
|
||
" trial_id: ed2010ec\n",
|
||
" warmup_time: 0.0034351348876953125\n",
|
||
" \n",
|
||
"Result for objective_ed2322be:\n",
|
||
" date: 2022-07-22_15-31-11\n",
|
||
" done: false\n",
|
||
" experiment_id: e95895ab00b54841933d324c3de8f58e\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 0\n",
|
||
" iterations_since_restore: 1\n",
|
||
" mean_loss: 11.99564236159307\n",
|
||
" neg_mean_loss: -11.99564236159307\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47163\n",
|
||
" time_since_restore: 0.10490107536315918\n",
|
||
" time_this_iter_s: 0.10490107536315918\n",
|
||
" time_total_s: 0.10490107536315918\n",
|
||
" timestamp: 1658500271\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 1\n",
|
||
" trial_id: ed2322be\n",
|
||
" warmup_time: 0.0032410621643066406\n",
|
||
" \n",
|
||
"Result for objective_ed217cf2:\n",
|
||
" date: 2022-07-22_15-31-11\n",
|
||
" done: false\n",
|
||
" experiment_id: 52186ede891e429aac44318036e7a7bb\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 0\n",
|
||
" iterations_since_restore: 1\n",
|
||
" mean_loss: 7.939249801790311\n",
|
||
" neg_mean_loss: -7.939249801790311\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47162\n",
|
||
" time_since_restore: 0.10419487953186035\n",
|
||
" time_this_iter_s: 0.10419487953186035\n",
|
||
" time_total_s: 0.10419487953186035\n",
|
||
" timestamp: 1658500271\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 1\n",
|
||
" trial_id: ed217cf2\n",
|
||
" warmup_time: 0.0031850337982177734\n",
|
||
" \n",
|
||
"Result for objective_eb74f122:\n",
|
||
" date: 2022-07-22_15-31-13\n",
|
||
" done: false\n",
|
||
" experiment_id: 0ed1b6ba6d99477dba0632102d1bc531\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 46\n",
|
||
" iterations_since_restore: 47\n",
|
||
" mean_loss: 1.9857142857142855\n",
|
||
" neg_mean_loss: -1.9857142857142855\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47156\n",
|
||
" time_since_restore: 5.155240058898926\n",
|
||
" time_this_iter_s: 0.11003398895263672\n",
|
||
" time_total_s: 5.155240058898926\n",
|
||
" timestamp: 1658500273\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 47\n",
|
||
" trial_id: eb74f122\n",
|
||
" warmup_time: 0.002730131149291992\n",
|
||
" \n",
|
||
"Result for objective_ed2010ec:\n",
|
||
" date: 2022-07-22_15-31-16\n",
|
||
" done: false\n",
|
||
" experiment_id: 3acf6a8ccf8442a7adcf98cc92362216\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 47\n",
|
||
" iterations_since_restore: 48\n",
|
||
" mean_loss: 0.7050505050505051\n",
|
||
" neg_mean_loss: -0.7050505050505051\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47161\n",
|
||
" time_since_restore: 5.1636271476745605\n",
|
||
" time_this_iter_s: 0.1071469783782959\n",
|
||
" time_total_s: 5.1636271476745605\n",
|
||
" timestamp: 1658500276\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 48\n",
|
||
" trial_id: ed2010ec\n",
|
||
" warmup_time: 0.0034351348876953125\n",
|
||
" \n",
|
||
"Result for objective_ed2322be:\n",
|
||
" date: 2022-07-22_15-31-16\n",
|
||
" done: false\n",
|
||
" experiment_id: e95895ab00b54841933d324c3de8f58e\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 47\n",
|
||
" iterations_since_restore: 48\n",
|
||
" mean_loss: 2.1909053484385796\n",
|
||
" neg_mean_loss: -2.1909053484385796\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47163\n",
|
||
" time_since_restore: 5.168597936630249\n",
|
||
" time_this_iter_s: 0.10674691200256348\n",
|
||
" time_total_s: 5.168597936630249\n",
|
||
" timestamp: 1658500276\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 48\n",
|
||
" trial_id: ed2322be\n",
|
||
" warmup_time: 0.0032410621643066406\n",
|
||
" \n",
|
||
"Result for objective_ed217cf2:\n",
|
||
" date: 2022-07-22_15-31-16\n",
|
||
" done: false\n",
|
||
" experiment_id: 52186ede891e429aac44318036e7a7bb\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 47\n",
|
||
" iterations_since_restore: 48\n",
|
||
" mean_loss: -1.9004596703789314\n",
|
||
" neg_mean_loss: 1.9004596703789314\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47162\n",
|
||
" time_since_restore: 5.14047384262085\n",
|
||
" time_this_iter_s: 0.10724306106567383\n",
|
||
" time_total_s: 5.14047384262085\n",
|
||
" timestamp: 1658500276\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 48\n",
|
||
" trial_id: ed217cf2\n",
|
||
" warmup_time: 0.0031850337982177734\n",
|
||
" \n",
|
||
"Result for objective_eb74f122:\n",
|
||
" date: 2022-07-22_15-31-18\n",
|
||
" done: false\n",
|
||
" experiment_id: 0ed1b6ba6d99477dba0632102d1bc531\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 93\n",
|
||
" iterations_since_restore: 94\n",
|
||
" mean_loss: 1.170873786407767\n",
|
||
" neg_mean_loss: -1.170873786407767\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47156\n",
|
||
" time_since_restore: 10.253305196762085\n",
|
||
" time_this_iter_s: 0.10791301727294922\n",
|
||
" time_total_s: 10.253305196762085\n",
|
||
" timestamp: 1658500278\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 94\n",
|
||
" trial_id: eb74f122\n",
|
||
" warmup_time: 0.002730131149291992\n",
|
||
" \n",
|
||
"Result for objective_eb74f122:\n",
|
||
" date: 2022-07-22_15-31-19\n",
|
||
" done: true\n",
|
||
" experiment_id: 0ed1b6ba6d99477dba0632102d1bc531\n",
|
||
" experiment_tag: 1_activation=relu,height=2.0000,steps=100,width=1.0000\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 99\n",
|
||
" iterations_since_restore: 100\n",
|
||
" mean_loss: 1.1174311926605505\n",
|
||
" neg_mean_loss: -1.1174311926605505\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47156\n",
|
||
" time_since_restore: 10.900829076766968\n",
|
||
" time_this_iter_s: 0.10532379150390625\n",
|
||
" time_total_s: 10.900829076766968\n",
|
||
" timestamp: 1658500279\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 100\n",
|
||
" trial_id: eb74f122\n",
|
||
" warmup_time: 0.002730131149291992\n",
|
||
" \n",
|
||
"Result for objective_ed217cf2:\n",
|
||
" date: 2022-07-22_15-31-21\n",
|
||
" done: false\n",
|
||
" experiment_id: 52186ede891e429aac44318036e7a7bb\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 91\n",
|
||
" iterations_since_restore: 92\n",
|
||
" mean_loss: -1.9773161428398818\n",
|
||
" neg_mean_loss: 1.9773161428398818\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47162\n",
|
||
" time_since_restore: 9.86834192276001\n",
|
||
" time_this_iter_s: 0.10606908798217773\n",
|
||
" time_total_s: 9.86834192276001\n",
|
||
" timestamp: 1658500281\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 92\n",
|
||
" trial_id: ed217cf2\n",
|
||
" warmup_time: 0.0031850337982177734\n",
|
||
" \n",
|
||
"Result for objective_ed2322be:\n",
|
||
" date: 2022-07-22_15-31-21\n",
|
||
" done: false\n",
|
||
" experiment_id: e95895ab00b54841933d324c3de8f58e\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 91\n",
|
||
" iterations_since_restore: 92\n",
|
||
" mean_loss: 2.0974537058269864\n",
|
||
" neg_mean_loss: -2.0974537058269864\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47163\n",
|
||
" time_since_restore: 9.877221822738647\n",
|
||
" time_this_iter_s: 0.10554194450378418\n",
|
||
" time_total_s: 9.877221822738647\n",
|
||
" timestamp: 1658500281\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 92\n",
|
||
" trial_id: ed2322be\n",
|
||
" warmup_time: 0.0032410621643066406\n",
|
||
" \n",
|
||
"Result for objective_ed2010ec:\n",
|
||
" date: 2022-07-22_15-31-21\n",
|
||
" done: false\n",
|
||
" experiment_id: 3acf6a8ccf8442a7adcf98cc92362216\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 92\n",
|
||
" iterations_since_restore: 93\n",
|
||
" mean_loss: 0.46455026455026455\n",
|
||
" neg_mean_loss: -0.46455026455026455\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47161\n",
|
||
" time_since_restore: 10.010878086090088\n",
|
||
" time_this_iter_s: 0.12952017784118652\n",
|
||
" time_total_s: 10.010878086090088\n",
|
||
" timestamp: 1658500281\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 93\n",
|
||
" trial_id: ed2010ec\n",
|
||
" warmup_time: 0.0034351348876953125\n",
|
||
" \n",
|
||
"Result for objective_f3a0bef8:\n",
|
||
" date: 2022-07-22_15-31-22\n",
|
||
" done: false\n",
|
||
" experiment_id: 7bbba7b85dbc4029bc8b22b7866d091f\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 0\n",
|
||
" iterations_since_restore: 1\n",
|
||
" mean_loss: 9.256084922802092\n",
|
||
" neg_mean_loss: -9.256084922802092\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47180\n",
|
||
" time_since_restore: 0.10452008247375488\n",
|
||
" time_this_iter_s: 0.10452008247375488\n",
|
||
" time_total_s: 0.10452008247375488\n",
|
||
" timestamp: 1658500282\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 1\n",
|
||
" trial_id: f3a0bef8\n",
|
||
" warmup_time: 0.003498077392578125\n",
|
||
" \n",
|
||
"Result for objective_ed2010ec:\n",
|
||
" date: 2022-07-22_15-31-22\n",
|
||
" done: true\n",
|
||
" experiment_id: 3acf6a8ccf8442a7adcf98cc92362216\n",
|
||
" experiment_tag: 2_activation=tanh,height=2.0000,steps=100,width=4.0000\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 99\n",
|
||
" iterations_since_restore: 100\n",
|
||
" mean_loss: 0.44630541871921187\n",
|
||
" neg_mean_loss: -0.44630541871921187\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47161\n",
|
||
" time_since_restore: 11.509758949279785\n",
|
||
" time_this_iter_s: 0.10628509521484375\n",
|
||
" time_total_s: 11.509758949279785\n",
|
||
" timestamp: 1658500282\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 100\n",
|
||
" trial_id: ed2010ec\n",
|
||
" warmup_time: 0.0034351348876953125\n",
|
||
" \n",
|
||
"Result for objective_ed217cf2:\n",
|
||
" date: 2022-07-22_15-31-22\n",
|
||
" done: true\n",
|
||
" experiment_id: 52186ede891e429aac44318036e7a7bb\n",
|
||
" experiment_tag: 3_activation=tanh,height=-20.6075,steps=100,width=13.0610\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 99\n",
|
||
" iterations_since_restore: 100\n",
|
||
" mean_loss: -1.9840065470391304\n",
|
||
" neg_mean_loss: 1.9840065470391304\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47162\n",
|
||
" time_since_restore: 11.562283754348755\n",
|
||
" time_this_iter_s: 0.10557794570922852\n",
|
||
" time_total_s: 11.562283754348755\n",
|
||
" timestamp: 1658500282\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 100\n",
|
||
" trial_id: ed217cf2\n",
|
||
" warmup_time: 0.0031850337982177734\n",
|
||
" \n",
|
||
"Result for objective_ed2322be:\n",
|
||
" date: 2022-07-22_15-31-22\n",
|
||
" done: true\n",
|
||
" experiment_id: e95895ab00b54841933d324c3de8f58e\n",
|
||
" experiment_tag: 4_activation=tanh,height=19.9564,steps=100,width=10.6836\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 99\n",
|
||
" iterations_since_restore: 100\n",
|
||
" mean_loss: 2.0893035832616653\n",
|
||
" neg_mean_loss: -2.0893035832616653\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47163\n",
|
||
" time_since_restore: 11.605261087417603\n",
|
||
" time_this_iter_s: 0.10673737525939941\n",
|
||
" time_total_s: 11.605261087417603\n",
|
||
" timestamp: 1658500282\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 100\n",
|
||
" trial_id: ed2322be\n",
|
||
" warmup_time: 0.0032410621643066406\n",
|
||
" \n"
|
||
]
|
||
},
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Result for objective_f59fe9d6:\n",
|
||
" date: 2022-07-22_15-31-25\n",
|
||
" done: false\n",
|
||
" experiment_id: efbda212c15c4bd38cfc5f39dfae8b73\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 0\n",
|
||
" iterations_since_restore: 1\n",
|
||
" mean_loss: 7.250423019956081\n",
|
||
" neg_mean_loss: -7.250423019956081\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47185\n",
|
||
" time_since_restore: 0.10512709617614746\n",
|
||
" time_this_iter_s: 0.10512709617614746\n",
|
||
" time_total_s: 0.10512709617614746\n",
|
||
" timestamp: 1658500285\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 1\n",
|
||
" trial_id: f59fe9d6\n",
|
||
" warmup_time: 0.003314971923828125\n",
|
||
" \n",
|
||
"Result for objective_f5b1ec08:\n",
|
||
" date: 2022-07-22_15-31-25\n",
|
||
" done: false\n",
|
||
" experiment_id: dadb542868ba4b10adf1ef161c75ea17\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 0\n",
|
||
" iterations_since_restore: 1\n",
|
||
" mean_loss: 10.41409792482367\n",
|
||
" neg_mean_loss: -10.41409792482367\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47189\n",
|
||
" time_since_restore: 0.10109281539916992\n",
|
||
" time_this_iter_s: 0.10109281539916992\n",
|
||
" time_total_s: 0.10109281539916992\n",
|
||
" timestamp: 1658500285\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 1\n",
|
||
" trial_id: f5b1ec08\n",
|
||
" warmup_time: 0.0031630992889404297\n",
|
||
" \n",
|
||
"Result for objective_f5aedf9a:\n",
|
||
" date: 2022-07-22_15-31-25\n",
|
||
" done: false\n",
|
||
" experiment_id: 9f7847b7440648cb86e73bf1be0ccc05\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 0\n",
|
||
" iterations_since_restore: 1\n",
|
||
" mean_loss: 14.870604096990785\n",
|
||
" neg_mean_loss: -14.870604096990785\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47188\n",
|
||
" time_since_restore: 0.10486412048339844\n",
|
||
" time_this_iter_s: 0.10486412048339844\n",
|
||
" time_total_s: 0.10486412048339844\n",
|
||
" timestamp: 1658500285\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 1\n",
|
||
" trial_id: f5aedf9a\n",
|
||
" warmup_time: 0.002830028533935547\n",
|
||
" \n",
|
||
"Result for objective_f3a0bef8:\n",
|
||
" date: 2022-07-22_15-31-27\n",
|
||
" done: false\n",
|
||
" experiment_id: 7bbba7b85dbc4029bc8b22b7866d091f\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 47\n",
|
||
" iterations_since_restore: 48\n",
|
||
" mean_loss: 0.12364690126601663\n",
|
||
" neg_mean_loss: -0.12364690126601663\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47180\n",
|
||
" time_since_restore: 5.149861097335815\n",
|
||
" time_this_iter_s: 0.1075279712677002\n",
|
||
" time_total_s: 5.149861097335815\n",
|
||
" timestamp: 1658500287\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 48\n",
|
||
" trial_id: f3a0bef8\n",
|
||
" warmup_time: 0.003498077392578125\n",
|
||
" \n",
|
||
"Result for objective_f59fe9d6:\n",
|
||
" date: 2022-07-22_15-31-30\n",
|
||
" done: false\n",
|
||
" experiment_id: efbda212c15c4bd38cfc5f39dfae8b73\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 47\n",
|
||
" iterations_since_restore: 48\n",
|
||
" mean_loss: -2.3825537636804834\n",
|
||
" neg_mean_loss: 2.3825537636804834\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47185\n",
|
||
" time_since_restore: 5.170727014541626\n",
|
||
" time_this_iter_s: 0.10740494728088379\n",
|
||
" time_total_s: 5.170727014541626\n",
|
||
" timestamp: 1658500290\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 48\n",
|
||
" trial_id: f59fe9d6\n",
|
||
" warmup_time: 0.003314971923828125\n",
|
||
" \n",
|
||
"Result for objective_f5b1ec08:\n",
|
||
" date: 2022-07-22_15-31-30\n",
|
||
" done: false\n",
|
||
" experiment_id: dadb542868ba4b10adf1ef161c75ea17\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 47\n",
|
||
" iterations_since_restore: 48\n",
|
||
" mean_loss: 0.5431509247331814\n",
|
||
" neg_mean_loss: -0.5431509247331814\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47189\n",
|
||
" time_since_restore: 5.153015613555908\n",
|
||
" time_this_iter_s: 0.10737276077270508\n",
|
||
" time_total_s: 5.153015613555908\n",
|
||
" timestamp: 1658500290\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 48\n",
|
||
" trial_id: f5b1ec08\n",
|
||
" warmup_time: 0.0031630992889404297\n",
|
||
" \n",
|
||
"Result for objective_f5aedf9a:\n",
|
||
" date: 2022-07-22_15-31-30\n",
|
||
" done: false\n",
|
||
" experiment_id: 9f7847b7440648cb86e73bf1be0ccc05\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 47\n",
|
||
" iterations_since_restore: 48\n",
|
||
" mean_loss: 4.977264708542528\n",
|
||
" neg_mean_loss: -4.977264708542528\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47188\n",
|
||
" time_since_restore: 5.183044195175171\n",
|
||
" time_this_iter_s: 0.10844588279724121\n",
|
||
" time_total_s: 5.183044195175171\n",
|
||
" timestamp: 1658500290\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 48\n",
|
||
" trial_id: f5aedf9a\n",
|
||
" warmup_time: 0.002830028533935547\n",
|
||
" \n",
|
||
"Result for objective_f3a0bef8:\n",
|
||
" date: 2022-07-22_15-31-32\n",
|
||
" done: false\n",
|
||
" experiment_id: 7bbba7b85dbc4029bc8b22b7866d091f\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 94\n",
|
||
" iterations_since_restore: 95\n",
|
||
" mean_loss: -0.29046425326874115\n",
|
||
" neg_mean_loss: 0.29046425326874115\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47180\n",
|
||
" time_since_restore: 10.200733184814453\n",
|
||
" time_this_iter_s: 0.10860228538513184\n",
|
||
" time_total_s: 10.200733184814453\n",
|
||
" timestamp: 1658500292\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 95\n",
|
||
" trial_id: f3a0bef8\n",
|
||
" warmup_time: 0.003498077392578125\n",
|
||
" \n",
|
||
"Result for objective_f3a0bef8:\n",
|
||
" date: 2022-07-22_15-31-32\n",
|
||
" done: true\n",
|
||
" experiment_id: 7bbba7b85dbc4029bc8b22b7866d091f\n",
|
||
" experiment_tag: 5_activation=tanh,height=-7.4392,steps=100,width=2.2397\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 99\n",
|
||
" iterations_since_restore: 100\n",
|
||
" mean_loss: -0.3123775218508783\n",
|
||
" neg_mean_loss: 0.3123775218508783\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47180\n",
|
||
" time_since_restore: 10.737780094146729\n",
|
||
" time_this_iter_s: 0.10722112655639648\n",
|
||
" time_total_s: 10.737780094146729\n",
|
||
" timestamp: 1658500292\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 100\n",
|
||
" trial_id: f3a0bef8\n",
|
||
" warmup_time: 0.003498077392578125\n",
|
||
" \n",
|
||
"Result for objective_fb926d32:\n",
|
||
" date: 2022-07-22_15-31-35\n",
|
||
" done: false\n",
|
||
" experiment_id: c6af4463441e4a3a8ca1298a043e6151\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 0\n",
|
||
" iterations_since_restore: 1\n",
|
||
" mean_loss: 14.477795797697073\n",
|
||
" neg_mean_loss: -14.477795797697073\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47205\n",
|
||
" time_since_restore: 0.10373091697692871\n",
|
||
" time_this_iter_s: 0.10373091697692871\n",
|
||
" time_total_s: 0.10373091697692871\n",
|
||
" timestamp: 1658500295\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 1\n",
|
||
" trial_id: fb926d32\n",
|
||
" warmup_time: 0.0029900074005126953\n",
|
||
" \n",
|
||
"Result for objective_f59fe9d6:\n",
|
||
" date: 2022-07-22_15-31-35\n",
|
||
" done: false\n",
|
||
" experiment_id: efbda212c15c4bd38cfc5f39dfae8b73\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 94\n",
|
||
" iterations_since_restore: 95\n",
|
||
" mean_loss: -2.5626347652141552\n",
|
||
" neg_mean_loss: 2.5626347652141552\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47185\n",
|
||
" time_since_restore: 10.179347038269043\n",
|
||
" time_this_iter_s: 0.1074531078338623\n",
|
||
" time_total_s: 10.179347038269043\n",
|
||
" timestamp: 1658500295\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 95\n",
|
||
" trial_id: f59fe9d6\n",
|
||
" warmup_time: 0.003314971923828125\n",
|
||
" \n",
|
||
"Result for objective_f5b1ec08:\n",
|
||
" date: 2022-07-22_15-31-35\n",
|
||
" done: false\n",
|
||
" experiment_id: dadb542868ba4b10adf1ef161c75ea17\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 94\n",
|
||
" iterations_since_restore: 95\n",
|
||
" mean_loss: 0.4790434958168009\n",
|
||
" neg_mean_loss: -0.4790434958168009\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47189\n",
|
||
" time_since_restore: 10.170033931732178\n",
|
||
" time_this_iter_s: 0.10643315315246582\n",
|
||
" time_total_s: 10.170033931732178\n",
|
||
" timestamp: 1658500295\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 95\n",
|
||
" trial_id: f5b1ec08\n",
|
||
" warmup_time: 0.0031630992889404297\n",
|
||
" \n",
|
||
"Result for objective_f5aedf9a:\n",
|
||
" date: 2022-07-22_15-31-35\n",
|
||
" done: false\n",
|
||
" experiment_id: 9f7847b7440648cb86e73bf1be0ccc05\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 94\n",
|
||
" iterations_since_restore: 95\n",
|
||
" mean_loss: 4.924220339829169\n",
|
||
" neg_mean_loss: -4.924220339829169\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47188\n",
|
||
" time_since_restore: 10.189186096191406\n",
|
||
" time_this_iter_s: 0.1078641414642334\n",
|
||
" time_total_s: 10.189186096191406\n",
|
||
" timestamp: 1658500295\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 95\n",
|
||
" trial_id: f5aedf9a\n",
|
||
" warmup_time: 0.002830028533935547\n",
|
||
" \n",
|
||
"Result for objective_f59fe9d6:\n",
|
||
" date: 2022-07-22_15-31-36\n",
|
||
" done: true\n",
|
||
" experiment_id: efbda212c15c4bd38cfc5f39dfae8b73\n",
|
||
" experiment_tag: 6_activation=tanh,height=-27.4958,steps=100,width=5.5843\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 99\n",
|
||
" iterations_since_restore: 100\n",
|
||
" mean_loss: -2.5719085451008423\n",
|
||
" neg_mean_loss: 2.5719085451008423\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47185\n",
|
||
" time_since_restore: 10.714513063430786\n",
|
||
" time_this_iter_s: 0.1071622371673584\n",
|
||
" time_total_s: 10.714513063430786\n",
|
||
" timestamp: 1658500296\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 100\n",
|
||
" trial_id: f59fe9d6\n",
|
||
" warmup_time: 0.003314971923828125\n",
|
||
" \n",
|
||
"Result for objective_f5b1ec08:\n",
|
||
" date: 2022-07-22_15-31-36\n",
|
||
" done: true\n",
|
||
" experiment_id: dadb542868ba4b10adf1ef161c75ea17\n",
|
||
" experiment_tag: 8_activation=tanh,height=4.1410,steps=100,width=16.2739\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 99\n",
|
||
" iterations_since_restore: 100\n",
|
||
" mean_loss: 0.4757836498809659\n",
|
||
" neg_mean_loss: -0.4757836498809659\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47189\n",
|
||
" time_since_restore: 10.703420877456665\n",
|
||
" time_this_iter_s: 0.10684394836425781\n",
|
||
" time_total_s: 10.703420877456665\n",
|
||
" timestamp: 1658500296\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 100\n",
|
||
" trial_id: f5b1ec08\n",
|
||
" warmup_time: 0.0031630992889404297\n",
|
||
" \n"
|
||
]
|
||
},
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Result for objective_f5aedf9a:\n",
|
||
" date: 2022-07-22_15-31-36\n",
|
||
" done: true\n",
|
||
" experiment_id: 9f7847b7440648cb86e73bf1be0ccc05\n",
|
||
" experiment_tag: 7_activation=tanh,height=48.7060,steps=100,width=19.7352\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 99\n",
|
||
" iterations_since_restore: 100\n",
|
||
" mean_loss: 4.9215262379377105\n",
|
||
" neg_mean_loss: -4.9215262379377105\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47188\n",
|
||
" time_since_restore: 10.734119176864624\n",
|
||
" time_this_iter_s: 0.11568498611450195\n",
|
||
" time_total_s: 10.734119176864624\n",
|
||
" timestamp: 1658500296\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 100\n",
|
||
" trial_id: f5aedf9a\n",
|
||
" warmup_time: 0.002830028533935547\n",
|
||
" \n",
|
||
"Result for objective_fd91e28e:\n",
|
||
" date: 2022-07-22_15-31-38\n",
|
||
" done: false\n",
|
||
" experiment_id: 5c3693eeb55b476088ce1de9127c5a39\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 0\n",
|
||
" iterations_since_restore: 1\n",
|
||
" mean_loss: 9.70376987633604\n",
|
||
" neg_mean_loss: -9.70376987633604\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47214\n",
|
||
" time_since_restore: 0.10451984405517578\n",
|
||
" time_this_iter_s: 0.10451984405517578\n",
|
||
" time_total_s: 0.10451984405517578\n",
|
||
" timestamp: 1658500298\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 1\n",
|
||
" trial_id: fd91e28e\n",
|
||
" warmup_time: 0.00302886962890625\n",
|
||
" \n",
|
||
"Result for objective_fb926d32:\n",
|
||
" date: 2022-07-22_15-31-40\n",
|
||
" done: false\n",
|
||
" experiment_id: c6af4463441e4a3a8ca1298a043e6151\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 25\n",
|
||
" iterations_since_restore: 26\n",
|
||
" mean_loss: 4.859752718513368\n",
|
||
" neg_mean_loss: -4.859752718513368\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47205\n",
|
||
" time_since_restore: 5.151448011398315\n",
|
||
" time_this_iter_s: 0.11450004577636719\n",
|
||
" time_total_s: 5.151448011398315\n",
|
||
" timestamp: 1658500300\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 26\n",
|
||
" trial_id: fb926d32\n",
|
||
" warmup_time: 0.0029900074005126953\n",
|
||
" \n",
|
||
"Result for objective_fd91e28e:\n",
|
||
" date: 2022-07-22_15-31-43\n",
|
||
" done: false\n",
|
||
" experiment_id: 5c3693eeb55b476088ce1de9127c5a39\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 46\n",
|
||
" iterations_since_restore: 47\n",
|
||
" mean_loss: -0.11565561363589152\n",
|
||
" neg_mean_loss: 0.11565561363589152\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47214\n",
|
||
" time_since_restore: 5.126538991928101\n",
|
||
" time_this_iter_s: 0.10815834999084473\n",
|
||
" time_total_s: 5.126538991928101\n",
|
||
" timestamp: 1658500303\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 47\n",
|
||
" trial_id: fd91e28e\n",
|
||
" warmup_time: 0.00302886962890625\n",
|
||
" \n",
|
||
"Result for objective_fb926d32:\n",
|
||
" date: 2022-07-22_15-31-45\n",
|
||
" done: false\n",
|
||
" experiment_id: c6af4463441e4a3a8ca1298a043e6151\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 72\n",
|
||
" iterations_since_restore: 73\n",
|
||
" mean_loss: 4.613811037170093\n",
|
||
" neg_mean_loss: -4.613811037170093\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47205\n",
|
||
" time_since_restore: 10.208579063415527\n",
|
||
" time_this_iter_s: 0.10763001441955566\n",
|
||
" time_total_s: 10.208579063415527\n",
|
||
" timestamp: 1658500305\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 73\n",
|
||
" trial_id: fb926d32\n",
|
||
" warmup_time: 0.0029900074005126953\n",
|
||
" \n",
|
||
"Result for objective_fb926d32:\n",
|
||
" date: 2022-07-22_15-31-48\n",
|
||
" done: true\n",
|
||
" experiment_id: c6af4463441e4a3a8ca1298a043e6151\n",
|
||
" experiment_tag: 9_activation=tanh,height=44.7780,steps=100,width=10.0724\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 99\n",
|
||
" iterations_since_restore: 100\n",
|
||
" mean_loss: 4.577084283144497\n",
|
||
" neg_mean_loss: -4.577084283144497\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47205\n",
|
||
" time_since_restore: 13.110869884490967\n",
|
||
" time_this_iter_s: 0.10806989669799805\n",
|
||
" time_total_s: 13.110869884490967\n",
|
||
" timestamp: 1658500308\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 100\n",
|
||
" trial_id: fb926d32\n",
|
||
" warmup_time: 0.0029900074005126953\n",
|
||
" \n",
|
||
"Result for objective_fd91e28e:\n",
|
||
" date: 2022-07-22_15-31-48\n",
|
||
" done: false\n",
|
||
" experiment_id: 5c3693eeb55b476088ce1de9127c5a39\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 93\n",
|
||
" iterations_since_restore: 94\n",
|
||
" mean_loss: -0.20609110794676003\n",
|
||
" neg_mean_loss: 0.20609110794676003\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47214\n",
|
||
" time_since_restore: 10.14166784286499\n",
|
||
" time_this_iter_s: 0.10436320304870605\n",
|
||
" time_total_s: 10.14166784286499\n",
|
||
" timestamp: 1658500308\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 94\n",
|
||
" trial_id: fd91e28e\n",
|
||
" warmup_time: 0.00302886962890625\n",
|
||
" \n",
|
||
"Result for objective_fd91e28e:\n",
|
||
" date: 2022-07-22_15-31-49\n",
|
||
" done: true\n",
|
||
" experiment_id: 5c3693eeb55b476088ce1de9127c5a39\n",
|
||
" experiment_tag: 10_activation=relu,height=-2.9623,steps=100,width=11.8215\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 99\n",
|
||
" iterations_since_restore: 100\n",
|
||
" mean_loss: -0.21150779503682235\n",
|
||
" neg_mean_loss: 0.21150779503682235\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47214\n",
|
||
" time_since_restore: 10.793406963348389\n",
|
||
" time_this_iter_s: 0.10776925086975098\n",
|
||
" time_total_s: 10.793406963348389\n",
|
||
" timestamp: 1658500309\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 100\n",
|
||
" trial_id: fd91e28e\n",
|
||
" warmup_time: 0.00302886962890625\n",
|
||
" \n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"tuner = tune.Tuner(\n",
|
||
" objective,\n",
|
||
" tune_config=tune.TuneConfig(\n",
|
||
" metric=\"mean_loss\",\n",
|
||
" mode=\"min\",\n",
|
||
" search_alg=algo,\n",
|
||
" num_samples=num_samples,\n",
|
||
" ),\n",
|
||
" param_space=search_config,\n",
|
||
")\n",
|
||
"results = tuner.fit()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "49be6f01",
|
||
"metadata": {},
|
||
"source": [
|
||
"Here are the hyperparamters found to minimize the mean loss of the defined objective."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 11,
|
||
"id": "7036798c",
|
||
"metadata": {
|
||
"vscode": {
|
||
"languageId": "python"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Best hyperparameters found were: {'steps': 100, 'width': 5.584304853357766, 'height': -27.49576980043919, 'activation': 'tanh'}\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"print(\"Best hyperparameters found were: \", results.get_best_result().config)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "504e9d2a",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Conditional search spaces\n",
|
||
"\n",
|
||
"Sometimes we may want to build a more complicated search space that has conditional dependencies on other hyperparameters. In this case, we pass a nested dictionary to `objective_two`, which has been slightly adjusted from `objective` to deal with the conditional search space."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 12,
|
||
"id": "2f7b5449",
|
||
"metadata": {
|
||
"vscode": {
|
||
"languageId": "python"
|
||
}
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"def evaluation_fn(step, width, height, mult=1):\n",
|
||
" return (0.1 + width * step / 100) ** (-1) + height * 0.1 * mult"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 13,
|
||
"id": "4b83b81c",
|
||
"metadata": {
|
||
"vscode": {
|
||
"languageId": "python"
|
||
}
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"def objective_two(config):\n",
|
||
" width, height = config[\"width\"], config[\"height\"]\n",
|
||
" sub_dict = config[\"activation\"]\n",
|
||
" mult = sub_dict.get(\"mult\", 1)\n",
|
||
" \n",
|
||
" for step in range(config[\"steps\"]):\n",
|
||
" intermediate_score = evaluation_fn(step, width, height, mult)\n",
|
||
" session.report({\"iterations\": step, \"mean_loss\": intermediate_score})\n",
|
||
" time.sleep(0.1)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 14,
|
||
"id": "75cea99e",
|
||
"metadata": {
|
||
"vscode": {
|
||
"languageId": "python"
|
||
}
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"conditional_space = {\n",
|
||
" \"activation\": hp.choice(\n",
|
||
" \"activation\",\n",
|
||
" [\n",
|
||
" {\"activation\": \"relu\", \"mult\": hp.uniform(\"mult\", 1, 2)},\n",
|
||
" {\"activation\": \"tanh\"},\n",
|
||
" ],\n",
|
||
" ),\n",
|
||
" \"width\": hp.uniform(\"width\", 0, 20),\n",
|
||
" \"height\": hp.uniform(\"height\", -100, 100),\n",
|
||
" \"steps\": 100,\n",
|
||
"}"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "7df282c1",
|
||
"metadata": {},
|
||
"source": [
|
||
"Now we the define the search algorithm built from `HyperOptSearch` constrained by `ConcurrencyLimiter`. When the hyperparameter search space is conditional, we pass it (`conditional_space`) into `HyperOptSearch`."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 15,
|
||
"id": "ea2c71a6",
|
||
"metadata": {
|
||
"vscode": {
|
||
"languageId": "python"
|
||
}
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"algo = HyperOptSearch(space=conditional_space, metric=\"mean_loss\", mode=\"min\")\n",
|
||
"algo = ConcurrencyLimiter(algo, max_concurrent=4)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "630f84ab",
|
||
"metadata": {},
|
||
"source": [
|
||
"Now we run the experiment, this time with an empty `config` because we instead provided `space` to the `HyperOptSearch` `search_alg`."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 16,
|
||
"id": "14111e9e",
|
||
"metadata": {
|
||
"vscode": {
|
||
"languageId": "python"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/html": [
|
||
"== Status ==<br>Current time: 2022-07-22 15:32:33 (running for 00:00:44.21)<br>Memory usage on this node: 10.7/16.0 GiB<br>Using FIFO scheduling algorithm.<br>Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/5.29 GiB heap, 0.0/2.0 GiB objects<br>Current best trial: 0de7d38c with mean_loss=-9.200364093875208 and parameters={'activation': {'activation': 'relu', 'mult': 1.3982639549501585}, 'height': -66.3136247260571, 'steps': 100, 'width': 13.922128223483856}<br>Result logdir: /Users/kai/ray_results/objective_two_2022-07-22_15-31-49<br>Number of trials: 10/10 (10 TERMINATED)<br><table>\n",
|
||
"<thead>\n",
|
||
"<tr><th>Trial name </th><th>status </th><th>loc </th><th>activation/activa... </th><th style=\"text-align: right;\"> activation/mult</th><th style=\"text-align: right;\"> height</th><th style=\"text-align: right;\"> steps</th><th style=\"text-align: right;\"> width</th><th style=\"text-align: right;\"> loss</th><th style=\"text-align: right;\"> iter</th><th style=\"text-align: right;\"> total time (s)</th><th style=\"text-align: right;\"> iterations</th><th style=\"text-align: right;\"> neg_mean_loss</th></tr>\n",
|
||
"</thead>\n",
|
||
"<tbody>\n",
|
||
"<tr><td>objective_two_05a0ad52</td><td>TERMINATED</td><td>127.0.0.1:47229</td><td>tanh </td><td style=\"text-align: right;\"> </td><td style=\"text-align: right;\">-63.0091 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\">10.1954 </td><td style=\"text-align: right;\">-6.20281</td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 10.8547</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\"> 6.20281</td></tr>\n",
|
||
"<tr><td>objective_two_075f0d78</td><td>TERMINATED</td><td>127.0.0.1:47236</td><td>relu </td><td style=\"text-align: right;\"> 1.22098</td><td style=\"text-align: right;\"> 46.4977 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\">13.6093 </td><td style=\"text-align: right;\"> 5.75095</td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 12.0174</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\"> -5.75095</td></tr>\n",
|
||
"<tr><td>objective_two_07617f54</td><td>TERMINATED</td><td>127.0.0.1:47237</td><td>tanh </td><td style=\"text-align: right;\"> </td><td style=\"text-align: right;\">-41.2706 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 7.34134 </td><td style=\"text-align: right;\">-3.99133</td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 11.8677</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\"> 3.99133</td></tr>\n",
|
||
"<tr><td>objective_two_07636cd8</td><td>TERMINATED</td><td>127.0.0.1:47238</td><td>tanh </td><td style=\"text-align: right;\"> </td><td style=\"text-align: right;\"> 49.0313 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 0.828826</td><td style=\"text-align: right;\"> 5.98945</td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 12.0258</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\"> -5.98945</td></tr>\n",
|
||
"<tr><td>objective_two_0de7d38c</td><td>TERMINATED</td><td>127.0.0.1:47256</td><td>relu </td><td style=\"text-align: right;\"> 1.39826</td><td style=\"text-align: right;\">-66.3136 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\">13.9221 </td><td style=\"text-align: right;\">-9.20036</td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 10.7072</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\"> 9.20036</td></tr>\n",
|
||
"<tr><td>objective_two_100ebe3c</td><td>TERMINATED</td><td>127.0.0.1:47265</td><td>tanh </td><td style=\"text-align: right;\"> </td><td style=\"text-align: right;\">-38.8555 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\">15.9966 </td><td style=\"text-align: right;\">-3.82281</td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 10.6571</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\"> 3.82281</td></tr>\n",
|
||
"<tr><td>objective_two_101e702a</td><td>TERMINATED</td><td>127.0.0.1:47268</td><td>relu </td><td style=\"text-align: right;\"> 1.51022</td><td style=\"text-align: right;\"> 5.84901</td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 0.431039</td><td style=\"text-align: right;\"> 2.78184</td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 10.7202</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\"> -2.78184</td></tr>\n",
|
||
"<tr><td>objective_two_1022212a</td><td>TERMINATED</td><td>127.0.0.1:47269</td><td>relu </td><td style=\"text-align: right;\"> 1.32257</td><td style=\"text-align: right;\"> 71.2885 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\">11.4466 </td><td style=\"text-align: right;\"> 9.51591</td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 10.7072</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\"> -9.51591</td></tr>\n",
|
||
"<tr><td>objective_two_15fbf8dc</td><td>TERMINATED</td><td>127.0.0.1:47288</td><td>relu </td><td style=\"text-align: right;\"> 1.22166</td><td style=\"text-align: right;\">-32.1584 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 9.43222 </td><td style=\"text-align: right;\">-3.82272</td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 10.5991</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\"> 3.82272</td></tr>\n",
|
||
"<tr><td>objective_two_18090732</td><td>TERMINATED</td><td>127.0.0.1:47302</td><td>tanh </td><td style=\"text-align: right;\"> </td><td style=\"text-align: right;\"> 52.7613 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\">16.7268 </td><td style=\"text-align: right;\"> 5.33616</td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 10.6336</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\"> -5.33616</td></tr>\n",
|
||
"</tbody>\n",
|
||
"</table><br><br>"
|
||
],
|
||
"text/plain": [
|
||
"<IPython.core.display.HTML object>"
|
||
]
|
||
},
|
||
"metadata": {},
|
||
"output_type": "display_data"
|
||
},
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Result for objective_two_05a0ad52:\n",
|
||
" date: 2022-07-22_15-31-52\n",
|
||
" done: false\n",
|
||
" experiment_id: af7041e0b2c947aa8a30c2e30f94ba83\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 0\n",
|
||
" iterations_since_restore: 1\n",
|
||
" mean_loss: 3.699090470918877\n",
|
||
" neg_mean_loss: -3.699090470918877\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47229\n",
|
||
" time_since_restore: 0.00011491775512695312\n",
|
||
" time_this_iter_s: 0.00011491775512695312\n",
|
||
" time_total_s: 0.00011491775512695312\n",
|
||
" timestamp: 1658500312\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 1\n",
|
||
" trial_id: 05a0ad52\n",
|
||
" warmup_time: 0.0027589797973632812\n",
|
||
" \n",
|
||
"Result for objective_two_07636cd8:\n",
|
||
" date: 2022-07-22_15-31-55\n",
|
||
" done: false\n",
|
||
" experiment_id: 5c9d75b036ae410b9fdea303f14fe4cd\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 0\n",
|
||
" iterations_since_restore: 1\n",
|
||
" mean_loss: 14.903126957374846\n",
|
||
" neg_mean_loss: -14.903126957374846\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47238\n",
|
||
" time_since_restore: 0.00011110305786132812\n",
|
||
" time_this_iter_s: 0.00011110305786132812\n",
|
||
" time_total_s: 0.00011110305786132812\n",
|
||
" timestamp: 1658500315\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 1\n",
|
||
" trial_id: 07636cd8\n",
|
||
" warmup_time: 0.0037419795989990234\n",
|
||
" \n",
|
||
"Result for objective_two_07617f54:\n",
|
||
" date: 2022-07-22_15-31-55\n",
|
||
" done: false\n",
|
||
" experiment_id: 32155cb3b7f04dc3b7b5b679eecd8c46\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 0\n",
|
||
" iterations_since_restore: 1\n",
|
||
" mean_loss: 5.872941741395985\n",
|
||
" neg_mean_loss: -5.872941741395985\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47237\n",
|
||
" time_since_restore: 0.00015211105346679688\n",
|
||
" time_this_iter_s: 0.00015211105346679688\n",
|
||
" time_total_s: 0.00015211105346679688\n",
|
||
" timestamp: 1658500315\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 1\n",
|
||
" trial_id: 07617f54\n",
|
||
" warmup_time: 0.00436091423034668\n",
|
||
" \n",
|
||
"Result for objective_two_075f0d78:\n",
|
||
" date: 2022-07-22_15-31-55\n",
|
||
" done: false\n",
|
||
" experiment_id: 5a46a57ef1164628bac1618e39078e0e\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 0\n",
|
||
" iterations_since_restore: 1\n",
|
||
" mean_loss: 15.677277832165048\n",
|
||
" neg_mean_loss: -15.677277832165048\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47236\n",
|
||
" time_since_restore: 0.00011396408081054688\n",
|
||
" time_this_iter_s: 0.00011396408081054688\n",
|
||
" time_total_s: 0.00011396408081054688\n",
|
||
" timestamp: 1658500315\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 1\n",
|
||
" trial_id: 075f0d78\n",
|
||
" warmup_time: 0.0035758018493652344\n",
|
||
" \n",
|
||
"Result for objective_two_05a0ad52:\n",
|
||
" date: 2022-07-22_15-31-57\n",
|
||
" done: false\n",
|
||
" experiment_id: af7041e0b2c947aa8a30c2e30f94ba83\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 45\n",
|
||
" iterations_since_restore: 46\n",
|
||
" mean_loss: -6.087595964816368\n",
|
||
" neg_mean_loss: 6.087595964816368\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47229\n",
|
||
" time_since_restore: 5.055715799331665\n",
|
||
" time_this_iter_s: 0.10677886009216309\n",
|
||
" time_total_s: 5.055715799331665\n",
|
||
" timestamp: 1658500317\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 46\n",
|
||
" trial_id: 05a0ad52\n",
|
||
" warmup_time: 0.0027589797973632812\n",
|
||
" \n",
|
||
"Result for objective_two_07617f54:\n",
|
||
" date: 2022-07-22_15-32-00\n",
|
||
" done: false\n",
|
||
" experiment_id: 32155cb3b7f04dc3b7b5b679eecd8c46\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 47\n",
|
||
" iterations_since_restore: 48\n",
|
||
" mean_loss: -3.8454022145530584\n",
|
||
" neg_mean_loss: 3.8454022145530584\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47237\n",
|
||
" time_since_restore: 5.048717021942139\n",
|
||
" time_this_iter_s: 0.10686898231506348\n",
|
||
" time_total_s: 5.048717021942139\n",
|
||
" timestamp: 1658500320\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 48\n",
|
||
" trial_id: 07617f54\n",
|
||
" warmup_time: 0.00436091423034668\n",
|
||
" \n",
|
||
"Result for objective_two_07636cd8:\n",
|
||
" date: 2022-07-22_15-32-00\n",
|
||
" done: false\n",
|
||
" experiment_id: 5c9d75b036ae410b9fdea303f14fe4cd\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 47\n",
|
||
" iterations_since_restore: 48\n",
|
||
" mean_loss: 6.9458266379188185\n",
|
||
" neg_mean_loss: -6.9458266379188185\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47238\n",
|
||
" time_since_restore: 5.097726821899414\n",
|
||
" time_this_iter_s: 0.1087799072265625\n",
|
||
" time_total_s: 5.097726821899414\n",
|
||
" timestamp: 1658500320\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 48\n",
|
||
" trial_id: 07636cd8\n",
|
||
" warmup_time: 0.0037419795989990234\n",
|
||
" \n",
|
||
"Result for objective_two_075f0d78:\n",
|
||
" date: 2022-07-22_15-32-00\n",
|
||
" done: false\n",
|
||
" experiment_id: 5a46a57ef1164628bac1618e39078e0e\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 47\n",
|
||
" iterations_since_restore: 48\n",
|
||
" mean_loss: 5.83121027007688\n",
|
||
" neg_mean_loss: -5.83121027007688\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47236\n",
|
||
" time_since_restore: 5.118211269378662\n",
|
||
" time_this_iter_s: 0.10918712615966797\n",
|
||
" time_total_s: 5.118211269378662\n",
|
||
" timestamp: 1658500320\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 48\n",
|
||
" trial_id: 075f0d78\n",
|
||
" warmup_time: 0.0035758018493652344\n",
|
||
" \n",
|
||
"Result for objective_two_05a0ad52:\n",
|
||
" date: 2022-07-22_15-32-02\n",
|
||
" done: false\n",
|
||
" experiment_id: af7041e0b2c947aa8a30c2e30f94ba83\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 92\n",
|
||
" iterations_since_restore: 93\n",
|
||
" mean_loss: -6.195421815991769\n",
|
||
" neg_mean_loss: 6.195421815991769\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47229\n",
|
||
" time_since_restore: 10.10265588760376\n",
|
||
" time_this_iter_s: 0.10532593727111816\n",
|
||
" time_total_s: 10.10265588760376\n",
|
||
" timestamp: 1658500322\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 93\n",
|
||
" trial_id: 05a0ad52\n",
|
||
" warmup_time: 0.0027589797973632812\n",
|
||
" \n",
|
||
"Result for objective_two_05a0ad52:\n",
|
||
" date: 2022-07-22_15-32-03\n",
|
||
" done: true\n",
|
||
" experiment_id: af7041e0b2c947aa8a30c2e30f94ba83\n",
|
||
" experiment_tag: 1_activation=tanh,height=-63.0091,steps=100,width=10.1954\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 99\n",
|
||
" iterations_since_restore: 100\n",
|
||
" mean_loss: -6.202807371456877\n",
|
||
" neg_mean_loss: 6.202807371456877\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47229\n",
|
||
" time_since_restore: 10.854680061340332\n",
|
||
" time_this_iter_s: 0.10699224472045898\n",
|
||
" time_total_s: 10.854680061340332\n",
|
||
" timestamp: 1658500323\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 100\n",
|
||
" trial_id: 05a0ad52\n",
|
||
" warmup_time: 0.0027589797973632812\n",
|
||
" \n",
|
||
"Result for objective_two_075f0d78:\n",
|
||
" date: 2022-07-22_15-32-04\n",
|
||
" done: false\n",
|
||
" experiment_id: 5a46a57ef1164628bac1618e39078e0e\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 91\n",
|
||
" iterations_since_restore: 92\n",
|
||
" mean_loss: 5.757377572321986\n",
|
||
" neg_mean_loss: -5.757377572321986\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47236\n",
|
||
" time_since_restore: 9.820771932601929\n",
|
||
" time_this_iter_s: 0.10582685470581055\n",
|
||
" time_total_s: 9.820771932601929\n",
|
||
" timestamp: 1658500324\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 92\n",
|
||
" trial_id: 075f0d78\n",
|
||
" warmup_time: 0.0035758018493652344\n",
|
||
" \n",
|
||
"Result for objective_two_07617f54:\n",
|
||
" date: 2022-07-22_15-32-04\n",
|
||
" done: false\n",
|
||
" experiment_id: 32155cb3b7f04dc3b7b5b679eecd8c46\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 92\n",
|
||
" iterations_since_restore: 93\n",
|
||
" mean_loss: -3.981158750752472\n",
|
||
" neg_mean_loss: 3.981158750752472\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47237\n",
|
||
" time_since_restore: 9.865068197250366\n",
|
||
" time_this_iter_s: 0.1065821647644043\n",
|
||
" time_total_s: 9.865068197250366\n",
|
||
" timestamp: 1658500324\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 93\n",
|
||
" trial_id: 07617f54\n",
|
||
" warmup_time: 0.00436091423034668\n",
|
||
" \n",
|
||
"Result for objective_two_0de7d38c:\n",
|
||
" date: 2022-07-22_15-32-06\n",
|
||
" done: false\n",
|
||
" experiment_id: 81d62307312a447e97c5d1f1cdad2687\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 0\n",
|
||
" iterations_since_restore: 1\n",
|
||
" mean_loss: 0.7276048823462773\n",
|
||
" neg_mean_loss: -0.7276048823462773\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47256\n",
|
||
" time_since_restore: 0.00012803077697753906\n",
|
||
" time_this_iter_s: 0.00012803077697753906\n",
|
||
" time_total_s: 0.00012803077697753906\n",
|
||
" timestamp: 1658500326\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 1\n",
|
||
" trial_id: 0de7d38c\n",
|
||
" warmup_time: 0.0027358531951904297\n",
|
||
" \n",
|
||
"Result for objective_two_07636cd8:\n",
|
||
" date: 2022-07-22_15-32-04\n",
|
||
" done: false\n",
|
||
" experiment_id: 5c9d75b036ae410b9fdea303f14fe4cd\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 91\n",
|
||
" iterations_since_restore: 92\n",
|
||
" mean_loss: 6.073769581173668\n",
|
||
" neg_mean_loss: -6.073769581173668\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47238\n",
|
||
" time_since_restore: 9.813458919525146\n",
|
||
" time_this_iter_s: 0.10477709770202637\n",
|
||
" time_total_s: 9.813458919525146\n",
|
||
" timestamp: 1658500324\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 92\n",
|
||
" trial_id: 07636cd8\n",
|
||
" warmup_time: 0.0037419795989990234\n",
|
||
" \n",
|
||
"Result for objective_two_07617f54:\n",
|
||
" date: 2022-07-22_15-32-06\n",
|
||
" done: true\n",
|
||
" experiment_id: 32155cb3b7f04dc3b7b5b679eecd8c46\n",
|
||
" experiment_tag: 3_activation=tanh,height=-41.2706,steps=100,width=7.3413\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 99\n",
|
||
" iterations_since_restore: 100\n",
|
||
" mean_loss: -3.9913348635947523\n",
|
||
" neg_mean_loss: 3.9913348635947523\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47237\n",
|
||
" time_since_restore: 11.867748260498047\n",
|
||
" time_this_iter_s: 0.10774397850036621\n",
|
||
" time_total_s: 11.867748260498047\n",
|
||
" timestamp: 1658500326\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 100\n",
|
||
" trial_id: 07617f54\n",
|
||
" warmup_time: 0.00436091423034668\n",
|
||
" \n",
|
||
"Result for objective_two_075f0d78:\n",
|
||
" date: 2022-07-22_15-32-07\n",
|
||
" done: true\n",
|
||
" experiment_id: 5a46a57ef1164628bac1618e39078e0e\n",
|
||
" experiment_tag: 2_activation=relu,mult=1.2210,height=46.4977,steps=100,width=13.6093\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 99\n",
|
||
" iterations_since_restore: 100\n",
|
||
" mean_loss: 5.7509525535298085\n",
|
||
" neg_mean_loss: -5.7509525535298085\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47236\n",
|
||
" time_since_restore: 12.017354011535645\n",
|
||
" time_this_iter_s: 0.10736513137817383\n",
|
||
" time_total_s: 12.017354011535645\n",
|
||
" timestamp: 1658500327\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 100\n",
|
||
" trial_id: 075f0d78\n",
|
||
" warmup_time: 0.0035758018493652344\n",
|
||
" \n",
|
||
"Result for objective_two_07636cd8:\n",
|
||
" date: 2022-07-22_15-32-07\n",
|
||
" done: true\n",
|
||
" experiment_id: 5c9d75b036ae410b9fdea303f14fe4cd\n",
|
||
" experiment_tag: 4_activation=tanh,height=49.0313,steps=100,width=0.8288\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 99\n",
|
||
" iterations_since_restore: 100\n",
|
||
" mean_loss: 5.989448515159168\n",
|
||
" neg_mean_loss: -5.989448515159168\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47238\n",
|
||
" time_since_restore: 12.025846004486084\n",
|
||
" time_this_iter_s: 0.10526180267333984\n",
|
||
" time_total_s: 12.025846004486084\n",
|
||
" timestamp: 1658500327\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 100\n",
|
||
" trial_id: 07636cd8\n",
|
||
" warmup_time: 0.0037419795989990234\n",
|
||
" \n"
|
||
]
|
||
},
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Result for objective_two_100ebe3c:\n",
|
||
" date: 2022-07-22_15-32-09\n",
|
||
" done: false\n",
|
||
" experiment_id: f31e00c7c87c4884bef04183585473d1\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 0\n",
|
||
" iterations_since_restore: 1\n",
|
||
" mean_loss: 6.11444579492732\n",
|
||
" neg_mean_loss: -6.11444579492732\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47265\n",
|
||
" time_since_restore: 0.00012183189392089844\n",
|
||
" time_this_iter_s: 0.00012183189392089844\n",
|
||
" time_total_s: 0.00012183189392089844\n",
|
||
" timestamp: 1658500329\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 1\n",
|
||
" trial_id: 100ebe3c\n",
|
||
" warmup_time: 0.002948284149169922\n",
|
||
" \n",
|
||
"Result for objective_two_101e702a:\n",
|
||
" date: 2022-07-22_15-32-09\n",
|
||
" done: false\n",
|
||
" experiment_id: 8c013d0ff6434dfe9719fa317f95feb8\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 0\n",
|
||
" iterations_since_restore: 1\n",
|
||
" mean_loss: 10.883328850459264\n",
|
||
" neg_mean_loss: -10.883328850459264\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47268\n",
|
||
" time_since_restore: 0.00011587142944335938\n",
|
||
" time_this_iter_s: 0.00011587142944335938\n",
|
||
" time_total_s: 0.00011587142944335938\n",
|
||
" timestamp: 1658500329\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 1\n",
|
||
" trial_id: 101e702a\n",
|
||
" warmup_time: 0.003309965133666992\n",
|
||
" \n",
|
||
"Result for objective_two_1022212a:\n",
|
||
" date: 2022-07-22_15-32-09\n",
|
||
" done: false\n",
|
||
" experiment_id: c3c0bdb8b68146c68f2c34e08fa2f184\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 0\n",
|
||
" iterations_since_restore: 1\n",
|
||
" mean_loss: 19.42843735686162\n",
|
||
" neg_mean_loss: -19.42843735686162\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47269\n",
|
||
" time_since_restore: 0.00011324882507324219\n",
|
||
" time_this_iter_s: 0.00011324882507324219\n",
|
||
" time_total_s: 0.00011324882507324219\n",
|
||
" timestamp: 1658500329\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 1\n",
|
||
" trial_id: 1022212a\n",
|
||
" warmup_time: 0.0029020309448242188\n",
|
||
" \n",
|
||
"Result for objective_two_0de7d38c:\n",
|
||
" date: 2022-07-22_15-32-11\n",
|
||
" done: false\n",
|
||
" experiment_id: 81d62307312a447e97c5d1f1cdad2687\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 47\n",
|
||
" iterations_since_restore: 48\n",
|
||
" mean_loss: -9.121869790245293\n",
|
||
" neg_mean_loss: 9.121869790245293\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47256\n",
|
||
" time_since_restore: 5.047628164291382\n",
|
||
" time_this_iter_s: 0.10473132133483887\n",
|
||
" time_total_s: 5.047628164291382\n",
|
||
" timestamp: 1658500331\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 48\n",
|
||
" trial_id: 0de7d38c\n",
|
||
" warmup_time: 0.0027358531951904297\n",
|
||
" \n",
|
||
"Result for objective_two_100ebe3c:\n",
|
||
" date: 2022-07-22_15-32-14\n",
|
||
" done: false\n",
|
||
" experiment_id: f31e00c7c87c4884bef04183585473d1\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 47\n",
|
||
" iterations_since_restore: 48\n",
|
||
" mean_loss: -3.7542934113649324\n",
|
||
" neg_mean_loss: 3.7542934113649324\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47265\n",
|
||
" time_since_restore: 5.057713985443115\n",
|
||
" time_this_iter_s: 0.10337996482849121\n",
|
||
" time_total_s: 5.057713985443115\n",
|
||
" timestamp: 1658500334\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 48\n",
|
||
" trial_id: 100ebe3c\n",
|
||
" warmup_time: 0.002948284149169922\n",
|
||
" \n",
|
||
"Result for objective_two_101e702a:\n",
|
||
" date: 2022-07-22_15-32-14\n",
|
||
" done: false\n",
|
||
" experiment_id: 8c013d0ff6434dfe9719fa317f95feb8\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 47\n",
|
||
" iterations_since_restore: 48\n",
|
||
" mean_loss: 4.188150411466976\n",
|
||
" neg_mean_loss: -4.188150411466976\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47268\n",
|
||
" time_since_restore: 5.101155757904053\n",
|
||
" time_this_iter_s: 0.10826706886291504\n",
|
||
" time_total_s: 5.101155757904053\n",
|
||
" timestamp: 1658500334\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 48\n",
|
||
" trial_id: 101e702a\n",
|
||
" warmup_time: 0.003309965133666992\n",
|
||
" \n",
|
||
"Result for objective_two_1022212a:\n",
|
||
" date: 2022-07-22_15-32-14\n",
|
||
" done: false\n",
|
||
" experiment_id: c3c0bdb8b68146c68f2c34e08fa2f184\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 47\n",
|
||
" iterations_since_restore: 48\n",
|
||
" mean_loss: 9.610922254324745\n",
|
||
" neg_mean_loss: -9.610922254324745\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47269\n",
|
||
" time_since_restore: 5.09367823600769\n",
|
||
" time_this_iter_s: 0.11145401000976562\n",
|
||
" time_total_s: 5.09367823600769\n",
|
||
" timestamp: 1658500334\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 48\n",
|
||
" trial_id: 1022212a\n",
|
||
" warmup_time: 0.0029020309448242188\n",
|
||
" \n",
|
||
"Result for objective_two_0de7d38c:\n",
|
||
" date: 2022-07-22_15-32-16\n",
|
||
" done: false\n",
|
||
" experiment_id: 81d62307312a447e97c5d1f1cdad2687\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 93\n",
|
||
" iterations_since_restore: 94\n",
|
||
" mean_loss: -9.195752548100788\n",
|
||
" neg_mean_loss: 9.195752548100788\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47256\n",
|
||
" time_since_restore: 10.068511009216309\n",
|
||
" time_this_iter_s: 0.10715794563293457\n",
|
||
" time_total_s: 10.068511009216309\n",
|
||
" timestamp: 1658500336\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 94\n",
|
||
" trial_id: 0de7d38c\n",
|
||
" warmup_time: 0.0027358531951904297\n",
|
||
" \n",
|
||
"Result for objective_two_0de7d38c:\n",
|
||
" date: 2022-07-22_15-32-16\n",
|
||
" done: true\n",
|
||
" experiment_id: 81d62307312a447e97c5d1f1cdad2687\n",
|
||
" experiment_tag: 5_activation=relu,mult=1.3983,height=-66.3136,steps=100,width=13.9221\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 99\n",
|
||
" iterations_since_restore: 100\n",
|
||
" mean_loss: -9.200364093875208\n",
|
||
" neg_mean_loss: 9.200364093875208\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47256\n",
|
||
" time_since_restore: 10.707159996032715\n",
|
||
" time_this_iter_s: 0.1063990592956543\n",
|
||
" time_total_s: 10.707159996032715\n",
|
||
" timestamp: 1658500336\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 100\n",
|
||
" trial_id: 0de7d38c\n",
|
||
" warmup_time: 0.0027358531951904297\n",
|
||
" \n",
|
||
"Result for objective_two_100ebe3c:\n",
|
||
" date: 2022-07-22_15-32-19\n",
|
||
" done: false\n",
|
||
" experiment_id: f31e00c7c87c4884bef04183585473d1\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 94\n",
|
||
" iterations_since_restore: 95\n",
|
||
" mean_loss: -3.8194902277136236\n",
|
||
" neg_mean_loss: 3.8194902277136236\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47265\n",
|
||
" time_since_restore: 10.118366956710815\n",
|
||
" time_this_iter_s: 0.10590505599975586\n",
|
||
" time_total_s: 10.118366956710815\n",
|
||
" timestamp: 1658500339\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 95\n",
|
||
" trial_id: 100ebe3c\n",
|
||
" warmup_time: 0.002948284149169922\n",
|
||
" \n",
|
||
"Result for objective_two_15fbf8dc:\n",
|
||
" date: 2022-07-22_15-32-19\n",
|
||
" done: false\n",
|
||
" experiment_id: 825ceb15d1134810ba19f09bb90c2b35\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 0\n",
|
||
" iterations_since_restore: 1\n",
|
||
" mean_loss: 6.071327317808535\n",
|
||
" neg_mean_loss: -6.071327317808535\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47288\n",
|
||
" time_since_restore: 0.0002067089080810547\n",
|
||
" time_this_iter_s: 0.0002067089080810547\n",
|
||
" time_total_s: 0.0002067089080810547\n",
|
||
" timestamp: 1658500339\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 1\n",
|
||
" trial_id: 15fbf8dc\n",
|
||
" warmup_time: 0.005033969879150391\n",
|
||
" \n",
|
||
"Result for objective_two_101e702a:\n",
|
||
" date: 2022-07-22_15-32-19\n",
|
||
" done: false\n",
|
||
" experiment_id: 8c013d0ff6434dfe9719fa317f95feb8\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 94\n",
|
||
" iterations_since_restore: 95\n",
|
||
" mean_loss: 2.86283543260466\n",
|
||
" neg_mean_loss: -2.86283543260466\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47268\n",
|
||
" time_since_restore: 10.181840896606445\n",
|
||
" time_this_iter_s: 0.10625004768371582\n",
|
||
" time_total_s: 10.181840896606445\n",
|
||
" timestamp: 1658500339\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 95\n",
|
||
" trial_id: 101e702a\n",
|
||
" warmup_time: 0.003309965133666992\n",
|
||
" \n",
|
||
"Result for objective_two_1022212a:\n",
|
||
" date: 2022-07-22_15-32-19\n",
|
||
" done: false\n",
|
||
" experiment_id: c3c0bdb8b68146c68f2c34e08fa2f184\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 94\n",
|
||
" iterations_since_restore: 95\n",
|
||
" mean_loss: 9.520519990087298\n",
|
||
" neg_mean_loss: -9.520519990087298\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47269\n",
|
||
" time_since_restore: 10.166353225708008\n",
|
||
" time_this_iter_s: 0.10583710670471191\n",
|
||
" time_total_s: 10.166353225708008\n",
|
||
" timestamp: 1658500339\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 95\n",
|
||
" trial_id: 1022212a\n",
|
||
" warmup_time: 0.0029020309448242188\n",
|
||
" \n",
|
||
"Result for objective_two_100ebe3c:\n",
|
||
" date: 2022-07-22_15-32-20\n",
|
||
" done: true\n",
|
||
" experiment_id: f31e00c7c87c4884bef04183585473d1\n",
|
||
" experiment_tag: 6_activation=tanh,height=-38.8555,steps=100,width=15.9966\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 99\n",
|
||
" iterations_since_restore: 100\n",
|
||
" mean_loss: -3.8228058558351754\n",
|
||
" neg_mean_loss: 3.8228058558351754\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47265\n",
|
||
" time_since_restore: 10.657065153121948\n",
|
||
" time_this_iter_s: 0.10721731185913086\n",
|
||
" time_total_s: 10.657065153121948\n",
|
||
" timestamp: 1658500340\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 100\n",
|
||
" trial_id: 100ebe3c\n",
|
||
" warmup_time: 0.002948284149169922\n",
|
||
" \n",
|
||
"Result for objective_two_101e702a:\n",
|
||
" date: 2022-07-22_15-32-20\n",
|
||
" done: true\n",
|
||
" experiment_id: 8c013d0ff6434dfe9719fa317f95feb8\n",
|
||
" experiment_tag: 7_activation=relu,mult=1.5102,height=5.8490,steps=100,width=0.4310\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 99\n",
|
||
" iterations_since_restore: 100\n",
|
||
" mean_loss: 2.781840740489214\n",
|
||
" neg_mean_loss: -2.781840740489214\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47268\n",
|
||
" time_since_restore: 10.72019100189209\n",
|
||
" time_this_iter_s: 0.1079869270324707\n",
|
||
" time_total_s: 10.72019100189209\n",
|
||
" timestamp: 1658500340\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 100\n",
|
||
" trial_id: 101e702a\n",
|
||
" warmup_time: 0.003309965133666992\n",
|
||
" \n",
|
||
"Result for objective_two_1022212a:\n",
|
||
" date: 2022-07-22_15-32-20\n",
|
||
" done: true\n",
|
||
" experiment_id: c3c0bdb8b68146c68f2c34e08fa2f184\n",
|
||
" experiment_tag: 8_activation=relu,mult=1.3226,height=71.2885,steps=100,width=11.4466\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 99\n",
|
||
" iterations_since_restore: 100\n",
|
||
" mean_loss: 9.515910032420853\n",
|
||
" neg_mean_loss: -9.515910032420853\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47269\n",
|
||
" time_since_restore: 10.707203149795532\n",
|
||
" time_this_iter_s: 0.10514593124389648\n",
|
||
" time_total_s: 10.707203149795532\n",
|
||
" timestamp: 1658500340\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 100\n",
|
||
" trial_id: 1022212a\n",
|
||
" warmup_time: 0.0029020309448242188\n",
|
||
" \n"
|
||
]
|
||
},
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Result for objective_two_18090732:\n",
|
||
" date: 2022-07-22_15-32-23\n",
|
||
" done: false\n",
|
||
" experiment_id: 7cb1145f46214bc4a5dd35e796969e53\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 0\n",
|
||
" iterations_since_restore: 1\n",
|
||
" mean_loss: 15.276134004304078\n",
|
||
" neg_mean_loss: -15.276134004304078\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47302\n",
|
||
" time_since_restore: 0.00011110305786132812\n",
|
||
" time_this_iter_s: 0.00011110305786132812\n",
|
||
" time_total_s: 0.00011110305786132812\n",
|
||
" timestamp: 1658500343\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 1\n",
|
||
" trial_id: '18090732'\n",
|
||
" warmup_time: 0.0028650760650634766\n",
|
||
" \n",
|
||
"Result for objective_two_15fbf8dc:\n",
|
||
" date: 2022-07-22_15-32-24\n",
|
||
" done: false\n",
|
||
" experiment_id: 825ceb15d1134810ba19f09bb90c2b35\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 47\n",
|
||
" iterations_since_restore: 48\n",
|
||
" mean_loss: -3.708075105876967\n",
|
||
" neg_mean_loss: 3.708075105876967\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47288\n",
|
||
" time_since_restore: 5.015958786010742\n",
|
||
" time_this_iter_s: 0.10610795021057129\n",
|
||
" time_total_s: 5.015958786010742\n",
|
||
" timestamp: 1658500344\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 48\n",
|
||
" trial_id: 15fbf8dc\n",
|
||
" warmup_time: 0.005033969879150391\n",
|
||
" \n",
|
||
"Result for objective_two_18090732:\n",
|
||
" date: 2022-07-22_15-32-28\n",
|
||
" done: false\n",
|
||
" experiment_id: 7cb1145f46214bc4a5dd35e796969e53\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 47\n",
|
||
" iterations_since_restore: 48\n",
|
||
" mean_loss: 5.4017373166361775\n",
|
||
" neg_mean_loss: -5.4017373166361775\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47302\n",
|
||
" time_since_restore: 5.047500133514404\n",
|
||
" time_this_iter_s: 0.10670995712280273\n",
|
||
" time_total_s: 5.047500133514404\n",
|
||
" timestamp: 1658500348\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 48\n",
|
||
" trial_id: '18090732'\n",
|
||
" warmup_time: 0.0028650760650634766\n",
|
||
" \n",
|
||
"Result for objective_two_15fbf8dc:\n",
|
||
" date: 2022-07-22_15-32-29\n",
|
||
" done: false\n",
|
||
" experiment_id: 825ceb15d1134810ba19f09bb90c2b35\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 94\n",
|
||
" iterations_since_restore: 95\n",
|
||
" mean_loss: -3.8171437433543964\n",
|
||
" neg_mean_loss: 3.8171437433543964\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47288\n",
|
||
" time_since_restore: 10.057979822158813\n",
|
||
" time_this_iter_s: 0.10869002342224121\n",
|
||
" time_total_s: 10.057979822158813\n",
|
||
" timestamp: 1658500349\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 95\n",
|
||
" trial_id: 15fbf8dc\n",
|
||
" warmup_time: 0.005033969879150391\n",
|
||
" \n",
|
||
"Result for objective_two_15fbf8dc:\n",
|
||
" date: 2022-07-22_15-32-30\n",
|
||
" done: true\n",
|
||
" experiment_id: 825ceb15d1134810ba19f09bb90c2b35\n",
|
||
" experiment_tag: 9_activation=relu,mult=1.2217,height=-32.1584,steps=100,width=9.4322\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 99\n",
|
||
" iterations_since_restore: 100\n",
|
||
" mean_loss: -3.8227168355020016\n",
|
||
" neg_mean_loss: 3.8227168355020016\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47288\n",
|
||
" time_since_restore: 10.599114894866943\n",
|
||
" time_this_iter_s: 0.10727405548095703\n",
|
||
" time_total_s: 10.599114894866943\n",
|
||
" timestamp: 1658500350\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 100\n",
|
||
" trial_id: 15fbf8dc\n",
|
||
" warmup_time: 0.005033969879150391\n",
|
||
" \n",
|
||
"Result for objective_two_18090732:\n",
|
||
" date: 2022-07-22_15-32-33\n",
|
||
" done: false\n",
|
||
" experiment_id: 7cb1145f46214bc4a5dd35e796969e53\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 94\n",
|
||
" iterations_since_restore: 95\n",
|
||
" mean_loss: 5.339332557853146\n",
|
||
" neg_mean_loss: -5.339332557853146\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47302\n",
|
||
" time_since_restore: 10.092173099517822\n",
|
||
" time_this_iter_s: 0.1067051887512207\n",
|
||
" time_total_s: 10.092173099517822\n",
|
||
" timestamp: 1658500353\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 95\n",
|
||
" trial_id: '18090732'\n",
|
||
" warmup_time: 0.0028650760650634766\n",
|
||
" \n",
|
||
"Result for objective_two_18090732:\n",
|
||
" date: 2022-07-22_15-32-33\n",
|
||
" done: true\n",
|
||
" experiment_id: 7cb1145f46214bc4a5dd35e796969e53\n",
|
||
" experiment_tag: 10_activation=tanh,height=52.7613,steps=100,width=16.7268\n",
|
||
" hostname: Kais-MacBook-Pro.local\n",
|
||
" iterations: 99\n",
|
||
" iterations_since_restore: 100\n",
|
||
" mean_loss: 5.336159871047403\n",
|
||
" neg_mean_loss: -5.336159871047403\n",
|
||
" node_ip: 127.0.0.1\n",
|
||
" pid: 47302\n",
|
||
" time_since_restore: 10.633639097213745\n",
|
||
" time_this_iter_s: 0.1067359447479248\n",
|
||
" time_total_s: 10.633639097213745\n",
|
||
" timestamp: 1658500353\n",
|
||
" timesteps_since_restore: 0\n",
|
||
" training_iteration: 100\n",
|
||
" trial_id: '18090732'\n",
|
||
" warmup_time: 0.0028650760650634766\n",
|
||
" \n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"tuner = tune.Tuner(\n",
|
||
" objective_two,\n",
|
||
" tune_config=tune.TuneConfig(\n",
|
||
" metric=\"mean_loss\",\n",
|
||
" mode=\"min\",\n",
|
||
" search_alg=algo,\n",
|
||
" num_samples=num_samples,\n",
|
||
" ),\n",
|
||
")\n",
|
||
"results = tuner.fit()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "e6172afa",
|
||
"metadata": {},
|
||
"source": [
|
||
"Finally, we again show the hyperparameters that minimize the mean loss defined by the score of the objective function above. "
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 17,
|
||
"id": "03c3fc49",
|
||
"metadata": {
|
||
"vscode": {
|
||
"languageId": "python"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Best hyperparameters found were: {'activation': {'activation': 'relu', 'mult': 1.3982639549501585}, 'height': -66.3136247260571, 'steps': 100, 'width': 13.922128223483856}\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"print(\"Best hyperparameters found were: \", results.get_best_result().config)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 18,
|
||
"id": "2f7b72d3",
|
||
"metadata": {
|
||
"tags": [
|
||
"remove-cell"
|
||
],
|
||
"vscode": {
|
||
"languageId": "python"
|
||
}
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"ray.shutdown()"
|
||
]
|
||
}
|
||
],
|
||
"metadata": {
|
||
"kernelspec": {
|
||
"display_name": "Python 3 (ipykernel)",
|
||
"language": "python",
|
||
"name": "python3"
|
||
},
|
||
"language_info": {
|
||
"codemirror_mode": {
|
||
"name": "ipython",
|
||
"version": 3
|
||
},
|
||
"file_extension": ".py",
|
||
"mimetype": "text/x-python",
|
||
"name": "python",
|
||
"nbconvert_exporter": "python",
|
||
"pygments_lexer": "ipython3",
|
||
"version": "3.7.7"
|
||
},
|
||
"orphan": true
|
||
},
|
||
"nbformat": 4,
|
||
"nbformat_minor": 5
|
||
}
|