{ "cells": [ { "cell_type": "markdown", "id": "a587ce4e", "metadata": {}, "source": [ "# Running Tune experiments with Optuna\n", "\n", "In this tutorial we introduce Optuna, while running a simple Ray Tune experiment. Tune’s Search Algorithms integrate with Optuna and, as a result, allow you to seamlessly scale up a Optuna optimization process - without sacrificing performance.\n", "\n", "Similar to Ray Tune, Optuna is an automatic hyperparameter optimization software framework, particularly designed for machine learning. It features an imperative (\"how\" over \"what\" emphasis), define-by-run style user API. With Optuna, a user has the ability to dynamically construct the search spaces for the hyperparameters. Optuna falls in the domain of \"derivative-free optimization\" and \"black-box optimization\".\n", "\n", "In this example we minimize a simple objective to briefly demonstrate the usage of Optuna with Ray Tune via `OptunaSearch`, including examples of conditional search spaces (string together relationships between hyperparameters), and the multi-objective problem (measure trade-offs among all important metrics). It's useful to keep in mind that despite the emphasis on machine learning experiments, Ray Tune optimizes any implicit or explicit objective. Here we assume `optuna==2.9.1` library is installed. To learn more, please refer to [Optuna website](https://optuna.org/).\n", "\n", "Please note that sophisticated schedulers, such as `AsyncHyperBandScheduler`, may not work correctly with multi-objective optimization, since they typically expect a scalar score to compare fitness among trials.\n" ] }, { "cell_type": "code", "execution_count": 1, "id": "aeaf9ff0", "metadata": { "tags": [ "remove-cell" ] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: optuna==2.9.1 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (2.9.1)\n", "Requirement already satisfied: alembic in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from optuna==2.9.1) (1.7.6)\n", "Requirement already satisfied: colorlog in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from optuna==2.9.1) (6.6.0)\n", "Requirement already satisfied: PyYAML in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from optuna==2.9.1) (5.4.1)\n", "Requirement already satisfied: scipy!=1.4.0 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from optuna==2.9.1) (1.4.1)\n", "Requirement already satisfied: tqdm in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from optuna==2.9.1) (4.64.0)\n", "Requirement already satisfied: numpy in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from optuna==2.9.1) (1.21.6)\n", "Requirement already satisfied: cliff in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from optuna==2.9.1) (3.10.1)\n", "Requirement already satisfied: sqlalchemy>=1.1.0 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from optuna==2.9.1) (1.4.32)\n", "Requirement already satisfied: packaging>=20.0 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from optuna==2.9.1) (21.3)\n", "Requirement already satisfied: cmaes>=0.8.2 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from optuna==2.9.1) (0.8.2)\n", "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from packaging>=20.0->optuna==2.9.1) (2.4.7)\n", "Requirement already satisfied: importlib-metadata in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from sqlalchemy>=1.1.0->optuna==2.9.1) (4.10.1)\n", "Requirement already satisfied: greenlet!=0.4.17 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from sqlalchemy>=1.1.0->optuna==2.9.1) (1.1.2)\n", "Requirement already satisfied: importlib-resources in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from alembic->optuna==2.9.1) (5.4.0)\n", "Requirement already satisfied: Mako in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from alembic->optuna==2.9.1) (1.1.6)\n", "Requirement already satisfied: autopage>=0.4.0 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from cliff->optuna==2.9.1) (0.5.0)\n", "Requirement already satisfied: cmd2>=1.0.0 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from cliff->optuna==2.9.1) (2.4.0)\n", "Requirement already satisfied: PrettyTable>=0.7.2 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from cliff->optuna==2.9.1) (3.2.0)\n", "Requirement already satisfied: stevedore>=2.0.1 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from cliff->optuna==2.9.1) (3.5.0)\n", "Requirement already satisfied: pbr!=2.1.0,>=2.0.0 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from cliff->optuna==2.9.1) (5.8.1)\n", "Requirement already satisfied: typing-extensions in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from cmd2>=1.0.0->cliff->optuna==2.9.1) (4.1.1)\n", "Requirement already satisfied: attrs>=16.3.0 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from cmd2>=1.0.0->cliff->optuna==2.9.1) (21.2.0)\n", "Requirement already satisfied: pyperclip>=1.6 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from cmd2>=1.0.0->cliff->optuna==2.9.1) (1.8.2)\n", "Requirement already satisfied: wcwidth>=0.1.7 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from cmd2>=1.0.0->cliff->optuna==2.9.1) (0.2.5)\n", "Requirement already satisfied: zipp>=0.5 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from importlib-metadata->sqlalchemy>=1.1.0->optuna==2.9.1) (3.7.0)\n", "Requirement already satisfied: MarkupSafe>=0.9.2 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from Mako->alembic->optuna==2.9.1) (2.0.1)\n", "\u001b[33mWARNING: There was an error checking the latest version of pip.\u001b[0m\u001b[33m\n", "\u001b[0m" ] } ], "source": [ "# !pip install ray[tune]\n", "!pip install optuna==2.9.1" ] }, { "cell_type": "markdown", "id": "467466a3", "metadata": {}, "source": [ "Click below to see all the imports we need for this example.\n", "You can also launch directly into a Binder instance to run this notebook yourself.\n", "Just click on the rocket symbol at the top of the navigation." ] }, { "cell_type": "code", "execution_count": 2, "id": "fb1ad624", "metadata": { "tags": [ "hide-input" ] }, "outputs": [], "source": [ "import time\n", "from typing import Dict, Optional, Any\n", "\n", "import ray\n", "from ray import tune\n", "from ray.air import session\n", "from ray.tune.search import ConcurrencyLimiter\n", "from ray.tune.search.optuna import OptunaSearch" ] }, { "cell_type": "code", "execution_count": 3, "id": "e64f0b44", "metadata": { "tags": [ "remove-cell" ] }, "outputs": [ { "data": { "text/html": [ "
\n", "
\n", "

Ray

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", "\n", "
Python version:3.7.7
Ray version: 2.0.0rc0
Dashboard:http://127.0.0.1:8265
\n", "
\n", "
\n" ], "text/plain": [ "RayContext(dashboard_url='127.0.0.1:8265', python_version='3.7.7', ray_version='2.0.0rc0', ray_commit='{{RAY_COMMIT_SHA}}', address_info={'node_ip_address': '127.0.0.1', 'raylet_ip_address': '127.0.0.1', 'redis_address': None, 'object_store_address': '/tmp/ray/session_2022-07-22_15-21-42_643152_46061/sockets/plasma_store', 'raylet_socket_name': '/tmp/ray/session_2022-07-22_15-21-42_643152_46061/sockets/raylet', 'webui_url': '127.0.0.1:8265', 'session_dir': '/tmp/ray/session_2022-07-22_15-21-42_643152_46061', 'metrics_export_port': 60351, 'gcs_address': '127.0.0.1:65028', 'address': '127.0.0.1:65028', 'dashboard_agent_listen_port': 52365, 'node_id': 'b7abfa8f9f137b294759a2e68a71edb9920c011a97e390fd1e0b71bf'})" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "ray.init(configure_logging=False)" ] }, { "cell_type": "markdown", "id": "56b0c685", "metadata": {}, "source": [ "Let's start by defining a simple evaluation function.\n", "An explicit math formula is queried here for demonstration, yet in practice this is typically a black-box function-- e.g. the performance results after training an ML model.\n", "We artificially sleep for a bit (`0.1` seconds) to simulate a long-running ML experiment.\n", "This setup assumes that we're running multiple `step`s of an experiment while tuning three hyperparameters,\n", "namely `width`, `height`, and `activation`." ] }, { "cell_type": "code", "execution_count": 4, "id": "90a11f98", "metadata": {}, "outputs": [], "source": [ "def evaluate(step, width, height, activation):\n", " time.sleep(0.1)\n", " activation_boost = 10 if activation==\"relu\" else 0\n", " return (0.1 + width * step / 100) ** (-1) + height * 0.1 + activation_boost" ] }, { "cell_type": "markdown", "id": "bc579b83", "metadata": {}, "source": [ "Next, our `objective` function to be optimized takes a Tune `config`, evaluates the `score` of your experiment in a training loop,\n", "and uses `session.report` to report the `score` back to Tune." ] }, { "cell_type": "code", "execution_count": 5, "id": "3a11d0e0", "metadata": { "lines_to_next_cell": 0 }, "outputs": [], "source": [ "def objective(config):\n", " for step in range(config[\"steps\"]):\n", " score = evaluate(step, config[\"width\"], config[\"height\"], config[\"activation\"])\n", " session.report({\"iterations\": step, \"mean_loss\": score})\n", " " ] }, { "cell_type": "markdown", "id": "c58bd20b", "metadata": {}, "source": [ "Next we define a search space. The critical assumption is that the optimal hyperparamters live within this space. Yet, if the space is very large, then those hyperparamters may be difficult to find in a short amount of time.\n", "\n", "The simplest case is a search space with independent dimensions. In this case, a config dictionary will suffice." ] }, { "cell_type": "code", "execution_count": 6, "id": "c3e4eecb", "metadata": {}, "outputs": [], "source": [ "search_space = {\n", " \"steps\": 100,\n", " \"width\": tune.uniform(0, 20),\n", " \"height\": tune.uniform(-100, 100),\n", " \"activation\": tune.choice([\"relu\", \"tanh\"]),\n", "}" ] }, { "cell_type": "markdown", "id": "ef0c666d", "metadata": {}, "source": [ "Here we define the Optuna search algorithm:" ] }, { "cell_type": "code", "execution_count": 7, "id": "f23cadc8", "metadata": {}, "outputs": [], "source": [ "algo = OptunaSearch()" ] }, { "cell_type": "markdown", "id": "4287fa79", "metadata": {}, "source": [ "We also constrain the the number of concurrent trials to `4` with a `ConcurrencyLimiter`." ] }, { "cell_type": "code", "execution_count": 8, "id": "68022ea4", "metadata": {}, "outputs": [], "source": [ "algo = ConcurrencyLimiter(algo, max_concurrent=4)\n" ] }, { "cell_type": "markdown", "id": "4c250f74", "metadata": {}, "source": [ "The number of samples is the number of hyperparameter combinations that will be tried out. This Tune run is set to `1000` samples.\n", "(you can decrease this if it takes too long on your machine)." ] }, { "cell_type": "code", "execution_count": 9, "id": "f6c21314", "metadata": {}, "outputs": [], "source": [ "num_samples = 1000" ] }, { "cell_type": "code", "execution_count": 10, "id": "9533aabf", "metadata": { "tags": [ "remove-cell" ] }, "outputs": [], "source": [ "# We override here for our smoke tests.\n", "num_samples = 10" ] }, { "cell_type": "markdown", "id": "92942b88", "metadata": {}, "source": [ "Finally, we run the experiment to `\"min\"`imize the \"mean_loss\" of the `objective` by searching `search_space` via `algo`, `num_samples` times. This previous sentence is fully characterizes the search problem we aim to solve. With this in mind, notice how efficient it is to execute `tuner.fit()`." ] }, { "cell_type": "code", "execution_count": 11, "id": "4e224bb2", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Function checkpointing is disabled. This may result in unexpected behavior when using checkpointing features or certain schedulers. To enable, set the train function arguments to be `func(config, checkpoint_dir=None)`.\n", "\u001b[32m[I 2022-07-22 15:21:47,769]\u001b[0m A new study created in memory with name: optuna\u001b[0m\n" ] }, { "data": { "text/html": [ "== Status ==
Current time: 2022-07-22 15:22:32 (running for 00:00:43.89)
Memory usage on this node: 10.1/16.0 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/5.07 GiB heap, 0.0/2.0 GiB objects
Current best trial: b1bad2fe with mean_loss=-1.5166144695235813 and parameters={'steps': 100, 'width': 7.688465886501621, 'height': -16.46289560854555, 'activation': 'tanh'}
Result logdir: /Users/kai/ray_results/optuna_example
Number of trials: 10/10 (10 TERMINATED)
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
Trial name status loc activation height width loss iter total time (s) iterations neg_mean_loss
objective_9f689668TERMINATED127.0.0.1:46141tanh 32.9806 16.0406 3.36064 100 11.0409 99 -3.36064
objective_a11d2104TERMINATED127.0.0.1:46149relu 72.2627 0.76388118.3942 100 11.7018 99 -18.3942
objective_a11eef8eTERMINATED127.0.0.1:46150relu -32.8474 13.1144 6.79169 100 11.8071 99 -6.79169
objective_a1209e56TERMINATED127.0.0.1:46151tanh 75.6408 10.4415 7.65989 100 11.6987 99 -7.65989
objective_a7b383e6TERMINATED127.0.0.1:46174relu 50.1501 3.4612 15.2986 100 10.7015 99 -15.2986
objective_a9c844c8TERMINATED127.0.0.1:46183relu -40.3931 14.0525 6.03205 100 10.6687 99 -6.03205
objective_a9cb308eTERMINATED127.0.0.1:46184tanh 48.4802 12.7746 4.92647 100 10.7122 99 -4.92647
objective_a9d8332eTERMINATED127.0.0.1:46189tanh -1.2668210.9788 -0.0355157 100 10.7089 99 0.0355157
objective_af948a1aTERMINATED127.0.0.1:46204tanh 61.6381 11.0105 6.25472 100 10.744 99 -6.25472
objective_b1bad2feTERMINATED127.0.0.1:46209tanh -16.4629 7.68847 -1.51661 100 10.7274 99 1.51661


" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "Result for objective_9f689668:\n", " date: 2022-07-22_15-21-51\n", " done: false\n", " experiment_id: 7d29e5c95aa44d77becc24b23a326b9b\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 13.298063134872326\n", " neg_mean_loss: -13.298063134872326\n", " node_ip: 127.0.0.1\n", " pid: 46141\n", " time_since_restore: 0.10468196868896484\n", " time_this_iter_s: 0.10468196868896484\n", " time_total_s: 0.10468196868896484\n", " timestamp: 1658499711\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 9f689668\n", " warmup_time: 0.003930091857910156\n", " \n", "Result for objective_a1209e56:\n", " date: 2022-07-22_15-21-54\n", " done: false\n", " experiment_id: 70d9788c327c43fb94450e5084416b81\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 17.564083106485064\n", " neg_mean_loss: -17.564083106485064\n", " node_ip: 127.0.0.1\n", " pid: 46151\n", " time_since_restore: 0.10416483879089355\n", " time_this_iter_s: 0.10416483879089355\n", " time_total_s: 0.10416483879089355\n", " timestamp: 1658499714\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: a1209e56\n", " warmup_time: 0.0026938915252685547\n", " \n", "Result for objective_a11d2104:\n", " date: 2022-07-22_15-21-54\n", " done: false\n", " experiment_id: af2598364a004cd28f37ede19afd1ffb\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 27.2262748487915\n", " neg_mean_loss: -27.2262748487915\n", " node_ip: 127.0.0.1\n", " pid: 46149\n", " time_since_restore: 0.10507392883300781\n", " time_this_iter_s: 0.10507392883300781\n", " time_total_s: 0.10507392883300781\n", " timestamp: 1658499714\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: a11d2104\n", " warmup_time: 0.0031232833862304688\n", " \n", "Result for objective_a11eef8e:\n", " date: 2022-07-22_15-21-54\n", " done: false\n", " experiment_id: e4a5922603d44b459c23607a42d5e574\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 16.71525618736072\n", " neg_mean_loss: -16.71525618736072\n", " node_ip: 127.0.0.1\n", " pid: 46150\n", " time_since_restore: 0.10363006591796875\n", " time_this_iter_s: 0.10363006591796875\n", " time_total_s: 0.10363006591796875\n", " timestamp: 1658499714\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: a11eef8e\n", " warmup_time: 0.0028891563415527344\n", " \n", "Result for objective_9f689668:\n", " date: 2022-07-22_15-21-56\n", " done: false\n", " experiment_id: 7d29e5c95aa44d77becc24b23a326b9b\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 45\n", " iterations_since_restore: 46\n", " mean_loss: 3.434707064844201\n", " neg_mean_loss: -3.434707064844201\n", " node_ip: 127.0.0.1\n", " pid: 46141\n", " time_since_restore: 5.202661752700806\n", " time_this_iter_s: 0.10527586936950684\n", " time_total_s: 5.202661752700806\n", " timestamp: 1658499716\n", " timesteps_since_restore: 0\n", " training_iteration: 46\n", " trial_id: 9f689668\n", " warmup_time: 0.003930091857910156\n", " \n", "Result for objective_a1209e56:\n", " date: 2022-07-22_15-21-59\n", " done: false\n", " experiment_id: 70d9788c327c43fb94450e5084416b81\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: 7.763782433167051\n", " neg_mean_loss: -7.763782433167051\n", " node_ip: 127.0.0.1\n", " pid: 46151\n", " time_since_restore: 5.146381139755249\n", " time_this_iter_s: 0.10691022872924805\n", " time_total_s: 5.146381139755249\n", " timestamp: 1658499719\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: a1209e56\n", " warmup_time: 0.0026938915252685547\n", " \n", "Result for objective_a11eef8e:\n", " date: 2022-07-22_15-21-59\n", " done: false\n", " experiment_id: e4a5922603d44b459c23607a42d5e574\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: 6.874905099790281\n", " neg_mean_loss: -6.874905099790281\n", " node_ip: 127.0.0.1\n", " pid: 46150\n", " time_since_restore: 5.1618921756744385\n", " time_this_iter_s: 0.10776615142822266\n", " time_total_s: 5.1618921756744385\n", " timestamp: 1658499719\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: a11eef8e\n", " warmup_time: 0.0028891563415527344\n", " \n", "Result for objective_a11d2104:\n", " date: 2022-07-22_15-21-59\n", " done: false\n", " experiment_id: af2598364a004cd28f37ede19afd1ffb\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: 19.4048102011043\n", " neg_mean_loss: -19.4048102011043\n", " node_ip: 127.0.0.1\n", " pid: 46149\n", " time_since_restore: 5.170748233795166\n", " time_this_iter_s: 0.11556220054626465\n", " time_total_s: 5.170748233795166\n", " timestamp: 1658499719\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: a11d2104\n", " warmup_time: 0.0031232833862304688\n", " \n", "Result for objective_9f689668:\n", " date: 2022-07-22_15-22-01\n", " done: false\n", " experiment_id: 7d29e5c95aa44d77becc24b23a326b9b\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 92\n", " iterations_since_restore: 93\n", " mean_loss: 3.3653696881211697\n", " neg_mean_loss: -3.3653696881211697\n", " node_ip: 127.0.0.1\n", " pid: 46141\n", " time_since_restore: 10.248206853866577\n", " time_this_iter_s: 0.10745501518249512\n", " time_total_s: 10.248206853866577\n", " timestamp: 1658499721\n", " timesteps_since_restore: 0\n", " training_iteration: 93\n", " trial_id: 9f689668\n", " warmup_time: 0.003930091857910156\n", " \n", "Result for objective_9f689668:\n", " date: 2022-07-22_15-22-02\n", " done: true\n", " experiment_id: 7d29e5c95aa44d77becc24b23a326b9b\n", " experiment_tag: 1_activation=tanh,height=32.9806,steps=100,width=16.0406\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 3.3606404197394255\n", " neg_mean_loss: -3.3606404197394255\n", " node_ip: 127.0.0.1\n", " pid: 46141\n", " time_since_restore: 11.04089903831482\n", " time_this_iter_s: 0.10824704170227051\n", " time_total_s: 11.04089903831482\n", " timestamp: 1658499722\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 9f689668\n", " warmup_time: 0.003930091857910156\n", " \n", "Result for objective_a1209e56:\n", " date: 2022-07-22_15-22-04\n", " done: false\n", " experiment_id: 70d9788c327c43fb94450e5084416b81\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 90\n", " iterations_since_restore: 91\n", " mean_loss: 7.669375143981996\n", " neg_mean_loss: -7.669375143981996\n", " node_ip: 127.0.0.1\n", " pid: 46151\n", " time_since_restore: 9.724387168884277\n", " time_this_iter_s: 0.10670638084411621\n", " time_total_s: 9.724387168884277\n", " timestamp: 1658499724\n", " timesteps_since_restore: 0\n", " training_iteration: 91\n", " trial_id: a1209e56\n", " warmup_time: 0.0026938915252685547\n", " \n", "Result for objective_a11d2104:\n", " date: 2022-07-22_15-22-04\n", " done: false\n", " experiment_id: af2598364a004cd28f37ede19afd1ffb\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 90\n", " iterations_since_restore: 91\n", " mean_loss: 18.49612779999574\n", " neg_mean_loss: -18.49612779999574\n", " node_ip: 127.0.0.1\n", " pid: 46149\n", " time_since_restore: 9.818917274475098\n", " time_this_iter_s: 0.10725140571594238\n", " time_total_s: 9.818917274475098\n", " timestamp: 1658499724\n", " timesteps_since_restore: 0\n", " training_iteration: 91\n", " trial_id: a11d2104\n", " warmup_time: 0.0031232833862304688\n", " \n", "Result for objective_a11eef8e:\n", " date: 2022-07-22_15-22-03\n", " done: false\n", " experiment_id: e4a5922603d44b459c23607a42d5e574\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 89\n", " iterations_since_restore: 90\n", " mean_loss: 6.800205168670805\n", " neg_mean_loss: -6.800205168670805\n", " node_ip: 127.0.0.1\n", " pid: 46150\n", " time_since_restore: 9.714852094650269\n", " time_this_iter_s: 0.10286402702331543\n", " time_total_s: 9.714852094650269\n", " timestamp: 1658499723\n", " timesteps_since_restore: 0\n", " training_iteration: 90\n", " trial_id: a11eef8e\n", " warmup_time: 0.0028891563415527344\n", " \n", "Result for objective_a7b383e6:\n", " date: 2022-07-22_15-22-05\n", " done: false\n", " experiment_id: d2b2d4c7590142b3bdd660dcf6c947aa\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 25.015010793208496\n", " neg_mean_loss: -25.015010793208496\n", " node_ip: 127.0.0.1\n", " pid: 46174\n", " time_since_restore: 0.10384607315063477\n", " time_this_iter_s: 0.10384607315063477\n", " time_total_s: 0.10384607315063477\n", " timestamp: 1658499725\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: a7b383e6\n", " warmup_time: 0.002817869186401367\n", " \n", "Result for objective_a1209e56:\n", " date: 2022-07-22_15-22-05\n", " done: true\n", " experiment_id: 70d9788c327c43fb94450e5084416b81\n", " experiment_tag: 4_activation=tanh,height=75.6408,steps=100,width=10.4415\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 7.659894851608276\n", " neg_mean_loss: -7.659894851608276\n", " node_ip: 127.0.0.1\n", " pid: 46151\n", " time_since_restore: 11.698678970336914\n", " time_this_iter_s: 0.10874581336975098\n", " time_total_s: 11.698678970336914\n", " timestamp: 1658499725\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: a1209e56\n", " warmup_time: 0.0026938915252685547\n", " \n", "Result for objective_a11d2104:\n", " date: 2022-07-22_15-22-05\n", " done: true\n", " experiment_id: af2598364a004cd28f37ede19afd1ffb\n", " experiment_tag: 2_activation=relu,height=72.2627,steps=100,width=0.7639\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 18.394168929385685\n", " neg_mean_loss: -18.394168929385685\n", " node_ip: 127.0.0.1\n", " pid: 46149\n", " time_since_restore: 11.701792001724243\n", " time_this_iter_s: 0.10734391212463379\n", " time_total_s: 11.701792001724243\n", " timestamp: 1658499725\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: a11d2104\n", " warmup_time: 0.0031232833862304688\n", " \n", "Result for objective_a11eef8e:\n", " date: 2022-07-22_15-22-06\n", " done: true\n", " experiment_id: e4a5922603d44b459c23607a42d5e574\n", " experiment_tag: 3_activation=relu,height=-32.8474,steps=100,width=13.1144\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 6.791690049131352\n", " neg_mean_loss: -6.791690049131352\n", " node_ip: 127.0.0.1\n", " pid: 46150\n", " time_since_restore: 11.807135105133057\n", " time_this_iter_s: 0.10672497749328613\n", " time_total_s: 11.807135105133057\n", " timestamp: 1658499726\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: a11eef8e\n", " warmup_time: 0.0028891563415527344\n", " \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Result for objective_a9cb308e:\n", " date: 2022-07-22_15-22-08\n", " done: false\n", " experiment_id: fcc8b8f4880c471c8e8545228d7e591a\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 14.84802392036395\n", " neg_mean_loss: -14.84802392036395\n", " node_ip: 127.0.0.1\n", " pid: 46184\n", " time_since_restore: 0.1039888858795166\n", " time_this_iter_s: 0.1039888858795166\n", " time_total_s: 0.1039888858795166\n", " timestamp: 1658499728\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: a9cb308e\n", " warmup_time: 0.0027310848236083984\n", " \n", "Result for objective_a9c844c8:\n", " date: 2022-07-22_15-22-08\n", " done: false\n", " experiment_id: 3ef3f95b10fb4a65a21c80130f69ed3e\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 15.960686077942514\n", " neg_mean_loss: -15.960686077942514\n", " node_ip: 127.0.0.1\n", " pid: 46183\n", " time_since_restore: 0.1004641056060791\n", " time_this_iter_s: 0.1004641056060791\n", " time_total_s: 0.1004641056060791\n", " timestamp: 1658499728\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: a9c844c8\n", " warmup_time: 0.002599954605102539\n", " \n", "Result for objective_a9d8332e:\n", " date: 2022-07-22_15-22-08\n", " done: false\n", " experiment_id: 9c2f8aa7e4ed45c8867025673b146195\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 9.873318211243287\n", " neg_mean_loss: -9.873318211243287\n", " node_ip: 127.0.0.1\n", " pid: 46189\n", " time_since_restore: 0.10463905334472656\n", " time_this_iter_s: 0.10463905334472656\n", " time_total_s: 0.10463905334472656\n", " timestamp: 1658499728\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: a9d8332e\n", " warmup_time: 0.002753019332885742\n", " \n", "Result for objective_a7b383e6:\n", " date: 2022-07-22_15-22-10\n", " done: false\n", " experiment_id: d2b2d4c7590142b3bdd660dcf6c947aa\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: 15.594128340598196\n", " neg_mean_loss: -15.594128340598196\n", " node_ip: 127.0.0.1\n", " pid: 46174\n", " time_since_restore: 5.12034797668457\n", " time_this_iter_s: 0.10577988624572754\n", " time_total_s: 5.12034797668457\n", " timestamp: 1658499730\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: a7b383e6\n", " warmup_time: 0.002817869186401367\n", " \n", "Result for objective_a9cb308e:\n", " date: 2022-07-22_15-22-13\n", " done: false\n", " experiment_id: fcc8b8f4880c471c8e8545228d7e591a\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: 5.011849709111575\n", " neg_mean_loss: -5.011849709111575\n", " node_ip: 127.0.0.1\n", " pid: 46184\n", " time_since_restore: 5.150436162948608\n", " time_this_iter_s: 0.10827016830444336\n", " time_total_s: 5.150436162948608\n", " timestamp: 1658499733\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: a9cb308e\n", " warmup_time: 0.0027310848236083984\n", " \n", "Result for objective_a9c844c8:\n", " date: 2022-07-22_15-22-13\n", " done: false\n", " experiment_id: 3ef3f95b10fb4a65a21c80130f69ed3e\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: 6.109835222238134\n", " neg_mean_loss: -6.109835222238134\n", " node_ip: 127.0.0.1\n", " pid: 46183\n", " time_since_restore: 5.136260032653809\n", " time_this_iter_s: 0.10730218887329102\n", " time_total_s: 5.136260032653809\n", " timestamp: 1658499733\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: a9c844c8\n", " warmup_time: 0.002599954605102539\n", " \n", "Result for objective_a9d8332e:\n", " date: 2022-07-22_15-22-13\n", " done: false\n", " experiment_id: 9c2f8aa7e4ed45c8867025673b146195\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: 0.06343129971586822\n", " neg_mean_loss: -0.06343129971586822\n", " node_ip: 127.0.0.1\n", " pid: 46189\n", " time_since_restore: 5.141700983047485\n", " time_this_iter_s: 0.1074991226196289\n", " time_total_s: 5.141700983047485\n", " timestamp: 1658499733\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: a9d8332e\n", " warmup_time: 0.002753019332885742\n", " \n", "Result for objective_a7b383e6:\n", " date: 2022-07-22_15-22-15\n", " done: false\n", " experiment_id: d2b2d4c7590142b3bdd660dcf6c947aa\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " mean_loss: 15.313204013214245\n", " neg_mean_loss: -15.313204013214245\n", " node_ip: 127.0.0.1\n", " pid: 46174\n", " time_since_restore: 10.163710832595825\n", " time_this_iter_s: 0.10754895210266113\n", " time_total_s: 10.163710832595825\n", " timestamp: 1658499735\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: a7b383e6\n", " warmup_time: 0.002817869186401367\n", " \n", "Result for objective_a7b383e6:\n", " date: 2022-07-22_15-22-15\n", " done: true\n", " experiment_id: d2b2d4c7590142b3bdd660dcf6c947aa\n", " experiment_tag: 5_activation=relu,height=50.1501,steps=100,width=3.4612\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 15.298570798421121\n", " neg_mean_loss: -15.298570798421121\n", " node_ip: 127.0.0.1\n", " pid: 46174\n", " time_since_restore: 10.701509952545166\n", " time_this_iter_s: 0.10684585571289062\n", " time_total_s: 10.701509952545166\n", " timestamp: 1658499735\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: a7b383e6\n", " warmup_time: 0.002817869186401367\n", " \n", "Result for objective_af948a1a:\n", " date: 2022-07-22_15-22-18\n", " done: false\n", " experiment_id: d27f58d6527a4d47ae4c12496f5275cf\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 16.16381214127525\n", " neg_mean_loss: -16.16381214127525\n", " node_ip: 127.0.0.1\n", " pid: 46204\n", " time_since_restore: 0.10407209396362305\n", " time_this_iter_s: 0.10407209396362305\n", " time_total_s: 0.10407209396362305\n", " timestamp: 1658499738\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: af948a1a\n", " warmup_time: 0.002669095993041992\n", " \n", "Result for objective_a9cb308e:\n", " date: 2022-07-22_15-22-18\n", " done: false\n", " experiment_id: fcc8b8f4880c471c8e8545228d7e591a\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " mean_loss: 4.930613328484136\n", " neg_mean_loss: -4.930613328484136\n", " node_ip: 127.0.0.1\n", " pid: 46184\n", " time_since_restore: 10.175731897354126\n", " time_this_iter_s: 0.10618400573730469\n", " time_total_s: 10.175731897354126\n", " timestamp: 1658499738\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: a9cb308e\n", " warmup_time: 0.0027310848236083984\n", " \n", "Result for objective_a9d8332e:\n", " date: 2022-07-22_15-22-18\n", " done: false\n", " experiment_id: 9c2f8aa7e4ed45c8867025673b146195\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " mean_loss: -0.030712998363490673\n", " neg_mean_loss: 0.030712998363490673\n", " node_ip: 127.0.0.1\n", " pid: 46189\n", " time_since_restore: 10.171820163726807\n", " time_this_iter_s: 0.10807609558105469\n", " time_total_s: 10.171820163726807\n", " timestamp: 1658499738\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: a9d8332e\n", " warmup_time: 0.002753019332885742\n", " \n", "Result for objective_a9c844c8:\n", " date: 2022-07-22_15-22-18\n", " done: false\n", " experiment_id: 3ef3f95b10fb4a65a21c80130f69ed3e\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 95\n", " iterations_since_restore: 96\n", " mean_loss: 6.035035952030598\n", " neg_mean_loss: -6.035035952030598\n", " node_ip: 127.0.0.1\n", " pid: 46183\n", " time_since_restore: 10.23954701423645\n", " time_this_iter_s: 0.10863399505615234\n", " time_total_s: 10.23954701423645\n", " timestamp: 1658499738\n", " timesteps_since_restore: 0\n", " training_iteration: 96\n", " trial_id: a9c844c8\n", " warmup_time: 0.002599954605102539\n", " \n", "Result for objective_a9cb308e:\n", " date: 2022-07-22_15-22-19\n", " done: true\n", " experiment_id: fcc8b8f4880c471c8e8545228d7e591a\n", " experiment_tag: 7_activation=tanh,height=48.4802,steps=100,width=12.7746\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 4.926474869576359\n", " neg_mean_loss: -4.926474869576359\n", " node_ip: 127.0.0.1\n", " pid: 46184\n", " time_since_restore: 10.712197065353394\n", " time_this_iter_s: 0.10726809501647949\n", " time_total_s: 10.712197065353394\n", " timestamp: 1658499739\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: a9cb308e\n", " warmup_time: 0.0027310848236083984\n", " \n", "Result for objective_a9c844c8:\n", " date: 2022-07-22_15-22-19\n", " done: true\n", " experiment_id: 3ef3f95b10fb4a65a21c80130f69ed3e\n", " experiment_tag: 6_activation=relu,height=-40.3931,steps=100,width=14.0525\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 6.03205335569619\n", " neg_mean_loss: -6.03205335569619\n", " node_ip: 127.0.0.1\n", " pid: 46183\n", " time_since_restore: 10.668684959411621\n", " time_this_iter_s: 0.1074681282043457\n", " time_total_s: 10.668684959411621\n", " timestamp: 1658499739\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: a9c844c8\n", " warmup_time: 0.002599954605102539\n", " \n", "Result for objective_a9d8332e:\n", " date: 2022-07-22_15-22-19\n", " done: true\n", " experiment_id: 9c2f8aa7e4ed45c8867025673b146195\n", " experiment_tag: 8_activation=tanh,height=-1.2668,steps=100,width=10.9788\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: -0.035515719608698634\n", " neg_mean_loss: 0.035515719608698634\n", " node_ip: 127.0.0.1\n", " pid: 46189\n", " time_since_restore: 10.708936929702759\n", " time_this_iter_s: 0.10948991775512695\n", " time_total_s: 10.708936929702759\n", " timestamp: 1658499739\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: a9d8332e\n", " warmup_time: 0.002753019332885742\n", " \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Result for objective_b1bad2fe:\n", " date: 2022-07-22_15-22-21\n", " done: false\n", " experiment_id: f58c8e48e2324dd69c28f526bf96497c\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 8.353710439145445\n", " neg_mean_loss: -8.353710439145445\n", " node_ip: 127.0.0.1\n", " pid: 46209\n", " time_since_restore: 0.10369992256164551\n", " time_this_iter_s: 0.10369992256164551\n", " time_total_s: 0.10369992256164551\n", " timestamp: 1658499741\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: b1bad2fe\n", " warmup_time: 0.0026597976684570312\n", " \n", "Result for objective_af948a1a:\n", " date: 2022-07-22_15-22-23\n", " done: false\n", " experiment_id: d27f58d6527a4d47ae4c12496f5275cf\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: 6.353387644558537\n", " neg_mean_loss: -6.353387644558537\n", " node_ip: 127.0.0.1\n", " pid: 46204\n", " time_since_restore: 5.161527872085571\n", " time_this_iter_s: 0.10784792900085449\n", " time_total_s: 5.161527872085571\n", " timestamp: 1658499743\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: af948a1a\n", " warmup_time: 0.002669095993041992\n", " \n", "Result for objective_b1bad2fe:\n", " date: 2022-07-22_15-22-26\n", " done: false\n", " experiment_id: f58c8e48e2324dd69c28f526bf96497c\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: -1.3770075531258914\n", " neg_mean_loss: 1.3770075531258914\n", " node_ip: 127.0.0.1\n", " pid: 46209\n", " time_since_restore: 5.158898115158081\n", " time_this_iter_s: 0.10701417922973633\n", " time_total_s: 5.158898115158081\n", " timestamp: 1658499746\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: b1bad2fe\n", " warmup_time: 0.0026597976684570312\n", " \n", "Result for objective_af948a1a:\n", " date: 2022-07-22_15-22-28\n", " done: false\n", " experiment_id: d27f58d6527a4d47ae4c12496f5275cf\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " mean_loss: 6.259506962612728\n", " neg_mean_loss: -6.259506962612728\n", " node_ip: 127.0.0.1\n", " pid: 46204\n", " time_since_restore: 10.20361876487732\n", " time_this_iter_s: 0.10824179649353027\n", " time_total_s: 10.20361876487732\n", " timestamp: 1658499748\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: af948a1a\n", " warmup_time: 0.002669095993041992\n", " \n", "Result for objective_af948a1a:\n", " date: 2022-07-22_15-22-28\n", " done: true\n", " experiment_id: d27f58d6527a4d47ae4c12496f5275cf\n", " experiment_tag: 9_activation=tanh,height=61.6381,steps=100,width=11.0105\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 6.254717826198094\n", " neg_mean_loss: -6.254717826198094\n", " node_ip: 127.0.0.1\n", " pid: 46204\n", " time_since_restore: 10.744001865386963\n", " time_this_iter_s: 0.10536718368530273\n", " time_total_s: 10.744001865386963\n", " timestamp: 1658499748\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: af948a1a\n", " warmup_time: 0.002669095993041992\n", " \n", "Result for objective_b1bad2fe:\n", " date: 2022-07-22_15-22-31\n", " done: false\n", " experiment_id: f58c8e48e2324dd69c28f526bf96497c\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " mean_loss: -1.5098109958909571\n", " neg_mean_loss: 1.5098109958909571\n", " node_ip: 127.0.0.1\n", " pid: 46209\n", " time_since_restore: 10.190011024475098\n", " time_this_iter_s: 0.10639405250549316\n", " time_total_s: 10.190011024475098\n", " timestamp: 1658499751\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: b1bad2fe\n", " warmup_time: 0.0026597976684570312\n", " \n", "Result for objective_b1bad2fe:\n", " date: 2022-07-22_15-22-32\n", " done: true\n", " experiment_id: f58c8e48e2324dd69c28f526bf96497c\n", " experiment_tag: 10_activation=tanh,height=-16.4629,steps=100,width=7.6885\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: -1.5166144695235813\n", " neg_mean_loss: 1.5166144695235813\n", " node_ip: 127.0.0.1\n", " pid: 46209\n", " time_since_restore: 10.727396011352539\n", " time_this_iter_s: 0.10742807388305664\n", " time_total_s: 10.727396011352539\n", " timestamp: 1658499752\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: b1bad2fe\n", " warmup_time: 0.0026597976684570312\n", " \n" ] } ], "source": [ "tuner = tune.Tuner(\n", " objective,\n", " tune_config=tune.TuneConfig(\n", " metric=\"mean_loss\",\n", " mode=\"min\",\n", " search_alg=algo,\n", " num_samples=num_samples,\n", " ),\n", " param_space=search_space,\n", ")\n", "results = tuner.fit()" ] }, { "cell_type": "markdown", "id": "b66aab6a", "metadata": {}, "source": [ "And now we have the hyperparameters found to minimize the mean loss." ] }, { "cell_type": "code", "execution_count": 12, "id": "e69db02e", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Best hyperparameters found were: {'steps': 100, 'width': 7.688465886501621, 'height': -16.46289560854555, 'activation': 'tanh'}\n" ] } ], "source": [ "print(\"Best hyperparameters found were: \", results.get_best_result().config)" ] }, { "cell_type": "markdown", "id": "d545d30b", "metadata": {}, "source": [ "## Providing an initial set of hyperparameters\n", "\n", "While defining the search algorithm, we may choose to provide an initial set of hyperparameters that we believe are especially promising or informative, and\n", "pass this information as a helpful starting point for the `OptunaSearch` object." ] }, { "cell_type": "code", "execution_count": 13, "id": "7596b7f4", "metadata": {}, "outputs": [], "source": [ "initial_params = [\n", " {\"width\": 1, \"height\": 2, \"activation\": \"relu\"},\n", " {\"width\": 4, \"height\": 2, \"activation\": \"relu\"},\n", "]" ] }, { "cell_type": "markdown", "id": "f84bbff0", "metadata": {}, "source": [ "Now the `search_alg` built using `OptunaSearch` takes `points_to_evaluate`." ] }, { "cell_type": "code", "execution_count": 14, "id": "320d1935", "metadata": { "lines_to_next_cell": 0 }, "outputs": [], "source": [ "searcher = OptunaSearch(points_to_evaluate=initial_params)\n", "algo = ConcurrencyLimiter(searcher, max_concurrent=4)" ] }, { "cell_type": "markdown", "id": "9147d9a2", "metadata": {}, "source": [ "And run the experiment with initial hyperparameter evaluations:" ] }, { "cell_type": "code", "execution_count": 15, "id": "ee442efd", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "\u001b[32m[I 2022-07-22 15:22:32,644]\u001b[0m A new study created in memory with name: optuna\u001b[0m\n", "/Users/kai/coding/ray/python/ray/tune/search/optuna/optuna_search.py:389: ExperimentalWarning: enqueue_trial is experimental (supported from v1.2.0). The interface can change in the future.\n", " self._ot_study.enqueue_trial(point)\n", "/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/optuna/study/study.py:857: ExperimentalWarning: create_trial is experimental (supported from v2.0.0). The interface can change in the future.\n", " create_trial(state=TrialState.WAITING, system_attrs={\"fixed_params\": params})\n", "/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/optuna/study/study.py:857: ExperimentalWarning: add_trial is experimental (supported from v2.0.0). The interface can change in the future.\n", " create_trial(state=TrialState.WAITING, system_attrs={\"fixed_params\": params})\n" ] }, { "data": { "text/html": [ "== Status ==
Current time: 2022-07-22 15:23:15 (running for 00:00:42.97)
Memory usage on this node: 10.1/16.0 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/5.07 GiB heap, 0.0/2.0 GiB objects
Current best trial: c3762426 with mean_loss=-8.621481572769046 and parameters={'steps': 100, 'width': 0.7236705053153614, 'height': -98.46320622400306, 'activation': 'tanh'}
Result logdir: /Users/kai/ray_results/optuna_example_2
Number of trials: 10/10 (10 TERMINATED)
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
Trial name status loc activation height width loss iter total time (s) iterations neg_mean_loss
objective_b9acff32TERMINATED127.0.0.1:46220relu 2 1 11.1174 100 10.6837 99 -11.1174
objective_bb3b2702TERMINATED127.0.0.1:46227relu 2 4 10.4463 100 11.2816 99 -10.4463
objective_bb3c8714TERMINATED127.0.0.1:46228relu -90.7016 0.566597 2.44285 100 11.2766 99 -2.44285
objective_bb3e0210TERMINATED127.0.0.1:46229tanh -81.4501 14.783 -8.07715 100 11.2196 99 8.07715
objective_c19b9ddeTERMINATED127.0.0.1:46248relu 86.466 0.30865921.1122 100 10.7226 99 -21.1122
objective_c3762426TERMINATED127.0.0.1:46253tanh -98.4632 0.723671-8.62148 100 10.7271 99 8.62148
objective_c385528eTERMINATED127.0.0.1:46256relu -54.8368 11.0765 4.60669 100 10.7498 99 -4.60669
objective_c387a7c8TERMINATED127.0.0.1:46257tanh 42.9466 8.01708 4.41909 100 10.7214 99 -4.41909
objective_c97678f8TERMINATED127.0.0.1:46272tanh 9.2521415.5288 0.989841 100 12.4591 99 -0.989841
objective_cb6180feTERMINATED127.0.0.1:46277relu 41.6647 8.0585 14.2903 100 10.752 99 -14.2903


" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "Result for objective_b9acff32:\n", " date: 2022-07-22_15-22-35\n", " done: false\n", " experiment_id: ec3c260f0ad348fa9b188f9c93c01eba\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 20.2\n", " neg_mean_loss: -20.2\n", " node_ip: 127.0.0.1\n", " pid: 46220\n", " time_since_restore: 0.10049605369567871\n", " time_this_iter_s: 0.10049605369567871\n", " time_total_s: 0.10049605369567871\n", " timestamp: 1658499755\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: b9acff32\n", " warmup_time: 0.0029239654541015625\n", " \n", "Result for objective_bb3e0210:\n", " date: 2022-07-22_15-22-37\n", " done: false\n", " experiment_id: 08a7bc47c140405296e58a049cd2f6b4\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 1.8549886153795079\n", " neg_mean_loss: -1.8549886153795079\n", " node_ip: 127.0.0.1\n", " pid: 46229\n", " time_since_restore: 0.10361814498901367\n", " time_this_iter_s: 0.10361814498901367\n", " time_total_s: 0.10361814498901367\n", " timestamp: 1658499757\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: bb3e0210\n", " warmup_time: 0.002833843231201172\n", " \n", "Result for objective_bb3b2702:\n", " date: 2022-07-22_15-22-37\n", " done: false\n", " experiment_id: 2f984120624b4e9aa3392fcc1b16b811\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 20.2\n", " neg_mean_loss: -20.2\n", " node_ip: 127.0.0.1\n", " pid: 46227\n", " time_since_restore: 0.10420107841491699\n", " time_this_iter_s: 0.10420107841491699\n", " time_total_s: 0.10420107841491699\n", " timestamp: 1658499757\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: bb3b2702\n", " warmup_time: 0.002672910690307617\n", " \n", "Result for objective_bb3c8714:\n", " date: 2022-07-22_15-22-37\n", " done: false\n", " experiment_id: 5fdcb12c4e234f1d9904a09b8705c0bf\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 10.929835533284928\n", " neg_mean_loss: -10.929835533284928\n", " node_ip: 127.0.0.1\n", " pid: 46228\n", " time_since_restore: 0.1048588752746582\n", " time_this_iter_s: 0.1048588752746582\n", " time_total_s: 0.1048588752746582\n", " timestamp: 1658499757\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: bb3c8714\n", " warmup_time: 0.0027511119842529297\n", " \n", "Result for objective_b9acff32:\n", " date: 2022-07-22_15-22-40\n", " done: false\n", " experiment_id: ec3c260f0ad348fa9b188f9c93c01eba\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 48\n", " iterations_since_restore: 49\n", " mean_loss: 11.924137931034483\n", " neg_mean_loss: -11.924137931034483\n", " node_ip: 127.0.0.1\n", " pid: 46220\n", " time_since_restore: 5.207761287689209\n", " time_this_iter_s: 0.10753417015075684\n", " time_total_s: 5.207761287689209\n", " timestamp: 1658499760\n", " timesteps_since_restore: 0\n", " training_iteration: 49\n", " trial_id: b9acff32\n", " warmup_time: 0.0029239654541015625\n", " \n", "Result for objective_bb3e0210:\n", " date: 2022-07-22_15-22-43\n", " done: false\n", " experiment_id: 08a7bc47c140405296e58a049cd2f6b4\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: -8.003127819283387\n", " neg_mean_loss: 8.003127819283387\n", " node_ip: 127.0.0.1\n", " pid: 46229\n", " time_since_restore: 5.188760042190552\n", " time_this_iter_s: 0.10853385925292969\n", " time_total_s: 5.188760042190552\n", " timestamp: 1658499763\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: bb3e0210\n", " warmup_time: 0.002833843231201172\n", " \n", "Result for objective_bb3b2702:\n", " date: 2022-07-22_15-22-43\n", " done: false\n", " experiment_id: 2f984120624b4e9aa3392fcc1b16b811\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: 10.705050505050504\n", " neg_mean_loss: -10.705050505050504\n", " node_ip: 127.0.0.1\n", " pid: 46227\n", " time_since_restore: 5.1574530601501465\n", " time_this_iter_s: 0.10719585418701172\n", " time_total_s: 5.1574530601501465\n", " timestamp: 1658499763\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: bb3b2702\n", " warmup_time: 0.002672910690307617\n", " \n", "Result for objective_bb3c8714:\n", " date: 2022-07-22_15-22-43\n", " done: false\n", " experiment_id: 5fdcb12c4e234f1d9904a09b8705c0bf\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: 3.6598337799021596\n", " neg_mean_loss: -3.6598337799021596\n", " node_ip: 127.0.0.1\n", " pid: 46228\n", " time_since_restore: 5.163677930831909\n", " time_this_iter_s: 0.10699272155761719\n", " time_total_s: 5.163677930831909\n", " timestamp: 1658499763\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: bb3c8714\n", " warmup_time: 0.0027511119842529297\n", " \n", "Result for objective_b9acff32:\n", " date: 2022-07-22_15-22-45\n", " done: false\n", " experiment_id: ec3c260f0ad348fa9b188f9c93c01eba\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 95\n", " iterations_since_restore: 96\n", " mean_loss: 11.152380952380952\n", " neg_mean_loss: -11.152380952380952\n", " node_ip: 127.0.0.1\n", " pid: 46220\n", " time_since_restore: 10.250982284545898\n", " time_this_iter_s: 0.1073312759399414\n", " time_total_s: 10.250982284545898\n", " timestamp: 1658499765\n", " timesteps_since_restore: 0\n", " training_iteration: 96\n", " trial_id: b9acff32\n", " warmup_time: 0.0029239654541015625\n", " \n", "Result for objective_b9acff32:\n", " date: 2022-07-22_15-22-45\n", " done: true\n", " experiment_id: ec3c260f0ad348fa9b188f9c93c01eba\n", " experiment_tag: 1_activation=relu,height=2,steps=100,width=1\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 11.11743119266055\n", " neg_mean_loss: -11.11743119266055\n", " node_ip: 127.0.0.1\n", " pid: 46220\n", " time_since_restore: 10.68371295928955\n", " time_this_iter_s: 0.10785365104675293\n", " time_total_s: 10.68371295928955\n", " timestamp: 1658499765\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: b9acff32\n", " warmup_time: 0.0029239654541015625\n", " \n", "Result for objective_bb3e0210:\n", " date: 2022-07-22_15-22-47\n", " done: false\n", " experiment_id: 08a7bc47c140405296e58a049cd2f6b4\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 93\n", " iterations_since_restore: 94\n", " mean_loss: -8.072800015147221\n", " neg_mean_loss: 8.072800015147221\n", " node_ip: 127.0.0.1\n", " pid: 46229\n", " time_since_restore: 10.099772214889526\n", " time_this_iter_s: 0.10250687599182129\n", " time_total_s: 10.099772214889526\n", " timestamp: 1658499767\n", " timesteps_since_restore: 0\n", " training_iteration: 94\n", " trial_id: bb3e0210\n", " warmup_time: 0.002833843231201172\n", " \n", "Result for objective_bb3c8714:\n", " date: 2022-07-22_15-22-47\n", " done: false\n", " experiment_id: 5fdcb12c4e234f1d9904a09b8705c0bf\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 92\n", " iterations_since_restore: 93\n", " mean_loss: 2.53944357306219\n", " neg_mean_loss: -2.53944357306219\n", " node_ip: 127.0.0.1\n", " pid: 46228\n", " time_since_restore: 9.980368852615356\n", " time_this_iter_s: 0.10658979415893555\n", " time_total_s: 9.980368852615356\n", " timestamp: 1658499767\n", " timesteps_since_restore: 0\n", " training_iteration: 93\n", " trial_id: bb3c8714\n", " warmup_time: 0.0027511119842529297\n", " \n", "Result for objective_bb3b2702:\n", " date: 2022-07-22_15-22-47\n", " done: false\n", " experiment_id: 2f984120624b4e9aa3392fcc1b16b811\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 92\n", " iterations_since_restore: 93\n", " mean_loss: 10.464550264550265\n", " neg_mean_loss: -10.464550264550265\n", " node_ip: 127.0.0.1\n", " pid: 46227\n", " time_since_restore: 9.964121103286743\n", " time_this_iter_s: 0.10547995567321777\n", " time_total_s: 9.964121103286743\n", " timestamp: 1658499767\n", " timesteps_since_restore: 0\n", " training_iteration: 93\n", " trial_id: bb3b2702\n", " warmup_time: 0.002672910690307617\n", " \n", "Result for objective_c19b9dde:\n", " date: 2022-07-22_15-22-48\n", " done: false\n", " experiment_id: fab35a2f68c347c4bb815c268e9500a6\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 28.646596578422436\n", " neg_mean_loss: -28.646596578422436\n", " node_ip: 127.0.0.1\n", " pid: 46248\n", " time_since_restore: 0.10230708122253418\n", " time_this_iter_s: 0.10230708122253418\n", " time_total_s: 0.10230708122253418\n", " timestamp: 1658499768\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: c19b9dde\n", " warmup_time: 0.0026540756225585938\n", " \n", "Result for objective_bb3e0210:\n", " date: 2022-07-22_15-22-49\n", " done: true\n", " experiment_id: 08a7bc47c140405296e58a049cd2f6b4\n", " experiment_tag: 4_activation=tanh,height=-81.4501,steps=100,width=14.7830\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: -8.077146761191683\n", " neg_mean_loss: 8.077146761191683\n", " node_ip: 127.0.0.1\n", " pid: 46229\n", " time_since_restore: 11.219618082046509\n", " time_this_iter_s: 0.1108238697052002\n", " time_total_s: 11.219618082046509\n", " timestamp: 1658499769\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: bb3e0210\n", " warmup_time: 0.002833843231201172\n", " \n", "Result for objective_bb3c8714:\n", " date: 2022-07-22_15-22-49\n", " done: true\n", " experiment_id: 5fdcb12c4e234f1d9904a09b8705c0bf\n", " experiment_tag: 3_activation=relu,height=-90.7016,steps=100,width=0.5666\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 2.442852652712544\n", " neg_mean_loss: -2.442852652712544\n", " node_ip: 127.0.0.1\n", " pid: 46228\n", " time_since_restore: 11.276630163192749\n", " time_this_iter_s: 0.10824012756347656\n", " time_total_s: 11.276630163192749\n", " timestamp: 1658499769\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: bb3c8714\n", " warmup_time: 0.0027511119842529297\n", " \n", "Result for objective_bb3b2702:\n", " date: 2022-07-22_15-22-49\n", " done: true\n", " experiment_id: 2f984120624b4e9aa3392fcc1b16b811\n", " experiment_tag: 2_activation=relu,height=2,steps=100,width=4\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 10.446305418719211\n", " neg_mean_loss: -10.446305418719211\n", " node_ip: 127.0.0.1\n", " pid: 46227\n", " time_since_restore: 11.281576871871948\n", " time_this_iter_s: 0.1068716049194336\n", " time_total_s: 11.281576871871948\n", " timestamp: 1658499769\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: bb3b2702\n", " warmup_time: 0.002672910690307617\n", " \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Result for objective_c3762426:\n", " date: 2022-07-22_15-22-51\n", " done: false\n", " experiment_id: 034596724b2a4c9f86726e9a15fb386b\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 0.15367937759969408\n", " neg_mean_loss: -0.15367937759969408\n", " node_ip: 127.0.0.1\n", " pid: 46253\n", " time_since_restore: 0.1050560474395752\n", " time_this_iter_s: 0.1050560474395752\n", " time_total_s: 0.1050560474395752\n", " timestamp: 1658499771\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: c3762426\n", " warmup_time: 0.0028579235076904297\n", " \n", "Result for objective_c385528e:\n", " date: 2022-07-22_15-22-51\n", " done: false\n", " experiment_id: 3044e8f1500f4b16aba04b72105c67be\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 14.516317633001044\n", " neg_mean_loss: -14.516317633001044\n", " node_ip: 127.0.0.1\n", " pid: 46256\n", " time_since_restore: 0.10402488708496094\n", " time_this_iter_s: 0.10402488708496094\n", " time_total_s: 0.10402488708496094\n", " timestamp: 1658499771\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: c385528e\n", " warmup_time: 0.0027010440826416016\n", " \n", "Result for objective_c387a7c8:\n", " date: 2022-07-22_15-22-51\n", " done: false\n", " experiment_id: fc7f3d46b136437cab526c2fa6ea2944\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 14.294664350689382\n", " neg_mean_loss: -14.294664350689382\n", " node_ip: 127.0.0.1\n", " pid: 46257\n", " time_since_restore: 0.10461091995239258\n", " time_this_iter_s: 0.10461091995239258\n", " time_total_s: 0.10461091995239258\n", " timestamp: 1658499771\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: c387a7c8\n", " warmup_time: 0.0025899410247802734\n", " \n", "Result for objective_c19b9dde:\n", " date: 2022-07-22_15-22-53\n", " done: false\n", " experiment_id: fab35a2f68c347c4bb815c268e9500a6\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: 22.72706638821166\n", " neg_mean_loss: -22.72706638821166\n", " node_ip: 127.0.0.1\n", " pid: 46248\n", " time_since_restore: 5.118642330169678\n", " time_this_iter_s: 0.10826611518859863\n", " time_total_s: 5.118642330169678\n", " timestamp: 1658499773\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: c19b9dde\n", " warmup_time: 0.0026540756225585938\n", " \n", "Result for objective_c3762426:\n", " date: 2022-07-22_15-22-56\n", " done: false\n", " experiment_id: 034596724b2a4c9f86726e9a15fb386b\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: -7.5742395372701825\n", " neg_mean_loss: 7.5742395372701825\n", " node_ip: 127.0.0.1\n", " pid: 46253\n", " time_since_restore: 5.14322304725647\n", " time_this_iter_s: 0.10359907150268555\n", " time_total_s: 5.14322304725647\n", " timestamp: 1658499776\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: c3762426\n", " warmup_time: 0.0028579235076904297\n", " \n", "Result for objective_c385528e:\n", " date: 2022-07-22_15-22-56\n", " done: false\n", " experiment_id: 3044e8f1500f4b16aba04b72105c67be\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: 4.704784733826337\n", " neg_mean_loss: -4.704784733826337\n", " node_ip: 127.0.0.1\n", " pid: 46256\n", " time_since_restore: 5.16935396194458\n", " time_this_iter_s: 0.10719513893127441\n", " time_total_s: 5.16935396194458\n", " timestamp: 1658499776\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: c385528e\n", " warmup_time: 0.0027010440826416016\n", " \n", "Result for objective_c387a7c8:\n", " date: 2022-07-22_15-22-56\n", " done: false\n", " experiment_id: fc7f3d46b136437cab526c2fa6ea2944\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: 4.553194134463865\n", " neg_mean_loss: -4.553194134463865\n", " node_ip: 127.0.0.1\n", " pid: 46257\n", " time_since_restore: 5.1458728313446045\n", " time_this_iter_s: 0.10374999046325684\n", " time_total_s: 5.1458728313446045\n", " timestamp: 1658499776\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: c387a7c8\n", " warmup_time: 0.0025899410247802734\n", " \n", "Result for objective_c19b9dde:\n", " date: 2022-07-22_15-22-58\n", " done: false\n", " experiment_id: fab35a2f68c347c4bb815c268e9500a6\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " mean_loss: 21.20978139671461\n", " neg_mean_loss: -21.20978139671461\n", " node_ip: 127.0.0.1\n", " pid: 46248\n", " time_since_restore: 10.183105230331421\n", " time_this_iter_s: 0.10718917846679688\n", " time_total_s: 10.183105230331421\n", " timestamp: 1658499778\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: c19b9dde\n", " warmup_time: 0.0026540756225585938\n", " \n", "Result for objective_c19b9dde:\n", " date: 2022-07-22_15-22-59\n", " done: true\n", " experiment_id: fab35a2f68c347c4bb815c268e9500a6\n", " experiment_tag: 5_activation=relu,height=86.4660,steps=100,width=0.3087\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 21.112246386473707\n", " neg_mean_loss: -21.112246386473707\n", " node_ip: 127.0.0.1\n", " pid: 46248\n", " time_since_restore: 10.722583055496216\n", " time_this_iter_s: 0.10788774490356445\n", " time_total_s: 10.722583055496216\n", " timestamp: 1658499779\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: c19b9dde\n", " warmup_time: 0.0026540756225585938\n", " \n", "Result for objective_c97678f8:\n", " date: 2022-07-22_15-23-01\n", " done: false\n", " experiment_id: 7c4187f99b9246bdbae534ce20f33bfc\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 10.925214276212655\n", " neg_mean_loss: -10.925214276212655\n", " node_ip: 127.0.0.1\n", " pid: 46272\n", " time_since_restore: 0.1039581298828125\n", " time_this_iter_s: 0.1039581298828125\n", " time_total_s: 0.1039581298828125\n", " timestamp: 1658499781\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: c97678f8\n", " warmup_time: 0.002711057662963867\n", " \n", "Result for objective_c3762426:\n", " date: 2022-07-22_15-23-01\n", " done: false\n", " experiment_id: 034596724b2a4c9f86726e9a15fb386b\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " mean_loss: -8.564680574270156\n", " neg_mean_loss: 8.564680574270156\n", " node_ip: 127.0.0.1\n", " pid: 46253\n", " time_since_restore: 10.190189123153687\n", " time_this_iter_s: 0.10848307609558105\n", " time_total_s: 10.190189123153687\n", " timestamp: 1658499781\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: c3762426\n", " warmup_time: 0.0028579235076904297\n", " \n", "Result for objective_c387a7c8:\n", " date: 2022-07-22_15-23-01\n", " done: false\n", " experiment_id: fc7f3d46b136437cab526c2fa6ea2944\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " mean_loss: 4.425622066068654\n", " neg_mean_loss: -4.425622066068654\n", " node_ip: 127.0.0.1\n", " pid: 46257\n", " time_since_restore: 10.181554794311523\n", " time_this_iter_s: 0.10594582557678223\n", " time_total_s: 10.181554794311523\n", " timestamp: 1658499781\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: c387a7c8\n", " warmup_time: 0.0025899410247802734\n", " \n", "Result for objective_c385528e:\n", " date: 2022-07-22_15-23-01\n", " done: false\n", " experiment_id: 3044e8f1500f4b16aba04b72105c67be\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " mean_loss: 4.611447627123378\n", " neg_mean_loss: -4.611447627123378\n", " node_ip: 127.0.0.1\n", " pid: 46256\n", " time_since_restore: 10.214791059494019\n", " time_this_iter_s: 0.10866808891296387\n", " time_total_s: 10.214791059494019\n", " timestamp: 1658499781\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: c385528e\n", " warmup_time: 0.0027010440826416016\n", " \n", "Result for objective_c3762426:\n", " date: 2022-07-22_15-23-02\n", " done: true\n", " experiment_id: 034596724b2a4c9f86726e9a15fb386b\n", " experiment_tag: 6_activation=tanh,height=-98.4632,steps=100,width=0.7237\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: -8.621481572769046\n", " neg_mean_loss: 8.621481572769046\n", " node_ip: 127.0.0.1\n", " pid: 46253\n", " time_since_restore: 10.72708010673523\n", " time_this_iter_s: 0.10719490051269531\n", " time_total_s: 10.72708010673523\n", " timestamp: 1658499782\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: c3762426\n", " warmup_time: 0.0028579235076904297\n", " \n", "Result for objective_c387a7c8:\n", " date: 2022-07-22_15-23-02\n", " done: true\n", " experiment_id: fc7f3d46b136437cab526c2fa6ea2944\n", " experiment_tag: 8_activation=tanh,height=42.9466,steps=100,width=8.0171\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 4.419090335709762\n", " neg_mean_loss: -4.419090335709762\n", " node_ip: 127.0.0.1\n", " pid: 46257\n", " time_since_restore: 10.721372842788696\n", " time_this_iter_s: 0.10873699188232422\n", " time_total_s: 10.721372842788696\n", " timestamp: 1658499782\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: c387a7c8\n", " warmup_time: 0.0025899410247802734\n", " \n", "Result for objective_c385528e:\n", " date: 2022-07-22_15-23-02\n", " done: true\n", " experiment_id: 3044e8f1500f4b16aba04b72105c67be\n", " experiment_tag: 7_activation=relu,height=-54.8368,steps=100,width=11.0765\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 4.606686500095965\n", " neg_mean_loss: -4.606686500095965\n", " node_ip: 127.0.0.1\n", " pid: 46256\n", " time_since_restore: 10.749844074249268\n", " time_this_iter_s: 0.10710501670837402\n", " time_total_s: 10.749844074249268\n", " timestamp: 1658499782\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: c385528e\n", " warmup_time: 0.0027010440826416016\n", " \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Result for objective_cb6180fe:\n", " date: 2022-07-22_15-23-04\n", " done: false\n", " experiment_id: 0cfd73afdca24f8f894b2a3e7ebb8cea\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 24.166473702046588\n", " neg_mean_loss: -24.166473702046588\n", " node_ip: 127.0.0.1\n", " pid: 46277\n", " time_since_restore: 0.10389995574951172\n", " time_this_iter_s: 0.10389995574951172\n", " time_total_s: 0.10389995574951172\n", " timestamp: 1658499784\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: cb6180fe\n", " warmup_time: 0.0027260780334472656\n", " \n", "Result for objective_c97678f8:\n", " date: 2022-07-22_15-23-06\n", " done: false\n", " experiment_id: 7c4187f99b9246bdbae534ce20f33bfc\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 31\n", " iterations_since_restore: 32\n", " mean_loss: 1.1287174007116527\n", " neg_mean_loss: -1.1287174007116527\n", " node_ip: 127.0.0.1\n", " pid: 46272\n", " time_since_restore: 5.140314340591431\n", " time_this_iter_s: 0.10663819313049316\n", " time_total_s: 5.140314340591431\n", " timestamp: 1658499786\n", " timesteps_since_restore: 0\n", " training_iteration: 32\n", " trial_id: c97678f8\n", " warmup_time: 0.002711057662963867\n", " \n", "Result for objective_cb6180fe:\n", " date: 2022-07-22_15-23-09\n", " done: false\n", " experiment_id: 0cfd73afdca24f8f894b2a3e7ebb8cea\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: 14.42370878193022\n", " neg_mean_loss: -14.42370878193022\n", " node_ip: 127.0.0.1\n", " pid: 46277\n", " time_since_restore: 5.139087915420532\n", " time_this_iter_s: 0.10686802864074707\n", " time_total_s: 5.139087915420532\n", " timestamp: 1658499789\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: cb6180fe\n", " warmup_time: 0.0027260780334472656\n", " \n", "Result for objective_c97678f8:\n", " date: 2022-07-22_15-23-11\n", " done: false\n", " experiment_id: 7c4187f99b9246bdbae534ce20f33bfc\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 78\n", " iterations_since_restore: 79\n", " mean_loss: 1.0070978091228147\n", " neg_mean_loss: -1.0070978091228147\n", " node_ip: 127.0.0.1\n", " pid: 46272\n", " time_since_restore: 10.206932067871094\n", " time_this_iter_s: 0.10774993896484375\n", " time_total_s: 10.206932067871094\n", " timestamp: 1658499791\n", " timesteps_since_restore: 0\n", " training_iteration: 79\n", " trial_id: c97678f8\n", " warmup_time: 0.002711057662963867\n", " \n", "Result for objective_c97678f8:\n", " date: 2022-07-22_15-23-14\n", " done: true\n", " experiment_id: 7c4187f99b9246bdbae534ce20f33bfc\n", " experiment_tag: 9_activation=tanh,height=9.2521,steps=100,width=15.5288\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 0.9898408262377837\n", " neg_mean_loss: -0.9898408262377837\n", " node_ip: 127.0.0.1\n", " pid: 46272\n", " time_since_restore: 12.459095239639282\n", " time_this_iter_s: 0.10785508155822754\n", " time_total_s: 12.459095239639282\n", " timestamp: 1658499794\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: c97678f8\n", " warmup_time: 0.002711057662963867\n", " \n", "Result for objective_cb6180fe:\n", " date: 2022-07-22_15-23-15\n", " done: false\n", " experiment_id: 0cfd73afdca24f8f894b2a3e7ebb8cea\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " mean_loss: 14.296767042885353\n", " neg_mean_loss: -14.296767042885353\n", " node_ip: 127.0.0.1\n", " pid: 46277\n", " time_since_restore: 10.220713138580322\n", " time_this_iter_s: 0.10750126838684082\n", " time_total_s: 10.220713138580322\n", " timestamp: 1658499795\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: cb6180fe\n", " warmup_time: 0.0027260780334472656\n", " \n", "Result for objective_cb6180fe:\n", " date: 2022-07-22_15-23-15\n", " done: true\n", " experiment_id: 0cfd73afdca24f8f894b2a3e7ebb8cea\n", " experiment_tag: 10_activation=relu,height=41.6647,steps=100,width=8.0585\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 14.29026803363612\n", " neg_mean_loss: -14.29026803363612\n", " node_ip: 127.0.0.1\n", " pid: 46277\n", " time_since_restore: 10.752035856246948\n", " time_this_iter_s: 0.1066899299621582\n", " time_total_s: 10.752035856246948\n", " timestamp: 1658499795\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: cb6180fe\n", " warmup_time: 0.0027260780334472656\n", " \n" ] } ], "source": [ "tuner = tune.Tuner(\n", " objective,\n", " tune_config=tune.TuneConfig(\n", " metric=\"mean_loss\",\n", " mode=\"min\",\n", " search_alg=algo,\n", " num_samples=num_samples,\n", " ),\n", " param_space=search_space,\n", ")\n", "results = tuner.fit()" ] }, { "cell_type": "markdown", "id": "ccfe15e2", "metadata": {}, "source": [ "We take another look at the optimal hyperparamters." ] }, { "cell_type": "code", "execution_count": 16, "id": "fcfa0c2e", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Best hyperparameters found were: {'steps': 100, 'width': 0.7236705053153614, 'height': -98.46320622400306, 'activation': 'tanh'}\n" ] } ], "source": [ "print(\"Best hyperparameters found were: \", results.get_best_result().config)" ] }, { "cell_type": "markdown", "id": "88080576", "metadata": {}, "source": [ "## Conditional search spaces \n", "\n", "Sometimes we may want to build a more complicated search space that has conditional dependencies on other hyperparameters. In this case, we pass a define-by-run function to the `search_alg` argument in `ray.tune()`." ] }, { "cell_type": "code", "execution_count": 17, "id": "f0acc2fc", "metadata": { "lines_to_next_cell": 0 }, "outputs": [], "source": [ "def define_by_run_func(trial) -> Optional[Dict[str, Any]]:\n", " \"\"\"Define-by-run function to create the search space.\n", "\n", " Ensure no actual computation takes place here. That should go into\n", " the trainable passed to ``Tuner()`` (in this example, that's\n", " ``objective``).\n", "\n", " For more information, see https://optuna.readthedocs.io/en/stable\\\n", " /tutorial/10_key_features/002_configurations.html\n", "\n", " This function should either return None or a dict with constant values.\n", " \"\"\"\n", "\n", " activation = trial.suggest_categorical(\"activation\", [\"relu\", \"tanh\"])\n", "\n", " # Define-by-run allows for conditional search spaces.\n", " if activation == \"relu\":\n", " trial.suggest_float(\"width\", 0, 20)\n", " trial.suggest_float(\"height\", -100, 100)\n", " else:\n", " trial.suggest_float(\"width\", -1, 21)\n", " trial.suggest_float(\"height\", -101, 101)\n", " \n", " # Return all constants in a dictionary.\n", " return {\"steps\": 100}" ] }, { "cell_type": "markdown", "id": "4c9d0945", "metadata": {}, "source": [ "As before, we create the `search_alg` from `OptunaSearch` and `ConcurrencyLimiter`, this time we define the scope of search via the `space` argument and provide no initialization. We also must specific metric and mode when using `space`. " ] }, { "cell_type": "code", "execution_count": 18, "id": "906f9ffc", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "\u001b[32m[I 2022-07-22 15:23:15,784]\u001b[0m A new study created in memory with name: optuna\u001b[0m\n" ] } ], "source": [ "searcher = OptunaSearch(space=define_by_run_func, metric=\"mean_loss\", mode=\"min\")\n", "algo = ConcurrencyLimiter(searcher, max_concurrent=4)" ] }, { "cell_type": "markdown", "id": "fea9399c", "metadata": {}, "source": [ "Running the experiment with a define-by-run search space:" ] }, { "cell_type": "code", "execution_count": 19, "id": "bf0ee932", "metadata": {}, "outputs": [ { "data": { "text/html": [ "== Status ==
Current time: 2022-07-22 15:23:58 (running for 00:00:43.15)
Memory usage on this node: 10.4/16.0 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/5.07 GiB heap, 0.0/2.0 GiB objects
Result logdir: /Users/kai/ray_results/optuna_example_3
Number of trials: 10/10 (10 TERMINATED)
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
Trial name status loc activation height steps width loss iter total time (s) iterations neg_mean_loss
objective_d363eed6TERMINATED127.0.0.1:46290relu 96.9804 100 6.47293 19.8517 100 10.735 99 -19.8517
objective_d4f3a700TERMINATED127.0.0.1:46298tanh -76.4387 10017.7358 -7.58724 100 11.396 99 7.58724
objective_d4f50ce4TERMINATED127.0.0.1:46299relu -89.8609 100 6.28321 1.17212 100 11.4182 99 -1.17212
objective_d4f67908TERMINATED127.0.0.1:46300relu 67.9266 10014.0599 16.864 100 11.341 99 -16.864
objective_db5c0402TERMINATED127.0.0.1:46317tanh -54.13 10011.4712 -5.32571 100 10.705 99 5.32571
objective_dd487b56TERMINATED127.0.0.1:46322relu -25.9818 100 0.758497 8.57703 100 10.6916 99 -8.57703
objective_dd4b4e94TERMINATED127.0.0.1:46323relu 54.6085 100 1.2361 16.2163 100 10.6711 99 -16.2163
objective_dd5a8bcaTERMINATED127.0.0.1:46328tanh 5.22131 10017.0952 0.580871 100 10.7017 99 -0.580871
objective_e3482178TERMINATED127.0.0.1:46341tanh 78.844 10015.5079 7.94912 100 12.6417 99 -7.94912
objective_e532d6e0TERMINATED127.0.0.1:46353tanh -66.9988 100 7.15087 -6.56059 100 10.7486 99 6.56059


" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "Result for objective_d363eed6:\n", " date: 2022-07-22_15-23-18\n", " done: false\n", " experiment_id: af8f846e60254bc794827e76909df4f0\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 29.698044266420816\n", " neg_mean_loss: -29.698044266420816\n", " node_ip: 127.0.0.1\n", " pid: 46290\n", " time_since_restore: 0.10497498512268066\n", " time_this_iter_s: 0.10497498512268066\n", " time_total_s: 0.10497498512268066\n", " timestamp: 1658499798\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: d363eed6\n", " warmup_time: 0.0027971267700195312\n", " \n", "Result for objective_d4f3a700:\n", " date: 2022-07-22_15-23-21\n", " done: false\n", " experiment_id: 52e5f2a557cf4466939b6c80bb9ff905\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 2.3561338711040003\n", " neg_mean_loss: -2.3561338711040003\n", " node_ip: 127.0.0.1\n", " pid: 46298\n", " time_since_restore: 0.10145115852355957\n", " time_this_iter_s: 0.10145115852355957\n", " time_total_s: 0.10145115852355957\n", " timestamp: 1658499801\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: d4f3a700\n", " warmup_time: 0.0033049583435058594\n", " \n", "Result for objective_d4f67908:\n", " date: 2022-07-22_15-23-21\n", " done: false\n", " experiment_id: 230c6da41a834386be865bb19780b3a5\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 26.792658178391687\n", " neg_mean_loss: -26.792658178391687\n", " node_ip: 127.0.0.1\n", " pid: 46300\n", " time_since_restore: 0.10428690910339355\n", " time_this_iter_s: 0.10428690910339355\n", " time_total_s: 0.10428690910339355\n", " timestamp: 1658499801\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: d4f67908\n", " warmup_time: 0.0030798912048339844\n", " \n", "Result for objective_d4f50ce4:\n", " date: 2022-07-22_15-23-21\n", " done: false\n", " experiment_id: f2396d75f50b417f9fae16d1c9ddcda4\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 11.013906266700227\n", " neg_mean_loss: -11.013906266700227\n", " node_ip: 127.0.0.1\n", " pid: 46299\n", " time_since_restore: 0.1020050048828125\n", " time_this_iter_s: 0.1020050048828125\n", " time_total_s: 0.1020050048828125\n", " timestamp: 1658499801\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: d4f50ce4\n", " warmup_time: 0.0027348995208740234\n", " \n", "Result for objective_d363eed6:\n", " date: 2022-07-22_15-23-23\n", " done: false\n", " experiment_id: af8f846e60254bc794827e76909df4f0\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: 20.016284953435374\n", " neg_mean_loss: -20.016284953435374\n", " node_ip: 127.0.0.1\n", " pid: 46290\n", " time_since_restore: 5.138504981994629\n", " time_this_iter_s: 0.10720086097717285\n", " time_total_s: 5.138504981994629\n", " timestamp: 1658499803\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: d363eed6\n", " warmup_time: 0.0027971267700195312\n", " \n", "Result for objective_d4f3a700:\n", " date: 2022-07-22_15-23-26\n", " done: false\n", " experiment_id: 52e5f2a557cf4466939b6c80bb9ff905\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: -7.525324371061028\n", " neg_mean_loss: 7.525324371061028\n", " node_ip: 127.0.0.1\n", " pid: 46298\n", " time_since_restore: 5.153075218200684\n", " time_this_iter_s: 0.10515189170837402\n", " time_total_s: 5.153075218200684\n", " timestamp: 1658499806\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: d4f3a700\n", " warmup_time: 0.0033049583435058594\n", " \n", "Result for objective_d4f67908:\n", " date: 2022-07-22_15-23-26\n", " done: false\n", " experiment_id: 230c6da41a834386be865bb19780b3a5\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: 16.94173032408916\n", " neg_mean_loss: -16.94173032408916\n", " node_ip: 127.0.0.1\n", " pid: 46300\n", " time_since_restore: 5.153704881668091\n", " time_this_iter_s: 0.10537075996398926\n", " time_total_s: 5.153704881668091\n", " timestamp: 1658499806\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: d4f67908\n", " warmup_time: 0.0030798912048339844\n", " \n", "Result for objective_d4f50ce4:\n", " date: 2022-07-22_15-23-26\n", " done: false\n", " experiment_id: f2396d75f50b417f9fae16d1c9ddcda4\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: 1.3414410369095506\n", " neg_mean_loss: -1.3414410369095506\n", " node_ip: 127.0.0.1\n", " pid: 46299\n", " time_since_restore: 5.1622138023376465\n", " time_this_iter_s: 0.10337996482849121\n", " time_total_s: 5.1622138023376465\n", " timestamp: 1658499806\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: d4f50ce4\n", " warmup_time: 0.0027348995208740234\n", " \n", "Result for objective_d363eed6:\n", " date: 2022-07-22_15-23-28\n", " done: false\n", " experiment_id: af8f846e60254bc794827e76909df4f0\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " mean_loss: 19.85973747786284\n", " neg_mean_loss: -19.85973747786284\n", " node_ip: 127.0.0.1\n", " pid: 46290\n", " time_since_restore: 10.1943039894104\n", " time_this_iter_s: 0.12109899520874023\n", " time_total_s: 10.1943039894104\n", " timestamp: 1658499808\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: d363eed6\n", " warmup_time: 0.0027971267700195312\n", " \n", "Result for objective_d363eed6:\n", " date: 2022-07-22_15-23-29\n", " done: true\n", " experiment_id: af8f846e60254bc794827e76909df4f0\n", " experiment_tag: 1_activation=relu,height=96.9804,steps=100,width=6.4729\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 19.851696631549423\n", " neg_mean_loss: -19.851696631549423\n", " node_ip: 127.0.0.1\n", " pid: 46290\n", " time_since_restore: 10.734967947006226\n", " time_this_iter_s: 0.10534882545471191\n", " time_total_s: 10.734967947006226\n", " timestamp: 1658499809\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: d363eed6\n", " warmup_time: 0.0027971267700195312\n", " \n", "Result for objective_d4f67908:\n", " date: 2022-07-22_15-23-31\n", " done: false\n", " experiment_id: 230c6da41a834386be865bb19780b3a5\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 93\n", " iterations_since_restore: 94\n", " mean_loss: 16.86855533914186\n", " neg_mean_loss: -16.86855533914186\n", " node_ip: 127.0.0.1\n", " pid: 46300\n", " time_since_restore: 10.096334218978882\n", " time_this_iter_s: 0.10534310340881348\n", " time_total_s: 10.096334218978882\n", " timestamp: 1658499811\n", " timesteps_since_restore: 0\n", " training_iteration: 94\n", " trial_id: d4f67908\n", " warmup_time: 0.0030798912048339844\n", " \n", "Result for objective_d4f50ce4:\n", " date: 2022-07-22_15-23-31\n", " done: false\n", " experiment_id: f2396d75f50b417f9fae16d1c9ddcda4\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 92\n", " iterations_since_restore: 93\n", " mean_loss: 1.183958166642718\n", " neg_mean_loss: -1.183958166642718\n", " node_ip: 127.0.0.1\n", " pid: 46299\n", " time_since_restore: 9.976859092712402\n", " time_this_iter_s: 0.1057121753692627\n", " time_total_s: 9.976859092712402\n", " timestamp: 1658499811\n", " timesteps_since_restore: 0\n", " training_iteration: 93\n", " trial_id: d4f50ce4\n", " warmup_time: 0.0027348995208740234\n", " \n", "Result for objective_d4f3a700:\n", " date: 2022-07-22_15-23-31\n", " done: false\n", " experiment_id: 52e5f2a557cf4466939b6c80bb9ff905\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 93\n", " iterations_since_restore: 94\n", " mean_loss: -7.583604593881161\n", " neg_mean_loss: 7.583604593881161\n", " node_ip: 127.0.0.1\n", " pid: 46298\n", " time_since_restore: 10.091833114624023\n", " time_this_iter_s: 0.10643887519836426\n", " time_total_s: 10.091833114624023\n", " timestamp: 1658499811\n", " timesteps_since_restore: 0\n", " training_iteration: 94\n", " trial_id: d4f3a700\n", " warmup_time: 0.0033049583435058594\n", " \n", "Result for objective_db5c0402:\n", " date: 2022-07-22_15-23-31\n", " done: false\n", " experiment_id: 69a34f42e51f4b97a5237fa5ae2be8d8\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 4.587004175501654\n", " neg_mean_loss: -4.587004175501654\n", " node_ip: 127.0.0.1\n", " pid: 46317\n", " time_since_restore: 0.10449004173278809\n", " time_this_iter_s: 0.10449004173278809\n", " time_total_s: 0.10449004173278809\n", " timestamp: 1658499811\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: db5c0402\n", " warmup_time: 0.003406047821044922\n", " \n", "Result for objective_d4f67908:\n", " date: 2022-07-22_15-23-32\n", " done: true\n", " experiment_id: 230c6da41a834386be865bb19780b3a5\n", " experiment_tag: 4_activation=relu,height=67.9266,steps=100,width=14.0599\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 16.86398832185618\n", " neg_mean_loss: -16.86398832185618\n", " node_ip: 127.0.0.1\n", " pid: 46300\n", " time_since_restore: 11.341041088104248\n", " time_this_iter_s: 0.10513019561767578\n", " time_total_s: 11.341041088104248\n", " timestamp: 1658499812\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: d4f67908\n", " warmup_time: 0.0030798912048339844\n", " \n", "Result for objective_d4f3a700:\n", " date: 2022-07-22_15-23-32\n", " done: true\n", " experiment_id: 52e5f2a557cf4466939b6c80bb9ff905\n", " experiment_tag: 2_activation=tanh,height=-76.4387,steps=100,width=17.7358\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: -7.587236125634034\n", " neg_mean_loss: 7.587236125634034\n", " node_ip: 127.0.0.1\n", " pid: 46298\n", " time_since_restore: 11.396040916442871\n", " time_this_iter_s: 0.10666394233703613\n", " time_total_s: 11.396040916442871\n", " timestamp: 1658499812\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: d4f3a700\n", " warmup_time: 0.0033049583435058594\n", " \n", "Result for objective_d4f50ce4:\n", " date: 2022-07-22_15-23-32\n", " done: true\n", " experiment_id: f2396d75f50b417f9fae16d1c9ddcda4\n", " experiment_tag: 3_activation=relu,height=-89.8609,steps=100,width=6.2832\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 1.1721245345451354\n", " neg_mean_loss: -1.1721245345451354\n", " node_ip: 127.0.0.1\n", " pid: 46299\n", " time_since_restore: 11.418162822723389\n", " time_this_iter_s: 0.11828303337097168\n", " time_total_s: 11.418162822723389\n", " timestamp: 1658499812\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: d4f50ce4\n", " warmup_time: 0.0027348995208740234\n", " \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Result for objective_dd4b4e94:\n", " date: 2022-07-22_15-23-35\n", " done: false\n", " experiment_id: c6036d4a26674f86946fd7c8c895e85c\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 25.460850093028082\n", " neg_mean_loss: -25.460850093028082\n", " node_ip: 127.0.0.1\n", " pid: 46323\n", " time_since_restore: 0.10411691665649414\n", " time_this_iter_s: 0.10411691665649414\n", " time_total_s: 0.10411691665649414\n", " timestamp: 1658499815\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: dd4b4e94\n", " warmup_time: 0.003142118453979492\n", " \n", "Result for objective_dd487b56:\n", " date: 2022-07-22_15-23-35\n", " done: false\n", " experiment_id: 75e118a5733e454fab54f4a20c036852\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 17.401819651158622\n", " neg_mean_loss: -17.401819651158622\n", " node_ip: 127.0.0.1\n", " pid: 46322\n", " time_since_restore: 0.1048891544342041\n", " time_this_iter_s: 0.1048891544342041\n", " time_total_s: 0.1048891544342041\n", " timestamp: 1658499815\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: dd487b56\n", " warmup_time: 0.0028679370880126953\n", " \n", "Result for objective_dd5a8bca:\n", " date: 2022-07-22_15-23-35\n", " done: false\n", " experiment_id: 775683e6d95a4705846a37e809b474d7\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 10.522131438315931\n", " neg_mean_loss: -10.522131438315931\n", " node_ip: 127.0.0.1\n", " pid: 46328\n", " time_since_restore: 0.10494732856750488\n", " time_this_iter_s: 0.10494732856750488\n", " time_total_s: 0.10494732856750488\n", " timestamp: 1658499815\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: dd5a8bca\n", " warmup_time: 0.002931833267211914\n", " \n", "Result for objective_db5c0402:\n", " date: 2022-07-22_15-23-36\n", " done: false\n", " experiment_id: 69a34f42e51f4b97a5237fa5ae2be8d8\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: -5.230895562358142\n", " neg_mean_loss: 5.230895562358142\n", " node_ip: 127.0.0.1\n", " pid: 46317\n", " time_since_restore: 5.120880842208862\n", " time_this_iter_s: 0.1076059341430664\n", " time_total_s: 5.120880842208862\n", " timestamp: 1658499816\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: db5c0402\n", " warmup_time: 0.003406047821044922\n", " \n", "Result for objective_dd4b4e94:\n", " date: 2022-07-22_15-23-40\n", " done: false\n", " experiment_id: c6036d4a26674f86946fd7c8c895e85c\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: 16.929346028466988\n", " neg_mean_loss: -16.929346028466988\n", " node_ip: 127.0.0.1\n", " pid: 46323\n", " time_since_restore: 5.137623071670532\n", " time_this_iter_s: 0.10670709609985352\n", " time_total_s: 5.137623071670532\n", " timestamp: 1658499820\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: dd4b4e94\n", " warmup_time: 0.003142118453979492\n", " \n", "Result for objective_dd487b56:\n", " date: 2022-07-22_15-23-40\n", " done: false\n", " experiment_id: 75e118a5733e454fab54f4a20c036852\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: 9.592430453048937\n", " neg_mean_loss: -9.592430453048937\n", " node_ip: 127.0.0.1\n", " pid: 46322\n", " time_since_restore: 5.140692949295044\n", " time_this_iter_s: 0.1086587905883789\n", " time_total_s: 5.140692949295044\n", " timestamp: 1658499820\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: dd487b56\n", " warmup_time: 0.0028679370880126953\n", " \n", "Result for objective_dd5a8bca:\n", " date: 2022-07-22_15-23-40\n", " done: false\n", " experiment_id: 775683e6d95a4705846a37e809b474d7\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: 0.6450610360030851\n", " neg_mean_loss: -0.6450610360030851\n", " node_ip: 127.0.0.1\n", " pid: 46328\n", " time_since_restore: 5.138496160507202\n", " time_this_iter_s: 0.10809993743896484\n", " time_total_s: 5.138496160507202\n", " timestamp: 1658499820\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: dd5a8bca\n", " warmup_time: 0.002931833267211914\n", " \n", "Result for objective_db5c0402:\n", " date: 2022-07-22_15-23-41\n", " done: false\n", " experiment_id: 69a34f42e51f4b97a5237fa5ae2be8d8\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " mean_loss: -5.321109063262834\n", " neg_mean_loss: 5.321109063262834\n", " node_ip: 127.0.0.1\n", " pid: 46317\n", " time_since_restore: 10.162945985794067\n", " time_this_iter_s: 0.10671615600585938\n", " time_total_s: 10.162945985794067\n", " timestamp: 1658499821\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: db5c0402\n", " warmup_time: 0.003406047821044922\n", " \n", "Result for objective_db5c0402:\n", " date: 2022-07-22_15-23-42\n", " done: true\n", " experiment_id: 69a34f42e51f4b97a5237fa5ae2be8d8\n", " experiment_tag: 5_activation=tanh,height=-54.1300,steps=100,width=11.4712\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: -5.325709301325668\n", " neg_mean_loss: 5.325709301325668\n", " node_ip: 127.0.0.1\n", " pid: 46317\n", " time_since_restore: 10.704953908920288\n", " time_this_iter_s: 0.1082158088684082\n", " time_total_s: 10.704953908920288\n", " timestamp: 1658499822\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: db5c0402\n", " warmup_time: 0.003406047821044922\n", " \n", "Result for objective_dd4b4e94:\n", " date: 2022-07-22_15-23-45\n", " done: false\n", " experiment_id: c6036d4a26674f86946fd7c8c895e85c\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " mean_loss: 16.253282229309562\n", " neg_mean_loss: -16.253282229309562\n", " node_ip: 127.0.0.1\n", " pid: 46323\n", " time_since_restore: 10.140441179275513\n", " time_this_iter_s: 0.10501408576965332\n", " time_total_s: 10.140441179275513\n", " timestamp: 1658499825\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: dd4b4e94\n", " warmup_time: 0.003142118453979492\n", " \n", "Result for objective_dd487b56:\n", " date: 2022-07-22_15-23-45\n", " done: false\n", " experiment_id: 75e118a5733e454fab54f4a20c036852\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " mean_loss: 8.631851056235014\n", " neg_mean_loss: -8.631851056235014\n", " node_ip: 127.0.0.1\n", " pid: 46322\n", " time_since_restore: 10.156053066253662\n", " time_this_iter_s: 0.10682201385498047\n", " time_total_s: 10.156053066253662\n", " timestamp: 1658499825\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: dd487b56\n", " warmup_time: 0.0028679370880126953\n", " \n", "Result for objective_e3482178:\n", " date: 2022-07-22_15-23-45\n", " done: false\n", " experiment_id: 6709b686be234eb996457210f3efbe08\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 17.884403838898738\n", " neg_mean_loss: -17.884403838898738\n", " node_ip: 127.0.0.1\n", " pid: 46341\n", " time_since_restore: 0.10499191284179688\n", " time_this_iter_s: 0.10499191284179688\n", " time_total_s: 0.10499191284179688\n", " timestamp: 1658499825\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: e3482178\n", " warmup_time: 0.002877950668334961\n", " \n", "Result for objective_dd5a8bca:\n", " date: 2022-07-22_15-23-45\n", " done: false\n", " experiment_id: 775683e6d95a4705846a37e809b474d7\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " mean_loss: 0.583976365762005\n", " neg_mean_loss: -0.583976365762005\n", " node_ip: 127.0.0.1\n", " pid: 46328\n", " time_since_restore: 10.165286302566528\n", " time_this_iter_s: 0.10989904403686523\n", " time_total_s: 10.165286302566528\n", " timestamp: 1658499825\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: dd5a8bca\n", " warmup_time: 0.002931833267211914\n", " \n", "Result for objective_dd4b4e94:\n", " date: 2022-07-22_15-23-45\n", " done: true\n", " experiment_id: c6036d4a26674f86946fd7c8c895e85c\n", " experiment_tag: 7_activation=relu,height=54.6085,steps=100,width=1.2361\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 16.216283787762706\n", " neg_mean_loss: -16.216283787762706\n", " node_ip: 127.0.0.1\n", " pid: 46323\n", " time_since_restore: 10.67109203338623\n", " time_this_iter_s: 0.1058969497680664\n", " time_total_s: 10.67109203338623\n", " timestamp: 1658499825\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: dd4b4e94\n", " warmup_time: 0.003142118453979492\n", " \n", "Result for objective_dd487b56:\n", " date: 2022-07-22_15-23-45\n", " done: true\n", " experiment_id: 75e118a5733e454fab54f4a20c036852\n", " experiment_tag: 6_activation=relu,height=-25.9818,steps=100,width=0.7585\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 8.577028987245935\n", " neg_mean_loss: -8.577028987245935\n", " node_ip: 127.0.0.1\n", " pid: 46322\n", " time_since_restore: 10.691627979278564\n", " time_this_iter_s: 0.10821914672851562\n", " time_total_s: 10.691627979278564\n", " timestamp: 1658499825\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: dd487b56\n", " warmup_time: 0.0028679370880126953\n", " \n", "Result for objective_dd5a8bca:\n", " date: 2022-07-22_15-23-45\n", " done: true\n", " experiment_id: 775683e6d95a4705846a37e809b474d7\n", " experiment_tag: 8_activation=tanh,height=5.2213,steps=100,width=17.0952\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 0.5808712318418436\n", " neg_mean_loss: -0.5808712318418436\n", " node_ip: 127.0.0.1\n", " pid: 46328\n", " time_since_restore: 10.701741218566895\n", " time_this_iter_s: 0.11321616172790527\n", " time_total_s: 10.701741218566895\n", " timestamp: 1658499825\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: dd5a8bca\n", " warmup_time: 0.002931833267211914\n", " \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Result for objective_e532d6e0:\n", " date: 2022-07-22_15-23-48\n", " done: false\n", " experiment_id: 34d05fd425474b82a450c22c2a063843\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 3.30012197692201\n", " neg_mean_loss: -3.30012197692201\n", " node_ip: 127.0.0.1\n", " pid: 46353\n", " time_since_restore: 0.10400509834289551\n", " time_this_iter_s: 0.10400509834289551\n", " time_total_s: 0.10400509834289551\n", " timestamp: 1658499828\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: e532d6e0\n", " warmup_time: 0.0028901100158691406\n", " \n", "Result for objective_e3482178:\n", " date: 2022-07-22_15-23-50\n", " done: false\n", " experiment_id: 6709b686be234eb996457210f3efbe08\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 29\n", " iterations_since_restore: 30\n", " mean_loss: 8.10192352359083\n", " neg_mean_loss: -8.10192352359083\n", " node_ip: 127.0.0.1\n", " pid: 46341\n", " time_since_restore: 5.183844089508057\n", " time_this_iter_s: 0.10574007034301758\n", " time_total_s: 5.183844089508057\n", " timestamp: 1658499830\n", " timesteps_since_restore: 0\n", " training_iteration: 30\n", " trial_id: e3482178\n", " warmup_time: 0.002877950668334961\n", " \n", "Result for objective_e532d6e0:\n", " date: 2022-07-22_15-23-53\n", " done: false\n", " experiment_id: 34d05fd425474b82a450c22c2a063843\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 48\n", " iterations_since_restore: 49\n", " mean_loss: -6.416785967274913\n", " neg_mean_loss: 6.416785967274913\n", " node_ip: 127.0.0.1\n", " pid: 46353\n", " time_since_restore: 5.212116003036499\n", " time_this_iter_s: 0.10714387893676758\n", " time_total_s: 5.212116003036499\n", " timestamp: 1658499833\n", " timesteps_since_restore: 0\n", " training_iteration: 49\n", " trial_id: e532d6e0\n", " warmup_time: 0.0028901100158691406\n", " \n", "Result for objective_e3482178:\n", " date: 2022-07-22_15-23-55\n", " done: false\n", " experiment_id: 6709b686be234eb996457210f3efbe08\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 76\n", " iterations_since_restore: 77\n", " mean_loss: 7.968536513457016\n", " neg_mean_loss: -7.968536513457016\n", " node_ip: 127.0.0.1\n", " pid: 46341\n", " time_since_restore: 10.190052032470703\n", " time_this_iter_s: 0.10843491554260254\n", " time_total_s: 10.190052032470703\n", " timestamp: 1658499835\n", " timesteps_since_restore: 0\n", " training_iteration: 77\n", " trial_id: e3482178\n", " warmup_time: 0.002877950668334961\n", " \n", "Result for objective_e3482178:\n", " date: 2022-07-22_15-23-57\n", " done: true\n", " experiment_id: 6709b686be234eb996457210f3efbe08\n", " experiment_tag: 9_activation=tanh,height=78.8440,steps=100,width=15.5079\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 7.9491170267942515\n", " neg_mean_loss: -7.9491170267942515\n", " node_ip: 127.0.0.1\n", " pid: 46341\n", " time_since_restore: 12.64172887802124\n", " time_this_iter_s: 0.10761594772338867\n", " time_total_s: 12.64172887802124\n", " timestamp: 1658499837\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: e3482178\n", " warmup_time: 0.002877950668334961\n", " \n", "Result for objective_e532d6e0:\n", " date: 2022-07-22_15-23-58\n", " done: false\n", " experiment_id: 34d05fd425474b82a450c22c2a063843\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 95\n", " iterations_since_restore: 96\n", " mean_loss: -6.554810275125985\n", " neg_mean_loss: 6.554810275125985\n", " node_ip: 127.0.0.1\n", " pid: 46353\n", " time_since_restore: 10.305040121078491\n", " time_this_iter_s: 0.10702300071716309\n", " time_total_s: 10.305040121078491\n", " timestamp: 1658499838\n", " timesteps_since_restore: 0\n", " training_iteration: 96\n", " trial_id: e532d6e0\n", " warmup_time: 0.0028901100158691406\n", " \n", "Result for objective_e532d6e0:\n", " date: 2022-07-22_15-23-58\n", " done: true\n", " experiment_id: 34d05fd425474b82a450c22c2a063843\n", " experiment_tag: 10_activation=tanh,height=-66.9988,steps=100,width=7.1509\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: -6.560589957038967\n", " neg_mean_loss: 6.560589957038967\n", " node_ip: 127.0.0.1\n", " pid: 46353\n", " time_since_restore: 10.74857211112976\n", " time_this_iter_s: 0.1075899600982666\n", " time_total_s: 10.74857211112976\n", " timestamp: 1658499838\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: e532d6e0\n", " warmup_time: 0.0028901100158691406\n", " \n" ] } ], "source": [ "tuner = tune.Tuner(\n", " objective,\n", " tune_config=tune.TuneConfig(\n", " search_alg=algo,\n", " num_samples=num_samples,\n", " ),\n", ")\n", "results = tuner.fit()" ] }, { "cell_type": "markdown", "id": "11e1ee04", "metadata": {}, "source": [ "We take a look again at the optimal hyperparameters." ] }, { "cell_type": "code", "execution_count": 21, "id": "13e4ce18", "metadata": { "lines_to_next_cell": 0 }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Best hyperparameters for loss found were: {'activation': 'tanh', 'width': 17.73584230792165, 'height': -76.43866128895999, 'steps': 100}\n" ] } ], "source": [ "print(\"Best hyperparameters for loss found were: \", results.get_best_result(\"mean_loss\", \"min\").config)" ] }, { "cell_type": "markdown", "id": "34bbd066", "metadata": {}, "source": [ "## Multi-objective optimization\n", "\n", "Finally, let's take a look at the multi-objective case." ] }, { "cell_type": "code", "execution_count": 22, "id": "b233cbea", "metadata": {}, "outputs": [], "source": [ "def multi_objective(config):\n", " # Hyperparameters\n", " width, height = config[\"width\"], config[\"height\"]\n", "\n", " for step in range(config[\"steps\"]):\n", " # Iterative training function - can be any arbitrary training procedure\n", " intermediate_score = evaluate(step, config[\"width\"], config[\"height\"], config[\"activation\"])\n", " # Feed the score back back to Tune.\n", " session.report({\n", " \"iterations\": step, \"loss\": intermediate_score, \"gain\": intermediate_score * width\n", " })" ] }, { "cell_type": "markdown", "id": "338e5108", "metadata": {}, "source": [ "We define the `OptunaSearch` object this time with metric and mode as list arguments." ] }, { "cell_type": "code", "execution_count": 23, "id": "624d0bc8", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "\u001b[32m[I 2022-07-22 15:26:50,680]\u001b[0m A new study created in memory with name: optuna\u001b[0m\n" ] }, { "data": { "text/html": [ "== Status ==
Current time: 2022-07-22 15:27:34 (running for 00:00:43.55)
Memory usage on this node: 8.2/16.0 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/5.07 GiB heap, 0.0/2.0 GiB objects
Result logdir: /Users/kai/ray_results/multi_objective_2022-07-22_15-26-50
Number of trials: 10/10 (10 TERMINATED)
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
Trial name status loc activation height width iter total time (s) iterations loss gain
multi_objective_5378f1e8TERMINATED127.0.0.1:46621tanh -47.523610.9256 100 10.7064 99-4.66076-50.9216
multi_objective_550962f4TERMINATED127.0.0.1:46631relu -40.6828 4.34953 100 11.3948 99 6.15869 26.7874
multi_objective_550ad9a4TERMINATED127.0.0.1:46632tanh -42.0563 2.02124 100 11.2755 99-3.72967 -7.53857
multi_objective_550c4faaTERMINATED127.0.0.1:46633relu -56.2838 5.65237 100 11.407 99 4.54719 25.7024
multi_objective_5b6c4ba2TERMINATED127.0.0.1:46679relu 54.913316.9039 100 10.6698 9915.5507 262.867
multi_objective_5d51f2d2TERMINATED127.0.0.1:46686tanh 16.5793 5.16709 100 10.7451 99 1.84967 9.55744
multi_objective_5d636760TERMINATED127.0.0.1:46689tanh -13.889513.6043 100 10.6897 99-1.31525-17.8931
multi_objective_5d66093eTERMINATED127.0.0.1:46690relu -51.671419.8211 100 10.7246 99 4.88357 96.7974
multi_objective_634c9444TERMINATED127.0.0.1:46710relu -12.338415.1097 100 13.1986 99 8.83256133.457
multi_objective_65469a2eTERMINATED127.0.0.1:46719tanh 19.677413.6172 100 10.7279 99 2.04137 27.7978


" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "Result for multi_objective_5378f1e8:\n", " date: 2022-07-22_15-26-53\n", " done: false\n", " experiment_id: 4ce03cab420a449a9f4827c3b6b76c38\n", " gain: 57.33359096413655\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " loss: 5.247638036988612\n", " node_ip: 127.0.0.1\n", " pid: 46621\n", " time_since_restore: 0.10418081283569336\n", " time_this_iter_s: 0.10418081283569336\n", " time_total_s: 0.10418081283569336\n", " timestamp: 1658500013\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 5378f1e8\n", " warmup_time: 0.002785921096801758\n", " \n", "Result for multi_objective_550ad9a4:\n", " date: 2022-07-22_15-26-56\n", " done: false\n", " experiment_id: 102a4b06449b428b9870a75d02812be6\n", " gain: 11.711822825618256\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " loss: 5.794371166191576\n", " node_ip: 127.0.0.1\n", " pid: 46632\n", " time_since_restore: 0.10288000106811523\n", " time_this_iter_s: 0.10288000106811523\n", " time_total_s: 0.10288000106811523\n", " timestamp: 1658500016\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 550ad9a4\n", " warmup_time: 0.0030989646911621094\n", " \n", "Result for multi_objective_550c4faa:\n", " date: 2022-07-22_15-26-56\n", " done: false\n", " experiment_id: 538d9621144b4ecea8c91ec7b70c8a9d\n", " gain: 81.23378320141829\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " loss: 14.371624912716872\n", " node_ip: 127.0.0.1\n", " pid: 46633\n", " time_since_restore: 0.10215306282043457\n", " time_this_iter_s: 0.10215306282043457\n", " time_total_s: 0.10215306282043457\n", " timestamp: 1658500016\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 550c4faa\n", " warmup_time: 0.0036542415618896484\n", " \n", "Result for multi_objective_550962f4:\n", " date: 2022-07-22_15-26-56\n", " done: false\n", " experiment_id: b916abdd07bd4e7cb4e9a1451cd76d64\n", " gain: 69.29543163212045\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " loss: 15.93172358423837\n", " node_ip: 127.0.0.1\n", " pid: 46631\n", " time_since_restore: 0.10244321823120117\n", " time_this_iter_s: 0.10244321823120117\n", " time_total_s: 0.10244321823120117\n", " timestamp: 1658500016\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 550962f4\n", " warmup_time: 0.003384828567504883\n", " \n", "Result for multi_objective_5378f1e8:\n", " date: 2022-07-22_15-26-58\n", " done: false\n", " experiment_id: 4ce03cab420a449a9f4827c3b6b76c38\n", " gain: -49.83538614085625\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " loss: -4.561341152769549\n", " node_ip: 127.0.0.1\n", " pid: 46621\n", " time_since_restore: 5.1368408203125\n", " time_this_iter_s: 0.1180417537689209\n", " time_total_s: 5.1368408203125\n", " timestamp: 1658500018\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: 5378f1e8\n", " warmup_time: 0.002785921096801758\n", " \n", "Result for multi_objective_550ad9a4:\n", " date: 2022-07-22_15-27-01\n", " done: false\n", " experiment_id: 102a4b06449b428b9870a75d02812be6\n", " gain: -6.575568930702184\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " loss: -3.2532328725144097\n", " node_ip: 127.0.0.1\n", " pid: 46632\n", " time_since_restore: 5.146106719970703\n", " time_this_iter_s: 0.10509181022644043\n", " time_total_s: 5.146106719970703\n", " timestamp: 1658500021\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: 550ad9a4\n", " warmup_time: 0.0030989646911621094\n", " \n", "Result for multi_objective_550c4faa:\n", " date: 2022-07-22_15-27-01\n", " done: false\n", " experiment_id: 538d9621144b4ecea8c91ec7b70c8a9d\n", " gain: 26.760529995257745\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " loss: 4.73438862995905\n", " node_ip: 127.0.0.1\n", " pid: 46633\n", " time_since_restore: 5.1661481857299805\n", " time_this_iter_s: 0.10656309127807617\n", " time_total_s: 5.1661481857299805\n", " timestamp: 1658500021\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: 550c4faa\n", " warmup_time: 0.0036542415618896484\n", " \n", "Result for multi_objective_550962f4:\n", " date: 2022-07-22_15-27-01\n", " done: false\n", " experiment_id: b916abdd07bd4e7cb4e9a1451cd76d64\n", " gain: 27.828615153405433\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " loss: 6.398081286367129\n", " node_ip: 127.0.0.1\n", " pid: 46631\n", " time_since_restore: 5.169018983840942\n", " time_this_iter_s: 0.10616683959960938\n", " time_total_s: 5.169018983840942\n", " timestamp: 1658500021\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: 550962f4\n", " warmup_time: 0.003384828567504883\n", " \n", "Result for multi_objective_5378f1e8:\n", " date: 2022-07-22_15-27-03\n", " done: false\n", " experiment_id: 4ce03cab420a449a9f4827c3b6b76c38\n", " gain: -50.86883186480195\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " loss: -4.65593053743818\n", " node_ip: 127.0.0.1\n", " pid: 46621\n", " time_since_restore: 10.17048692703247\n", " time_this_iter_s: 0.10630917549133301\n", " time_total_s: 10.17048692703247\n", " timestamp: 1658500023\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: 5378f1e8\n", " warmup_time: 0.002785921096801758\n", " \n", "Result for multi_objective_5378f1e8:\n", " date: 2022-07-22_15-27-04\n", " done: true\n", " experiment_id: 4ce03cab420a449a9f4827c3b6b76c38\n", " experiment_tag: 1_activation=tanh,height=-47.5236,steps=100,width=10.9256\n", " gain: -50.92155508732405\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " loss: -4.66075619693878\n", " node_ip: 127.0.0.1\n", " pid: 46621\n", " time_since_restore: 10.706397771835327\n", " time_this_iter_s: 0.10694766044616699\n", " time_total_s: 10.706397771835327\n", " timestamp: 1658500024\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 5378f1e8\n", " warmup_time: 0.002785921096801758\n", " \n", "Result for multi_objective_550ad9a4:\n", " date: 2022-07-22_15-27-05\n", " done: false\n", " experiment_id: 102a4b06449b428b9870a75d02812be6\n", " gain: -7.479635489825483\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 93\n", " iterations_since_restore: 94\n", " loss: -3.700515697784228\n", " node_ip: 127.0.0.1\n", " pid: 46632\n", " time_since_restore: 10.056212902069092\n", " time_this_iter_s: 0.10589313507080078\n", " time_total_s: 10.056212902069092\n", " timestamp: 1658500025\n", " timesteps_since_restore: 0\n", " training_iteration: 94\n", " trial_id: 550ad9a4\n", " warmup_time: 0.0030989646911621094\n", " \n", "Result for multi_objective_550c4faa:\n", " date: 2022-07-22_15-27-05\n", " done: false\n", " experiment_id: 538d9621144b4ecea8c91ec7b70c8a9d\n", " gain: 25.776502806907246\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 92\n", " iterations_since_restore: 93\n", " loss: 4.5602976409942295\n", " node_ip: 127.0.0.1\n", " pid: 46633\n", " time_since_restore: 9.99274206161499\n", " time_this_iter_s: 0.10583209991455078\n", " time_total_s: 9.99274206161499\n", " timestamp: 1658500025\n", " timesteps_since_restore: 0\n", " training_iteration: 93\n", " trial_id: 550c4faa\n", " warmup_time: 0.0036542415618896484\n", " \n", "Result for multi_objective_550962f4:\n", " date: 2022-07-22_15-27-05\n", " done: false\n", " experiment_id: b916abdd07bd4e7cb4e9a1451cd76d64\n", " gain: 26.860636112316993\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 92\n", " iterations_since_restore: 93\n", " loss: 6.175533072801938\n", " node_ip: 127.0.0.1\n", " pid: 46631\n", " time_since_restore: 9.971143007278442\n", " time_this_iter_s: 0.1054527759552002\n", " time_total_s: 9.971143007278442\n", " timestamp: 1658500025\n", " timesteps_since_restore: 0\n", " training_iteration: 93\n", " trial_id: 550962f4\n", " warmup_time: 0.003384828567504883\n", " \n", "Result for multi_objective_5b6c4ba2:\n", " date: 2022-07-22_15-27-06\n", " done: false\n", " experiment_id: f143c125a0734166ac832270ac08dbb6\n", " gain: 430.90195437776714\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " loss: 25.491329340230966\n", " node_ip: 127.0.0.1\n", " pid: 46679\n", " time_since_restore: 0.10215902328491211\n", " time_this_iter_s: 0.10215902328491211\n", " time_total_s: 0.10215902328491211\n", " timestamp: 1658500026\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 5b6c4ba2\n", " warmup_time: 0.002663135528564453\n", " \n", "Result for multi_objective_550ad9a4:\n", " date: 2022-07-22_15-27-07\n", " done: true\n", " experiment_id: 102a4b06449b428b9870a75d02812be6\n", " experiment_tag: 3_activation=tanh,height=-42.0563,steps=100,width=2.0212\n", " gain: -7.538566547680232\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " loss: -3.729671571085163\n", " node_ip: 127.0.0.1\n", " pid: 46632\n", " time_since_restore: 11.27550482749939\n", " time_this_iter_s: 0.10642290115356445\n", " time_total_s: 11.27550482749939\n", " timestamp: 1658500027\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 550ad9a4\n", " warmup_time: 0.0030989646911621094\n", " \n", "Result for multi_objective_550962f4:\n", " date: 2022-07-22_15-27-07\n", " done: true\n", " experiment_id: b916abdd07bd4e7cb4e9a1451cd76d64\n", " experiment_tag: 2_activation=relu,height=-40.6828,steps=100,width=4.3495\n", " gain: 26.787356208375382\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " loss: 6.15868527856241\n", " node_ip: 127.0.0.1\n", " pid: 46631\n", " time_since_restore: 11.394789934158325\n", " time_this_iter_s: 0.10491585731506348\n", " time_total_s: 11.394789934158325\n", " timestamp: 1658500027\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 550962f4\n", " warmup_time: 0.003384828567504883\n", " \n", "Result for multi_objective_550c4faa:\n", " date: 2022-07-22_15-27-07\n", " done: true\n", " experiment_id: 538d9621144b4ecea8c91ec7b70c8a9d\n", " experiment_tag: 4_activation=relu,height=-56.2838,steps=100,width=5.6524\n", " gain: 25.702421216002982\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " loss: 4.547191359402326\n", " node_ip: 127.0.0.1\n", " pid: 46633\n", " time_since_restore: 11.406961917877197\n", " time_this_iter_s: 0.12177181243896484\n", " time_total_s: 11.406961917877197\n", " timestamp: 1658500027\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 550c4faa\n", " warmup_time: 0.0036542415618896484\n", " \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Result for multi_objective_5d51f2d2:\n", " date: 2022-07-22_15-27-09\n", " done: false\n", " experiment_id: 1433194530b14db4b77ce40a69d65407\n", " gain: 60.23765295445784\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " loss: 11.657934184433229\n", " node_ip: 127.0.0.1\n", " pid: 46686\n", " time_since_restore: 0.10274410247802734\n", " time_this_iter_s: 0.10274410247802734\n", " time_total_s: 0.10274410247802734\n", " timestamp: 1658500029\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 5d51f2d2\n", " warmup_time: 0.0028769969940185547\n", " \n", "Result for multi_objective_5d636760:\n", " date: 2022-07-22_15-27-10\n", " done: false\n", " experiment_id: b8dccac5d8914569b1c2ca061ad47a56\n", " gain: 117.14725547845826\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " loss: 8.611046741256526\n", " node_ip: 127.0.0.1\n", " pid: 46689\n", " time_since_restore: 0.10460686683654785\n", " time_this_iter_s: 0.10460686683654785\n", " time_total_s: 0.10460686683654785\n", " timestamp: 1658500030\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 5d636760\n", " warmup_time: 0.0028090476989746094\n", " \n", "Result for multi_objective_5d66093e:\n", " date: 2022-07-22_15-27-10\n", " done: false\n", " experiment_id: e7259e986278458899b46b485e069ec9\n", " gain: 294.00300634729234\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " loss: 14.832864636325777\n", " node_ip: 127.0.0.1\n", " pid: 46690\n", " time_since_restore: 0.10294318199157715\n", " time_this_iter_s: 0.10294318199157715\n", " time_total_s: 0.10294318199157715\n", " timestamp: 1658500030\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 5d66093e\n", " warmup_time: 0.0027511119842529297\n", " \n", "Result for multi_objective_5b6c4ba2:\n", " date: 2022-07-22_15-27-11\n", " done: false\n", " experiment_id: f143c125a0734166ac832270ac08dbb6\n", " gain: 263.92128797150474\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 48\n", " iterations_since_restore: 49\n", " loss: 15.6130748612977\n", " node_ip: 127.0.0.1\n", " pid: 46679\n", " time_since_restore: 5.2078468799591064\n", " time_this_iter_s: 0.10803008079528809\n", " time_total_s: 5.2078468799591064\n", " timestamp: 1658500031\n", " timesteps_since_restore: 0\n", " training_iteration: 49\n", " trial_id: 5b6c4ba2\n", " warmup_time: 0.002663135528564453\n", " \n", "Result for multi_objective_5d51f2d2:\n", " date: 2022-07-22_15-27-14\n", " done: false\n", " experiment_id: 1433194530b14db4b77ce40a69d65407\n", " gain: 10.610216987384797\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " loss: 2.053420165872076\n", " node_ip: 127.0.0.1\n", " pid: 46686\n", " time_since_restore: 5.178762912750244\n", " time_this_iter_s: 0.10710978507995605\n", " time_total_s: 5.178762912750244\n", " timestamp: 1658500034\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: 5d51f2d2\n", " warmup_time: 0.0028769969940185547\n", " \n", "Result for multi_objective_5d636760:\n", " date: 2022-07-22_15-27-15\n", " done: false\n", " experiment_id: b8dccac5d8914569b1c2ca061ad47a56\n", " gain: -16.800839364465826\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " loss: -1.234965449842409\n", " node_ip: 127.0.0.1\n", " pid: 46689\n", " time_since_restore: 5.146665811538696\n", " time_this_iter_s: 0.10526585578918457\n", " time_total_s: 5.146665811538696\n", " timestamp: 1658500035\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: 5d636760\n", " warmup_time: 0.0028090476989746094\n", " \n", "Result for multi_objective_5d66093e:\n", " date: 2022-07-22_15-27-15\n", " done: false\n", " experiment_id: e7259e986278458899b46b485e069ec9\n", " gain: 97.89753254610552\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " loss: 4.939068026982622\n", " node_ip: 127.0.0.1\n", " pid: 46690\n", " time_since_restore: 5.1532862186431885\n", " time_this_iter_s: 0.10752320289611816\n", " time_total_s: 5.1532862186431885\n", " timestamp: 1658500035\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: 5d66093e\n", " warmup_time: 0.0027511119842529297\n", " \n", "Result for multi_objective_5b6c4ba2:\n", " date: 2022-07-22_15-27-16\n", " done: false\n", " experiment_id: f143c125a0734166ac832270ac08dbb6\n", " gain: 262.90943552209455\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 95\n", " iterations_since_restore: 96\n", " loss: 15.553215620072208\n", " node_ip: 127.0.0.1\n", " pid: 46679\n", " time_since_restore: 10.240519762039185\n", " time_this_iter_s: 0.10443782806396484\n", " time_total_s: 10.240519762039185\n", " timestamp: 1658500036\n", " timesteps_since_restore: 0\n", " training_iteration: 96\n", " trial_id: 5b6c4ba2\n", " warmup_time: 0.002663135528564453\n", " \n", "Result for multi_objective_5b6c4ba2:\n", " date: 2022-07-22_15-27-17\n", " done: true\n", " experiment_id: f143c125a0734166ac832270ac08dbb6\n", " experiment_tag: 5_activation=relu,height=54.9133,steps=100,width=16.9039\n", " gain: 262.86741923011425\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " loss: 15.550730017197484\n", " node_ip: 127.0.0.1\n", " pid: 46679\n", " time_since_restore: 10.669770956039429\n", " time_this_iter_s: 0.10598993301391602\n", " time_total_s: 10.669770956039429\n", " timestamp: 1658500037\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 5b6c4ba2\n", " warmup_time: 0.002663135528564453\n", " \n", "Result for multi_objective_634c9444:\n", " date: 2022-07-22_15-27-19\n", " done: false\n", " experiment_id: 1c6a5d9e38394f1ca770da6a92b6a0ee\n", " gain: 283.55076351521427\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " loss: 18.766157678691147\n", " node_ip: 127.0.0.1\n", " pid: 46710\n", " time_since_restore: 0.10465264320373535\n", " time_this_iter_s: 0.10465264320373535\n", " time_total_s: 0.10465264320373535\n", " timestamp: 1658500039\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 634c9444\n", " warmup_time: 0.004335880279541016\n", " \n", "Result for multi_objective_5d51f2d2:\n", " date: 2022-07-22_15-27-20\n", " done: false\n", " experiment_id: 1433194530b14db4b77ce40a69d65407\n", " gain: 9.60907229177291\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " loss: 1.8596662860626851\n", " node_ip: 127.0.0.1\n", " pid: 46686\n", " time_since_restore: 10.218497037887573\n", " time_this_iter_s: 0.10553288459777832\n", " time_total_s: 10.218497037887573\n", " timestamp: 1658500040\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: 5d51f2d2\n", " warmup_time: 0.0028769969940185547\n", " \n", "Result for multi_objective_5d636760:\n", " date: 2022-07-22_15-27-20\n", " done: false\n", " experiment_id: b8dccac5d8914569b1c2ca061ad47a56\n", " gain: -17.840160182810624\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " loss: -1.311361948500235\n", " node_ip: 127.0.0.1\n", " pid: 46689\n", " time_since_restore: 10.16199278831482\n", " time_this_iter_s: 0.10700488090515137\n", " time_total_s: 10.16199278831482\n", " timestamp: 1658500040\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: 5d636760\n", " warmup_time: 0.0028090476989746094\n", " \n", "Result for multi_objective_5d66093e:\n", " date: 2022-07-22_15-27-20\n", " done: false\n", " experiment_id: e7259e986278458899b46b485e069ec9\n", " gain: 96.85061995053432\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " loss: 4.886249816008858\n", " node_ip: 127.0.0.1\n", " pid: 46690\n", " time_since_restore: 10.197196006774902\n", " time_this_iter_s: 0.10463690757751465\n", " time_total_s: 10.197196006774902\n", " timestamp: 1658500040\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: 5d66093e\n", " warmup_time: 0.0027511119842529297\n", " \n", "Result for multi_objective_5d51f2d2:\n", " date: 2022-07-22_15-27-20\n", " done: true\n", " experiment_id: 1433194530b14db4b77ce40a69d65407\n", " experiment_tag: 6_activation=tanh,height=16.5793,steps=100,width=5.1671\n", " gain: 9.557436804323405\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " loss: 1.8496731491334866\n", " node_ip: 127.0.0.1\n", " pid: 46686\n", " time_since_restore: 10.745142936706543\n", " time_this_iter_s: 0.10509181022644043\n", " time_total_s: 10.745142936706543\n", " timestamp: 1658500040\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 5d51f2d2\n", " warmup_time: 0.0028769969940185547\n", " \n", "Result for multi_objective_5d636760:\n", " date: 2022-07-22_15-27-20\n", " done: true\n", " experiment_id: b8dccac5d8914569b1c2ca061ad47a56\n", " experiment_tag: 7_activation=tanh,height=-13.8895,steps=100,width=13.6043\n", " gain: -17.893079155032332\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " loss: -1.3152518197690004\n", " node_ip: 127.0.0.1\n", " pid: 46689\n", " time_since_restore: 10.689707040786743\n", " time_this_iter_s: 0.10322403907775879\n", " time_total_s: 10.689707040786743\n", " timestamp: 1658500040\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 5d636760\n", " warmup_time: 0.0028090476989746094\n", " \n", "Result for multi_objective_5d66093e:\n", " date: 2022-07-22_15-27-20\n", " done: true\n", " experiment_id: e7259e986278458899b46b485e069ec9\n", " experiment_tag: 8_activation=relu,height=-51.6714,steps=100,width=19.8211\n", " gain: 96.79744897015512\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " loss: 4.8835672653630535\n", " node_ip: 127.0.0.1\n", " pid: 46690\n", " time_since_restore: 10.72464108467102\n", " time_this_iter_s: 0.10328292846679688\n", " time_total_s: 10.72464108467102\n", " timestamp: 1658500040\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 5d66093e\n", " warmup_time: 0.0027511119842529297\n", " \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Result for multi_objective_65469a2e:\n", " date: 2022-07-22_15-27-23\n", " done: false\n", " experiment_id: 015ee29ddb8b4c239d85a022a23d4093\n", " gain: 162.96737941207124\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " loss: 11.96774070054979\n", " node_ip: 127.0.0.1\n", " pid: 46719\n", " time_since_restore: 0.10403180122375488\n", " time_this_iter_s: 0.10403180122375488\n", " time_total_s: 0.10403180122375488\n", " timestamp: 1658500043\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 65469a2e\n", " warmup_time: 0.0033850669860839844\n", " \n", "Result for multi_objective_634c9444:\n", " date: 2022-07-22_15-27-24\n", " done: false\n", " experiment_id: 1c6a5d9e38394f1ca770da6a92b6a0ee\n", " gain: 136.50874589912112\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 24\n", " iterations_since_restore: 25\n", " loss: 9.034518610724309\n", " node_ip: 127.0.0.1\n", " pid: 46710\n", " time_since_restore: 5.125189781188965\n", " time_this_iter_s: 0.11187601089477539\n", " time_total_s: 5.125189781188965\n", " timestamp: 1658500044\n", " timesteps_since_restore: 0\n", " training_iteration: 25\n", " trial_id: 634c9444\n", " warmup_time: 0.004335880279541016\n", " \n", "Result for multi_objective_65469a2e:\n", " date: 2022-07-22_15-27-28\n", " done: false\n", " experiment_id: 015ee29ddb8b4c239d85a022a23d4093\n", " gain: 28.890088364068774\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " loss: 2.1215846238952016\n", " node_ip: 127.0.0.1\n", " pid: 46719\n", " time_since_restore: 5.174077749252319\n", " time_this_iter_s: 0.10734677314758301\n", " time_total_s: 5.174077749252319\n", " timestamp: 1658500048\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: 65469a2e\n", " warmup_time: 0.0033850669860839844\n", " \n", "Result for multi_objective_634c9444:\n", " date: 2022-07-22_15-27-30\n", " done: false\n", " experiment_id: 1c6a5d9e38394f1ca770da6a92b6a0ee\n", " gain: 133.84933935751317\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 71\n", " iterations_since_restore: 72\n", " loss: 8.85851188137237\n", " node_ip: 127.0.0.1\n", " pid: 46710\n", " time_since_restore: 10.204095840454102\n", " time_this_iter_s: 0.10942888259887695\n", " time_total_s: 10.204095840454102\n", " timestamp: 1658500050\n", " timesteps_since_restore: 0\n", " training_iteration: 72\n", " trial_id: 634c9444\n", " warmup_time: 0.004335880279541016\n", " \n", "Result for multi_objective_634c9444:\n", " date: 2022-07-22_15-27-33\n", " done: true\n", " experiment_id: 1c6a5d9e38394f1ca770da6a92b6a0ee\n", " experiment_tag: 9_activation=relu,height=-12.3384,steps=100,width=15.1097\n", " gain: 133.45728949146132\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " loss: 8.83256495915982\n", " node_ip: 127.0.0.1\n", " pid: 46710\n", " time_since_restore: 13.198630809783936\n", " time_this_iter_s: 0.1067969799041748\n", " time_total_s: 13.198630809783936\n", " timestamp: 1658500053\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 634c9444\n", " warmup_time: 0.004335880279541016\n", " \n", "Result for multi_objective_65469a2e:\n", " date: 2022-07-22_15-27-33\n", " done: false\n", " experiment_id: 015ee29ddb8b4c239d85a022a23d4093\n", " gain: 27.850744704436202\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " loss: 2.0452589477867855\n", " node_ip: 127.0.0.1\n", " pid: 46719\n", " time_since_restore: 10.19201397895813\n", " time_this_iter_s: 0.10735702514648438\n", " time_total_s: 10.19201397895813\n", " timestamp: 1658500053\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: 65469a2e\n", " warmup_time: 0.0033850669860839844\n", " \n", "Result for multi_objective_65469a2e:\n", " date: 2022-07-22_15-27-34\n", " done: true\n", " experiment_id: 015ee29ddb8b4c239d85a022a23d4093\n", " experiment_tag: 10_activation=tanh,height=19.6774,steps=100,width=13.6172\n", " gain: 27.797824972416166\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " loss: 2.041372712191397\n", " node_ip: 127.0.0.1\n", " pid: 46719\n", " time_since_restore: 10.727944135665894\n", " time_this_iter_s: 0.10885024070739746\n", " time_total_s: 10.727944135665894\n", " timestamp: 1658500054\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 65469a2e\n", " warmup_time: 0.0033850669860839844\n", " \n" ] } ], "source": [ "searcher = OptunaSearch(metric=[\"loss\", \"gain\"], mode=[\"min\", \"max\"])\n", "algo = ConcurrencyLimiter(searcher, max_concurrent=4)\n", "\n", "tuner = tune.Tuner(\n", " multi_objective,\n", " tune_config=tune.TuneConfig(\n", " search_alg=algo,\n", " num_samples=num_samples,\n", " ),\n", " param_space=search_space\n", ")\n", "results = tuner.fit()" ] }, { "cell_type": "markdown", "id": "df42b8b3", "metadata": {}, "source": [ "Now there are two hyperparameter sets for the two objectives." ] }, { "cell_type": "code", "execution_count": 24, "id": "183fef1a", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Best hyperparameters for loss found were: {'steps': 100, 'width': 10.925599395387751, 'height': -47.52361963011387, 'activation': 'tanh'}\n", "Best hyperparameters for gain found were: {'steps': 100, 'width': 16.90386360893735, 'height': 54.91329340230965, 'activation': 'relu'}\n" ] } ], "source": [ "print(\"Best hyperparameters for loss found were: \", results.get_best_result(\"loss\", \"min\").config)\n", "print(\"Best hyperparameters for gain found were: \", results.get_best_result(\"gain\", \"max\").config)" ] }, { "cell_type": "markdown", "id": "cdf4d49a", "metadata": {}, "source": [ "We can mix-and-match the use of initial hyperparameter evaluations, conditional search spaces via define-by-run functions, and multi-objective tasks. This is also true of scheduler usage, with the exception of multi-objective optimization-- schedulers typically rely on a single scalar score, rather than the two scores we use here: loss, gain." ] }, { "cell_type": "code", "execution_count": 25, "id": "a058fdb3", "metadata": { "tags": [ "remove-cell" ] }, "outputs": [], "source": [ "ray.shutdown()" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.7" }, "orphan": true }, "nbformat": 4, "nbformat_minor": 5 }