{ "cells": [ { "cell_type": "markdown", "id": "d3d9d39e", "metadata": {}, "source": [ "# Running Tune experiments with BlendSearch and CFO\n", "\n", "In this tutorial we introduce BlendSearch and CFO, while running a simple Ray Tune\n", "experiment. Tune’s Search Algorithms integrate with FLAML and, as a result, allow\n", "you to seamlessly scale up a BlendSearch and CFO optimization\n", "process - without sacrificing performance.\n", "\n", "Fast Library for Automated Machine Learning & Tuning (FLAML) does not rely on the\n", "gradient of the objective function, but instead, learns from samples of the\n", "search space. It is suitable for optimizing functions that are non-differentiable,\n", "with many local minima, or even unknown but only testable. Therefore, it is\n", "necessarily belongs to the domain of \"derivative-free optimization\"\n", "and \"black-box optimization\".\n", "\n", "FLAML has two primary algorithms: (1) Frugal Optimization for Cost-related\n", "Hyperparameters (CFO) begins with a low-cost initial point and gradually moves to\n", "a high-cost region as needed. It is a local search method that leverages randomized\n", "direct search method with an adaptive step-size and random restarts.\n", "As a local search method, it has an appealing provable convergence rate and bounded\n", "cost but may get trapped in suboptimal local minima. (2) Economical Hyperparameter\n", "Optimization With Blended Search Strategy (BlendSearch) combines CFO's local search\n", "with global search, making it less suspectable to local minima traps.\n", "It leverages the frugality of CFO and the space exploration ability of global search\n", "methods such as Bayesian optimization.\n", "\n", "In this example we minimize a simple objective to briefly demonstrate the usage of\n", "FLAML with Ray Tune via `BlendSearch` and `CFO`. It's useful to keep in mind that\n", "despite the emphasis on machine learning experiments, Ray Tune optimizes any implicit\n", "or explicit objective. Here we assume `flaml==0.4.1` and `optuna==2.9.1` libraries\n", "are installed. To learn more, please refer to\n", "the [FLAML website](https://github.com/microsoft/FLAML/tree/main/flaml/tune).\n", " \n", "Click below to see all the imports we need for this example.\n", "You can also launch directly into a Binder instance to run this notebook yourself.\n", "Just click on the rocket symbol at the top of the navigation." ] }, { "cell_type": "code", "execution_count": 1, "id": "1a9f10f0", "metadata": { "tags": [ "hide-input" ] }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/flaml/searcher/blendsearch.py:14: DeprecationWarning: The module `ray.tune.suggest` has been moved to `ray.tune.search` and the old location will be deprecated soon. Please adjust your imports to point to the new location. Example: Do a global search and replace `ray.tune.suggest` with `ray.tune.search`.\n", " from ray.tune.suggest import Searcher\n", "/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/flaml/searcher/blendsearch.py:15: DeprecationWarning: The module `ray.tune.suggest.optuna` has been moved to `ray.tune.search.optuna` and the old location will be deprecated soon. Please adjust your imports to point to the new location. Example: Do a global search and replace `ray.tune.suggest.optuna` with `ray.tune.search.optuna`.\n", " from ray.tune.suggest.optuna import OptunaSearch as GlobalSearch\n", "/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/flaml/tune/sample.py:39: DeprecationWarning: The module `ray.tune.sample` has been moved to `ray.tune.search.sample` and the old location will be deprecated soon. Please adjust your imports to point to the new location. Example: Do a global search and replace `ray.tune.sample` with `ray.tune.search.sample`.\n", " from ray.tune.sample import _BackwardsCompatibleNumpyRng\n", "/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/flaml/tune/space.py:6: DeprecationWarning: The module `ray.tune.suggest.variant_generator` has been moved to `ray.tune.search.variant_generator` and the old location will be deprecated soon. Please adjust your imports to point to the new location. Example: Do a global search and replace `ray.tune.suggest.variant_generator` with `ray.tune.search.variant_generator`.\n", " from ray.tune.suggest.variant_generator import generate_variants\n" ] } ], "source": [ "import time\n", "\n", "import ray\n", "from ray import tune\n", "from ray.air import session\n", "from ray.tune.search import ConcurrencyLimiter\n", "from ray.tune.search.flaml import BlendSearch, CFO" ] }, { "cell_type": "markdown", "id": "f5a4bce8", "metadata": {}, "source": [ "Let's start by defining a simple evaluation function.\n", "We artificially sleep for a bit (`0.1` seconds) to simulate a long-running ML experiment.\n", "This setup assumes that we're running multiple `step`s of an experiment and try to\n", "tune three hyperparameters, namely `width` and `height`, and `activation`." ] }, { "cell_type": "code", "execution_count": 2, "id": "9b80ac32", "metadata": {}, "outputs": [], "source": [ "def evaluate(step, width, height, activation):\n", " time.sleep(0.1)\n", " activation_boost = 10 if activation==\"relu\" else 1\n", " return (0.1 + width * step / 100) ** (-1) + height * 0.1 + activation_boost" ] }, { "cell_type": "markdown", "id": "391dcd0e", "metadata": {}, "source": [ "Next, our `objective` function takes a Tune `config`, evaluates the `score` of your\n", "experiment in a training loop, and uses `session.report` to report the `score` back to Tune." ] }, { "cell_type": "code", "execution_count": 3, "id": "0773e711", "metadata": {}, "outputs": [], "source": [ "def objective(config):\n", " for step in range(config[\"steps\"]):\n", " score = evaluate(step, config[\"width\"], config[\"height\"], config[\"activation\"])\n", " session.report({\"iterations\": step, \"mean_loss\": score})" ] }, { "cell_type": "code", "execution_count": 4, "id": "8dde8596", "metadata": { "lines_to_next_cell": 0, "tags": [ "remove-cell" ] }, "outputs": [ { "data": { "text/html": [ "
\n", "
\n", "

Ray

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", "\n", "
Python version:3.7.7
Ray version: 2.0.0rc0
Dashboard:http://127.0.0.1:8265
\n", "
\n", "
\n" ], "text/plain": [ "RayContext(dashboard_url='127.0.0.1:8265', python_version='3.7.7', ray_version='2.0.0rc0', ray_commit='{{RAY_COMMIT_SHA}}', address_info={'node_ip_address': '127.0.0.1', 'raylet_ip_address': '127.0.0.1', 'redis_address': None, 'object_store_address': '/tmp/ray/session_2022-07-22_15-18-05_176636_45728/sockets/plasma_store', 'raylet_socket_name': '/tmp/ray/session_2022-07-22_15-18-05_176636_45728/sockets/raylet', 'webui_url': '127.0.0.1:8265', 'session_dir': '/tmp/ray/session_2022-07-22_15-18-05_176636_45728', 'metrics_export_port': 63113, 'gcs_address': '127.0.0.1:63688', 'address': '127.0.0.1:63688', 'dashboard_agent_listen_port': 52365, 'node_id': '35eafee40228cabb8ae4ddafa3e39f41262af0440bf69a0d776f8fe8'})" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "ray.init(configure_logging=False)" ] }, { "cell_type": "markdown", "id": "a1007f97", "metadata": {}, "source": [ "## Running Tune experiments with BlendSearch\n", "\n", "This example demonstrates the usage of Economical Hyperparameter Optimization\n", "With Blended Search Strategy (BlendSearch) with Ray Tune.\n", "\n", "Now we define the search algorithm built from `BlendSearch`, constrained to a\n", "maximum of `4` concurrent trials with a `ConcurrencyLimiter`." ] }, { "cell_type": "code", "execution_count": 5, "id": "37b5070e", "metadata": {}, "outputs": [], "source": [ "algo = BlendSearch()\n", "algo = ConcurrencyLimiter(algo, max_concurrent=4)" ] }, { "cell_type": "markdown", "id": "6bb24c73", "metadata": {}, "source": [ "The number of samples this Tune run is set to `1000`.\n", "(you can decrease this if it takes too long on your machine)." ] }, { "cell_type": "code", "execution_count": 6, "id": "f219c33a", "metadata": {}, "outputs": [], "source": [ "num_samples = 1000" ] }, { "cell_type": "code", "execution_count": 7, "id": "7ad72495", "metadata": { "tags": [ "remove-cell" ] }, "outputs": [], "source": [ "# If 1000 samples take too long, you can reduce this number.\n", "# We override this number here for our smoke tests.\n", "num_samples = 10" ] }, { "cell_type": "markdown", "id": "6a206833", "metadata": {}, "source": [ "Next we define a search space. The critical assumption is that the optimal\n", "hyperparameters live within this space. Yet, if the space is very large, then those\n", "hyperparameters may be difficult to find in a short amount of time." ] }, { "cell_type": "code", "execution_count": 8, "id": "f2236f81", "metadata": {}, "outputs": [], "source": [ "search_config = {\n", " \"steps\": 100,\n", " \"width\": tune.uniform(0, 20),\n", " \"height\": tune.uniform(-100, 100),\n", " \"activation\": tune.choice([\"relu, tanh\"])\n", "}" ] }, { "cell_type": "markdown", "id": "f77303bc", "metadata": {}, "source": [ "Finally, we run the experiment to `\"min\"`imize the \"mean_loss\" of the `objective` by\n", "searching `search_config` via `algo`, `num_samples` times. This previous sentence is\n", "fully characterizes the search problem we aim to solve. With this in mind, observe\n", "how efficient it is to execute `tuner.fit()`." ] }, { "cell_type": "code", "execution_count": 9, "id": "7a95373a", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Function checkpointing is disabled. This may result in unexpected behavior when using checkpointing features or certain schedulers. To enable, set the train function arguments to be `func(config, checkpoint_dir=None)`.\n", "\u001b[32m[I 2022-07-22 15:18:10,659]\u001b[0m A new study created in memory with name: optuna\u001b[0m\n" ] }, { "data": { "text/html": [ "== Status ==
Current time: 2022-07-22 15:18:55 (running for 00:00:43.79)
Memory usage on this node: 10.0/16.0 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/4.69 GiB heap, 0.0/2.0 GiB objects
Current best trial: 1fc231f8 with mean_loss=-8.519908298508359 and parameters={'steps': 100, 'width': 15.42641286533492, 'height': -95.8496101281197, 'activation': 'relu, tanh'}
Result logdir: /Users/kai/ray_results/blendsearch_exp
Number of trials: 10/10 (10 TERMINATED)
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
Trial name status loc activation height steps width loss iter total time (s) iterations neg_mean_loss
objective_1e0f814eTERMINATED127.0.0.1:45801relu, tanh 29.4532 100 1.94864 4.43814 100 11.061 99 -4.43814
objective_1fc231f8TERMINATED127.0.0.1:45809relu, tanh -95.8496 10015.4264 -8.51991 100 11.6807 99 8.51991
objective_1fc3b668TERMINATED127.0.0.1:45810relu, tanh 49.7608 10012.673 6.05515 100 11.7372 99 -6.05515
objective_1fc58b1eTERMINATED127.0.0.1:45811relu, tanh -55.0407 100 9.97014 -4.40377 100 11.6335 99 4.40377
objective_265c09eeTERMINATED127.0.0.1:45837relu, tanh 52.1061 100 3.96126 6.45927 100 10.7539 99 -6.45927
objective_28587494TERMINATED127.0.0.1:45842relu, tanh -82.332 100 3.38222 -6.94321 100 10.7116 99 6.94321
objective_28682bfaTERMINATED127.0.0.1:45845relu, tanh -94.9771 10017.681 -8.4409 100 10.7471 99 8.4409
objective_28788388TERMINATED127.0.0.1:45848relu, tanh 90.6787 10013.7072 10.141 100 10.7525 99 -10.141
objective_2e3cd63eTERMINATED127.0.0.1:45864relu, tanh 2.43845 100 0.0789653 6.85628 100 10.7006 99 -6.85628
objective_3046455aTERMINATED127.0.0.1:45869relu, tanh 22.5052 10016.2524 3.31229 100 10.7176 99 -3.31229


" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "Result for objective_1e0f814e:\n", " date: 2022-07-22_15-18-14\n", " done: false\n", " experiment_id: 623da6c1aa28400a9d85da19bdc5721d\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 13.945321626643842\n", " neg_mean_loss: -13.945321626643842\n", " node_ip: 127.0.0.1\n", " pid: 45801\n", " time_since_restore: 0.10402607917785645\n", " time_this_iter_s: 0.10402607917785645\n", " time_total_s: 0.10402607917785645\n", " timestamp: 1658499494\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 1e0f814e\n", " warmup_time: 0.003920078277587891\n", " \n", "Result for objective_1fc58b1e:\n", " date: 2022-07-22_15-18-17\n", " done: false\n", " experiment_id: da508e209a1946e89ec15a3f1a5ce5ef\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 5.495932910616953\n", " neg_mean_loss: -5.495932910616953\n", " node_ip: 127.0.0.1\n", " pid: 45811\n", " time_since_restore: 0.10269379615783691\n", " time_this_iter_s: 0.10269379615783691\n", " time_total_s: 0.10269379615783691\n", " timestamp: 1658499497\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 1fc58b1e\n", " warmup_time: 0.004099130630493164\n", " \n", "Result for objective_1fc231f8:\n", " date: 2022-07-22_15-18-17\n", " done: false\n", " experiment_id: 24c1fa443be246229e2c2ed9581452de\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 1.4150389871880282\n", " neg_mean_loss: -1.4150389871880282\n", " node_ip: 127.0.0.1\n", " pid: 45809\n", " time_since_restore: 0.10376811027526855\n", " time_this_iter_s: 0.10376811027526855\n", " time_total_s: 0.10376811027526855\n", " timestamp: 1658499497\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 1fc231f8\n", " warmup_time: 0.004148244857788086\n", " \n", "Result for objective_1fc3b668:\n", " date: 2022-07-22_15-18-17\n", " done: false\n", " experiment_id: 5cdd489afb0e43b7a44aa4484765dc92\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 15.976077650772236\n", " neg_mean_loss: -15.976077650772236\n", " node_ip: 127.0.0.1\n", " pid: 45810\n", " time_since_restore: 0.1012420654296875\n", " time_this_iter_s: 0.1012420654296875\n", " time_total_s: 0.1012420654296875\n", " timestamp: 1658499497\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 1fc3b668\n", " warmup_time: 0.002666950225830078\n", " \n", "Result for objective_1e0f814e:\n", " date: 2022-07-22_15-18-19\n", " done: false\n", " experiment_id: 623da6c1aa28400a9d85da19bdc5721d\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 44\n", " iterations_since_restore: 45\n", " mean_loss: 4.989814690087961\n", " neg_mean_loss: -4.989814690087961\n", " node_ip: 127.0.0.1\n", " pid: 45801\n", " time_since_restore: 5.1339499950408936\n", " time_this_iter_s: 0.10842394828796387\n", " time_total_s: 5.1339499950408936\n", " timestamp: 1658499499\n", " timesteps_since_restore: 0\n", " training_iteration: 45\n", " trial_id: 1e0f814e\n", " warmup_time: 0.003920078277587891\n", " \n", "Result for objective_1fc58b1e:\n", " date: 2022-07-22_15-18-22\n", " done: false\n", " experiment_id: da508e209a1946e89ec15a3f1a5ce5ef\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: -4.295122851662673\n", " neg_mean_loss: 4.295122851662673\n", " node_ip: 127.0.0.1\n", " pid: 45811\n", " time_since_restore: 5.167006015777588\n", " time_this_iter_s: 0.10817217826843262\n", " time_total_s: 5.167006015777588\n", " timestamp: 1658499502\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: 1fc58b1e\n", " warmup_time: 0.004099130630493164\n", " \n", "Result for objective_1fc231f8:\n", " date: 2022-07-22_15-18-22\n", " done: false\n", " experiment_id: 24c1fa443be246229e2c2ed9581452de\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: -8.44891425494968\n", " neg_mean_loss: 8.44891425494968\n", " node_ip: 127.0.0.1\n", " pid: 45809\n", " time_since_restore: 5.138737916946411\n", " time_this_iter_s: 0.10868191719055176\n", " time_total_s: 5.138737916946411\n", " timestamp: 1658499502\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: 1fc231f8\n", " warmup_time: 0.004148244857788086\n", " \n", "Result for objective_1fc3b668:\n", " date: 2022-07-22_15-18-22\n", " done: false\n", " experiment_id: 5cdd489afb0e43b7a44aa4484765dc92\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: 6.141195146339472\n", " neg_mean_loss: -6.141195146339472\n", " node_ip: 127.0.0.1\n", " pid: 45810\n", " time_since_restore: 5.181504011154175\n", " time_this_iter_s: 0.10758399963378906\n", " time_total_s: 5.181504011154175\n", " timestamp: 1658499502\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: 1fc3b668\n", " warmup_time: 0.002666950225830078\n", " \n", "Result for objective_1e0f814e:\n", " date: 2022-07-22_15-18-24\n", " done: false\n", " experiment_id: 623da6c1aa28400a9d85da19bdc5721d\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 91\n", " iterations_since_restore: 92\n", " mean_loss: 4.479149291114477\n", " neg_mean_loss: -4.479149291114477\n", " node_ip: 127.0.0.1\n", " pid: 45801\n", " time_since_restore: 10.196602821350098\n", " time_this_iter_s: 0.10757303237915039\n", " time_total_s: 10.196602821350098\n", " timestamp: 1658499504\n", " timesteps_since_restore: 0\n", " training_iteration: 92\n", " trial_id: 1e0f814e\n", " warmup_time: 0.003920078277587891\n", " \n", "Result for objective_1e0f814e:\n", " date: 2022-07-22_15-18-25\n", " done: true\n", " experiment_id: 623da6c1aa28400a9d85da19bdc5721d\n", " experiment_tag: 1_activation=relu_tanh,height=29.4532,steps=100,width=1.9486\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 4.438137591322411\n", " neg_mean_loss: -4.438137591322411\n", " node_ip: 127.0.0.1\n", " pid: 45801\n", " time_since_restore: 11.060971975326538\n", " time_this_iter_s: 0.10748600959777832\n", " time_total_s: 11.060971975326538\n", " timestamp: 1658499505\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 1e0f814e\n", " warmup_time: 0.003920078277587891\n", " \n", "Result for objective_1fc231f8:\n", " date: 2022-07-22_15-18-26\n", " done: false\n", " experiment_id: 24c1fa443be246229e2c2ed9581452de\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 90\n", " iterations_since_restore: 91\n", " mean_loss: -8.513449547227397\n", " neg_mean_loss: 8.513449547227397\n", " node_ip: 127.0.0.1\n", " pid: 45809\n", " time_since_restore: 9.72878909111023\n", " time_this_iter_s: 0.10644912719726562\n", " time_total_s: 9.72878909111023\n", " timestamp: 1658499506\n", " timesteps_since_restore: 0\n", " training_iteration: 91\n", " trial_id: 1fc231f8\n", " warmup_time: 0.004148244857788086\n", " \n", "Result for objective_1fc3b668:\n", " date: 2022-07-22_15-18-26\n", " done: false\n", " experiment_id: 5cdd489afb0e43b7a44aa4484765dc92\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 89\n", " iterations_since_restore: 90\n", " mean_loss: 6.063959309753095\n", " neg_mean_loss: -6.063959309753095\n", " node_ip: 127.0.0.1\n", " pid: 45810\n", " time_since_restore: 9.6911301612854\n", " time_this_iter_s: 0.10597920417785645\n", " time_total_s: 9.6911301612854\n", " timestamp: 1658499506\n", " timesteps_since_restore: 0\n", " training_iteration: 90\n", " trial_id: 1fc3b668\n", " warmup_time: 0.002666950225830078\n", " \n", "Result for objective_1fc58b1e:\n", " date: 2022-07-22_15-18-27\n", " done: false\n", " experiment_id: da508e209a1946e89ec15a3f1a5ce5ef\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 91\n", " iterations_since_restore: 92\n", " mean_loss: -4.395049451529077\n", " neg_mean_loss: 4.395049451529077\n", " node_ip: 127.0.0.1\n", " pid: 45811\n", " time_since_restore: 9.873695850372314\n", " time_this_iter_s: 0.10703802108764648\n", " time_total_s: 9.873695850372314\n", " timestamp: 1658499507\n", " timesteps_since_restore: 0\n", " training_iteration: 92\n", " trial_id: 1fc58b1e\n", " warmup_time: 0.004099130630493164\n", " \n", "Result for objective_265c09ee:\n", " date: 2022-07-22_15-18-28\n", " done: false\n", " experiment_id: a7543e8d696745edbfe36a6f4ebe4071\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 16.210614243979172\n", " neg_mean_loss: -16.210614243979172\n", " node_ip: 127.0.0.1\n", " pid: 45837\n", " time_since_restore: 0.1049339771270752\n", " time_this_iter_s: 0.1049339771270752\n", " time_total_s: 0.1049339771270752\n", " timestamp: 1658499508\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 265c09ee\n", " warmup_time: 0.0026078224182128906\n", " \n", "Result for objective_1fc58b1e:\n", " date: 2022-07-22_15-18-28\n", " done: true\n", " experiment_id: da508e209a1946e89ec15a3f1a5ce5ef\n", " experiment_tag: 4_activation=relu_tanh,height=-55.0407,steps=100,width=9.9701\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: -4.403770601366095\n", " neg_mean_loss: 4.403770601366095\n", " node_ip: 127.0.0.1\n", " pid: 45811\n", " time_since_restore: 11.633522987365723\n", " time_this_iter_s: 0.10734701156616211\n", " time_total_s: 11.633522987365723\n", " timestamp: 1658499508\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 1fc58b1e\n", " warmup_time: 0.004099130630493164\n", " \n", "Result for objective_1fc231f8:\n", " date: 2022-07-22_15-18-28\n", " done: true\n", " experiment_id: 24c1fa443be246229e2c2ed9581452de\n", " experiment_tag: 2_activation=relu_tanh,height=-95.8496,steps=100,width=15.4264\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: -8.519908298508359\n", " neg_mean_loss: 8.519908298508359\n", " node_ip: 127.0.0.1\n", " pid: 45809\n", " time_since_restore: 11.680676937103271\n", " time_this_iter_s: 0.10814285278320312\n", " time_total_s: 11.680676937103271\n", " timestamp: 1658499508\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 1fc231f8\n", " warmup_time: 0.004148244857788086\n", " \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Result for objective_1fc3b668:\n", " date: 2022-07-22_15-18-29\n", " done: true\n", " experiment_id: 5cdd489afb0e43b7a44aa4484765dc92\n", " experiment_tag: 3_activation=relu_tanh,height=49.7608,steps=100,width=12.6730\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 6.055152568795227\n", " neg_mean_loss: -6.055152568795227\n", " node_ip: 127.0.0.1\n", " pid: 45810\n", " time_since_restore: 11.73720097541809\n", " time_this_iter_s: 0.10816287994384766\n", " time_total_s: 11.73720097541809\n", " timestamp: 1658499509\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 1fc3b668\n", " warmup_time: 0.002666950225830078\n", " \n", "Result for objective_28587494:\n", " date: 2022-07-22_15-18-31\n", " done: false\n", " experiment_id: 750bd707fda147f0bd615c32243d64e5\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 2.766796283480206\n", " neg_mean_loss: -2.766796283480206\n", " node_ip: 127.0.0.1\n", " pid: 45842\n", " time_since_restore: 0.10265803337097168\n", " time_this_iter_s: 0.10265803337097168\n", " time_total_s: 0.10265803337097168\n", " timestamp: 1658499511\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: '28587494'\n", " warmup_time: 0.0028159618377685547\n", " \n", "Result for objective_28682bfa:\n", " date: 2022-07-22_15-18-31\n", " done: false\n", " experiment_id: 5487e6a1cf70423096cf6285197fb824\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 1.5022935169945555\n", " neg_mean_loss: -1.5022935169945555\n", " node_ip: 127.0.0.1\n", " pid: 45845\n", " time_since_restore: 0.10289216041564941\n", " time_this_iter_s: 0.10289216041564941\n", " time_total_s: 0.10289216041564941\n", " timestamp: 1658499511\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 28682bfa\n", " warmup_time: 0.003072023391723633\n", " \n", "Result for objective_28788388:\n", " date: 2022-07-22_15-18-31\n", " done: false\n", " experiment_id: e959dae493a240c489a3193352c46f3a\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 20.067866923898734\n", " neg_mean_loss: -20.067866923898734\n", " node_ip: 127.0.0.1\n", " pid: 45848\n", " time_since_restore: 0.1038503646850586\n", " time_this_iter_s: 0.1038503646850586\n", " time_total_s: 0.1038503646850586\n", " timestamp: 1658499511\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: '28788388'\n", " warmup_time: 0.0027070045471191406\n", " \n", "Result for objective_265c09ee:\n", " date: 2022-07-22_15-18-33\n", " done: false\n", " experiment_id: a7543e8d696745edbfe36a6f4ebe4071\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: 6.720352557756592\n", " neg_mean_loss: -6.720352557756592\n", " node_ip: 127.0.0.1\n", " pid: 45837\n", " time_since_restore: 5.148331165313721\n", " time_this_iter_s: 0.10472607612609863\n", " time_total_s: 5.148331165313721\n", " timestamp: 1658499513\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: 265c09ee\n", " warmup_time: 0.0026078224182128906\n", " \n", "Result for objective_28587494:\n", " date: 2022-07-22_15-18-36\n", " done: false\n", " experiment_id: 750bd707fda147f0bd615c32243d64e5\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: -6.641362320132167\n", " neg_mean_loss: 6.641362320132167\n", " node_ip: 127.0.0.1\n", " pid: 45842\n", " time_since_restore: 5.175055980682373\n", " time_this_iter_s: 0.10797286033630371\n", " time_total_s: 5.175055980682373\n", " timestamp: 1658499516\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: '28587494'\n", " warmup_time: 0.0028159618377685547\n", " \n", "Result for objective_28682bfa:\n", " date: 2022-07-22_15-18-36\n", " done: false\n", " experiment_id: 5487e6a1cf70423096cf6285197fb824\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: -8.378801379858526\n", " neg_mean_loss: 8.378801379858526\n", " node_ip: 127.0.0.1\n", " pid: 45845\n", " time_since_restore: 5.192234039306641\n", " time_this_iter_s: 0.10880398750305176\n", " time_total_s: 5.192234039306641\n", " timestamp: 1658499516\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: 28682bfa\n", " warmup_time: 0.003072023391723633\n", " \n", "Result for objective_28788388:\n", " date: 2022-07-22_15-18-36\n", " done: false\n", " experiment_id: e959dae493a240c489a3193352c46f3a\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: 10.22071644495505\n", " neg_mean_loss: -10.22071644495505\n", " node_ip: 127.0.0.1\n", " pid: 45848\n", " time_since_restore: 5.175441265106201\n", " time_this_iter_s: 0.10689902305603027\n", " time_total_s: 5.175441265106201\n", " timestamp: 1658499516\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: '28788388'\n", " warmup_time: 0.0027070045471191406\n", " \n", "Result for objective_265c09ee:\n", " date: 2022-07-22_15-18-38\n", " done: false\n", " experiment_id: a7543e8d696745edbfe36a6f4ebe4071\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " mean_loss: 6.472149118155721\n", " neg_mean_loss: -6.472149118155721\n", " node_ip: 127.0.0.1\n", " pid: 45837\n", " time_since_restore: 10.2187819480896\n", " time_this_iter_s: 0.10822892189025879\n", " time_total_s: 10.2187819480896\n", " timestamp: 1658499518\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: 265c09ee\n", " warmup_time: 0.0026078224182128906\n", " \n", "Result for objective_265c09ee:\n", " date: 2022-07-22_15-18-38\n", " done: true\n", " experiment_id: a7543e8d696745edbfe36a6f4ebe4071\n", " experiment_tag: 5_activation=relu_tanh,height=52.1061,steps=100,width=3.9613\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 6.459268729660066\n", " neg_mean_loss: -6.459268729660066\n", " node_ip: 127.0.0.1\n", " pid: 45837\n", " time_since_restore: 10.753926277160645\n", " time_this_iter_s: 0.10714435577392578\n", " time_total_s: 10.753926277160645\n", " timestamp: 1658499518\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 265c09ee\n", " warmup_time: 0.0026078224182128906\n", " \n", "Result for objective_2e3cd63e:\n", " date: 2022-07-22_15-18-41\n", " done: false\n", " experiment_id: a2f52d042c5d43bcbb59c6a16249fdb9\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 11.243845267715532\n", " neg_mean_loss: -11.243845267715532\n", " node_ip: 127.0.0.1\n", " pid: 45864\n", " time_since_restore: 0.10467910766601562\n", " time_this_iter_s: 0.10467910766601562\n", " time_total_s: 0.10467910766601562\n", " timestamp: 1658499521\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 2e3cd63e\n", " warmup_time: 0.002730131149291992\n", " \n", "Result for objective_28587494:\n", " date: 2022-07-22_15-18-41\n", " done: false\n", " experiment_id: 750bd707fda147f0bd615c32243d64e5\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " mean_loss: -6.928259075209273\n", " neg_mean_loss: 6.928259075209273\n", " node_ip: 127.0.0.1\n", " pid: 45842\n", " time_since_restore: 10.179664850234985\n", " time_this_iter_s: 0.10827279090881348\n", " time_total_s: 10.179664850234985\n", " timestamp: 1658499521\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: '28587494'\n", " warmup_time: 0.0028159618377685547\n", " \n", "Result for objective_28682bfa:\n", " date: 2022-07-22_15-18-41\n", " done: false\n", " experiment_id: 5487e6a1cf70423096cf6285197fb824\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " mean_loss: -8.437898356861577\n", " neg_mean_loss: 8.437898356861577\n", " node_ip: 127.0.0.1\n", " pid: 45845\n", " time_since_restore: 10.208577156066895\n", " time_this_iter_s: 0.10761713981628418\n", " time_total_s: 10.208577156066895\n", " timestamp: 1658499521\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: 28682bfa\n", " warmup_time: 0.003072023391723633\n", " \n", "Result for objective_28788388:\n", " date: 2022-07-22_15-18-41\n", " done: false\n", " experiment_id: e959dae493a240c489a3193352c46f3a\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " mean_loss: 10.144880256980718\n", " neg_mean_loss: -10.144880256980718\n", " node_ip: 127.0.0.1\n", " pid: 45848\n", " time_since_restore: 10.215306997299194\n", " time_this_iter_s: 0.10723686218261719\n", " time_total_s: 10.215306997299194\n", " timestamp: 1658499521\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: '28788388'\n", " warmup_time: 0.0027070045471191406\n", " \n", "Result for objective_28587494:\n", " date: 2022-07-22_15-18-42\n", " done: true\n", " experiment_id: 750bd707fda147f0bd615c32243d64e5\n", " experiment_tag: 6_activation=relu_tanh,height=-82.3320,steps=100,width=3.3822\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: -6.943213699003365\n", " neg_mean_loss: 6.943213699003365\n", " node_ip: 127.0.0.1\n", " pid: 45842\n", " time_since_restore: 10.711625099182129\n", " time_this_iter_s: 0.10703706741333008\n", " time_total_s: 10.711625099182129\n", " timestamp: 1658499522\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: '28587494'\n", " warmup_time: 0.0028159618377685547\n", " \n", "Result for objective_28682bfa:\n", " date: 2022-07-22_15-18-42\n", " done: true\n", " experiment_id: 5487e6a1cf70423096cf6285197fb824\n", " experiment_tag: 7_activation=relu_tanh,height=-94.9771,steps=100,width=17.6810\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: -8.440901810803183\n", " neg_mean_loss: 8.440901810803183\n", " node_ip: 127.0.0.1\n", " pid: 45845\n", " time_since_restore: 10.747072219848633\n", " time_this_iter_s: 0.1084749698638916\n", " time_total_s: 10.747072219848633\n", " timestamp: 1658499522\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 28682bfa\n", " warmup_time: 0.003072023391723633\n", " \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Result for objective_28788388:\n", " date: 2022-07-22_15-18-42\n", " done: true\n", " experiment_id: e959dae493a240c489a3193352c46f3a\n", " experiment_tag: 8_activation=relu_tanh,height=90.6787,steps=100,width=13.7072\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 10.141019147716873\n", " neg_mean_loss: -10.141019147716873\n", " node_ip: 127.0.0.1\n", " pid: 45848\n", " time_since_restore: 10.752525091171265\n", " time_this_iter_s: 0.10643196105957031\n", " time_total_s: 10.752525091171265\n", " timestamp: 1658499522\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: '28788388'\n", " warmup_time: 0.0027070045471191406\n", " \n", "Result for objective_3046455a:\n", " date: 2022-07-22_15-18-44\n", " done: false\n", " experiment_id: f282080699d848f69c4b8626bd4c2d91\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 13.250521336587763\n", " neg_mean_loss: -13.250521336587763\n", " node_ip: 127.0.0.1\n", " pid: 45869\n", " time_since_restore: 0.10329103469848633\n", " time_this_iter_s: 0.10329103469848633\n", " time_total_s: 0.10329103469848633\n", " timestamp: 1658499524\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 3046455a\n", " warmup_time: 0.0027010440826416016\n", " \n", "Result for objective_2e3cd63e:\n", " date: 2022-07-22_15-18-46\n", " done: false\n", " experiment_id: a2f52d042c5d43bcbb59c6a16249fdb9\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: 8.5370623175226\n", " neg_mean_loss: -8.5370623175226\n", " node_ip: 127.0.0.1\n", " pid: 45864\n", " time_since_restore: 5.129793882369995\n", " time_this_iter_s: 0.10869097709655762\n", " time_total_s: 5.129793882369995\n", " timestamp: 1658499526\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: 2e3cd63e\n", " warmup_time: 0.002730131149291992\n", " \n", "Result for objective_3046455a:\n", " date: 2022-07-22_15-18-49\n", " done: false\n", " experiment_id: f282080699d848f69c4b8626bd4c2d91\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: 3.3797430580474837\n", " neg_mean_loss: -3.3797430580474837\n", " node_ip: 127.0.0.1\n", " pid: 45869\n", " time_since_restore: 5.139970064163208\n", " time_this_iter_s: 0.10870695114135742\n", " time_total_s: 5.139970064163208\n", " timestamp: 1658499529\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: 3046455a\n", " warmup_time: 0.0027010440826416016\n", " \n", "Result for objective_2e3cd63e:\n", " date: 2022-07-22_15-18-51\n", " done: false\n", " experiment_id: a2f52d042c5d43bcbb59c6a16249fdb9\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " mean_loss: 6.983470378488619\n", " neg_mean_loss: -6.983470378488619\n", " node_ip: 127.0.0.1\n", " pid: 45864\n", " time_since_restore: 10.16285490989685\n", " time_this_iter_s: 0.1078939437866211\n", " time_total_s: 10.16285490989685\n", " timestamp: 1658499531\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: 2e3cd63e\n", " warmup_time: 0.002730131149291992\n", " \n", "Result for objective_2e3cd63e:\n", " date: 2022-07-22_15-18-51\n", " done: true\n", " experiment_id: a2f52d042c5d43bcbb59c6a16249fdb9\n", " experiment_tag: 9_activation=relu_tanh,height=2.4385,steps=100,width=0.0790\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 6.8562837197212945\n", " neg_mean_loss: -6.8562837197212945\n", " node_ip: 127.0.0.1\n", " pid: 45864\n", " time_since_restore: 10.700610876083374\n", " time_this_iter_s: 0.10786271095275879\n", " time_total_s: 10.700610876083374\n", " timestamp: 1658499531\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 2e3cd63e\n", " warmup_time: 0.002730131149291992\n", " \n", "Result for objective_3046455a:\n", " date: 2022-07-22_15-18-54\n", " done: false\n", " experiment_id: f282080699d848f69c4b8626bd4c2d91\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " mean_loss: 3.315552368411653\n", " neg_mean_loss: -3.315552368411653\n", " node_ip: 127.0.0.1\n", " pid: 45869\n", " time_since_restore: 10.18196415901184\n", " time_this_iter_s: 0.10727214813232422\n", " time_total_s: 10.18196415901184\n", " timestamp: 1658499534\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: 3046455a\n", " warmup_time: 0.0027010440826416016\n", " \n", "Result for objective_3046455a:\n", " date: 2022-07-22_15-18-55\n", " done: true\n", " experiment_id: f282080699d848f69c4b8626bd4c2d91\n", " experiment_tag: 10_activation=relu_tanh,height=22.5052,steps=100,width=16.2524\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 3.3122882595656677\n", " neg_mean_loss: -3.3122882595656677\n", " node_ip: 127.0.0.1\n", " pid: 45869\n", " time_since_restore: 10.717580080032349\n", " time_this_iter_s: 0.1054232120513916\n", " time_total_s: 10.717580080032349\n", " timestamp: 1658499535\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 3046455a\n", " warmup_time: 0.0027010440826416016\n", " \n" ] } ], "source": [ "tuner = tune.Tuner(\n", " objective,\n", " tune_config=tune.TuneConfig(\n", " metric=\"mean_loss\",\n", " mode=\"min\",\n", " search_alg=algo,\n", " num_samples=num_samples,\n", " ),\n", " param_space=search_config,\n", ")\n", "results = tuner.fit()" ] }, { "cell_type": "markdown", "id": "77a49ffb", "metadata": {}, "source": [ "Here are the hyperparamters found to minimize the mean loss of the defined objective." ] }, { "cell_type": "code", "execution_count": 10, "id": "59cdf197", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Best hyperparameters found were: {'steps': 100, 'width': 15.42641286533492, 'height': -95.8496101281197, 'activation': 'relu, tanh'}\n" ] } ], "source": [ "print(\"Best hyperparameters found were: \", results.get_best_result().config)" ] }, { "cell_type": "markdown", "id": "1dca41c4", "metadata": {}, "source": [ "## Incorporating a time budget to the experiment\n", "\n", "Define the time budget in seconds:" ] }, { "cell_type": "code", "execution_count": 11, "id": "878bd08f", "metadata": {}, "outputs": [], "source": [ "time_budget_s = 30" ] }, { "cell_type": "markdown", "id": "5c9d1bdc", "metadata": {}, "source": [ "Similarly we define a search space, but this time we feed it as an argument to\n", "`BlendSearch` rather than `Tuner()`'s `param_space` argument.\n", "\n", "We next define the time budget via `set_search_properties`.\n", "And once again include the `ConcurrencyLimiter`." ] }, { "cell_type": "code", "execution_count": 12, "id": "5846724f", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "You passed a `space` parameter to OptunaSearch that contained unresolved search space definitions. OptunaSearch should however be instantiated with fully configured search spaces only. To use Ray Tune's automatic search space conversion, pass the space definition as part of the `param_space` argument to `Tuner()` instead.\n", "\u001b[32m[I 2022-07-22 15:18:55,531]\u001b[0m A new study created in memory with name: optuna\u001b[0m\n" ] } ], "source": [ "algo = BlendSearch(\n", " metric=\"mean_loss\",\n", " mode=\"min\",\n", " space={\n", " \"width\": tune.uniform(0, 20),\n", " \"height\": tune.uniform(-100, 100),\n", " \"activation\": tune.choice([\"relu\", \"tanh\"]),\n", " },\n", ")\n", "algo.set_search_properties(config={\"time_budget_s\": time_budget_s})\n", "algo = ConcurrencyLimiter(algo, max_concurrent=4)" ] }, { "cell_type": "markdown", "id": "13564767", "metadata": {}, "source": [ "Now we run the experiment, this time with the `time_budget` included as an argument.\n", "Note: We allow for virtually infinite `num_samples` by passing `-1`, so that the\n", "experiment is stopped according to the time budget rather than a sample limit." ] }, { "cell_type": "code", "execution_count": 13, "id": "b878b2be", "metadata": {}, "outputs": [ { "data": { "text/html": [ "== Status ==
Current time: 2022-07-22 15:19:27 (running for 00:00:32.42)
Memory usage on this node: 10.2/16.0 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/4.69 GiB heap, 0.0/2.0 GiB objects
Current best trial: 421d81ee with mean_loss=-8.596112689018389 and parameters={'steps': 100, 'width': 13.171830039895717, 'height': -96.72215542618497, 'activation': 'tanh'}
Result logdir: /Users/kai/ray_results/blendsearch_exp
Number of trials: 12/infinite (12 TERMINATED)
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
Trial name status loc activation height steps width loss iter total time (s) iterations neg_mean_loss
objective_38449df6TERMINATED127.0.0.1:45883relu 29.4532 100 1.94864 13.4381 100 10.6555 99 -13.4381
objective_39d44feaTERMINATED127.0.0.1:45889tanh -95.8496 10015.4264 -8.51991 100 11.2945 99 8.51991
objective_39d5c046TERMINATED127.0.0.1:45890tanh -55.0407 100 9.97014 -4.40377 100 11.169 99 4.40377
objective_39d71608TERMINATED127.0.0.1:45891tanh -82.332 100 3.38222 -6.94321 100 11.3021 99 6.94321
objective_402f9534TERMINATED127.0.0.1:45909relu 2.43845 100 0.078965315.8563 100 10.7275 99 -15.8563
objective_4208c6aaTERMINATED127.0.0.1:45914relu -41.6248 10014.4351 5.90701 100 10.811 99 -5.90701
objective_421aeb64TERMINATED127.0.0.1:45917relu -94.9771 10017.681 0.559098 100 10.8502 99 -0.559098
objective_421d81eeTERMINATED127.0.0.1:45918tanh -96.7222 10013.1718 -8.59611 100 10.8456 99 8.59611
objective_481b0de6TERMINATED127.0.0.1:45933relu -81.3401 10015.2514 2.48132 11 1.17882 10 -2.48132
objective_4a00fc10TERMINATED127.0.0.1:45940tanh -100 10011.0922
objective_4a167946TERMINATED127.0.0.1:45945tanh -71.566 10010.8509
objective_4a1908a0TERMINATED127.0.0.1:45946relu -100 10012.4575


" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "Result for objective_38449df6:\n", " date: 2022-07-22_15-18-58\n", " done: false\n", " experiment_id: 22ed7d2e0f154e3d89fddc8f7f0b743f\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 22.945321626643842\n", " neg_mean_loss: -22.945321626643842\n", " node_ip: 127.0.0.1\n", " pid: 45883\n", " time_since_restore: 0.1004023551940918\n", " time_this_iter_s: 0.1004023551940918\n", " time_total_s: 0.1004023551940918\n", " timestamp: 1658499538\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 38449df6\n", " warmup_time: 0.0025548934936523438\n", " \n", "Result for objective_39d5c046:\n", " date: 2022-07-22_15-19-00\n", " done: false\n", " experiment_id: faa00d9b1a2b4014bf01bab455bf9dbc\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 5.495932910616953\n", " neg_mean_loss: -5.495932910616953\n", " node_ip: 127.0.0.1\n", " pid: 45890\n", " time_since_restore: 0.10018515586853027\n", " time_this_iter_s: 0.10018515586853027\n", " time_total_s: 0.10018515586853027\n", " timestamp: 1658499540\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 39d5c046\n", " warmup_time: 0.0029740333557128906\n", " \n", "Result for objective_39d71608:\n", " date: 2022-07-22_15-19-00\n", " done: false\n", " experiment_id: 43ea2d3839fb42cabf83009ddd82acbd\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 2.766796283480206\n", " neg_mean_loss: -2.766796283480206\n", " node_ip: 127.0.0.1\n", " pid: 45891\n", " time_since_restore: 0.10022497177124023\n", " time_this_iter_s: 0.10022497177124023\n", " time_total_s: 0.10022497177124023\n", " timestamp: 1658499540\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 39d71608\n", " warmup_time: 0.003571033477783203\n", " \n", "Result for objective_39d44fea:\n", " date: 2022-07-22_15-19-00\n", " done: false\n", " experiment_id: d45c2138630140c3915a785b0e544a10\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 1.4150389871880282\n", " neg_mean_loss: -1.4150389871880282\n", " node_ip: 127.0.0.1\n", " pid: 45889\n", " time_since_restore: 0.10021686553955078\n", " time_this_iter_s: 0.10021686553955078\n", " time_total_s: 0.10021686553955078\n", " timestamp: 1658499540\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 39d44fea\n", " warmup_time: 0.003281831741333008\n", " \n", "Result for objective_38449df6:\n", " date: 2022-07-22_15-19-03\n", " done: false\n", " experiment_id: 22ed7d2e0f154e3d89fddc8f7f0b743f\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 48\n", " iterations_since_restore: 49\n", " mean_loss: 13.91118054261762\n", " neg_mean_loss: -13.91118054261762\n", " node_ip: 127.0.0.1\n", " pid: 45883\n", " time_since_restore: 5.199498176574707\n", " time_this_iter_s: 0.10724401473999023\n", " time_total_s: 5.199498176574707\n", " timestamp: 1658499543\n", " timesteps_since_restore: 0\n", " training_iteration: 49\n", " trial_id: 38449df6\n", " warmup_time: 0.0025548934936523438\n", " \n", "Result for objective_39d5c046:\n", " date: 2022-07-22_15-19-05\n", " done: false\n", " experiment_id: faa00d9b1a2b4014bf01bab455bf9dbc\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: -4.295122851662673\n", " neg_mean_loss: 4.295122851662673\n", " node_ip: 127.0.0.1\n", " pid: 45890\n", " time_since_restore: 5.137373924255371\n", " time_this_iter_s: 0.1063838005065918\n", " time_total_s: 5.137373924255371\n", " timestamp: 1658499545\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: 39d5c046\n", " warmup_time: 0.0029740333557128906\n", " \n", "Result for objective_39d44fea:\n", " date: 2022-07-22_15-19-05\n", " done: false\n", " experiment_id: d45c2138630140c3915a785b0e544a10\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: -8.44891425494968\n", " neg_mean_loss: 8.44891425494968\n", " node_ip: 127.0.0.1\n", " pid: 45889\n", " time_since_restore: 5.140375852584839\n", " time_this_iter_s: 0.10918521881103516\n", " time_total_s: 5.140375852584839\n", " timestamp: 1658499545\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: 39d44fea\n", " warmup_time: 0.003281831741333008\n", " \n", "Result for objective_39d71608:\n", " date: 2022-07-22_15-19-05\n", " done: false\n", " experiment_id: 43ea2d3839fb42cabf83009ddd82acbd\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: -6.641362320132167\n", " neg_mean_loss: 6.641362320132167\n", " node_ip: 127.0.0.1\n", " pid: 45891\n", " time_since_restore: 5.144212007522583\n", " time_this_iter_s: 0.1064291000366211\n", " time_total_s: 5.144212007522583\n", " timestamp: 1658499545\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: 39d71608\n", " warmup_time: 0.003571033477783203\n", " \n", "Result for objective_38449df6:\n", " date: 2022-07-22_15-19-08\n", " done: false\n", " experiment_id: 22ed7d2e0f154e3d89fddc8f7f0b743f\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 95\n", " iterations_since_restore: 96\n", " mean_loss: 13.457824286707531\n", " neg_mean_loss: -13.457824286707531\n", " node_ip: 127.0.0.1\n", " pid: 45883\n", " time_since_restore: 10.220572233200073\n", " time_this_iter_s: 0.10792016983032227\n", " time_total_s: 10.220572233200073\n", " timestamp: 1658499548\n", " timesteps_since_restore: 0\n", " training_iteration: 96\n", " trial_id: 38449df6\n", " warmup_time: 0.0025548934936523438\n", " \n", "Result for objective_38449df6:\n", " date: 2022-07-22_15-19-08\n", " done: true\n", " experiment_id: 22ed7d2e0f154e3d89fddc8f7f0b743f\n", " experiment_tag: 1_activation=relu,height=29.4532,steps=100,width=1.9486\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 13.43813759132241\n", " neg_mean_loss: -13.43813759132241\n", " node_ip: 127.0.0.1\n", " pid: 45883\n", " time_since_restore: 10.655535221099854\n", " time_this_iter_s: 0.10872602462768555\n", " time_total_s: 10.655535221099854\n", " timestamp: 1658499548\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 38449df6\n", " warmup_time: 0.0025548934936523438\n", " \n", "Result for objective_39d5c046:\n", " date: 2022-07-22_15-19-10\n", " done: false\n", " experiment_id: faa00d9b1a2b4014bf01bab455bf9dbc\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " mean_loss: -4.398492005249027\n", " neg_mean_loss: 4.398492005249027\n", " node_ip: 127.0.0.1\n", " pid: 45890\n", " time_since_restore: 10.1424720287323\n", " time_this_iter_s: 0.10685110092163086\n", " time_total_s: 10.1424720287323\n", " timestamp: 1658499550\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: 39d5c046\n", " warmup_time: 0.0029740333557128906\n", " \n", "Result for objective_39d44fea:\n", " date: 2022-07-22_15-19-10\n", " done: false\n", " experiment_id: d45c2138630140c3915a785b0e544a10\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 93\n", " iterations_since_restore: 94\n", " mean_loss: -8.515740400738455\n", " neg_mean_loss: 8.515740400738455\n", " node_ip: 127.0.0.1\n", " pid: 45889\n", " time_since_restore: 10.051257848739624\n", " time_this_iter_s: 0.10633492469787598\n", " time_total_s: 10.051257848739624\n", " timestamp: 1658499550\n", " timesteps_since_restore: 0\n", " training_iteration: 94\n", " trial_id: 39d44fea\n", " warmup_time: 0.003281831741333008\n", " \n", "Result for objective_39d71608:\n", " date: 2022-07-22_15-19-10\n", " done: false\n", " experiment_id: 43ea2d3839fb42cabf83009ddd82acbd\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 93\n", " iterations_since_restore: 94\n", " mean_loss: -6.925081133185133\n", " neg_mean_loss: 6.925081133185133\n", " node_ip: 127.0.0.1\n", " pid: 45891\n", " time_since_restore: 10.069098949432373\n", " time_this_iter_s: 0.10451602935791016\n", " time_total_s: 10.069098949432373\n", " timestamp: 1658499550\n", " timesteps_since_restore: 0\n", " training_iteration: 94\n", " trial_id: 39d71608\n", " warmup_time: 0.003571033477783203\n", " \n", "Result for objective_402f9534:\n", " date: 2022-07-22_15-19-11\n", " done: false\n", " experiment_id: 66af4092dc134e049ed47a56532133ce\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 20.243845267715532\n", " neg_mean_loss: -20.243845267715532\n", " node_ip: 127.0.0.1\n", " pid: 45909\n", " time_since_restore: 0.10260891914367676\n", " time_this_iter_s: 0.10260891914367676\n", " time_total_s: 0.10260891914367676\n", " timestamp: 1658499551\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 402f9534\n", " warmup_time: 0.0026721954345703125\n", " \n", "Result for objective_39d5c046:\n", " date: 2022-07-22_15-19-11\n", " done: true\n", " experiment_id: faa00d9b1a2b4014bf01bab455bf9dbc\n", " experiment_tag: 3_activation=tanh,height=-55.0407,steps=100,width=9.9701\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: -4.403770601366095\n", " neg_mean_loss: 4.403770601366095\n", " node_ip: 127.0.0.1\n", " pid: 45890\n", " time_since_restore: 11.169048070907593\n", " time_this_iter_s: 0.1081700325012207\n", " time_total_s: 11.169048070907593\n", " timestamp: 1658499551\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 39d5c046\n", " warmup_time: 0.0029740333557128906\n", " \n", "Result for objective_39d44fea:\n", " date: 2022-07-22_15-19-12\n", " done: true\n", " experiment_id: d45c2138630140c3915a785b0e544a10\n", " experiment_tag: 2_activation=tanh,height=-95.8496,steps=100,width=15.4264\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: -8.519908298508359\n", " neg_mean_loss: 8.519908298508359\n", " node_ip: 127.0.0.1\n", " pid: 45889\n", " time_since_restore: 11.29450273513794\n", " time_this_iter_s: 0.1167898178100586\n", " time_total_s: 11.29450273513794\n", " timestamp: 1658499552\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 39d44fea\n", " warmup_time: 0.003281831741333008\n", " \n", "Result for objective_39d71608:\n", " date: 2022-07-22_15-19-12\n", " done: true\n", " experiment_id: 43ea2d3839fb42cabf83009ddd82acbd\n", " experiment_tag: 4_activation=tanh,height=-82.3320,steps=100,width=3.3822\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: -6.943213699003365\n", " neg_mean_loss: 6.943213699003365\n", " node_ip: 127.0.0.1\n", " pid: 45891\n", " time_since_restore: 11.302148818969727\n", " time_this_iter_s: 0.12157797813415527\n", " time_total_s: 11.302148818969727\n", " timestamp: 1658499552\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 39d71608\n", " warmup_time: 0.003571033477783203\n", " \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Result for objective_4208c6aa:\n", " date: 2022-07-22_15-19-14\n", " done: false\n", " experiment_id: ce292c59f7c74e1ab42de6004eb3d846\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 15.837521363412662\n", " neg_mean_loss: -15.837521363412662\n", " node_ip: 127.0.0.1\n", " pid: 45914\n", " time_since_restore: 0.10442900657653809\n", " time_this_iter_s: 0.10442900657653809\n", " time_total_s: 0.10442900657653809\n", " timestamp: 1658499554\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 4208c6aa\n", " warmup_time: 0.002707958221435547\n", " \n", "Result for objective_421aeb64:\n", " date: 2022-07-22_15-19-14\n", " done: false\n", " experiment_id: a4b264c05660423ea645cef7a9cd38bb\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 10.502293516994555\n", " neg_mean_loss: -10.502293516994555\n", " node_ip: 127.0.0.1\n", " pid: 45917\n", " time_since_restore: 0.1030268669128418\n", " time_this_iter_s: 0.1030268669128418\n", " time_total_s: 0.1030268669128418\n", " timestamp: 1658499554\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 421aeb64\n", " warmup_time: 0.0031020641326904297\n", " \n", "Result for objective_421d81ee:\n", " date: 2022-07-22_15-19-14\n", " done: false\n", " experiment_id: 836a62c82d6c467fb232414586872fd0\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 1.3277844573815027\n", " neg_mean_loss: -1.3277844573815027\n", " node_ip: 127.0.0.1\n", " pid: 45918\n", " time_since_restore: 0.10497617721557617\n", " time_this_iter_s: 0.10497617721557617\n", " time_total_s: 0.10497617721557617\n", " timestamp: 1658499554\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 421d81ee\n", " warmup_time: 0.002989053726196289\n", " \n", "Result for objective_402f9534:\n", " date: 2022-07-22_15-19-16\n", " done: false\n", " experiment_id: 66af4092dc134e049ed47a56532133ce\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: 17.5370623175226\n", " neg_mean_loss: -17.5370623175226\n", " node_ip: 127.0.0.1\n", " pid: 45909\n", " time_since_restore: 5.13571572303772\n", " time_this_iter_s: 0.1079568862915039\n", " time_total_s: 5.13571572303772\n", " timestamp: 1658499556\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: 402f9534\n", " warmup_time: 0.0026721954345703125\n", " \n", "Result for objective_4208c6aa:\n", " date: 2022-07-22_15-19-19\n", " done: false\n", " experiment_id: ce292c59f7c74e1ab42de6004eb3d846\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 46\n", " iterations_since_restore: 47\n", " mean_loss: 5.98588603781584\n", " neg_mean_loss: -5.98588603781584\n", " node_ip: 127.0.0.1\n", " pid: 45914\n", " time_since_restore: 5.11784291267395\n", " time_this_iter_s: 0.106842041015625\n", " time_total_s: 5.11784291267395\n", " timestamp: 1658499559\n", " timesteps_since_restore: 0\n", " training_iteration: 47\n", " trial_id: 4208c6aa\n", " warmup_time: 0.002707958221435547\n", " \n", "Result for objective_421aeb64:\n", " date: 2022-07-22_15-19-19\n", " done: false\n", " experiment_id: a4b264c05660423ea645cef7a9cd38bb\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 46\n", " iterations_since_restore: 47\n", " mean_loss: 0.6237521179327707\n", " neg_mean_loss: -0.6237521179327707\n", " node_ip: 127.0.0.1\n", " pid: 45917\n", " time_since_restore: 5.119860887527466\n", " time_this_iter_s: 0.10904288291931152\n", " time_total_s: 5.119860887527466\n", " timestamp: 1658499559\n", " timesteps_since_restore: 0\n", " training_iteration: 47\n", " trial_id: 421aeb64\n", " warmup_time: 0.0031020641326904297\n", " \n", "Result for objective_421d81ee:\n", " date: 2022-07-22_15-19-19\n", " done: false\n", " experiment_id: 836a62c82d6c467fb232414586872fd0\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 46\n", " iterations_since_restore: 47\n", " mean_loss: -8.509852624896407\n", " neg_mean_loss: 8.509852624896407\n", " node_ip: 127.0.0.1\n", " pid: 45918\n", " time_since_restore: 5.103626012802124\n", " time_this_iter_s: 0.10568881034851074\n", " time_total_s: 5.103626012802124\n", " timestamp: 1658499559\n", " timesteps_since_restore: 0\n", " training_iteration: 47\n", " trial_id: 421d81ee\n", " warmup_time: 0.002989053726196289\n", " \n", "Result for objective_402f9534:\n", " date: 2022-07-22_15-19-21\n", " done: false\n", " experiment_id: 66af4092dc134e049ed47a56532133ce\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " mean_loss: 15.983470378488619\n", " neg_mean_loss: -15.983470378488619\n", " node_ip: 127.0.0.1\n", " pid: 45909\n", " time_since_restore: 10.194732666015625\n", " time_this_iter_s: 0.11167168617248535\n", " time_total_s: 10.194732666015625\n", " timestamp: 1658499561\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: 402f9534\n", " warmup_time: 0.0026721954345703125\n", " \n", "Result for objective_402f9534:\n", " date: 2022-07-22_15-19-22\n", " done: true\n", " experiment_id: 66af4092dc134e049ed47a56532133ce\n", " experiment_tag: 5_activation=relu,height=2.4385,steps=100,width=0.0790\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 15.856283719721294\n", " neg_mean_loss: -15.856283719721294\n", " node_ip: 127.0.0.1\n", " pid: 45909\n", " time_since_restore: 10.727468013763428\n", " time_this_iter_s: 0.10604405403137207\n", " time_total_s: 10.727468013763428\n", " timestamp: 1658499562\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 402f9534\n", " warmup_time: 0.0026721954345703125\n", " \n", "Result for objective_4208c6aa:\n", " date: 2022-07-22_15-19-24\n", " done: false\n", " experiment_id: ce292c59f7c74e1ab42de6004eb3d846\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 93\n", " iterations_since_restore: 94\n", " mean_loss: 5.911460436218253\n", " neg_mean_loss: -5.911460436218253\n", " node_ip: 127.0.0.1\n", " pid: 45914\n", " time_since_restore: 10.168487071990967\n", " time_this_iter_s: 0.10619020462036133\n", " time_total_s: 10.168487071990967\n", " timestamp: 1658499564\n", " timesteps_since_restore: 0\n", " training_iteration: 94\n", " trial_id: 4208c6aa\n", " warmup_time: 0.002707958221435547\n", " \n", "Result for objective_481b0de6:\n", " date: 2022-07-22_15-19-24\n", " done: false\n", " experiment_id: ceafc8f530054af6bc676fd081f84c46\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 11.865986796947288\n", " neg_mean_loss: -11.865986796947288\n", " node_ip: 127.0.0.1\n", " pid: 45933\n", " time_since_restore: 0.10329604148864746\n", " time_this_iter_s: 0.10329604148864746\n", " time_total_s: 0.10329604148864746\n", " timestamp: 1658499564\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 481b0de6\n", " warmup_time: 0.0028421878814697266\n", " \n", "Result for objective_421aeb64:\n", " date: 2022-07-22_15-19-24\n", " done: false\n", " experiment_id: a4b264c05660423ea645cef7a9cd38bb\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 93\n", " iterations_since_restore: 94\n", " mean_loss: 0.5627408539120644\n", " neg_mean_loss: -0.5627408539120644\n", " node_ip: 127.0.0.1\n", " pid: 45917\n", " time_since_restore: 10.207238912582397\n", " time_this_iter_s: 0.1081080436706543\n", " time_total_s: 10.207238912582397\n", " timestamp: 1658499564\n", " timesteps_since_restore: 0\n", " training_iteration: 94\n", " trial_id: 421aeb64\n", " warmup_time: 0.0031020641326904297\n", " \n", "Result for objective_421d81ee:\n", " date: 2022-07-22_15-19-24\n", " done: false\n", " experiment_id: 836a62c82d6c467fb232414586872fd0\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 93\n", " iterations_since_restore: 94\n", " mean_loss: -8.591242584097142\n", " neg_mean_loss: 8.591242584097142\n", " node_ip: 127.0.0.1\n", " pid: 45918\n", " time_since_restore: 10.2047119140625\n", " time_this_iter_s: 0.1085958480834961\n", " time_total_s: 10.2047119140625\n", " timestamp: 1658499564\n", " timesteps_since_restore: 0\n", " training_iteration: 94\n", " trial_id: 421d81ee\n", " warmup_time: 0.002989053726196289\n", " \n", "Result for objective_4208c6aa:\n", " date: 2022-07-22_15-19-25\n", " done: true\n", " experiment_id: ce292c59f7c74e1ab42de6004eb3d846\n", " experiment_tag: 6_activation=relu,height=-41.6248,steps=100,width=14.4351\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 5.907010419420164\n", " neg_mean_loss: -5.907010419420164\n", " node_ip: 127.0.0.1\n", " pid: 45914\n", " time_since_restore: 10.811041831970215\n", " time_this_iter_s: 0.10398101806640625\n", " time_total_s: 10.811041831970215\n", " timestamp: 1658499565\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 4208c6aa\n", " warmup_time: 0.002707958221435547\n", " \n", "Result for objective_421aeb64:\n", " date: 2022-07-22_15-19-25\n", " done: true\n", " experiment_id: a4b264c05660423ea645cef7a9cd38bb\n", " experiment_tag: 7_activation=relu,height=-94.9771,steps=100,width=17.6810\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 0.559098189196817\n", " neg_mean_loss: -0.559098189196817\n", " node_ip: 127.0.0.1\n", " pid: 45917\n", " time_since_restore: 10.850210905075073\n", " time_this_iter_s: 0.10677385330200195\n", " time_total_s: 10.850210905075073\n", " timestamp: 1658499565\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 421aeb64\n", " warmup_time: 0.0031020641326904297\n", " \n", "Result for objective_421d81ee:\n", " date: 2022-07-22_15-19-25\n", " done: true\n", " experiment_id: 836a62c82d6c467fb232414586872fd0\n", " experiment_tag: 8_activation=tanh,height=-96.7222,steps=100,width=13.1718\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: -8.596112689018389\n", " neg_mean_loss: 8.596112689018389\n", " node_ip: 127.0.0.1\n", " pid: 45918\n", " time_since_restore: 10.845621109008789\n", " time_this_iter_s: 0.10427713394165039\n", " time_total_s: 10.845621109008789\n", " timestamp: 1658499565\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 421d81ee\n", " warmup_time: 0.002989053726196289\n", " \n" ] } ], "source": [ "tuner = tune.Tuner(\n", " objective,\n", " tune_config=tune.TuneConfig(\n", " metric=\"mean_loss\",\n", " mode=\"min\",\n", " search_alg=algo,\n", " num_samples=-1,\n", " time_budget_s=time_budget_s,\n", " ),\n", " param_space={\"steps\": 100},\n", ")\n", "results = tuner.fit()" ] }, { "cell_type": "code", "execution_count": 14, "id": "48f06c5f", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Best hyperparameters found were: {'steps': 100, 'width': 13.171830039895717, 'height': -96.72215542618497, 'activation': 'tanh'}\n" ] } ], "source": [ "print(\"Best hyperparameters found were: \", results.get_best_result().config)" ] }, { "cell_type": "markdown", "id": "47b906b7", "metadata": {}, "source": [ "## Running Tune experiments with CFO\n", "\n", "This example demonstrates the usage of Frugal Optimization for Cost-related\n", "Hyperparameters (CFO) with Ray Tune.\n", "\n", "We now define the search algorithm as built from `CFO`, constrained to a maximum of `4`\n", "concurrent trials with a `ConcurrencyLimiter`." ] }, { "cell_type": "code", "execution_count": 15, "id": "29f2c0be", "metadata": {}, "outputs": [], "source": [ "algo = CFO()\n", "algo = ConcurrencyLimiter(algo, max_concurrent=4)" ] }, { "cell_type": "markdown", "id": "a89cf9bb", "metadata": {}, "source": [ "The number of samples is the number of hyperparameter combinations that will be\n", "tried out. This Tune run is set to `1000` samples.\n", "(you can decrease this if it takes too long on your machine)." ] }, { "cell_type": "code", "execution_count": 16, "id": "3d15f10d", "metadata": {}, "outputs": [], "source": [ "num_samples = 1000" ] }, { "cell_type": "code", "execution_count": 17, "id": "137f3ec0", "metadata": { "tags": [ "remove-cell" ] }, "outputs": [], "source": [ "# If 1000 samples take too long, you can reduce this number.\n", "# We override this number here for our smoke tests.\n", "num_samples = 10" ] }, { "cell_type": "markdown", "id": "cfa2e413", "metadata": {}, "source": [ "Next we define a search space. The critical assumption is that the optimal\n", "hyperparameters live within this space. Yet, if the space is very large, then\n", "those hyperparameters may be difficult to find in a short amount of time." ] }, { "cell_type": "code", "execution_count": 18, "id": "ab4fbcbe", "metadata": {}, "outputs": [], "source": [ "search_config = {\n", " \"steps\": 100,\n", " \"width\": tune.uniform(0, 20),\n", " \"height\": tune.uniform(-100, 100),\n", " \"activation\": tune.choice([\"relu, tanh\"])\n", "}" ] }, { "cell_type": "markdown", "id": "b19390a2", "metadata": {}, "source": [ "Finally, we run the experiment to `\"min\"`imize the \"mean_loss\" of the `objective`\n", "by searching `search_config` via `algo`, `num_samples` times. This previous sentence\n", "is fully characterizes the search problem we aim to solve. With this in mind,\n", "notice how efficient it is to execute `tuner.fit()`." ] }, { "cell_type": "code", "execution_count": 19, "id": "2acf19f5", "metadata": {}, "outputs": [ { "data": { "text/html": [ "== Status ==
Current time: 2022-07-22 15:20:11 (running for 00:00:43.01)
Memory usage on this node: 10.0/16.0 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/4.69 GiB heap, 0.0/2.0 GiB objects
Current best trial: 4d51b38c with mean_loss=-8.661424748129757 and parameters={'steps': 100, 'width': 15.124213652112319, 'height': -97.27768667042203, 'activation': 'relu, tanh'}
Result logdir: /Users/kai/ray_results/cfo_exp
Number of trials: 10/10 (10 TERMINATED)
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
Trial name status loc activation height steps width loss iter total time (s) iterations neg_mean_loss
objective_4bc281feTERMINATED127.0.0.1:45958relu, tanh 29.4532 100 1.94864 4.43814 100 10.7193 99 -4.43814
objective_4d4ed536TERMINATED127.0.0.1:45963relu, tanh -26.2681 100 7.25618 -1.48952 100 11.1932 99 1.48952
objective_4d5064aaTERMINATED127.0.0.1:45964relu, tanh -46.7215 10015.1097 -3.60574 100 11.1983 99 3.60574
objective_4d51b38cTERMINATED127.0.0.1:45965relu, tanh -97.2777 10015.1242 -8.66142 100 11.1977 99 8.66142
objective_53b39e84TERMINATED127.0.0.1:45983relu, tanh 25.7812 100 1.77704 4.11597 100 10.7964 99 -4.11597
objective_558c6132TERMINATED127.0.0.1:45990relu, tanh 33.1252 100 2.12024 4.76727 100 10.7659 99 -4.76727
objective_558f27aaTERMINATED127.0.0.1:45991relu, tanh 40.929 100 0.013803213.8907 100 10.785 99 -13.8907
objective_5592ca68TERMINATED127.0.0.1:45992relu, tanh 17.9775 100 3.88348 3.05125 100 10.7802 99 -3.05125
objective_5b999414TERMINATED127.0.0.1:46011relu, tanh 11.0354 100 3.27423 2.40281 100 12.6262 99 -2.40281
objective_5d8462c2TERMINATED127.0.0.1:46018relu, tanh 24.9195 100 4.49273 3.71184 100 10.7339 99 -3.71184


" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "Result for objective_4bc281fe:\n", " date: 2022-07-22_15-19-30\n", " done: false\n", " experiment_id: 129b24a058ee40fd991fb40844988057\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 13.945321626643842\n", " neg_mean_loss: -13.945321626643842\n", " node_ip: 127.0.0.1\n", " pid: 45958\n", " time_since_restore: 0.10440444946289062\n", " time_this_iter_s: 0.10440444946289062\n", " time_total_s: 0.10440444946289062\n", " timestamp: 1658499570\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 4bc281fe\n", " warmup_time: 0.003987789154052734\n", " \n", "Result for objective_4d4ed536:\n", " date: 2022-07-22_15-19-33\n", " done: false\n", " experiment_id: 6c18c359d1574792ac99ed37265d5d87\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 8.373189005362395\n", " neg_mean_loss: -8.373189005362395\n", " node_ip: 127.0.0.1\n", " pid: 45963\n", " time_since_restore: 0.10463309288024902\n", " time_this_iter_s: 0.10463309288024902\n", " time_total_s: 0.10463309288024902\n", " timestamp: 1658499573\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 4d4ed536\n", " warmup_time: 0.0028281211853027344\n", " \n", "Result for objective_4d51b38c:\n", " date: 2022-07-22_15-19-33\n", " done: false\n", " experiment_id: 5c9cf6fc35c4411b8ec13e0b90b791a4\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 1.2722313329577961\n", " neg_mean_loss: -1.2722313329577961\n", " node_ip: 127.0.0.1\n", " pid: 45965\n", " time_since_restore: 0.10346817970275879\n", " time_this_iter_s: 0.10346817970275879\n", " time_total_s: 0.10346817970275879\n", " timestamp: 1658499573\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 4d51b38c\n", " warmup_time: 0.0027761459350585938\n", " \n", "Result for objective_4d5064aa:\n", " date: 2022-07-22_15-19-33\n", " done: false\n", " experiment_id: 93ed3aa2df8c4f71b0b1e9437c062ccf\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 6.327848408616047\n", " neg_mean_loss: -6.327848408616047\n", " node_ip: 127.0.0.1\n", " pid: 45964\n", " time_since_restore: 0.10395598411560059\n", " time_this_iter_s: 0.10395598411560059\n", " time_total_s: 0.10395598411560059\n", " timestamp: 1658499573\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 4d5064aa\n", " warmup_time: 0.002546072006225586\n", " \n", "Result for objective_4bc281fe:\n", " date: 2022-07-22_15-19-35\n", " done: false\n", " experiment_id: 129b24a058ee40fd991fb40844988057\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: 4.9297078000588614\n", " neg_mean_loss: -4.9297078000588614\n", " node_ip: 127.0.0.1\n", " pid: 45958\n", " time_since_restore: 5.122281312942505\n", " time_this_iter_s: 0.10893416404724121\n", " time_total_s: 5.122281312942505\n", " timestamp: 1658499575\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: 4bc281fe\n", " warmup_time: 0.003987789154052734\n", " \n", "Result for objective_4d4ed536:\n", " date: 2022-07-22_15-19-38\n", " done: false\n", " experiment_id: 6c18c359d1574792ac99ed37265d5d87\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: -1.3419437208788234\n", " neg_mean_loss: 1.3419437208788234\n", " node_ip: 127.0.0.1\n", " pid: 45963\n", " time_since_restore: 5.132510185241699\n", " time_this_iter_s: 0.10718894004821777\n", " time_total_s: 5.132510185241699\n", " timestamp: 1658499578\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: 4d4ed536\n", " warmup_time: 0.0028281211853027344\n", " \n", "Result for objective_4d51b38c:\n", " date: 2022-07-22_15-19-38\n", " done: false\n", " experiment_id: 5c9cf6fc35c4411b8ec13e0b90b791a4\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: -8.58904124947806\n", " neg_mean_loss: 8.58904124947806\n", " node_ip: 127.0.0.1\n", " pid: 45965\n", " time_since_restore: 5.1296000480651855\n", " time_this_iter_s: 0.10642719268798828\n", " time_total_s: 5.1296000480651855\n", " timestamp: 1658499578\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: 4d51b38c\n", " warmup_time: 0.0027761459350585938\n", " \n", "Result for objective_4d5064aa:\n", " date: 2022-07-22_15-19-38\n", " done: false\n", " experiment_id: 93ed3aa2df8c4f71b0b1e9437c062ccf\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: -3.5332931133029772\n", " neg_mean_loss: 3.5332931133029772\n", " node_ip: 127.0.0.1\n", " pid: 45964\n", " time_since_restore: 5.1527419090271\n", " time_this_iter_s: 0.10794281959533691\n", " time_total_s: 5.1527419090271\n", " timestamp: 1658499578\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: 4d5064aa\n", " warmup_time: 0.002546072006225586\n", " \n", "Result for objective_4bc281fe:\n", " date: 2022-07-22_15-19-41\n", " done: false\n", " experiment_id: 129b24a058ee40fd991fb40844988057\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " mean_loss: 4.462994199505408\n", " neg_mean_loss: -4.462994199505408\n", " node_ip: 127.0.0.1\n", " pid: 45958\n", " time_since_restore: 10.180925369262695\n", " time_this_iter_s: 0.10581827163696289\n", " time_total_s: 10.180925369262695\n", " timestamp: 1658499581\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: 4bc281fe\n", " warmup_time: 0.003987789154052734\n", " \n", "Result for objective_4bc281fe:\n", " date: 2022-07-22_15-19-41\n", " done: true\n", " experiment_id: 129b24a058ee40fd991fb40844988057\n", " experiment_tag: 1_activation=relu_tanh,height=29.4532,steps=100,width=1.9486\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 4.438137591322411\n", " neg_mean_loss: -4.438137591322411\n", " node_ip: 127.0.0.1\n", " pid: 45958\n", " time_since_restore: 10.71930718421936\n", " time_this_iter_s: 0.10692596435546875\n", " time_total_s: 10.71930718421936\n", " timestamp: 1658499581\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 4bc281fe\n", " warmup_time: 0.003987789154052734\n", " \n", "Result for objective_4d4ed536:\n", " date: 2022-07-22_15-19-43\n", " done: false\n", " experiment_id: 6c18c359d1574792ac99ed37265d5d87\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 93\n", " iterations_since_restore: 94\n", " mean_loss: -1.48078832328251\n", " neg_mean_loss: 1.48078832328251\n", " node_ip: 127.0.0.1\n", " pid: 45963\n", " time_since_restore: 10.033550262451172\n", " time_this_iter_s: 0.10465312004089355\n", " time_total_s: 10.033550262451172\n", " timestamp: 1658499583\n", " timesteps_since_restore: 0\n", " training_iteration: 94\n", " trial_id: 4d4ed536\n", " warmup_time: 0.0028281211853027344\n", " \n", "Result for objective_4d51b38c:\n", " date: 2022-07-22_15-19-43\n", " done: false\n", " experiment_id: 5c9cf6fc35c4411b8ec13e0b90b791a4\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 93\n", " iterations_since_restore: 94\n", " mean_loss: -8.657174711294605\n", " neg_mean_loss: 8.657174711294605\n", " node_ip: 127.0.0.1\n", " pid: 45965\n", " time_since_restore: 10.026285171508789\n", " time_this_iter_s: 0.10726714134216309\n", " time_total_s: 10.026285171508789\n", " timestamp: 1658499583\n", " timesteps_since_restore: 0\n", " training_iteration: 94\n", " trial_id: 4d51b38c\n", " warmup_time: 0.0027761459350585938\n", " \n", "Result for objective_4d5064aa:\n", " date: 2022-07-22_15-19-43\n", " done: false\n", " experiment_id: 93ed3aa2df8c4f71b0b1e9437c062ccf\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 93\n", " iterations_since_restore: 94\n", " mean_loss: -3.601490481891755\n", " neg_mean_loss: 3.601490481891755\n", " node_ip: 127.0.0.1\n", " pid: 45964\n", " time_since_restore: 10.049538135528564\n", " time_this_iter_s: 0.10654830932617188\n", " time_total_s: 10.049538135528564\n", " timestamp: 1658499583\n", " timesteps_since_restore: 0\n", " training_iteration: 94\n", " trial_id: 4d5064aa\n", " warmup_time: 0.002546072006225586\n", " \n", "Result for objective_53b39e84:\n", " date: 2022-07-22_15-19-44\n", " done: false\n", " experiment_id: abfeeaaa7eae4ad798c655bed2d1e223\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 13.57812082128205\n", " neg_mean_loss: -13.57812082128205\n", " node_ip: 127.0.0.1\n", " pid: 45983\n", " time_since_restore: 0.10458230972290039\n", " time_this_iter_s: 0.10458230972290039\n", " time_total_s: 0.10458230972290039\n", " timestamp: 1658499584\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 53b39e84\n", " warmup_time: 0.002878904342651367\n", " \n", "Result for objective_4d4ed536:\n", " date: 2022-07-22_15-19-44\n", " done: true\n", " experiment_id: 6c18c359d1574792ac99ed37265d5d87\n", " experiment_tag: 2_activation=relu_tanh,height=-26.2681,steps=100,width=7.2562\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: -1.489516678620475\n", " neg_mean_loss: 1.489516678620475\n", " node_ip: 127.0.0.1\n", " pid: 45963\n", " time_since_restore: 11.193175077438354\n", " time_this_iter_s: 0.11231780052185059\n", " time_total_s: 11.193175077438354\n", " timestamp: 1658499584\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 4d4ed536\n", " warmup_time: 0.0028281211853027344\n", " \n", "Result for objective_4d51b38c:\n", " date: 2022-07-22_15-19-44\n", " done: true\n", " experiment_id: 5c9cf6fc35c4411b8ec13e0b90b791a4\n", " experiment_tag: 4_activation=relu_tanh,height=-97.2777,steps=100,width=15.1242\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: -8.661424748129757\n", " neg_mean_loss: 8.661424748129757\n", " node_ip: 127.0.0.1\n", " pid: 45965\n", " time_since_restore: 11.19774603843689\n", " time_this_iter_s: 0.10802578926086426\n", " time_total_s: 11.19774603843689\n", " timestamp: 1658499584\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 4d51b38c\n", " warmup_time: 0.0027761459350585938\n", " \n", "Result for objective_4d5064aa:\n", " date: 2022-07-22_15-19-44\n", " done: true\n", " experiment_id: 93ed3aa2df8c4f71b0b1e9437c062ccf\n", " experiment_tag: 3_activation=relu_tanh,height=-46.7215,steps=100,width=15.1097\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: -3.6057445346106167\n", " neg_mean_loss: 3.6057445346106167\n", " node_ip: 127.0.0.1\n", " pid: 45964\n", " time_since_restore: 11.198285102844238\n", " time_this_iter_s: 0.10615801811218262\n", " time_total_s: 11.198285102844238\n", " timestamp: 1658499584\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 4d5064aa\n", " warmup_time: 0.002546072006225586\n", " \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Result for objective_558c6132:\n", " date: 2022-07-22_15-19-47\n", " done: false\n", " experiment_id: 98df2ca60c5f4a33a3da6870c3d8db4c\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 14.31252243200564\n", " neg_mean_loss: -14.31252243200564\n", " node_ip: 127.0.0.1\n", " pid: 45990\n", " time_since_restore: 0.10317206382751465\n", " time_this_iter_s: 0.10317206382751465\n", " time_total_s: 0.10317206382751465\n", " timestamp: 1658499587\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 558c6132\n", " warmup_time: 0.003049135208129883\n", " \n", "Result for objective_558f27aa:\n", " date: 2022-07-22_15-19-47\n", " done: false\n", " experiment_id: 73def1ffc7f443669328b88695453248\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 15.092898207127158\n", " neg_mean_loss: -15.092898207127158\n", " node_ip: 127.0.0.1\n", " pid: 45991\n", " time_since_restore: 0.10495114326477051\n", " time_this_iter_s: 0.10495114326477051\n", " time_total_s: 0.10495114326477051\n", " timestamp: 1658499587\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 558f27aa\n", " warmup_time: 0.0029478073120117188\n", " \n", "Result for objective_5592ca68:\n", " date: 2022-07-22_15-19-47\n", " done: false\n", " experiment_id: 1de409fb58a84c3bba522620ec7e1fe4\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 12.79774504616053\n", " neg_mean_loss: -12.79774504616053\n", " node_ip: 127.0.0.1\n", " pid: 45992\n", " time_since_restore: 0.10448837280273438\n", " time_this_iter_s: 0.10448837280273438\n", " time_total_s: 0.10448837280273438\n", " timestamp: 1658499587\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 5592ca68\n", " warmup_time: 0.0027618408203125\n", " \n", "Result for objective_53b39e84:\n", " date: 2022-07-22_15-19-49\n", " done: false\n", " experiment_id: abfeeaaa7eae4ad798c655bed2d1e223\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 46\n", " iterations_since_restore: 47\n", " mean_loss: 4.668111815725785\n", " neg_mean_loss: -4.668111815725785\n", " node_ip: 127.0.0.1\n", " pid: 45983\n", " time_since_restore: 5.113715171813965\n", " time_this_iter_s: 0.10550904273986816\n", " time_total_s: 5.113715171813965\n", " timestamp: 1658499589\n", " timesteps_since_restore: 0\n", " training_iteration: 47\n", " trial_id: 53b39e84\n", " warmup_time: 0.002878904342651367\n", " \n", "Result for objective_558c6132:\n", " date: 2022-07-22_15-19-52\n", " done: false\n", " experiment_id: 98df2ca60c5f4a33a3da6870c3d8db4c\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: 5.2245036670526375\n", " neg_mean_loss: -5.2245036670526375\n", " node_ip: 127.0.0.1\n", " pid: 45990\n", " time_since_restore: 5.1552488803863525\n", " time_this_iter_s: 0.10927176475524902\n", " time_total_s: 5.1552488803863525\n", " timestamp: 1658499592\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: 558c6132\n", " warmup_time: 0.003049135208129883\n", " \n", "Result for objective_558f27aa:\n", " date: 2022-07-22_15-19-52\n", " done: false\n", " experiment_id: 73def1ffc7f443669328b88695453248\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: 14.483670237907232\n", " neg_mean_loss: -14.483670237907232\n", " node_ip: 127.0.0.1\n", " pid: 45991\n", " time_since_restore: 5.174212217330933\n", " time_this_iter_s: 0.10701107978820801\n", " time_total_s: 5.174212217330933\n", " timestamp: 1658499592\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: 558f27aa\n", " warmup_time: 0.0029478073120117188\n", " \n", "Result for objective_5592ca68:\n", " date: 2022-07-22_15-19-52\n", " done: false\n", " experiment_id: 1de409fb58a84c3bba522620ec7e1fe4\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: 3.3171620341364583\n", " neg_mean_loss: -3.3171620341364583\n", " node_ip: 127.0.0.1\n", " pid: 45992\n", " time_since_restore: 5.171269178390503\n", " time_this_iter_s: 0.10331320762634277\n", " time_total_s: 5.171269178390503\n", " timestamp: 1658499592\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: 5592ca68\n", " warmup_time: 0.0027618408203125\n", " \n", "Result for objective_53b39e84:\n", " date: 2022-07-22_15-19-54\n", " done: false\n", " experiment_id: abfeeaaa7eae4ad798c655bed2d1e223\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 93\n", " iterations_since_restore: 94\n", " mean_loss: 4.148686061272061\n", " neg_mean_loss: -4.148686061272061\n", " node_ip: 127.0.0.1\n", " pid: 45983\n", " time_since_restore: 10.137071132659912\n", " time_this_iter_s: 0.10901093482971191\n", " time_total_s: 10.137071132659912\n", " timestamp: 1658499594\n", " timesteps_since_restore: 0\n", " training_iteration: 94\n", " trial_id: 53b39e84\n", " warmup_time: 0.002878904342651367\n", " \n", "Result for objective_53b39e84:\n", " date: 2022-07-22_15-19-54\n", " done: true\n", " experiment_id: abfeeaaa7eae4ad798c655bed2d1e223\n", " experiment_tag: 5_activation=relu_tanh,height=25.7812,steps=100,width=1.7770\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 4.1159662035734215\n", " neg_mean_loss: -4.1159662035734215\n", " node_ip: 127.0.0.1\n", " pid: 45983\n", " time_since_restore: 10.796404361724854\n", " time_this_iter_s: 0.10543107986450195\n", " time_total_s: 10.796404361724854\n", " timestamp: 1658499594\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 53b39e84\n", " warmup_time: 0.002878904342651367\n", " \n", "Result for objective_5b999414:\n", " date: 2022-07-22_15-19-57\n", " done: false\n", " experiment_id: 5548713c4f5046c0a44e1946db840d73\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 12.103540718376786\n", " neg_mean_loss: -12.103540718376786\n", " node_ip: 127.0.0.1\n", " pid: 46011\n", " time_since_restore: 0.10435795783996582\n", " time_this_iter_s: 0.10435795783996582\n", " time_total_s: 0.10435795783996582\n", " timestamp: 1658499597\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 5b999414\n", " warmup_time: 0.002862691879272461\n", " \n", "Result for objective_558c6132:\n", " date: 2022-07-22_15-19-57\n", " done: false\n", " experiment_id: 98df2ca60c5f4a33a3da6870c3d8db4c\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " mean_loss: 4.790299222921972\n", " neg_mean_loss: -4.790299222921972\n", " node_ip: 127.0.0.1\n", " pid: 45990\n", " time_since_restore: 10.228116989135742\n", " time_this_iter_s: 0.146165132522583\n", " time_total_s: 10.228116989135742\n", " timestamp: 1658499597\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: 558c6132\n", " warmup_time: 0.003049135208129883\n", " \n", "Result for objective_558f27aa:\n", " date: 2022-07-22_15-19-57\n", " done: false\n", " experiment_id: 73def1ffc7f443669328b88695453248\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " mean_loss: 13.944411303108408\n", " neg_mean_loss: -13.944411303108408\n", " node_ip: 127.0.0.1\n", " pid: 45991\n", " time_since_restore: 10.248456001281738\n", " time_this_iter_s: 0.13899707794189453\n", " time_total_s: 10.248456001281738\n", " timestamp: 1658499597\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: 558f27aa\n", " warmup_time: 0.0029478073120117188\n", " \n", "Result for objective_5592ca68:\n", " date: 2022-07-22_15-19-57\n", " done: false\n", " experiment_id: 1de409fb58a84c3bba522620ec7e1fe4\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " mean_loss: 3.064378230421662\n", " neg_mean_loss: -3.064378230421662\n", " node_ip: 127.0.0.1\n", " pid: 45992\n", " time_since_restore: 10.244185209274292\n", " time_this_iter_s: 0.16223502159118652\n", " time_total_s: 10.244185209274292\n", " timestamp: 1658499597\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: 5592ca68\n", " warmup_time: 0.0027618408203125\n", " \n", "Result for objective_558c6132:\n", " date: 2022-07-22_15-19-58\n", " done: true\n", " experiment_id: 98df2ca60c5f4a33a3da6870c3d8db4c\n", " experiment_tag: 6_activation=relu_tanh,height=33.1252,steps=100,width=2.1202\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 4.767266385536331\n", " neg_mean_loss: -4.767266385536331\n", " node_ip: 127.0.0.1\n", " pid: 45990\n", " time_since_restore: 10.765881776809692\n", " time_this_iter_s: 0.10750794410705566\n", " time_total_s: 10.765881776809692\n", " timestamp: 1658499598\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 558c6132\n", " warmup_time: 0.003049135208129883\n", " \n", "Result for objective_558f27aa:\n", " date: 2022-07-22_15-19-58\n", " done: true\n", " experiment_id: 73def1ffc7f443669328b88695453248\n", " experiment_tag: 7_activation=relu_tanh,height=40.9290,steps=100,width=0.0138\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 13.890665978269285\n", " neg_mean_loss: -13.890665978269285\n", " node_ip: 127.0.0.1\n", " pid: 45991\n", " time_since_restore: 10.784975051879883\n", " time_this_iter_s: 0.10729503631591797\n", " time_total_s: 10.784975051879883\n", " timestamp: 1658499598\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 558f27aa\n", " warmup_time: 0.0029478073120117188\n", " \n", "Result for objective_5592ca68:\n", " date: 2022-07-22_15-19-58\n", " done: true\n", " experiment_id: 1de409fb58a84c3bba522620ec7e1fe4\n", " experiment_tag: 8_activation=relu_tanh,height=17.9775,steps=100,width=3.8835\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 3.0512532903599245\n", " neg_mean_loss: -3.0512532903599245\n", " node_ip: 127.0.0.1\n", " pid: 45992\n", " time_since_restore: 10.780152082443237\n", " time_this_iter_s: 0.10820770263671875\n", " time_total_s: 10.780152082443237\n", " timestamp: 1658499598\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 5592ca68\n", " warmup_time: 0.0027618408203125\n", " \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Result for objective_5d8462c2:\n", " date: 2022-07-22_15-20-00\n", " done: false\n", " experiment_id: 37975742a6894daeb580d222effadf3c\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 13.491949373944276\n", " neg_mean_loss: -13.491949373944276\n", " node_ip: 127.0.0.1\n", " pid: 46018\n", " time_since_restore: 0.1021728515625\n", " time_this_iter_s: 0.1021728515625\n", " time_total_s: 0.1021728515625\n", " timestamp: 1658499600\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 5d8462c2\n", " warmup_time: 0.0030527114868164062\n", " \n", "Result for objective_5b999414:\n", " date: 2022-07-22_15-20-02\n", " done: false\n", " experiment_id: 5548713c4f5046c0a44e1946db840d73\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 29\n", " iterations_since_restore: 30\n", " mean_loss: 3.0563505542773735\n", " neg_mean_loss: -3.0563505542773735\n", " node_ip: 127.0.0.1\n", " pid: 46011\n", " time_since_restore: 5.116016149520874\n", " time_this_iter_s: 0.1060340404510498\n", " time_total_s: 5.116016149520874\n", " timestamp: 1658499602\n", " timesteps_since_restore: 0\n", " training_iteration: 30\n", " trial_id: 5b999414\n", " warmup_time: 0.002862691879272461\n", " \n", "Result for objective_5d8462c2:\n", " date: 2022-07-22_15-20-05\n", " done: false\n", " experiment_id: 37975742a6894daeb580d222effadf3c\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: 3.9441144421537513\n", " neg_mean_loss: -3.9441144421537513\n", " node_ip: 127.0.0.1\n", " pid: 46018\n", " time_since_restore: 5.143904685974121\n", " time_this_iter_s: 0.1046457290649414\n", " time_total_s: 5.143904685974121\n", " timestamp: 1658499605\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: 5d8462c2\n", " warmup_time: 0.0030527114868164062\n", " \n", "Result for objective_5b999414:\n", " date: 2022-07-22_15-20-07\n", " done: false\n", " experiment_id: 5548713c4f5046c0a44e1946db840d73\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 76\n", " iterations_since_restore: 77\n", " mean_loss: 2.4898772989369\n", " neg_mean_loss: -2.4898772989369\n", " node_ip: 127.0.0.1\n", " pid: 46011\n", " time_since_restore: 10.162328004837036\n", " time_this_iter_s: 0.10597825050354004\n", " time_total_s: 10.162328004837036\n", " timestamp: 1658499607\n", " timesteps_since_restore: 0\n", " training_iteration: 77\n", " trial_id: 5b999414\n", " warmup_time: 0.002862691879272461\n", " \n", "Result for objective_5b999414:\n", " date: 2022-07-22_15-20-09\n", " done: true\n", " experiment_id: 5548713c4f5046c0a44e1946db840d73\n", " experiment_tag: 9_activation=relu_tanh,height=11.0354,steps=100,width=3.2742\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 2.4028084118568103\n", " neg_mean_loss: -2.4028084118568103\n", " node_ip: 127.0.0.1\n", " pid: 46011\n", " time_since_restore: 12.626188039779663\n", " time_this_iter_s: 0.10697293281555176\n", " time_total_s: 12.626188039779663\n", " timestamp: 1658499609\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 5b999414\n", " warmup_time: 0.002862691879272461\n", " \n", "Result for objective_5d8462c2:\n", " date: 2022-07-22_15-20-10\n", " done: false\n", " experiment_id: 37975742a6894daeb580d222effadf3c\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " mean_loss: 3.723261470545891\n", " neg_mean_loss: -3.723261470545891\n", " node_ip: 127.0.0.1\n", " pid: 46018\n", " time_since_restore: 10.198927879333496\n", " time_this_iter_s: 0.10776281356811523\n", " time_total_s: 10.198927879333496\n", " timestamp: 1658499610\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: 5d8462c2\n", " warmup_time: 0.0030527114868164062\n", " \n", "Result for objective_5d8462c2:\n", " date: 2022-07-22_15-20-11\n", " done: true\n", " experiment_id: 37975742a6894daeb580d222effadf3c\n", " experiment_tag: 10_activation=relu_tanh,height=24.9195,steps=100,width=4.4927\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 3.711835922326217\n", " neg_mean_loss: -3.711835922326217\n", " node_ip: 127.0.0.1\n", " pid: 46018\n", " time_since_restore: 10.733852863311768\n", " time_this_iter_s: 0.10374593734741211\n", " time_total_s: 10.733852863311768\n", " timestamp: 1658499611\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 5d8462c2\n", " warmup_time: 0.0030527114868164062\n", " \n" ] } ], "source": [ "tuner = tune.Tuner(\n", " objective,\n", " tune_config=tune.TuneConfig(\n", " metric=\"mean_loss\",\n", " mode=\"min\",\n", " search_alg=algo,\n", " num_samples=num_samples,\n", " ),\n", " param_space=search_config,\n", ")\n", "results = tuner.fit()" ] }, { "cell_type": "markdown", "id": "71e6f6ec", "metadata": {}, "source": [ "Here are the hyperparameters found to minimize the mean loss of the defined objective." ] }, { "cell_type": "code", "execution_count": 20, "id": "b657dccf", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Best hyperparameters found were: {'steps': 100, 'width': 15.124213652112319, 'height': -97.27768667042203, 'activation': 'relu, tanh'}\n" ] } ], "source": [ "print(\"Best hyperparameters found were: \", results.get_best_result().config)" ] }, { "cell_type": "code", "execution_count": 21, "id": "505995af", "metadata": { "tags": [ "remove-cell" ] }, "outputs": [], "source": [ "ray.shutdown()" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.7" }, "orphan": true }, "nbformat": 4, "nbformat_minor": 5 }