{ "cells": [ { "cell_type": "markdown", "id": "a97c49a9", "metadata": {}, "source": [ "# Running Tune experiments with ZOOpt\n", "\n", "In this tutorial we introduce ZOOpt, while running a simple Ray Tune experiment. Tune’s Search Algorithms integrate with ZOOpt and, as a result, allow you to seamlessly scale up a ZOOpt optimization process - without sacrificing performance.\n", "\n", "Zeroth-order optimization (ZOOpt) does not rely on the gradient of the objective function, but instead, learns from samples of the search space. It is suitable for optimizing functions that are nondifferentiable, with many local minima, or even unknown but only testable. Therefore, zeroth-order optimization is commonly referred to as \"derivative-free optimization\" and \"black-box optimization\". In this example we minimize a simple objective to briefly demonstrate the usage of ZOOpt with Ray Tune via `ZOOptSearch`. It's useful to keep in mind that despite the emphasis on machine learning experiments, Ray Tune optimizes any implicit or explicit objective. Here we assume `zoopt==0.4.1` library is installed. To learn more, please refer to the [ZOOpt website](https://github.com/polixir/ZOOpt)." ] }, { "cell_type": "code", "execution_count": 1, "id": "58fee596", "metadata": { "tags": [ "remove-cell" ] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: zoopt==0.4.1 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (0.4.1)\n", "Requirement already satisfied: matplotlib in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from zoopt==0.4.1) (3.5.0)\n", "Requirement already satisfied: numpy in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from zoopt==0.4.1) (1.21.6)\n", "Requirement already satisfied: cycler>=0.10 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from matplotlib->zoopt==0.4.1) (0.11.0)\n", "Requirement already satisfied: python-dateutil>=2.7 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from matplotlib->zoopt==0.4.1) (2.8.2)\n", "Requirement already satisfied: fonttools>=4.22.0 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from matplotlib->zoopt==0.4.1) (4.28.2)\n", "Requirement already satisfied: packaging>=20.0 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from matplotlib->zoopt==0.4.1) (21.3)\n", "Requirement already satisfied: pillow>=6.2.0 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from matplotlib->zoopt==0.4.1) (9.1.0)\n", "Requirement already satisfied: setuptools-scm>=4 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from matplotlib->zoopt==0.4.1) (6.3.2)\n", "Requirement already satisfied: kiwisolver>=1.0.1 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from matplotlib->zoopt==0.4.1) (1.3.2)\n", "Requirement already satisfied: pyparsing>=2.2.1 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from matplotlib->zoopt==0.4.1) (2.4.7)\n", "Requirement already satisfied: six>=1.5 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from python-dateutil>=2.7->matplotlib->zoopt==0.4.1) (1.16.0)\n", "Requirement already satisfied: setuptools in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from setuptools-scm>=4->matplotlib->zoopt==0.4.1) (59.5.0)\n", "Requirement already satisfied: tomli>=1.0.0 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from setuptools-scm>=4->matplotlib->zoopt==0.4.1) (1.2.3)\n", "\u001b[33mWARNING: There was an error checking the latest version of pip.\u001b[0m\u001b[33m\n", "\u001b[0m" ] } ], "source": [ "# !pip install ray[tune]\n", "!pip install zoopt==0.4.1" ] }, { "cell_type": "markdown", "id": "a9c8a34b", "metadata": {}, "source": [ "Click below to see all the imports we need for this example.\n", "You can also launch directly into a Binder instance to run this notebook yourself.\n", "Just click on the rocket symbol at the top of the navigation." ] }, { "cell_type": "code", "execution_count": 2, "id": "d05017fe", "metadata": { "tags": [ "hide-input" ] }, "outputs": [], "source": [ "import time\n", "\n", "import ray\n", "from ray import tune\n", "from ray.air import session\n", "from ray.tune.search.zoopt import ZOOptSearch\n", "from zoopt import ValueType" ] }, { "cell_type": "markdown", "id": "5a41255a", "metadata": {}, "source": [ "Let's start by defining a simple evaluation function.\n", "We artificially sleep for a bit (`0.1` seconds) to simulate a long-running ML experiment.\n", "This setup assumes that we're running multiple `step`s of an experiment and try to tune two hyperparameters,\n", "namely `width` and `height`, and `activation`." ] }, { "cell_type": "code", "execution_count": 3, "id": "61db0806", "metadata": {}, "outputs": [], "source": [ "def evaluate(step, width, height):\n", " time.sleep(0.1)\n", " return (0.1 + width * step / 100) ** (-1) + height * 0.1" ] }, { "cell_type": "markdown", "id": "a979f791", "metadata": {}, "source": [ "Next, our ``objective`` function takes a Tune ``config``, evaluates the `score` of your experiment in a training loop,\n", "and uses `session.report` to report the `score` back to Tune." ] }, { "cell_type": "code", "execution_count": 4, "id": "3b451a2c", "metadata": {}, "outputs": [], "source": [ "def objective(config):\n", " for step in range(config[\"steps\"]):\n", " score = evaluate(step, config[\"width\"], config[\"height\"])\n", " session.report({\"iterations\": step, \"mean_loss\": score})" ] }, { "cell_type": "code", "execution_count": 5, "id": "ad4f9faf", "metadata": { "lines_to_next_cell": 0, "tags": [ "remove-cell" ] }, "outputs": [ { "data": { "text/html": [ "
\n", "
\n", "

Ray

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", "\n", "
Python version:3.7.7
Ray version: 2.0.0rc0
Dashboard:http://127.0.0.1:8266
\n", "
\n", "
\n" ], "text/plain": [ "RayContext(dashboard_url='127.0.0.1:8266', python_version='3.7.7', ray_version='2.0.0rc0', ray_commit='{{RAY_COMMIT_SHA}}', address_info={'node_ip_address': '127.0.0.1', 'raylet_ip_address': '127.0.0.1', 'redis_address': None, 'object_store_address': '/tmp/ray/session_2022-07-22_15-35-29_724425_47582/sockets/plasma_store', 'raylet_socket_name': '/tmp/ray/session_2022-07-22_15-35-29_724425_47582/sockets/raylet', 'webui_url': '127.0.0.1:8266', 'session_dir': '/tmp/ray/session_2022-07-22_15-35-29_724425_47582', 'metrics_export_port': 63508, 'gcs_address': '127.0.0.1:65155', 'address': '127.0.0.1:65155', 'dashboard_agent_listen_port': 52365, 'node_id': 'cecb81f61b8504b3ccfcd881ed1c79c3b7d184be1ba13216cb7e3957'})" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "ray.init(configure_logging=False)" ] }, { "cell_type": "markdown", "id": "036e0085", "metadata": {}, "source": [ "Next we define a search space. The critical assumption is that the optimal hyperparameters live within this space. Yet, if the space is very large, then those hyperparameters may be difficult to find in a short amount of time." ] }, { "cell_type": "code", "execution_count": 6, "id": "b28469ce", "metadata": {}, "outputs": [], "source": [ "search_config = {\n", " \"steps\": 100,\n", " \"width\": tune.randint(0, 10),\n", " \"height\": tune.quniform(-10, 10, 1e-2),\n", " \"activation\": tune.choice([\"relu, tanh\"])\n", "}" ] }, { "cell_type": "markdown", "id": "599b1ece", "metadata": {}, "source": [ "The number of samples is the number of hyperparameter combinations that will be tried out. This Tune run is set to `1000` samples.\n", "(you can decrease this if it takes too long on your machine)." ] }, { "cell_type": "code", "execution_count": 7, "id": "94fcbc63", "metadata": {}, "outputs": [], "source": [ "num_samples = 1000" ] }, { "cell_type": "code", "execution_count": 8, "id": "33f11052", "metadata": { "tags": [ "remove-cell" ] }, "outputs": [], "source": [ "# If 1000 samples take too long, you can reduce this number.\n", "# We override this number here for our smoke tests.\n", "num_samples = 10" ] }, { "cell_type": "markdown", "id": "c5b47448", "metadata": {}, "source": [ "Next we define the search algorithm built from `ZOOptSearch`, constrained to a maximum of `8` concurrent trials via ZOOpt's internal `\"parallel_num\"`." ] }, { "cell_type": "code", "execution_count": 9, "id": "22c15bc5", "metadata": {}, "outputs": [], "source": [ "zoopt_config = {\n", " \"parallel_num\": 8\n", "}\n", "algo = ZOOptSearch(\n", " algo=\"Asracos\", # only supports ASRacos currently\n", " budget=num_samples,\n", " **zoopt_config,\n", ")" ] }, { "cell_type": "markdown", "id": "f5147bec", "metadata": {}, "source": [ "Finally, we run the experiment to `\"min\"`imize the \"mean_loss\" of the `objective` by searching `search_config` via `algo`, `num_samples` times. This previous sentence is fully characterizes the search problem we aim to solve. With this in mind, notice how efficient it is to execute `tuner.fit()`." ] }, { "cell_type": "code", "execution_count": 10, "id": "b51e7a31", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Function checkpointing is disabled. This may result in unexpected behavior when using checkpointing features or certain schedulers. To enable, set the train function arguments to be `func(config, checkpoint_dir=None)`.\n" ] }, { "data": { "text/html": [ "== Status ==
Current time: 2022-07-22 15:36:05 (running for 00:00:29.94)
Memory usage on this node: 11.1/16.0 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/5.39 GiB heap, 0.0/2.0 GiB objects
Current best trial: 963faf2a with mean_loss=-0.39843708609271516 and parameters={'steps': 100, 'width': 6, 'height': -5.64, 'activation': 'relu, tanh'}
Result logdir: /Users/kai/ray_results/objective_2022-07-22_15-35-34
Number of trials: 7/10 (7 TERMINATED)
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
Trial name status loc activation height width loss iter total time (s) iterations neg_mean_loss
objective_8c72f588TERMINATED127.0.0.1:47662relu, tanh -3.94 0 9.606 100 10.9102 99 -9.606
objective_8e2f11aeTERMINATED127.0.0.1:47667relu, tanh -0.68 6 0.0975629 100 10.7479 99 -0.0975629
objective_8e30a596TERMINATED127.0.0.1:47668relu, tanh -5.84 0 9.416 100 10.7724 99 -9.416
objective_8e32324eTERMINATED127.0.0.1:47669relu, tanh -2.15 3 0.110733 100 10.7684 99 -0.110733
objective_963faf2aTERMINATED127.0.0.1:47689relu, tanh -5.64 6-0.398437 100 10.8267 99 0.398437
objective_96417df0TERMINATED127.0.0.1:47690relu, tanh 2.84 6 0.449563 100 10.821 99 -0.449563
objective_96435da0TERMINATED127.0.0.1:47691relu, tanh 2.6 6 0.425563 100 10.7694 99 -0.425563


" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "Result for objective_8c72f588:\n", " date: 2022-07-22_15-35-38\n", " done: false\n", " experiment_id: 3eb9bcef55e341b0970abd6c1f97eda7\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 9.606\n", " neg_mean_loss: -9.606\n", " node_ip: 127.0.0.1\n", " pid: 47662\n", " time_since_restore: 0.10410094261169434\n", " time_this_iter_s: 0.10410094261169434\n", " time_total_s: 0.10410094261169434\n", " timestamp: 1658500538\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 8c72f588\n", " warmup_time: 0.003092050552368164\n", " \n", "Result for objective_8e30a596:\n", " date: 2022-07-22_15-35-41\n", " done: false\n", " experiment_id: d58453075b71453ab615e10ae9713072\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 9.416\n", " neg_mean_loss: -9.416\n", " node_ip: 127.0.0.1\n", " pid: 47668\n", " time_since_restore: 0.1051950454711914\n", " time_this_iter_s: 0.1051950454711914\n", " time_total_s: 0.1051950454711914\n", " timestamp: 1658500541\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 8e30a596\n", " warmup_time: 0.004169940948486328\n", " \n", "Result for objective_8e32324e:\n", " date: 2022-07-22_15-35-41\n", " done: false\n", " experiment_id: 22c7ba8baa2644479b661e6d91e5fae8\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 9.785\n", " neg_mean_loss: -9.785\n", " node_ip: 127.0.0.1\n", " pid: 47669\n", " time_since_restore: 0.10500001907348633\n", " time_this_iter_s: 0.10500001907348633\n", " time_total_s: 0.10500001907348633\n", " timestamp: 1658500541\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 8e32324e\n", " warmup_time: 0.004729032516479492\n", " \n", "Result for objective_8e2f11ae:\n", " date: 2022-07-22_15-35-41\n", " done: false\n", " experiment_id: c4d325574058491c8af3a0c869f0ebe5\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 9.932\n", " neg_mean_loss: -9.932\n", " node_ip: 127.0.0.1\n", " pid: 47667\n", " time_since_restore: 0.10310196876525879\n", " time_this_iter_s: 0.10310196876525879\n", " time_total_s: 0.10310196876525879\n", " timestamp: 1658500541\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 8e2f11ae\n", " warmup_time: 0.0029730796813964844\n", " \n", "Result for objective_8c72f588:\n", " date: 2022-07-22_15-35-43\n", " done: false\n", " experiment_id: 3eb9bcef55e341b0970abd6c1f97eda7\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 45\n", " iterations_since_restore: 46\n", " mean_loss: 9.606\n", " neg_mean_loss: -9.606\n", " node_ip: 127.0.0.1\n", " pid: 47662\n", " time_since_restore: 5.112913131713867\n", " time_this_iter_s: 0.10695910453796387\n", " time_total_s: 5.112913131713867\n", " timestamp: 1658500543\n", " timesteps_since_restore: 0\n", " training_iteration: 46\n", " trial_id: 8c72f588\n", " warmup_time: 0.003092050552368164\n", " \n", "Result for objective_8e30a596:\n", " date: 2022-07-22_15-35-46\n", " done: false\n", " experiment_id: d58453075b71453ab615e10ae9713072\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: 9.416\n", " neg_mean_loss: -9.416\n", " node_ip: 127.0.0.1\n", " pid: 47668\n", " time_since_restore: 5.1615166664123535\n", " time_this_iter_s: 0.10595178604125977\n", " time_total_s: 5.1615166664123535\n", " timestamp: 1658500546\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: 8e30a596\n", " warmup_time: 0.004169940948486328\n", " \n", "Result for objective_8e2f11ae:\n", " date: 2022-07-22_15-35-46\n", " done: false\n", " experiment_id: c4d325574058491c8af3a0c869f0ebe5\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: 0.2744657534246575\n", " neg_mean_loss: -0.2744657534246575\n", " node_ip: 127.0.0.1\n", " pid: 47667\n", " time_since_restore: 5.1498119831085205\n", " time_this_iter_s: 0.10741090774536133\n", " time_total_s: 5.1498119831085205\n", " timestamp: 1658500546\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: 8e2f11ae\n", " warmup_time: 0.0029730796813964844\n", " \n", "Result for objective_8e32324e:\n", " date: 2022-07-22_15-35-46\n", " done: false\n", " experiment_id: 22c7ba8baa2644479b661e6d91e5fae8\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: 0.44725165562913916\n", " neg_mean_loss: -0.44725165562913916\n", " node_ip: 127.0.0.1\n", " pid: 47669\n", " time_since_restore: 5.166383981704712\n", " time_this_iter_s: 0.1064291000366211\n", " time_total_s: 5.166383981704712\n", " timestamp: 1658500546\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: 8e32324e\n", " warmup_time: 0.004729032516479492\n", " \n", "Result for objective_8c72f588:\n", " date: 2022-07-22_15-35-48\n", " done: false\n", " experiment_id: 3eb9bcef55e341b0970abd6c1f97eda7\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 92\n", " iterations_since_restore: 93\n", " mean_loss: 9.606\n", " neg_mean_loss: -9.606\n", " node_ip: 127.0.0.1\n", " pid: 47662\n", " time_since_restore: 10.156940937042236\n", " time_this_iter_s: 0.10845208168029785\n", " time_total_s: 10.156940937042236\n", " timestamp: 1658500548\n", " timesteps_since_restore: 0\n", " training_iteration: 93\n", " trial_id: 8c72f588\n", " warmup_time: 0.003092050552368164\n", " \n", "Result for objective_8c72f588:\n", " date: 2022-07-22_15-35-49\n", " done: true\n", " experiment_id: 3eb9bcef55e341b0970abd6c1f97eda7\n", " experiment_tag: 1_activation=relu_tanh,height=-3.9400,steps=100,width=0\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 9.606\n", " neg_mean_loss: -9.606\n", " node_ip: 127.0.0.1\n", " pid: 47662\n", " time_since_restore: 10.910246133804321\n", " time_this_iter_s: 0.1059122085571289\n", " time_total_s: 10.910246133804321\n", " timestamp: 1658500549\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 8c72f588\n", " warmup_time: 0.003092050552368164\n", " \n", "Result for objective_8e2f11ae:\n", " date: 2022-07-22_15-35-51\n", " done: false\n", " experiment_id: c4d325574058491c8af3a0c869f0ebe5\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " mean_loss: 0.10621602787456447\n", " neg_mean_loss: -0.10621602787456447\n", " node_ip: 127.0.0.1\n", " pid: 47667\n", " time_since_restore: 10.211436986923218\n", " time_this_iter_s: 0.10804891586303711\n", " time_total_s: 10.211436986923218\n", " timestamp: 1658500551\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: 8e2f11ae\n", " warmup_time: 0.0029730796813964844\n", " \n", "Result for objective_8e32324e:\n", " date: 2022-07-22_15-35-51\n", " done: false\n", " experiment_id: 22c7ba8baa2644479b661e6d91e5fae8\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " mean_loss: 0.12746575342465752\n", " neg_mean_loss: -0.12746575342465752\n", " node_ip: 127.0.0.1\n", " pid: 47669\n", " time_since_restore: 10.228847980499268\n", " time_this_iter_s: 0.10761308670043945\n", " time_total_s: 10.228847980499268\n", " timestamp: 1658500551\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: 8e32324e\n", " warmup_time: 0.004729032516479492\n", " \n", "Result for objective_8e30a596:\n", " date: 2022-07-22_15-35-51\n", " done: false\n", " experiment_id: d58453075b71453ab615e10ae9713072\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " mean_loss: 9.416\n", " neg_mean_loss: -9.416\n", " node_ip: 127.0.0.1\n", " pid: 47668\n", " time_since_restore: 10.231056928634644\n", " time_this_iter_s: 0.10677409172058105\n", " time_total_s: 10.231056928634644\n", " timestamp: 1658500551\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: 8e30a596\n", " warmup_time: 0.004169940948486328\n", " \n", "Result for objective_8e2f11ae:\n", " date: 2022-07-22_15-35-52\n", " done: true\n", " experiment_id: c4d325574058491c8af3a0c869f0ebe5\n", " experiment_tag: 2_activation=relu_tanh,height=-0.6800,steps=100,width=6\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 0.09756291390728478\n", " neg_mean_loss: -0.09756291390728478\n", " node_ip: 127.0.0.1\n", " pid: 47667\n", " time_since_restore: 10.747868061065674\n", " time_this_iter_s: 0.10819792747497559\n", " time_total_s: 10.747868061065674\n", " timestamp: 1658500552\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 8e2f11ae\n", " warmup_time: 0.0029730796813964844\n", " \n", "Result for objective_8e32324e:\n", " date: 2022-07-22_15-35-52\n", " done: true\n", " experiment_id: 22c7ba8baa2644479b661e6d91e5fae8\n", " experiment_tag: 4_activation=relu_tanh,height=-2.1500,steps=100,width=3\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 0.11073289902280128\n", " neg_mean_loss: -0.11073289902280128\n", " node_ip: 127.0.0.1\n", " pid: 47669\n", " time_since_restore: 10.768368005752563\n", " time_this_iter_s: 0.10648989677429199\n", " time_total_s: 10.768368005752563\n", " timestamp: 1658500552\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 8e32324e\n", " warmup_time: 0.004729032516479492\n", " \n", "Result for objective_8e30a596:\n", " date: 2022-07-22_15-35-52\n", " done: true\n", " experiment_id: d58453075b71453ab615e10ae9713072\n", " experiment_tag: 3_activation=relu_tanh,height=-5.8400,steps=100,width=0\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 9.416\n", " neg_mean_loss: -9.416\n", " node_ip: 127.0.0.1\n", " pid: 47668\n", " time_since_restore: 10.7723867893219\n", " time_this_iter_s: 0.10686278343200684\n", " time_total_s: 10.7723867893219\n", " timestamp: 1658500552\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 8e30a596\n", " warmup_time: 0.004169940948486328\n", " \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Result for objective_96417df0:\n", " date: 2022-07-22_15-35-54\n", " done: false\n", " experiment_id: e98b245717b4423d8917589cf5d42088\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 10.284\n", " neg_mean_loss: -10.284\n", " node_ip: 127.0.0.1\n", " pid: 47690\n", " time_since_restore: 0.10359907150268555\n", " time_this_iter_s: 0.10359907150268555\n", " time_total_s: 0.10359907150268555\n", " timestamp: 1658500554\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 96417df0\n", " warmup_time: 0.003325939178466797\n", " \n", "Result for objective_96435da0:\n", " date: 2022-07-22_15-35-54\n", " done: false\n", " experiment_id: 8680a80a099c452dadd3be44d3457ca5\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 10.26\n", " neg_mean_loss: -10.26\n", " node_ip: 127.0.0.1\n", " pid: 47691\n", " time_since_restore: 0.10206389427185059\n", " time_this_iter_s: 0.10206389427185059\n", " time_total_s: 0.10206389427185059\n", " timestamp: 1658500554\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 96435da0\n", " warmup_time: 0.002891063690185547\n", " \n", "Result for objective_963faf2a:\n", " date: 2022-07-22_15-35-54\n", " done: false\n", " experiment_id: f28968c2634348c2ae4b0118cc844687\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 9.436\n", " neg_mean_loss: -9.436\n", " node_ip: 127.0.0.1\n", " pid: 47689\n", " time_since_restore: 0.10424089431762695\n", " time_this_iter_s: 0.10424089431762695\n", " time_total_s: 0.10424089431762695\n", " timestamp: 1658500554\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 963faf2a\n", " warmup_time: 0.002835988998413086\n", " \n", "Result for objective_96417df0:\n", " date: 2022-07-22_15-35-59\n", " done: false\n", " experiment_id: e98b245717b4423d8917589cf5d42088\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 46\n", " iterations_since_restore: 47\n", " mean_loss: 0.6336503496503496\n", " neg_mean_loss: -0.6336503496503496\n", " node_ip: 127.0.0.1\n", " pid: 47690\n", " time_since_restore: 5.134061098098755\n", " time_this_iter_s: 0.1091001033782959\n", " time_total_s: 5.134061098098755\n", " timestamp: 1658500559\n", " timesteps_since_restore: 0\n", " training_iteration: 47\n", " trial_id: 96417df0\n", " warmup_time: 0.003325939178466797\n", " \n", "Result for objective_96435da0:\n", " date: 2022-07-22_15-35-59\n", " done: false\n", " experiment_id: 8680a80a099c452dadd3be44d3457ca5\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: 0.6024657534246576\n", " neg_mean_loss: -0.6024657534246576\n", " node_ip: 127.0.0.1\n", " pid: 47691\n", " time_since_restore: 5.1902501583099365\n", " time_this_iter_s: 0.10941314697265625\n", " time_total_s: 5.1902501583099365\n", " timestamp: 1658500559\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: 96435da0\n", " warmup_time: 0.002891063690185547\n", " \n", "Result for objective_963faf2a:\n", " date: 2022-07-22_15-35-59\n", " done: false\n", " experiment_id: f28968c2634348c2ae4b0118cc844687\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 46\n", " iterations_since_restore: 47\n", " mean_loss: -0.21434965034965026\n", " neg_mean_loss: 0.21434965034965026\n", " node_ip: 127.0.0.1\n", " pid: 47689\n", " time_since_restore: 5.137102127075195\n", " time_this_iter_s: 0.10854196548461914\n", " time_total_s: 5.137102127075195\n", " timestamp: 1658500559\n", " timesteps_since_restore: 0\n", " training_iteration: 47\n", " trial_id: 963faf2a\n", " warmup_time: 0.002835988998413086\n", " \n", "Result for objective_96417df0:\n", " date: 2022-07-22_15-36-04\n", " done: false\n", " experiment_id: e98b245717b4423d8917589cf5d42088\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 93\n", " iterations_since_restore: 94\n", " mean_loss: 0.46005633802816903\n", " neg_mean_loss: -0.46005633802816903\n", " node_ip: 127.0.0.1\n", " pid: 47690\n", " time_since_restore: 10.175445079803467\n", " time_this_iter_s: 0.10581207275390625\n", " time_total_s: 10.175445079803467\n", " timestamp: 1658500564\n", " timesteps_since_restore: 0\n", " training_iteration: 94\n", " trial_id: 96417df0\n", " warmup_time: 0.003325939178466797\n", " \n", "Result for objective_96435da0:\n", " date: 2022-07-22_15-36-05\n", " done: false\n", " experiment_id: 8680a80a099c452dadd3be44d3457ca5\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " mean_loss: 0.4342160278745645\n", " neg_mean_loss: -0.4342160278745645\n", " node_ip: 127.0.0.1\n", " pid: 47691\n", " time_since_restore: 10.23218584060669\n", " time_this_iter_s: 0.10732793807983398\n", " time_total_s: 10.23218584060669\n", " timestamp: 1658500565\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: 96435da0\n", " warmup_time: 0.002891063690185547\n", " \n", "Result for objective_963faf2a:\n", " date: 2022-07-22_15-36-05\n", " done: false\n", " experiment_id: f28968c2634348c2ae4b0118cc844687\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 93\n", " iterations_since_restore: 94\n", " mean_loss: -0.38794366197183094\n", " neg_mean_loss: 0.38794366197183094\n", " node_ip: 127.0.0.1\n", " pid: 47689\n", " time_since_restore: 10.181179761886597\n", " time_this_iter_s: 0.10726284980773926\n", " time_total_s: 10.181179761886597\n", " timestamp: 1658500565\n", " timesteps_since_restore: 0\n", " training_iteration: 94\n", " trial_id: 963faf2a\n", " warmup_time: 0.002835988998413086\n", " \n", "Result for objective_96417df0:\n", " date: 2022-07-22_15-36-05\n", " done: true\n", " experiment_id: e98b245717b4423d8917589cf5d42088\n", " experiment_tag: 6_activation=relu_tanh,height=2.8400,steps=100,width=6\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 0.44956291390728476\n", " neg_mean_loss: -0.44956291390728476\n", " node_ip: 127.0.0.1\n", " pid: 47690\n", " time_since_restore: 10.820996046066284\n", " time_this_iter_s: 0.10588788986206055\n", " time_total_s: 10.820996046066284\n", " timestamp: 1658500565\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 96417df0\n", " warmup_time: 0.003325939178466797\n", " \n", "Result for objective_96435da0:\n", " date: 2022-07-22_15-36-05\n", " done: true\n", " experiment_id: 8680a80a099c452dadd3be44d3457ca5\n", " experiment_tag: 7_activation=relu_tanh,height=2.6000,steps=100,width=6\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 0.4255629139072848\n", " neg_mean_loss: -0.4255629139072848\n", " node_ip: 127.0.0.1\n", " pid: 47691\n", " time_since_restore: 10.769440174102783\n", " time_this_iter_s: 0.10849618911743164\n", " time_total_s: 10.769440174102783\n", " timestamp: 1658500565\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 96435da0\n", " warmup_time: 0.002891063690185547\n", " \n", "Result for objective_963faf2a:\n", " date: 2022-07-22_15-36-05\n", " done: true\n", " experiment_id: f28968c2634348c2ae4b0118cc844687\n", " experiment_tag: 5_activation=relu_tanh,height=-5.6400,steps=100,width=6\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: -0.39843708609271516\n", " neg_mean_loss: 0.39843708609271516\n", " node_ip: 127.0.0.1\n", " pid: 47689\n", " time_since_restore: 10.826673984527588\n", " time_this_iter_s: 0.10800504684448242\n", " time_total_s: 10.826673984527588\n", " timestamp: 1658500565\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 963faf2a\n", " warmup_time: 0.002835988998413086\n", " \n" ] } ], "source": [ "tuner = tune.Tuner(\n", " objective,\n", " tune_config=tune.TuneConfig(\n", " metric=\"mean_loss\",\n", " mode=\"min\",\n", " search_alg=algo,\n", " num_samples=num_samples,\n", " ),\n", " param_space=search_config,\n", ")\n", "results = tuner.fit()" ] }, { "cell_type": "markdown", "id": "8b1080a3", "metadata": {}, "source": [ "Here are the hyperparamters found to minimize the mean loss of the defined objective." ] }, { "cell_type": "code", "execution_count": 11, "id": "c2fb0f9c", "metadata": { "lines_to_next_cell": 0 }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Best hyperparameters found were: {'steps': 100, 'width': 6, 'height': -5.64, 'activation': 'relu, tanh'}\n" ] } ], "source": [ "print(\"Best hyperparameters found were: \", results.get_best_result().config)" ] }, { "cell_type": "markdown", "id": "48d0547d", "metadata": {}, "source": [ "## Optional: passing the parameter space into the search algorithm\n", "\n", "We can also pass the parameter space ourselves in the following formats: \n", "- continuous dimensions: (continuous, search_range, precision)\n", "- discrete dimensions: (discrete, search_range, has_order)\n", "- grid dimensions: (grid, grid_list)" ] }, { "cell_type": "code", "execution_count": 12, "id": "00763339", "metadata": {}, "outputs": [], "source": [ "space = {\n", " \"height\": (ValueType.CONTINUOUS, [-10, 10], 1e-2),\n", " \"width\": (ValueType.DISCRETE, [0, 10], True),\n", " \"layers\": (ValueType.GRID, [4, 8, 16])\n", "}" ] }, { "cell_type": "markdown", "id": "54ecd17b", "metadata": {}, "source": [ "ZOOpt again handles constraining the amount of concurrent trials with `\"parallel_num\"`." ] }, { "cell_type": "code", "execution_count": 13, "id": "9ea6d7be", "metadata": {}, "outputs": [], "source": [ "zoopt_search_config = {\n", " \"parallel_num\": 8,\n", " \"metric\": \"mean_loss\",\n", " \"mode\": \"min\"\n", "}\n", "algo = ZOOptSearch(\n", " algo=\"Asracos\",\n", " budget=num_samples,\n", " dim_dict=space,\n", " **zoopt_search_config\n", ")" ] }, { "cell_type": "markdown", "id": "b0b0be8b", "metadata": {}, "source": [ "This time we pass only `\"steps\"` and `\"activation\"` to the Tune `config` because `\"height\"` and `\"width\"` have been passed into `ZOOptSearch` to create the `search_algo`. \n", "Again, we run the experiment to `\"min\"`imize the \"mean_loss\" of the `objective` by searching `search_config` via `algo`, `num_samples` times." ] }, { "cell_type": "code", "execution_count": 14, "id": "7995ede5", "metadata": {}, "outputs": [ { "data": { "text/html": [ "== Status ==
Current time: 2022-07-22 15:36:35 (running for 00:00:29.76)
Memory usage on this node: 8.2/16.0 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/5.39 GiB heap, 0.0/2.0 GiB objects
Current best trial: a7fcf02e with mean_loss=-0.7457524893314368 and parameters={'steps': 100, 'height': -8.88, 'width': 7, 'layers': 16}
Result logdir: /Users/kai/ray_results/objective_2022-07-22_15-36-05
Number of trials: 10/10 (10 TERMINATED)
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
Trial name status loc height layers width loss iter total time (s) iterations neg_mean_loss
objective_9e64c92eTERMINATED127.0.0.1:47713 -8.08 16 0 9.192 100 10.7118 99 -9.192
objective_9ff31930TERMINATED127.0.0.1:47718 0.38 16 7 0.180248 100 10.7315 99 -0.180248
objective_9ff47d0cTERMINATED127.0.0.1:47719 5.09 4 10 0.609 100 10.7924 99 -0.609
objective_9ff5c2b6TERMINATED127.0.0.1:47720 5.26 16 1 1.44343 100 10.7868 99 -1.44343
objective_a7f414d6TERMINATED127.0.0.1:47737 0.38 4 7 0.180248 100 10.7232 99 -0.180248
objective_a7f5c682TERMINATED127.0.0.1:47738 -2.38 16 7-0.0957525 100 10.7337 99 0.0957525
objective_a7f7c162TERMINATED127.0.0.1:47739 0.38 8 7 0.180248 100 10.7452 99 -0.180248
objective_a7f96fdaTERMINATED127.0.0.1:47740 0.38 16 4 0.284305 100 10.7079 99 -0.284305
objective_a7fb1844TERMINATED127.0.0.1:47741 0.38 8 7 0.180248 100 10.7157 99 -0.180248
objective_a7fcf02eTERMINATED127.0.0.1:47742 -8.88 16 7-0.745752 100 10.7305 99 0.745752


" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "Result for objective_9e64c92e:\n", " date: 2022-07-22_15-36-08\n", " done: false\n", " experiment_id: fafb7d88360d408286e616de3dcd4407\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 9.192\n", " neg_mean_loss: -9.192\n", " node_ip: 127.0.0.1\n", " pid: 47713\n", " time_since_restore: 0.1042020320892334\n", " time_this_iter_s: 0.1042020320892334\n", " time_total_s: 0.1042020320892334\n", " timestamp: 1658500568\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 9e64c92e\n", " warmup_time: 0.0030999183654785156\n", " \n", "Result for objective_9ff47d0c:\n", " date: 2022-07-22_15-36-11\n", " done: false\n", " experiment_id: de33a45d0c344d3aa344d8cff20b6c37\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 10.509\n", " neg_mean_loss: -10.509\n", " node_ip: 127.0.0.1\n", " pid: 47719\n", " time_since_restore: 0.10373878479003906\n", " time_this_iter_s: 0.10373878479003906\n", " time_total_s: 0.10373878479003906\n", " timestamp: 1658500571\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 9ff47d0c\n", " warmup_time: 0.003734111785888672\n", " \n", "Result for objective_9ff31930:\n", " date: 2022-07-22_15-36-11\n", " done: false\n", " experiment_id: 404b066d5e154070b5a3f6f6ef3acb33\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 10.038\n", " neg_mean_loss: -10.038\n", " node_ip: 127.0.0.1\n", " pid: 47718\n", " time_since_restore: 0.10415983200073242\n", " time_this_iter_s: 0.10415983200073242\n", " time_total_s: 0.10415983200073242\n", " timestamp: 1658500571\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 9ff31930\n", " warmup_time: 0.0033788681030273438\n", " \n", "Result for objective_9ff5c2b6:\n", " date: 2022-07-22_15-36-11\n", " done: false\n", " experiment_id: aeb7cb0b0c7c4f6692d47dcd1fe36462\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 10.526\n", " neg_mean_loss: -10.526\n", " node_ip: 127.0.0.1\n", " pid: 47720\n", " time_since_restore: 0.10468196868896484\n", " time_this_iter_s: 0.10468196868896484\n", " time_total_s: 0.10468196868896484\n", " timestamp: 1658500571\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 9ff5c2b6\n", " warmup_time: 0.0027132034301757812\n", " \n", "Result for objective_9e64c92e:\n", " date: 2022-07-22_15-36-13\n", " done: false\n", " experiment_id: fafb7d88360d408286e616de3dcd4407\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: 9.192\n", " neg_mean_loss: -9.192\n", " node_ip: 127.0.0.1\n", " pid: 47713\n", " time_since_restore: 5.111851215362549\n", " time_this_iter_s: 0.10771799087524414\n", " time_total_s: 5.111851215362549\n", " timestamp: 1658500573\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: 9e64c92e\n", " warmup_time: 0.0030999183654785156\n", " \n", "Result for objective_9ff47d0c:\n", " date: 2022-07-22_15-36-16\n", " done: false\n", " experiment_id: de33a45d0c344d3aa344d8cff20b6c37\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: 0.7173333333333334\n", " neg_mean_loss: -0.7173333333333334\n", " node_ip: 127.0.0.1\n", " pid: 47719\n", " time_since_restore: 5.179923057556152\n", " time_this_iter_s: 0.12629103660583496\n", " time_total_s: 5.179923057556152\n", " timestamp: 1658500576\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: 9ff47d0c\n", " warmup_time: 0.003734111785888672\n", " \n", "Result for objective_9ff31930:\n", " date: 2022-07-22_15-36-16\n", " done: false\n", " experiment_id: 404b066d5e154070b5a3f6f6ef3acb33\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: 0.3329852507374631\n", " neg_mean_loss: -0.3329852507374631\n", " node_ip: 127.0.0.1\n", " pid: 47718\n", " time_since_restore: 5.155292749404907\n", " time_this_iter_s: 0.10897374153137207\n", " time_total_s: 5.155292749404907\n", " timestamp: 1658500576\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: 9ff31930\n", " warmup_time: 0.0033788681030273438\n", " \n", "Result for objective_9ff5c2b6:\n", " date: 2022-07-22_15-36-16\n", " done: false\n", " experiment_id: aeb7cb0b0c7c4f6692d47dcd1fe36462\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: 2.2803859649122806\n", " neg_mean_loss: -2.2803859649122806\n", " node_ip: 127.0.0.1\n", " pid: 47720\n", " time_since_restore: 5.177261829376221\n", " time_this_iter_s: 0.10725784301757812\n", " time_total_s: 5.177261829376221\n", " timestamp: 1658500576\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: 9ff5c2b6\n", " warmup_time: 0.0027132034301757812\n", " \n", "Result for objective_9e64c92e:\n", " date: 2022-07-22_15-36-18\n", " done: false\n", " experiment_id: fafb7d88360d408286e616de3dcd4407\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " mean_loss: 9.192\n", " neg_mean_loss: -9.192\n", " node_ip: 127.0.0.1\n", " pid: 47713\n", " time_since_restore: 10.170606136322021\n", " time_this_iter_s: 0.10434508323669434\n", " time_total_s: 10.170606136322021\n", " timestamp: 1658500578\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: 9e64c92e\n", " warmup_time: 0.0030999183654785156\n", " \n", "Result for objective_9e64c92e:\n", " date: 2022-07-22_15-36-19\n", " done: true\n", " experiment_id: fafb7d88360d408286e616de3dcd4407\n", " experiment_tag: 1_height=-8.0800,layers=16,steps=100,width=0\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 9.192\n", " neg_mean_loss: -9.192\n", " node_ip: 127.0.0.1\n", " pid: 47713\n", " time_since_restore: 10.711803197860718\n", " time_this_iter_s: 0.10728001594543457\n", " time_total_s: 10.711803197860718\n", " timestamp: 1658500579\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 9e64c92e\n", " warmup_time: 0.0030999183654785156\n", " \n", "Result for objective_9ff31930:\n", " date: 2022-07-22_15-36-21\n", " done: false\n", " experiment_id: 404b066d5e154070b5a3f6f6ef3acb33\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " mean_loss: 0.18770059880239523\n", " neg_mean_loss: -0.18770059880239523\n", " node_ip: 127.0.0.1\n", " pid: 47718\n", " time_since_restore: 10.197859048843384\n", " time_this_iter_s: 0.10710310935974121\n", " time_total_s: 10.197859048843384\n", " timestamp: 1658500581\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: 9ff31930\n", " warmup_time: 0.0033788681030273438\n", " \n", "Result for objective_9ff47d0c:\n", " date: 2022-07-22_15-36-21\n", " done: false\n", " experiment_id: de33a45d0c344d3aa344d8cff20b6c37\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " mean_loss: 0.6142631578947368\n", " neg_mean_loss: -0.6142631578947368\n", " node_ip: 127.0.0.1\n", " pid: 47719\n", " time_since_restore: 10.256150722503662\n", " time_this_iter_s: 0.11135077476501465\n", " time_total_s: 10.256150722503662\n", " timestamp: 1658500581\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: 9ff47d0c\n", " warmup_time: 0.003734111785888672\n", " \n", "Result for objective_9ff5c2b6:\n", " date: 2022-07-22_15-36-21\n", " done: false\n", " experiment_id: aeb7cb0b0c7c4f6692d47dcd1fe36462\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " mean_loss: 1.4875384615384615\n", " neg_mean_loss: -1.4875384615384615\n", " node_ip: 127.0.0.1\n", " pid: 47720\n", " time_since_restore: 10.24931287765503\n", " time_this_iter_s: 0.10830807685852051\n", " time_total_s: 10.24931287765503\n", " timestamp: 1658500581\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: 9ff5c2b6\n", " warmup_time: 0.0027132034301757812\n", " \n", "Result for objective_9ff31930:\n", " date: 2022-07-22_15-36-21\n", " done: true\n", " experiment_id: 404b066d5e154070b5a3f6f6ef3acb33\n", " experiment_tag: 2_height=0.3800,layers=16,steps=100,width=7\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 0.18024751066856332\n", " neg_mean_loss: -0.18024751066856332\n", " node_ip: 127.0.0.1\n", " pid: 47718\n", " time_since_restore: 10.731510877609253\n", " time_this_iter_s: 0.1073448657989502\n", " time_total_s: 10.731510877609253\n", " timestamp: 1658500581\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 9ff31930\n", " warmup_time: 0.0033788681030273438\n", " \n", "Result for objective_9ff47d0c:\n", " date: 2022-07-22_15-36-21\n", " done: true\n", " experiment_id: de33a45d0c344d3aa344d8cff20b6c37\n", " experiment_tag: 3_height=5.0900,layers=4,steps=100,width=10\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 0.609\n", " neg_mean_loss: -0.609\n", " node_ip: 127.0.0.1\n", " pid: 47719\n", " time_since_restore: 10.792427778244019\n", " time_this_iter_s: 0.10576391220092773\n", " time_total_s: 10.792427778244019\n", " timestamp: 1658500581\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 9ff47d0c\n", " warmup_time: 0.003734111785888672\n", " \n", "Result for objective_9ff5c2b6:\n", " date: 2022-07-22_15-36-21\n", " done: true\n", " experiment_id: aeb7cb0b0c7c4f6692d47dcd1fe36462\n", " experiment_tag: 4_height=5.2600,layers=16,steps=100,width=1\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 1.4434311926605505\n", " neg_mean_loss: -1.4434311926605505\n", " node_ip: 127.0.0.1\n", " pid: 47720\n", " time_since_restore: 10.786811828613281\n", " time_this_iter_s: 0.10388994216918945\n", " time_total_s: 10.786811828613281\n", " timestamp: 1658500581\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 9ff5c2b6\n", " warmup_time: 0.0027132034301757812\n", " \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Result for objective_a7fb1844:\n", " date: 2022-07-22_15-36-24\n", " done: false\n", " experiment_id: 668080a52a3f4c4fad2258177b1fb755\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 10.038\n", " neg_mean_loss: -10.038\n", " node_ip: 127.0.0.1\n", " pid: 47741\n", " time_since_restore: 0.10408973693847656\n", " time_this_iter_s: 0.10408973693847656\n", " time_total_s: 0.10408973693847656\n", " timestamp: 1658500584\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: a7fb1844\n", " warmup_time: 0.004377126693725586\n", " \n", "Result for objective_a7f5c682:\n", " date: 2022-07-22_15-36-24\n", " done: false\n", " experiment_id: 4449a3ef690c4b76a0cfbf7c4de7df43\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 9.762\n", " neg_mean_loss: -9.762\n", " node_ip: 127.0.0.1\n", " pid: 47738\n", " time_since_restore: 0.10507631301879883\n", " time_this_iter_s: 0.10507631301879883\n", " time_total_s: 0.10507631301879883\n", " timestamp: 1658500584\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: a7f5c682\n", " warmup_time: 0.003899097442626953\n", " \n", "Result for objective_a7f414d6:\n", " date: 2022-07-22_15-36-24\n", " done: false\n", " experiment_id: 20575f602c234dd7a167a93ff353299b\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 10.038\n", " neg_mean_loss: -10.038\n", " node_ip: 127.0.0.1\n", " pid: 47737\n", " time_since_restore: 0.10182499885559082\n", " time_this_iter_s: 0.10182499885559082\n", " time_total_s: 0.10182499885559082\n", " timestamp: 1658500584\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: a7f414d6\n", " warmup_time: 0.0046689510345458984\n", " \n", "Result for objective_a7f7c162:\n", " date: 2022-07-22_15-36-24\n", " done: false\n", " experiment_id: f4b185b58be045f594bf127addecbace\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 10.038\n", " neg_mean_loss: -10.038\n", " node_ip: 127.0.0.1\n", " pid: 47739\n", " time_since_restore: 0.10427188873291016\n", " time_this_iter_s: 0.10427188873291016\n", " time_total_s: 0.10427188873291016\n", " timestamp: 1658500584\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: a7f7c162\n", " warmup_time: 0.004554033279418945\n", " \n", "Result for objective_a7fcf02e:\n", " date: 2022-07-22_15-36-24\n", " done: false\n", " experiment_id: 5646116e3b69493b920c031d27eeb11b\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 9.112\n", " neg_mean_loss: -9.112\n", " node_ip: 127.0.0.1\n", " pid: 47742\n", " time_since_restore: 0.10358119010925293\n", " time_this_iter_s: 0.10358119010925293\n", " time_total_s: 0.10358119010925293\n", " timestamp: 1658500584\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: a7fcf02e\n", " warmup_time: 0.003072977066040039\n", " \n", "Result for objective_a7f96fda:\n", " date: 2022-07-22_15-36-25\n", " done: false\n", " experiment_id: 1983d487678f485a8a8ae7da4b7495a0\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 10.038\n", " neg_mean_loss: -10.038\n", " node_ip: 127.0.0.1\n", " pid: 47740\n", " time_since_restore: 0.10406088829040527\n", " time_this_iter_s: 0.10406088829040527\n", " time_total_s: 0.10406088829040527\n", " timestamp: 1658500585\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: a7f96fda\n", " warmup_time: 0.0031909942626953125\n", " \n", "Result for objective_a7fb1844:\n", " date: 2022-07-22_15-36-29\n", " done: false\n", " experiment_id: 668080a52a3f4c4fad2258177b1fb755\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: 0.3329852507374631\n", " neg_mean_loss: -0.3329852507374631\n", " node_ip: 127.0.0.1\n", " pid: 47741\n", " time_since_restore: 5.134153842926025\n", " time_this_iter_s: 0.10844898223876953\n", " time_total_s: 5.134153842926025\n", " timestamp: 1658500589\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: a7fb1844\n", " warmup_time: 0.004377126693725586\n", " \n", "Result for objective_a7f5c682:\n", " date: 2022-07-22_15-36-29\n", " done: false\n", " experiment_id: 4449a3ef690c4b76a0cfbf7c4de7df43\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: 0.05698525073746313\n", " neg_mean_loss: -0.05698525073746313\n", " node_ip: 127.0.0.1\n", " pid: 47738\n", " time_since_restore: 5.151050090789795\n", " time_this_iter_s: 0.10810613632202148\n", " time_total_s: 5.151050090789795\n", " timestamp: 1658500589\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: a7f5c682\n", " warmup_time: 0.003899097442626953\n", " \n", "Result for objective_a7fcf02e:\n", " date: 2022-07-22_15-36-29\n", " done: false\n", " experiment_id: 5646116e3b69493b920c031d27eeb11b\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: -0.593014749262537\n", " neg_mean_loss: 0.593014749262537\n", " node_ip: 127.0.0.1\n", " pid: 47742\n", " time_since_restore: 5.1300742626190186\n", " time_this_iter_s: 0.10815715789794922\n", " time_total_s: 5.1300742626190186\n", " timestamp: 1658500589\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: a7fcf02e\n", " warmup_time: 0.003072977066040039\n", " \n", "Result for objective_a7f414d6:\n", " date: 2022-07-22_15-36-29\n", " done: false\n", " experiment_id: 20575f602c234dd7a167a93ff353299b\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: 0.3329852507374631\n", " neg_mean_loss: -0.3329852507374631\n", " node_ip: 127.0.0.1\n", " pid: 47737\n", " time_since_restore: 5.151458024978638\n", " time_this_iter_s: 0.10657620429992676\n", " time_total_s: 5.151458024978638\n", " timestamp: 1658500589\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: a7f414d6\n", " warmup_time: 0.0046689510345458984\n", " \n", "Result for objective_a7f7c162:\n", " date: 2022-07-22_15-36-29\n", " done: false\n", " experiment_id: f4b185b58be045f594bf127addecbace\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: 0.3329852507374631\n", " neg_mean_loss: -0.3329852507374631\n", " node_ip: 127.0.0.1\n", " pid: 47739\n", " time_since_restore: 5.154465675354004\n", " time_this_iter_s: 0.1063377857208252\n", " time_total_s: 5.154465675354004\n", " timestamp: 1658500589\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: a7f7c162\n", " warmup_time: 0.004554033279418945\n", " \n", "Result for objective_a7f96fda:\n", " date: 2022-07-22_15-36-30\n", " done: false\n", " experiment_id: 1983d487678f485a8a8ae7da4b7495a0\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: 0.5430505050505051\n", " neg_mean_loss: -0.5430505050505051\n", " node_ip: 127.0.0.1\n", " pid: 47740\n", " time_since_restore: 5.135071039199829\n", " time_this_iter_s: 0.10689830780029297\n", " time_total_s: 5.135071039199829\n", " timestamp: 1658500590\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: a7f96fda\n", " warmup_time: 0.0031909942626953125\n", " \n", "Result for objective_a7fb1844:\n", " date: 2022-07-22_15-36-35\n", " done: false\n", " experiment_id: 668080a52a3f4c4fad2258177b1fb755\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " mean_loss: 0.18770059880239523\n", " neg_mean_loss: -0.18770059880239523\n", " node_ip: 127.0.0.1\n", " pid: 47741\n", " time_since_restore: 10.180493831634521\n", " time_this_iter_s: 0.10809183120727539\n", " time_total_s: 10.180493831634521\n", " timestamp: 1658500595\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: a7fb1844\n", " warmup_time: 0.004377126693725586\n", " \n", "Result for objective_a7f414d6:\n", " date: 2022-07-22_15-36-35\n", " done: false\n", " experiment_id: 20575f602c234dd7a167a93ff353299b\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " mean_loss: 0.18770059880239523\n", " neg_mean_loss: -0.18770059880239523\n", " node_ip: 127.0.0.1\n", " pid: 47737\n", " time_since_restore: 10.188506126403809\n", " time_this_iter_s: 0.10684013366699219\n", " time_total_s: 10.188506126403809\n", " timestamp: 1658500595\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: a7f414d6\n", " warmup_time: 0.0046689510345458984\n", " \n", "Result for objective_a7f5c682:\n", " date: 2022-07-22_15-36-35\n", " done: false\n", " experiment_id: 4449a3ef690c4b76a0cfbf7c4de7df43\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " mean_loss: -0.08829940119760477\n", " neg_mean_loss: 0.08829940119760477\n", " node_ip: 127.0.0.1\n", " pid: 47738\n", " time_since_restore: 10.20038390159607\n", " time_this_iter_s: 0.10745882987976074\n", " time_total_s: 10.20038390159607\n", " timestamp: 1658500595\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: a7f5c682\n", " warmup_time: 0.003899097442626953\n", " \n", "Result for objective_a7f7c162:\n", " date: 2022-07-22_15-36-35\n", " done: false\n", " experiment_id: f4b185b58be045f594bf127addecbace\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " mean_loss: 0.18770059880239523\n", " neg_mean_loss: -0.18770059880239523\n", " node_ip: 127.0.0.1\n", " pid: 47739\n", " time_since_restore: 10.203772783279419\n", " time_this_iter_s: 0.1077718734741211\n", " time_total_s: 10.203772783279419\n", " timestamp: 1658500595\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: a7f7c162\n", " warmup_time: 0.004554033279418945\n", " \n", "Result for objective_a7fcf02e:\n", " date: 2022-07-22_15-36-35\n", " done: false\n", " experiment_id: 5646116e3b69493b920c031d27eeb11b\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " mean_loss: -0.738299401197605\n", " neg_mean_loss: 0.738299401197605\n", " node_ip: 127.0.0.1\n", " pid: 47742\n", " time_since_restore: 10.187027215957642\n", " time_this_iter_s: 0.10881304740905762\n", " time_total_s: 10.187027215957642\n", " timestamp: 1658500595\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: a7fcf02e\n", " warmup_time: 0.003072977066040039\n", " \n", "Result for objective_a7f96fda:\n", " date: 2022-07-22_15-36-35\n", " done: false\n", " experiment_id: 1983d487678f485a8a8ae7da4b7495a0\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " mean_loss: 0.29706735751295343\n", " neg_mean_loss: -0.29706735751295343\n", " node_ip: 127.0.0.1\n", " pid: 47740\n", " time_since_restore: 10.170713901519775\n", " time_this_iter_s: 0.10522603988647461\n", " time_total_s: 10.170713901519775\n", " timestamp: 1658500595\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: a7f96fda\n", " warmup_time: 0.0031909942626953125\n", " \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Result for objective_a7fb1844:\n", " date: 2022-07-22_15-36-35\n", " done: true\n", " experiment_id: 668080a52a3f4c4fad2258177b1fb755\n", " experiment_tag: 9_height=0.3800,layers=8,steps=100,width=7\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 0.18024751066856332\n", " neg_mean_loss: -0.18024751066856332\n", " node_ip: 127.0.0.1\n", " pid: 47741\n", " time_since_restore: 10.715673923492432\n", " time_this_iter_s: 0.10643696784973145\n", " time_total_s: 10.715673923492432\n", " timestamp: 1658500595\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: a7fb1844\n", " warmup_time: 0.004377126693725586\n", " \n", "Result for objective_a7f414d6:\n", " date: 2022-07-22_15-36-35\n", " done: true\n", " experiment_id: 20575f602c234dd7a167a93ff353299b\n", " experiment_tag: 5_height=0.3800,layers=4,steps=100,width=7\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 0.18024751066856332\n", " neg_mean_loss: -0.18024751066856332\n", " node_ip: 127.0.0.1\n", " pid: 47737\n", " time_since_restore: 10.723177909851074\n", " time_this_iter_s: 0.1052548885345459\n", " time_total_s: 10.723177909851074\n", " timestamp: 1658500595\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: a7f414d6\n", " warmup_time: 0.0046689510345458984\n", " \n", "Result for objective_a7f5c682:\n", " date: 2022-07-22_15-36-35\n", " done: true\n", " experiment_id: 4449a3ef690c4b76a0cfbf7c4de7df43\n", " experiment_tag: 6_height=-2.3800,layers=16,steps=100,width=7\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: -0.09575248933143668\n", " neg_mean_loss: 0.09575248933143668\n", " node_ip: 127.0.0.1\n", " pid: 47738\n", " time_since_restore: 10.733658075332642\n", " time_this_iter_s: 0.10680294036865234\n", " time_total_s: 10.733658075332642\n", " timestamp: 1658500595\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: a7f5c682\n", " warmup_time: 0.003899097442626953\n", " \n", "Result for objective_a7f7c162:\n", " date: 2022-07-22_15-36-35\n", " done: true\n", " experiment_id: f4b185b58be045f594bf127addecbace\n", " experiment_tag: 7_height=0.3800,layers=8,steps=100,width=7\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 0.18024751066856332\n", " neg_mean_loss: -0.18024751066856332\n", " node_ip: 127.0.0.1\n", " pid: 47739\n", " time_since_restore: 10.745207786560059\n", " time_this_iter_s: 0.10880804061889648\n", " time_total_s: 10.745207786560059\n", " timestamp: 1658500595\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: a7f7c162\n", " warmup_time: 0.004554033279418945\n", " \n", "Result for objective_a7fcf02e:\n", " date: 2022-07-22_15-36-35\n", " done: true\n", " experiment_id: 5646116e3b69493b920c031d27eeb11b\n", " experiment_tag: 10_height=-8.8800,layers=16,steps=100,width=7\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: -0.7457524893314368\n", " neg_mean_loss: 0.7457524893314368\n", " node_ip: 127.0.0.1\n", " pid: 47742\n", " time_since_restore: 10.730549097061157\n", " time_this_iter_s: 0.11018085479736328\n", " time_total_s: 10.730549097061157\n", " timestamp: 1658500595\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: a7fcf02e\n", " warmup_time: 0.003072977066040039\n", " \n", "Result for objective_a7f96fda:\n", " date: 2022-07-22_15-36-35\n", " done: true\n", " experiment_id: 1983d487678f485a8a8ae7da4b7495a0\n", " experiment_tag: 8_height=0.3800,layers=16,steps=100,width=4\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 0.2843054187192119\n", " neg_mean_loss: -0.2843054187192119\n", " node_ip: 127.0.0.1\n", " pid: 47740\n", " time_since_restore: 10.707929849624634\n", " time_this_iter_s: 0.10735177993774414\n", " time_total_s: 10.707929849624634\n", " timestamp: 1658500595\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: a7f96fda\n", " warmup_time: 0.0031909942626953125\n", " \n" ] } ], "source": [ "tuner = tune.Tuner(\n", " objective,\n", " tune_config=tune.TuneConfig(\n", " metric=\"mean_loss\",\n", " mode=\"min\",\n", " search_alg=algo,\n", " num_samples=num_samples,\n", " ),\n", " param_space={\"steps\": 100},\n", ")\n", "results = tuner.fit()" ] }, { "cell_type": "markdown", "id": "01e3b036", "metadata": {}, "source": [ "Here are the hyperparamters found to minimize the mean loss of the defined objective." ] }, { "cell_type": "code", "execution_count": 15, "id": "c11178dd", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Best hyperparameters found were: {'steps': 100, 'height': -8.88, 'width': 7, 'layers': 16}\n" ] } ], "source": [ "print(\"Best hyperparameters found were: \", results.get_best_result().config)" ] }, { "cell_type": "code", "execution_count": 16, "id": "23eab59f", "metadata": { "tags": [ "remove-cell" ] }, "outputs": [], "source": [ "ray.shutdown()" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.7" }, "orphan": true }, "nbformat": 4, "nbformat_minor": 5 }