{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "35712b90",
   "metadata": {},
   "source": [
    "# Running Tune experiments with HyperOpt\n",
    "\n",
    "This example demonstrates the usage of HyperOpt with Ray Tune, using a `AsyncHyperBandScheduler` scheduler\n",
    "together with `HyperOptSearch`.\n",
    "Click below to see all the imports we need for this example.\n",
    "You can also launch directly into a Binder instance to run this notebook yourself.\n",
    "Just click on the rocket symbol at the top of the navigation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "42ed664d",
   "metadata": {
    "tags": [
     "hide-input"
    ]
   },
   "outputs": [],
   "source": [
    "import time\n",
    "\n",
    "import ray\n",
    "from ray import tune\n",
    "from ray.tune.suggest import ConcurrencyLimiter\n",
    "from ray.tune.schedulers import AsyncHyperBandScheduler\n",
    "from ray.tune.suggest.hyperopt import HyperOptSearch"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e4932f5",
   "metadata": {},
   "source": [
    "Let's start by defining a simple evaluation function.\n",
    "We artificially sleep for a bit (`0.1` seconds) to simulate a long-running ML experiment.\n",
    "This setup assumes that we're running multiple `step`s of an experiment and try to tune two hyperparameters,\n",
    "namely `x` and `y`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0854c730",
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluate(step, x, y):\n",
    "    time.sleep(0.1)\n",
    "    return (0.1 + x * step / 100) ** (-1) + y * 0.1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f3915b1b",
   "metadata": {},
   "source": [
    "Next, our ``objective`` function takes a Tune ``config``, evaluates the `score` of your experiment in a training loop,\n",
    "and uses `tune.report` to report the `score` back to Tune."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9d7df36a",
   "metadata": {},
   "outputs": [],
   "source": [
    "def objective(config):\n",
    "    for step in range(config[\"steps\"]):\n",
    "        score = evaluate(step, config[\"x\"], config[\"y\"])\n",
    "        tune.report(iterations=step, mean_loss=score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5354d2a1",
   "metadata": {
    "tags": [
     "remove-cell"
    ]
   },
   "outputs": [],
   "source": [
    "ray.init(configure_logging=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e761e836",
   "metadata": {},
   "source": [
    "Let's say we have a hypothesis on what the best parameters currently are (`current_best_params`), then we can\n",
    "pass this belief into a `HyperOptSearch` searcher and set the maximum concurrent trials to `4` with a `ConcurrencyLimiter`.\n",
    "We can also define a `scheduler` to go along with our algorithm and set the number of samples for this Tune run to `1000`\n",
    "(your can decrease this if it takes too long on your machine)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4ea0667d",
   "metadata": {},
   "outputs": [],
   "source": [
    "current_best_params = [\n",
    "    {\"x\": 1, \"y\": 2},\n",
    "    {\"x\": 4, \"y\": 2},\n",
    "]\n",
    "searcher = HyperOptSearch(points_to_evaluate=current_best_params)\n",
    "\n",
    "algo = ConcurrencyLimiter(searcher, max_concurrent=4)\n",
    "scheduler = AsyncHyperBandScheduler()\n",
    "\n",
    "num_samples = 1000"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8fbd31df",
   "metadata": {
    "tags": [
     "remove-cell"
    ]
   },
   "outputs": [],
   "source": [
    "# If 1000 samples take too long, you can reduce this number.\n",
    "# We override this number here for our smoke tests.\n",
    "num_samples = 10"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5edc6d9b",
   "metadata": {},
   "source": [
    "Finally, all that's left is to define a search space, run the experiment and print the best parameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0da436cb",
   "metadata": {},
   "outputs": [],
   "source": [
    "search_space = {\n",
    "    \"steps\": 100,\n",
    "    \"x\": tune.uniform(0, 20),\n",
    "    \"y\": tune.uniform(-100, 100),\n",
    "}\n",
    "\n",
    "analysis = tune.run(\n",
    "    objective,\n",
    "    search_alg=algo,\n",
    "    scheduler=scheduler,\n",
    "    metric=\"mean_loss\",\n",
    "    mode=\"min\",\n",
    "    num_samples=num_samples,\n",
    "    config=search_space,\n",
    ")\n",
    "\n",
    "print(\"Best hyperparameters found were: \", analysis.best_config)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "81945658",
   "metadata": {
    "tags": [
     "remove-cell"
    ]
   },
   "outputs": [],
   "source": [
    "ray.shutdown()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "orphan": true
 },
 "nbformat": 4,
 "nbformat_minor": 5
}