mirror of
https://github.com/vale981/ray
synced 2025-03-07 02:51:39 -05:00
cd5218f831
Replaces #24225 and adds example navigation Signed-off-by: Max Pumperla <max.pumperla@googlemail.com>
506 lines
No EOL
14 KiB
Text
506 lines
No EOL
14 KiB
Text
{
|
||
"cells": [
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "d3d9d39e",
|
||
"metadata": {},
|
||
"source": [
|
||
"# Running Tune experiments with BlendSearch and CFO\n",
|
||
"\n",
|
||
"In this tutorial we introduce BlendSearch and CFO, while running a simple Ray Tune\n",
|
||
"experiment. Tune’s Search Algorithms integrate with FLAML and, as a result, allow\n",
|
||
"you to seamlessly scale up a BlendSearch and CFO optimization\n",
|
||
"process - without sacrificing performance.\n",
|
||
"\n",
|
||
"Fast Library for Automated Machine Learning & Tuning (FLAML) does not rely on the\n",
|
||
"gradient of the objective function, but instead, learns from samples of the\n",
|
||
"search space. It is suitable for optimizing functions that are non-differentiable,\n",
|
||
"with many local minima, or even unknown but only testable. Therefore, it is\n",
|
||
"necessarily belongs to the domain of \"derivative-free optimization\"\n",
|
||
"and \"black-box optimization\".\n",
|
||
"\n",
|
||
"FLAML has two primary algorithms: (1) Frugal Optimization for Cost-related\n",
|
||
"Hyperparameters (CFO) begins with a low-cost initial point and gradually moves to\n",
|
||
"a high-cost region as needed. It is a local search method that leverages randomized\n",
|
||
"direct search method with an adaptive step-size and random restarts.\n",
|
||
"As a local search method, it has an appealing provable convergence rate and bounded\n",
|
||
"cost but may get trapped in suboptimal local minima. (2) Economical Hyperparameter\n",
|
||
"Optimization With Blended Search Strategy (BlendSearch) combines CFO's local search\n",
|
||
"with global search, making it less suspectable to local minima traps.\n",
|
||
"It leverages the frugality of CFO and the space exploration ability of global search\n",
|
||
"methods such as Bayesian optimization.\n",
|
||
"\n",
|
||
"In this example we minimize a simple objective to briefly demonstrate the usage of\n",
|
||
"FLAML with Ray Tune via `BlendSearch` and `CFO`. It's useful to keep in mind that\n",
|
||
"despite the emphasis on machine learning experiments, Ray Tune optimizes any implicit\n",
|
||
"or explicit objective. Here we assume `flaml==0.4.1` and `optuna==2.9.1` libraries\n",
|
||
"are installed. To learn more, please refer to\n",
|
||
"the [FLAML website](https://github.com/microsoft/FLAML/tree/main/flaml/tune).\n",
|
||
" \n",
|
||
"Click below to see all the imports we need for this example.\n",
|
||
"You can also launch directly into a Binder instance to run this notebook yourself.\n",
|
||
"Just click on the rocket symbol at the top of the navigation."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "1a9f10f0",
|
||
"metadata": {
|
||
"tags": [
|
||
"hide-input"
|
||
]
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"import time\n",
|
||
"\n",
|
||
"import ray\n",
|
||
"from ray import tune\n",
|
||
"from ray.tune.suggest import ConcurrencyLimiter\n",
|
||
"from ray.tune.suggest.flaml import BlendSearch, CFO"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "f5a4bce8",
|
||
"metadata": {},
|
||
"source": [
|
||
"Let's start by defining a simple evaluation function.\n",
|
||
"We artificially sleep for a bit (`0.1` seconds) to simulate a long-running ML experiment.\n",
|
||
"This setup assumes that we're running multiple `step`s of an experiment and try to\n",
|
||
"tune three hyperparameters, namely `width` and `height`, and `activation`."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "9b80ac32",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"def evaluate(step, width, height, activation):\n",
|
||
" time.sleep(0.1)\n",
|
||
" activation_boost = 10 if activation==\"relu\" else 1\n",
|
||
" return (0.1 + width * step / 100) ** (-1) + height * 0.1 + activation_boost"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "391dcd0e",
|
||
"metadata": {},
|
||
"source": [
|
||
"Next, our `objective` function takes a Tune `config`, evaluates the `score` of your\n",
|
||
"experiment in a training loop, and uses `tune.report` to report the `score` back to Tune."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "0773e711",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"def objective(config):\n",
|
||
" for step in range(config[\"steps\"]):\n",
|
||
" score = evaluate(step, config[\"width\"], config[\"height\"], config[\"activation\"])\n",
|
||
" tune.report(iterations=step, mean_loss=score)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "8dde8596",
|
||
"metadata": {
|
||
"lines_to_next_cell": 0,
|
||
"tags": [
|
||
"remove-cell"
|
||
]
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"ray.init(configure_logging=False)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "a1007f97",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Running Tune experiments with BlendSearch\n",
|
||
"\n",
|
||
"This example demonstrates the usage of Economical Hyperparameter Optimization\n",
|
||
"With Blended Search Strategy (BlendSearch) with Ray Tune.\n",
|
||
"\n",
|
||
"Now we define the search algorithm built from `BlendSearch`, constrained to a\n",
|
||
"maximum of `4` concurrent trials with a `ConcurrencyLimiter`."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "37b5070e",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"algo = BlendSearch()\n",
|
||
"algo = ConcurrencyLimiter(algo, max_concurrent=4)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "6bb24c73",
|
||
"metadata": {},
|
||
"source": [
|
||
"The number of samples this Tune run is set to `1000`.\n",
|
||
"(you can decrease this if it takes too long on your machine)."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "f219c33a",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"num_samples = 1000"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "7ad72495",
|
||
"metadata": {
|
||
"tags": [
|
||
"remove-cell"
|
||
]
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"# If 1000 samples take too long, you can reduce this number.\n",
|
||
"# We override this number here for our smoke tests.\n",
|
||
"num_samples = 10"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "6a206833",
|
||
"metadata": {},
|
||
"source": [
|
||
"Next we define a search space. The critical assumption is that the optimal\n",
|
||
"hyperparameters live within this space. Yet, if the space is very large, then those\n",
|
||
"hyperparameters may be difficult to find in a short amount of time."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "f2236f81",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"search_config = {\n",
|
||
" \"steps\": 100,\n",
|
||
" \"width\": tune.uniform(0, 20),\n",
|
||
" \"height\": tune.uniform(-100, 100),\n",
|
||
" \"activation\": tune.choice([\"relu, tanh\"])\n",
|
||
"}"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "f77303bc",
|
||
"metadata": {},
|
||
"source": [
|
||
"Finally, we run the experiment to `\"min\"`imize the \"mean_loss\" of the `objective` by\n",
|
||
"searching `search_config` via `algo`, `num_samples` times. This previous sentence is\n",
|
||
"fully characterizes the search problem we aim to solve. With this in mind, observe\n",
|
||
"how efficient it is to execute `tune.run()`."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "7a95373a",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"analysis = tune.run(\n",
|
||
" objective,\n",
|
||
" search_alg=algo,\n",
|
||
" metric=\"mean_loss\",\n",
|
||
" mode=\"min\",\n",
|
||
" name=\"blendsearch_exp\",\n",
|
||
" num_samples=num_samples,\n",
|
||
" config=search_config,\n",
|
||
")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "77a49ffb",
|
||
"metadata": {},
|
||
"source": [
|
||
"Here are the hyperparamters found to minimize the mean loss of the defined objective."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "59cdf197",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"print(\"Best hyperparameters found were: \", analysis.best_config)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "1dca41c4",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Incorporating a time budget to the experiment\n",
|
||
"\n",
|
||
"Define the time budget in seconds:"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "878bd08f",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"time_budget_s = 30"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "5c9d1bdc",
|
||
"metadata": {},
|
||
"source": [
|
||
"Similarly we define a search space, but this time we feed it as an argument to\n",
|
||
"`BlendSearch` rather than `tune.run`'s `config` argument.\n",
|
||
"\n",
|
||
"We next define the time budget via `set_search_properties`.\n",
|
||
"And once again include the `ConcurrencyLimiter`."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "5846724f",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"algo = BlendSearch(\n",
|
||
" metric=\"mean_loss\",\n",
|
||
" mode=\"min\",\n",
|
||
" space={\n",
|
||
" \"width\": tune.uniform(0, 20),\n",
|
||
" \"height\": tune.uniform(-100, 100),\n",
|
||
" \"activation\": tune.choice([\"relu\", \"tanh\"]),\n",
|
||
" },\n",
|
||
")\n",
|
||
"algo.set_search_properties(config={\"time_budget_s\": time_budget_s})\n",
|
||
"algo = ConcurrencyLimiter(algo, max_concurrent=4)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "13564767",
|
||
"metadata": {},
|
||
"source": [
|
||
"Now we run the experiment, this time with the `time_budget` included as an argument.\n",
|
||
"Note: We allow for virtually infinite `num_samples` by passing `-1`, so that the\n",
|
||
"experiment is stopped according to the time budget rather than a sample limit."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "b878b2be",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"analysis = tune.run(\n",
|
||
" objective,\n",
|
||
" search_alg=algo,\n",
|
||
" time_budget_s=time_budget_s,\n",
|
||
" metric=\"mean_loss\",\n",
|
||
" mode=\"min\",\n",
|
||
" name=\"blendsearch_exp\",\n",
|
||
" num_samples=-1,\n",
|
||
" config={\"steps\": 100},\n",
|
||
")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "48f06c5f",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"print(\"Best hyperparameters found were: \", analysis.best_config)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "47b906b7",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Running Tune experiments with CFO\n",
|
||
"\n",
|
||
"This example demonstrates the usage of Frugal Optimization for Cost-related\n",
|
||
"Hyperparameters (CFO) with Ray Tune.\n",
|
||
"\n",
|
||
"We now define the search algorithm as built from `CFO`, constrained to a maximum of `4`\n",
|
||
"concurrent trials with a `ConcurrencyLimiter`."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "29f2c0be",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"algo = CFO()\n",
|
||
"algo = ConcurrencyLimiter(algo, max_concurrent=4)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "a89cf9bb",
|
||
"metadata": {},
|
||
"source": [
|
||
"The number of samples is the number of hyperparameter combinations that will be\n",
|
||
"tried out. This Tune run is set to `1000` samples.\n",
|
||
"(you can decrease this if it takes too long on your machine)."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "3d15f10d",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"num_samples = 1000"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "137f3ec0",
|
||
"metadata": {
|
||
"tags": [
|
||
"remove-cell"
|
||
]
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"# If 1000 samples take too long, you can reduce this number.\n",
|
||
"# We override this number here for our smoke tests.\n",
|
||
"num_samples = 10"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "cfa2e413",
|
||
"metadata": {},
|
||
"source": [
|
||
"Next we define a search space. The critical assumption is that the optimal\n",
|
||
"hyperparameters live within this space. Yet, if the space is very large, then\n",
|
||
"those hyperparameters may be difficult to find in a short amount of time."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "ab4fbcbe",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"search_config = {\n",
|
||
" \"steps\": 100,\n",
|
||
" \"width\": tune.uniform(0, 20),\n",
|
||
" \"height\": tune.uniform(-100, 100),\n",
|
||
" \"activation\": tune.choice([\"relu, tanh\"])\n",
|
||
"}"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "b19390a2",
|
||
"metadata": {},
|
||
"source": [
|
||
"Finally, we run the experiment to `\"min\"`imize the \"mean_loss\" of the `objective`\n",
|
||
"by searching `search_config` via `algo`, `num_samples` times. This previous sentence\n",
|
||
"is fully characterizes the search problem we aim to solve. With this in mind,\n",
|
||
"notice how efficient it is to execute `tune.run()`."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "2acf19f5",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"analysis = tune.run(\n",
|
||
" objective,\n",
|
||
" search_alg=algo,\n",
|
||
" metric=\"mean_loss\",\n",
|
||
" mode=\"min\",\n",
|
||
" name=\"cfo_exp\",\n",
|
||
" num_samples=num_samples,\n",
|
||
" config=search_config,\n",
|
||
")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "71e6f6ec",
|
||
"metadata": {},
|
||
"source": [
|
||
"Here are the hyperparameters found to minimize the mean loss of the defined objective."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "b657dccf",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"print(\"Best hyperparameters found were: \", analysis.best_config)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "505995af",
|
||
"metadata": {
|
||
"tags": [
|
||
"remove-cell"
|
||
]
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"ray.shutdown()"
|
||
]
|
||
}
|
||
],
|
||
"metadata": {
|
||
"kernelspec": {
|
||
"display_name": "Python 3 (ipykernel)",
|
||
"language": "python",
|
||
"name": "python3"
|
||
},
|
||
"orphan": true
|
||
},
|
||
"nbformat": 4,
|
||
"nbformat_minor": 5
|
||
} |