{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "3b05af3b",
   "metadata": {},
   "source": [
    "(tune-comet-ref)=\n",
    "\n",
    "# Using Comet with Tune\n",
    "\n",
    "[Comet](https://www.comet.ml/site/) is a tool to manage and optimize the\n",
    "entire ML lifecycle, from experiment tracking, model optimization and dataset\n",
    "versioning to model production monitoring.\n",
    "\n",
    "```{image} /images/comet_logo_full.png\n",
    ":align: center\n",
    ":alt: Comet\n",
    ":height: 120px\n",
    ":target: https://www.comet.ml/site/\n",
    "```\n",
    "\n",
    "```{contents}\n",
    ":backlinks: none\n",
    ":local: true\n",
    "```\n",
    "\n",
    "## Example\n",
    "\n",
    "To illustrate logging your trial results to Comet, we'll define a simple training function\n",
    "that simulates a `loss` metric:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "19e3c389",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from ray import tune\n",
    "\n",
    "\n",
    "def train_function(config, checkpoint_dir=None):\n",
    "    for i in range(30):\n",
    "        loss = config[\"mean\"] + config[\"sd\"] * np.random.randn()\n",
    "        tune.report(loss=loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6fb69a24",
   "metadata": {},
   "source": [
    "Now, given that you provide your Comet API key and your project name like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "993d5be6",
   "metadata": {},
   "outputs": [],
   "source": [
    "api_key = \"YOUR_COMET_API_KEY\"\n",
    "project_name = \"YOUR_COMET_PROJECT_NAME\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e9ce0d76",
   "metadata": {
    "tags": [
     "remove-cell"
    ]
   },
   "outputs": [],
   "source": [
    "# This cell is hidden from the rendered notebook. It makes the \n",
    "from unittest.mock import MagicMock\n",
    "from ray.tune.integration.comet import CometLoggerCallback\n",
    "\n",
    "CometLoggerCallback._logger_process_cls = MagicMock\n",
    "api_key = \"abc\"\n",
    "project_name = \"test\""
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "You can add a Comet logger by specifying the `callbacks` argument in your `tune.run` accordingly:"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dbb761e7",
   "metadata": {},
   "outputs": [],
   "source": [
    "from ray.tune.integration.comet import CometLoggerCallback\n",
    "\n",
    "analysis = tune.run(\n",
    "    train_function,\n",
    "    name=\"comet\",\n",
    "    metric=\"loss\",\n",
    "    mode=\"min\",\n",
    "    callbacks=[\n",
    "        CometLoggerCallback(\n",
    "            api_key=api_key, project_name=project_name, tags=[\"comet_example\"]\n",
    "        )\n",
    "    ],\n",
    "    config={\"mean\": tune.grid_search([1, 2, 3]), \"sd\": tune.uniform(0.2, 0.8)},\n",
    ")\n",
    "print(analysis.best_config)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d7e46189",
   "metadata": {},
   "source": [
    "## Tune Comet Logger\n",
    "\n",
    "Ray Tune offers an integration with Comet through the `CometLoggerCallback`,\n",
    "which automatically logs metrics and parameters reported to Tune to the Comet UI.\n",
    "\n",
    "Click on the following dropdown to see this callback API in detail:\n",
    "\n",
    "```{eval-rst}\n",
    ".. autoclass:: ray.tune.integration.comet.CometLoggerCallback\n",
    "   :noindex:\n",
    "```"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "orphan": true
 },
 "nbformat": 4,
 "nbformat_minor": 5
}