{ "cells": [ { "cell_type": "markdown", "id": "0d385409", "metadata": {}, "source": [ "# Training a model with distributed LightGBM\n", "In this example we will train a model in Ray AIR using distributed LightGBM." ] }, { "cell_type": "markdown", "id": "07d92cee", "metadata": {}, "source": [ "Let's start with installing our dependencies:" ] }, { "cell_type": "code", "execution_count": 1, "id": "86131abe", "metadata": {}, "outputs": [], "source": [ "!pip install -qU \"ray[tune]\" lightgbm_ray" ] }, { "cell_type": "markdown", "id": "135fc884", "metadata": {}, "source": [ "Then we need some imports:" ] }, { "cell_type": "code", "execution_count": 4, "id": "102ef1ac", "metadata": {}, "outputs": [], "source": [ "from typing import Tuple\n", "\n", "import ray\n", "from ray.train.batch_predictor import BatchPredictor\n", "from ray.train.lightgbm import LightGBMPredictor\n", "from ray.data.preprocessors.chain import Chain\n", "from ray.data.preprocessors.encoder import Categorizer\n", "from ray.train.lightgbm import LightGBMTrainer\n", "from ray.data.dataset import Dataset\n", "from ray.air.result import Result\n", "from ray.air.util.datasets import train_test_split\n", "from ray.data.preprocessors import StandardScaler" ] }, { "cell_type": "markdown", "id": "c7d102bd", "metadata": {}, "source": [ "Next we define a function to load our train, validation, and test datasets." ] }, { "cell_type": "code", "execution_count": 13, "id": "f1f35cd7", "metadata": {}, "outputs": [], "source": [ "def prepare_data() -> Tuple[Dataset, Dataset, Dataset]:\n", " import pandas as pd\n", " df = pd.read_csv(\"https://air-example-data.s3.us-east-2.amazonaws.com/breast_cancer_with_categorical.csv\")\n", " dataset = ray.data.from_pandas(df)\n", " # Optionally, read directly from s3\n", " # dataset = ray.data.read_csv(\"s3://air-example-data/breast_cancer_with_categorical.csv\")\n", " train_dataset, valid_dataset = train_test_split(dataset, test_size=0.3)\n", " test_dataset = valid_dataset.map_batches(lambda df: df.drop(\"target\", axis=1), batch_format=\"pandas\")\n", " return train_dataset, valid_dataset, test_dataset" ] }, { "cell_type": "markdown", "id": "8f7afbce", "metadata": {}, "source": [ "The following function will create a LightGBM trainer, train it, and return the result." ] }, { "cell_type": "code", "execution_count": 14, "id": "fefcbc8a", "metadata": {}, "outputs": [], "source": [ "def train_lightgbm(num_workers: int, use_gpu: bool = False) -> Result:\n", " train_dataset, valid_dataset, _ = prepare_data()\n", "\n", " # Scale some random columns, and categorify the categorical_column,\n", " # allowing LightGBM to use its built-in categorical feature support\n", " columns_to_scale = [\"mean radius\", \"mean texture\"]\n", " preprocessor = Chain(\n", " Categorizer([\"categorical_column\"]), StandardScaler(columns=columns_to_scale)\n", " )\n", "\n", " # LightGBM specific params\n", " params = {\n", " \"objective\": \"binary\",\n", " \"metric\": [\"binary_logloss\", \"binary_error\"],\n", " }\n", "\n", " trainer = LightGBMTrainer(\n", " scaling_config={\n", " \"num_workers\": num_workers,\n", " \"use_gpu\": use_gpu,\n", " },\n", " label_column=\"target\",\n", " params=params,\n", " datasets={\"train\": train_dataset, \"valid\": valid_dataset},\n", " preprocessor=preprocessor,\n", " num_boost_round=100,\n", " )\n", " result = trainer.fit()\n", " print(result.metrics)\n", "\n", " return result" ] }, { "cell_type": "markdown", "id": "04d278ae", "metadata": {}, "source": [ "Once we have the result, we can do batch inference on the obtained model. Let's define a utility function for this." ] }, { "cell_type": "code", "execution_count": 15, "id": "3f1d0c19", "metadata": {}, "outputs": [], "source": [ "def predict_lightgbm(result: Result):\n", " _, _, test_dataset = prepare_data()\n", " batch_predictor = BatchPredictor.from_checkpoint(\n", " result.checkpoint, LightGBMPredictor\n", " )\n", "\n", " predicted_labels = (\n", " batch_predictor.predict(test_dataset)\n", " .map_batches(lambda df: (df > 0.5).astype(int), batch_format=\"pandas\")\n", " )\n", " print(f\"PREDICTED LABELS\")\n", " predicted_labels.show()\n", "\n", " shap_values = batch_predictor.predict(test_dataset, pred_contrib=True)\n", " print(f\"SHAP VALUES\")\n", " shap_values.show()" ] }, { "cell_type": "markdown", "id": "2bb0e5df", "metadata": {}, "source": [ "Now we can run the training:" ] }, { "cell_type": "code", "execution_count": 16, "id": "8244ff3c", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2022-06-22 17:26:41,346\tWARNING read_api.py:260 -- The number of blocks in this dataset (1) limits its parallelism to 1 concurrent tasks. This is much less than the number of available CPU slots in the cluster. Use `.repartition(n)` to increase the number of dataset blocks.\n", "Map_Batches: 100%|██████████| 1/1 [00:00<00:00, 46.26it/s]\n" ] }, { "data": { "text/html": [ "== Status ==
Current time: 2022-06-22 17:26:56 (running for 00:00:14.07)
Memory usage on this node: 10.0/31.0 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/8 CPUs, 0/0 GPUs, 0.0/13.32 GiB heap, 0.0/6.66 GiB objects
Result logdir: /home/ubuntu/ray_results/LightGBMTrainer_2022-06-22_17-26-41
Number of trials: 1/1 (1 TERMINATED)
\n", "\n", "\n", "\n", "\n", "\n", "\n", "
Trial name status loc iter total time (s) train-binary_logloss train-binary_error valid-binary_logloss
LightGBMTrainer_7b049_00000TERMINATED172.31.43.110:1491578 100 10.9726 0.000574522 0 0.171898


" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stderr", "output_type": "stream", "text": [ "UserWarning: cpus_per_actor is set to less than 2. Distributed LightGBM needs at least 2 CPUs per actor to train efficiently. This may lead to a degradation of performance during training.\n", "\u001b[2m\u001b[36m(pid=1491578)\u001b[0m /home/ubuntu/ray/venv/lib/python3.8/site-packages/dask/dataframe/backends.py:181: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.\n", "\u001b[2m\u001b[36m(pid=1491578)\u001b[0m _numeric_index_types = (pd.Int64Index, pd.Float64Index, pd.UInt64Index)\n", "\u001b[2m\u001b[36m(pid=1491578)\u001b[0m /home/ubuntu/ray/venv/lib/python3.8/site-packages/dask/dataframe/backends.py:181: FutureWarning: pandas.Float64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.\n", "\u001b[2m\u001b[36m(pid=1491578)\u001b[0m _numeric_index_types = (pd.Int64Index, pd.Float64Index, pd.UInt64Index)\n", "\u001b[2m\u001b[36m(pid=1491578)\u001b[0m /home/ubuntu/ray/venv/lib/python3.8/site-packages/dask/dataframe/backends.py:181: FutureWarning: pandas.UInt64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.\n", "\u001b[2m\u001b[36m(pid=1491578)\u001b[0m _numeric_index_types = (pd.Int64Index, pd.Float64Index, pd.UInt64Index)\n", "\u001b[2m\u001b[36m(pid=1491578)\u001b[0m /home/ubuntu/ray/venv/lib/python3.8/site-packages/xgboost/compat.py:31: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.\n", "\u001b[2m\u001b[36m(pid=1491578)\u001b[0m from pandas import MultiIndex, Int64Index\n", "\u001b[2m\u001b[36m(LightGBMTrainer pid=1491578)\u001b[0m UserWarning: Dataset 'train' has 1 blocks, which is less than the `num_workers` 2. This dataset will be automatically repartitioned to 2 blocks.\n", "\u001b[2m\u001b[36m(LightGBMTrainer pid=1491578)\u001b[0m UserWarning: Dataset 'valid' has 1 blocks, which is less than the `num_workers` 2. This dataset will be automatically repartitioned to 2 blocks.\n", "\u001b[2m\u001b[36m(LightGBMTrainer pid=1491578)\u001b[0m UserWarning: cpus_per_actor is set to less than 2. Distributed LightGBM needs at least 2 CPUs per actor to train efficiently. This may lead to a degradation of performance during training.\n", "\u001b[2m\u001b[36m(pid=1491651)\u001b[0m /home/ubuntu/ray/venv/lib/python3.8/site-packages/xgboost/compat.py:31: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.\n", "\u001b[2m\u001b[36m(pid=1491651)\u001b[0m from pandas import MultiIndex, Int64Index\n", "\u001b[2m\u001b[36m(pid=1491651)\u001b[0m FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.\n", "\u001b[2m\u001b[36m(pid=1491651)\u001b[0m FutureWarning: pandas.Float64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.\n", "\u001b[2m\u001b[36m(pid=1491651)\u001b[0m FutureWarning: pandas.UInt64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.\n", "\u001b[2m\u001b[36m(pid=1491653)\u001b[0m /home/ubuntu/ray/venv/lib/python3.8/site-packages/dask/dataframe/backends.py:181: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.\n", "\u001b[2m\u001b[36m(pid=1491653)\u001b[0m _numeric_index_types = (pd.Int64Index, pd.Float64Index, pd.UInt64Index)\n", "\u001b[2m\u001b[36m(pid=1491653)\u001b[0m /home/ubuntu/ray/venv/lib/python3.8/site-packages/dask/dataframe/backends.py:181: FutureWarning: pandas.Float64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.\n", "\u001b[2m\u001b[36m(pid=1491653)\u001b[0m _numeric_index_types = (pd.Int64Index, pd.Float64Index, pd.UInt64Index)\n", "\u001b[2m\u001b[36m(pid=1491653)\u001b[0m /home/ubuntu/ray/venv/lib/python3.8/site-packages/dask/dataframe/backends.py:181: FutureWarning: pandas.UInt64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.\n", "\u001b[2m\u001b[36m(pid=1491653)\u001b[0m _numeric_index_types = (pd.Int64Index, pd.Float64Index, pd.UInt64Index)\n", "\u001b[2m\u001b[36m(pid=1491653)\u001b[0m /home/ubuntu/ray/venv/lib/python3.8/site-packages/xgboost/compat.py:31: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.\n", "\u001b[2m\u001b[36m(pid=1491653)\u001b[0m from pandas import MultiIndex, Int64Index\n", "\u001b[2m\u001b[36m(pid=1491652)\u001b[0m /home/ubuntu/ray/venv/lib/python3.8/site-packages/dask/dataframe/backends.py:181: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.\n", "\u001b[2m\u001b[36m(pid=1491652)\u001b[0m _numeric_index_types = (pd.Int64Index, pd.Float64Index, pd.UInt64Index)\n", "\u001b[2m\u001b[36m(pid=1491652)\u001b[0m /home/ubuntu/ray/venv/lib/python3.8/site-packages/dask/dataframe/backends.py:181: FutureWarning: pandas.Float64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.\n", "\u001b[2m\u001b[36m(pid=1491652)\u001b[0m _numeric_index_types = (pd.Int64Index, pd.Float64Index, pd.UInt64Index)\n", "\u001b[2m\u001b[36m(pid=1491652)\u001b[0m /home/ubuntu/ray/venv/lib/python3.8/site-packages/dask/dataframe/backends.py:181: FutureWarning: pandas.UInt64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.\n", "\u001b[2m\u001b[36m(pid=1491652)\u001b[0m _numeric_index_types = (pd.Int64Index, pd.Float64Index, pd.UInt64Index)\n", "\u001b[2m\u001b[36m(pid=1491652)\u001b[0m /home/ubuntu/ray/venv/lib/python3.8/site-packages/xgboost/compat.py:31: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.\n", "\u001b[2m\u001b[36m(pid=1491652)\u001b[0m from pandas import MultiIndex, Int64Index\n", "\u001b[2m\u001b[36m(_RemoteRayLightGBMActor pid=1491653)\u001b[0m 2022-06-22 17:26:50,509\tWARNING __init__.py:190 -- DeprecationWarning: `ray.worker.get_resource_ids` is a private attribute and access will be removed in a future Ray version.\n", "\u001b[2m\u001b[36m(_RemoteRayLightGBMActor pid=1491652)\u001b[0m 2022-06-22 17:26:50,658\tWARNING __init__.py:190 -- DeprecationWarning: `ray.worker.get_resource_ids` is a private attribute and access will be removed in a future Ray version.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\u001b[2m\u001b[36m(_RemoteRayLightGBMActor pid=1491653)\u001b[0m [LightGBM] [Info] Trying to bind port 59039...\n", "\u001b[2m\u001b[36m(_RemoteRayLightGBMActor pid=1491653)\u001b[0m [LightGBM] [Info] Binding port 59039 succeeded\n", "\u001b[2m\u001b[36m(_RemoteRayLightGBMActor pid=1491653)\u001b[0m [LightGBM] [Info] Listening...\n", "\u001b[2m\u001b[36m(_RemoteRayLightGBMActor pid=1491652)\u001b[0m [LightGBM] [Info] Trying to bind port 46955...\n", "\u001b[2m\u001b[36m(_RemoteRayLightGBMActor pid=1491652)\u001b[0m [LightGBM] [Info] Binding port 46955 succeeded\n", "\u001b[2m\u001b[36m(_RemoteRayLightGBMActor pid=1491652)\u001b[0m [LightGBM] [Info] Listening...\n", "\u001b[2m\u001b[36m(_RemoteRayLightGBMActor pid=1491652)\u001b[0m [LightGBM] [Warning] Connecting to rank 1 failed, waiting for 200 milliseconds\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[2m\u001b[36m(_RemoteRayLightGBMActor pid=1491653)\u001b[0m UserWarning: Overriding the parameters from Reference Dataset.\n", "\u001b[2m\u001b[36m(_RemoteRayLightGBMActor pid=1491653)\u001b[0m UserWarning: categorical_column in param dict is overridden.\n", "\u001b[2m\u001b[36m(_RemoteRayLightGBMActor pid=1491652)\u001b[0m UserWarning: Overriding the parameters from Reference Dataset.\n", "\u001b[2m\u001b[36m(_RemoteRayLightGBMActor pid=1491652)\u001b[0m UserWarning: categorical_column in param dict is overridden.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\u001b[2m\u001b[36m(_RemoteRayLightGBMActor pid=1491653)\u001b[0m [LightGBM] [Info] Connected to rank 0\n", "\u001b[2m\u001b[36m(_RemoteRayLightGBMActor pid=1491653)\u001b[0m [LightGBM] [Info] Local rank: 1, total number of machines: 2\n", "\u001b[2m\u001b[36m(_RemoteRayLightGBMActor pid=1491653)\u001b[0m [LightGBM] [Warning] num_threads is set=1, n_jobs=-1 will be ignored. Current value: num_threads=1\n", "\u001b[2m\u001b[36m(_RemoteRayLightGBMActor pid=1491652)\u001b[0m [LightGBM] [Info] Connected to rank 1\n", "\u001b[2m\u001b[36m(_RemoteRayLightGBMActor pid=1491652)\u001b[0m [LightGBM] [Info] Local rank: 0, total number of machines: 2\n", "\u001b[2m\u001b[36m(_RemoteRayLightGBMActor pid=1491652)\u001b[0m [LightGBM] [Warning] num_threads is set=1, n_jobs=-1 will be ignored. Current value: num_threads=1\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[2m\u001b[36m(_QueueActor pid=1491650)\u001b[0m /home/ubuntu/ray/venv/lib/python3.8/site-packages/dask/dataframe/backends.py:181: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.\n", "\u001b[2m\u001b[36m(_QueueActor pid=1491650)\u001b[0m _numeric_index_types = (pd.Int64Index, pd.Float64Index, pd.UInt64Index)\n", "\u001b[2m\u001b[36m(_QueueActor pid=1491650)\u001b[0m /home/ubuntu/ray/venv/lib/python3.8/site-packages/dask/dataframe/backends.py:181: FutureWarning: pandas.Float64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.\n", "\u001b[2m\u001b[36m(_QueueActor pid=1491650)\u001b[0m _numeric_index_types = (pd.Int64Index, pd.Float64Index, pd.UInt64Index)\n", "\u001b[2m\u001b[36m(_QueueActor pid=1491650)\u001b[0m /home/ubuntu/ray/venv/lib/python3.8/site-packages/dask/dataframe/backends.py:181: FutureWarning: pandas.UInt64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.\n", "\u001b[2m\u001b[36m(_QueueActor pid=1491650)\u001b[0m _numeric_index_types = (pd.Int64Index, pd.Float64Index, pd.UInt64Index)\n", "\u001b[2m\u001b[36m(_QueueActor pid=1491650)\u001b[0m /home/ubuntu/ray/venv/lib/python3.8/site-packages/xgboost/compat.py:31: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.\n", "\u001b[2m\u001b[36m(_QueueActor pid=1491650)\u001b[0m from pandas import MultiIndex, Int64Index\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Result for LightGBMTrainer_7b049_00000:\n", " date: 2022-06-22_17-26-53\n", " done: false\n", " experiment_id: b4a87c26a7604a43baf895755d4f16b3\n", " hostname: ip-172-31-43-110\n", " iterations_since_restore: 1\n", " node_ip: 172.31.43.110\n", " pid: 1491578\n", " should_checkpoint: true\n", " time_since_restore: 8.369545459747314\n", " time_this_iter_s: 8.369545459747314\n", " time_total_s: 8.369545459747314\n", " timestamp: 1655918813\n", " timesteps_since_restore: 0\n", " train-binary_error: 0.5175879396984925\n", " train-binary_logloss: 0.6302848981539763\n", " training_iteration: 1\n", " trial_id: 7b049_00000\n", " valid-binary_error: 0.2\n", " valid-binary_logloss: 0.558752017793943\n", " warmup_time: 0.008721590042114258\n", " \n", "Result for LightGBMTrainer_7b049_00000:\n", " date: 2022-06-22_17-26-56\n", " done: true\n", " experiment_id: b4a87c26a7604a43baf895755d4f16b3\n", " experiment_tag: '0'\n", " hostname: ip-172-31-43-110\n", " iterations_since_restore: 100\n", " node_ip: 172.31.43.110\n", " pid: 1491578\n", " should_checkpoint: true\n", " time_since_restore: 10.972588300704956\n", " time_this_iter_s: 0.027977466583251953\n", " time_total_s: 10.972588300704956\n", " timestamp: 1655918816\n", " timesteps_since_restore: 0\n", " train-binary_error: 0.0\n", " train-binary_logloss: 0.0005745220956391456\n", " training_iteration: 100\n", " trial_id: 7b049_00000\n", " valid-binary_error: 0.058823529411764705\n", " valid-binary_logloss: 0.17189847605331432\n", " warmup_time: 0.008721590042114258\n", " \n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "2022-06-22 17:26:56,406\tINFO tune.py:734 -- Total run time: 14.73 seconds (14.06 seconds for the tuning loop).\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "{'train-binary_logloss': 0.0005745220956391456, 'train-binary_error': 0.0, 'valid-binary_logloss': 0.17189847605331432, 'valid-binary_error': 0.058823529411764705, 'time_this_iter_s': 0.027977466583251953, 'should_checkpoint': True, 'done': True, 'timesteps_total': None, 'episodes_total': None, 'training_iteration': 100, 'trial_id': '7b049_00000', 'experiment_id': 'b4a87c26a7604a43baf895755d4f16b3', 'date': '2022-06-22_17-26-56', 'timestamp': 1655918816, 'time_total_s': 10.972588300704956, 'pid': 1491578, 'hostname': 'ip-172-31-43-110', 'node_ip': '172.31.43.110', 'config': {}, 'time_since_restore': 10.972588300704956, 'timesteps_since_restore': 0, 'iterations_since_restore': 100, 'warmup_time': 0.008721590042114258, 'experiment_tag': '0'}\n" ] } ], "source": [ "result = train_lightgbm(num_workers=2, use_gpu=False)" ] }, { "cell_type": "markdown", "id": "d7155d9b", "metadata": {}, "source": [ "And perform inference on the obtained model:" ] }, { "cell_type": "code", "execution_count": 17, "id": "871c9be6", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2022-06-22 17:26:57,517\tWARNING read_api.py:260 -- The number of blocks in this dataset (1) limits its parallelism to 1 concurrent tasks. This is much less than the number of available CPU slots in the cluster. Use `.repartition(n)` to increase the number of dataset blocks.\n", "Map_Batches: 100%|██████████| 1/1 [00:00<00:00, 50.96it/s]\n", "Map_Batches: 0%| | 0/1 [00:00