integrations/torchvision/tutorials/docs-torchvision-sparsify-from-scratch-mobilenetv2-beans.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "00e081b7",
   "metadata": {},
   "source": [
    "# Sparsifying MobileNetv2 from Scratch (Beans)\n",
    "\n",
    "In this example, we will demonstrate how to sparsify an image classification model from scratch using SparseML's PyTorch integration. We train and prune [MobileNetv2](https://pytorch.org/vision/main/models/generated/torchvision.models.mobilenet_v2.html) on the downstream [Beans dataset](https://huggingface.co/datasets/beans) using the Global Magnitude Pruning algorithm. \n",
    "\n",
    "## Agenda\n",
    "\n",
    "There are a few steps:\n",
    "\n",
    " 1. Setup the dataset\n",
    " 2. Setup the PyTorch training loop\n",
    " 3. Train a dense version of MobileNetv2\n",
    " 4. Run the GMP pruning algorithm on the dense model\n",
    " \n",
    "## Installation\n",
    "\n",
    "Install SparseML with `pip`:\n",
    "\n",
    "```\n",
    "pip install sparseml[torchvision]\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "1ad80edf",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import sparseml\n",
    "import torchvision\n",
    "from sparseml.pytorch.optim import ScheduledModifierManager\n",
    "from sparseml.pytorch.utils import TensorBoardLogger, ModuleExporter, get_prunable_layers, tensor_sparsity\n",
    "from torch.utils.data import DataLoader\n",
    "from torch.nn import CrossEntropyLoss\n",
    "from torch.optim import Adam\n",
    "from torchvision import transforms\n",
    "from tqdm.auto import tqdm\n",
    "import math\n",
    "import datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0d30f71c",
   "metadata": {},
   "source": [
    "## Step 1: Setup Dataset\n",
    "\n",
    "Beans leaf dataset is a set of images of diseased and healthy leaves. Based on a leaf image, the goal of this task is to predict the disease type (Angular Leaf Spot and Bean Rust), if any.\n",
    "\n",
    "We will use the Hugging Face `datasets` library to download the data and the torchvision `ImageFolder` in the training loop.\n",
    "\n",
    "[Checkout the dataset card](https://huggingface.co/datasets/beans)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "f8eb3642",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Found cached dataset beans (/home/ubuntu/.cache/huggingface/datasets/beans/default/0.0.0/90c755fb6db1c0ccdad02e897a37969dbf070bed3755d4391e269ff70642d791)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7a630bb60c4a4413a0d43039cd9ac5d9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/3 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "beans_dataset = datasets.load_dataset(\"beans\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "b06fc4fd",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/ubuntu/.cache/huggingface/datasets/downloads/extracted/7ad2d437b751e134577576a32849c44a9ade89297680ad5f6a64051e2108810b/train/angular_leaf_spot/angular_leaf_spot_train.0.jpg\n",
      "/home/ubuntu/.cache/huggingface/datasets/downloads/extracted/1141f86479bc0bb56c75616d153591cc8299d1ea4edc53bb1ab65edd2c65b240/validation/angular_leaf_spot/angular_leaf_spot_val.0.jpg\n"
     ]
    }
   ],
   "source": [
    "print(beans_dataset[\"train\"][0][\"image_file_path\"])\n",
    "print(beans_dataset[\"validation\"][0][\"image_file_path\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "4ecdebad",
   "metadata": {},
   "outputs": [],
   "source": [
    "train_path = \"/home/ubuntu/.cache/huggingface/datasets/downloads/extracted/eeb026374cf5ecfd5f40131a3159be9b9055ac21a3da11690e7eb4d117c99eee/train\"\n",
    "val_path = \"/home/ubuntu/.cache/huggingface/datasets/downloads/extracted/f287261265d2f9a3e8f87a5526a54d1847b17f7c3ec5714e5719432f2b3e4a73/validation\" "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "7d180bbc",
   "metadata": {},
   "outputs": [],
   "source": [
    "NUM_LABELS = 3\n",
    "BATCH_SIZE = 32\n",
    "\n",
    "# imagenet transforms\n",
    "imagenet_transform = transforms.Compose([\n",
    "   transforms.Resize(size=256, interpolation=transforms.InterpolationMode.BILINEAR, max_size=None, antialias=None),\n",
    "   transforms.CenterCrop(size=(224, 224)),\n",
    "   transforms.ToTensor(),\n",
    "   transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])\n",
    "])\n",
    "\n",
    "# datasets\n",
    "train_dataset = torchvision.datasets.ImageFolder(\n",
    "    root=train_path,\n",
    "    transform=imagenet_transform\n",
    ")\n",
    "\n",
    "val_dataset = torchvision.datasets.ImageFolder(\n",
    "    root=val_path,\n",
    "    transform=imagenet_transform\n",
    ")\n",
    "\n",
    "# dataloaders\n",
    "train_loader = DataLoader(train_dataset, BATCH_SIZE, shuffle=True, pin_memory=True, num_workers=16)\n",
    "val_loader = DataLoader(val_dataset, BATCH_SIZE, shuffle=False, pin_memory=True, num_workers=16)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "132fc514",
   "metadata": {},
   "source": [
    "## Step 2: Setup PyTorch Training Loop\n",
    "\n",
    "We will use this training loop below. This is standard PyTorch functionality."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "c968c098",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cuda\n"
     ]
    }
   ],
   "source": [
    "device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
    "print(device)\n",
    "\n",
    "def run_model_one_epoch(model, data_loader, criterion, device, train=False, optimizer=None):\n",
    "    if train:\n",
    "        model.train()\n",
    "    else:\n",
    "        model.eval()\n",
    "\n",
    "    running_loss = 0.0\n",
    "    total_correct = 0\n",
    "    total_predictions = 0\n",
    "\n",
    "    # loop through batches\n",
    "    for step, (inputs, labels) in tqdm(enumerate(data_loader), total=len(data_loader)):\n",
    "        inputs = inputs.to(device)\n",
    "        labels = labels.to(device)\n",
    "\n",
    "        if train:\n",
    "            optimizer.zero_grad()\n",
    "\n",
    "        # compute loss, run backpropogation\n",
    "        outputs = model(inputs)  # model returns logits\n",
    "        loss = criterion(outputs, labels)\n",
    "        if train:\n",
    "            loss.backward()\n",
    "            optimizer.step()\n",
    "\n",
    "        running_loss += loss.item()\n",
    "\n",
    "        # run evaluation\n",
    "        predictions = outputs.argmax(dim=1)\n",
    "        total_correct += torch.sum(predictions == labels).item()\n",
    "        total_predictions += inputs.size(0)\n",
    "\n",
    "    # return loss and evaluation metric\n",
    "    loss = running_loss / (step + 1.0)\n",
    "    accuracy = total_correct / total_predictions\n",
    "    return loss, accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a3521e78",
   "metadata": {},
   "source": [
    "## **Step 3: Train MobileNet-v2 on Beans**\n",
    "\n",
    "First, we will train a dense version of MobileNetv2 on the Beans dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "4d554578",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# download pre-trained model, setup classification head\n",
    "model = torchvision.models.mobilenet_v2(weights=torchvision.models.MobileNet_V2_Weights.DEFAULT)\n",
    "model.classifier[1] = torch.nn.Linear(model.classifier[1].in_features, NUM_LABELS)\n",
    "model.to(device)\n",
    "\n",
    "# setup loss function and optimizer\n",
    "criterion = CrossEntropyLoss()\n",
    "optimizer = Adam(model.parameters(), lr=8e-3) # lr will be override by sparseml"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31064d5c",
   "metadata": {},
   "source": [
    "Next, we will use SparseML's recipes to set the hyperparameters of training loop. In this case, we will use the following recipe:\n",
    "\n",
    "```yaml\n",
    "# Epoch and Learning-Rate variables\n",
    "num_epochs: 10.0\n",
    "init_lr: 0.0005\n",
    "\n",
    "training_modifiers:\n",
    "  - !EpochRangeModifier\n",
    "    start_epoch: 0.0\n",
    "    end_epoch: eval(num_epochs)\n",
    "\n",
    "  - !LearningRateFunctionModifier\n",
    "    final_lr: 0.0\n",
    "    init_lr: eval(init_lr)\n",
    "    lr_func: cosine\n",
    "    start_epoch: 0.0\n",
    "    end_epoch: eval(num_epochs)\n",
    "```\n",
    "\n",
    "As you can see, the recipe includes an `!EpochRangeModifier` and a `!LearningRateFunctionModifier`. These modifiers simply set the number of epochs to train for and the learning rate schedule. As a result, the final model will be dense."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "2723a40f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r\n",
      "# Epoch and Learning-Rate variables\r\n",
      "num_epochs: 10.0\r\n",
      "init_lr: 0.0005\r\n",
      "\r\n",
      "training_modifiers:\r\n",
      "  - !EpochRangeModifier\r\n",
      "    start_epoch: 0.0\r\n",
      "    end_epoch: eval(num_epochs)\r\n",
      "\r\n",
      "  - !LearningRateFunctionModifier\r\n",
      "    final_lr: 0.0\r\n",
      "    init_lr: eval(init_lr)\r\n",
      "    lr_func: cosine\r\n",
      "    start_epoch: 0.0\r\n",
      "    end_epoch: eval(num_epochs)"
     ]
    }
   ],
   "source": [
    "!cat ./recipes/mobilenetv2-beans-dense-recipe.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "cec41d57",
   "metadata": {},
   "outputs": [],
   "source": [
    "dense_recipe_path = \"./recipes/mobilenetv2-beans-dense-recipe.yaml\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f7cba342",
   "metadata": {},
   "source": [
    "Next, we use SparseML's `ScheduledModifierManager` to parse and apply the recipe. The `manager.modify` function modifies and wraps the `model` and `optimizer` with the instructions from the recipe. You can use the `model` and `optimizer` just like standard PyTorch objects."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "1749e00d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# create ScheduledModifierManager and Optimizer wrapper\n",
    "manager = ScheduledModifierManager.from_yaml(dense_recipe_path)\n",
    "optimizer = manager.modify(model, optimizer, steps_per_epoch=len(train_loader))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a00c4ceb",
   "metadata": {},
   "source": [
    "Kick off the transfer learning loop. Our run reached ~99% validation accuracy after 10 epochs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "d00d175b",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running Training Epoch 1/10\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5cc895557d43463fa06855d8cb6a04b6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/33 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training Epoch: 1/10\n",
      "Training Loss: 0.5145365252639308\n",
      "Top 1 Acc: 0.8268858800773694\n",
      "\n",
      "Running Validation Epoch 1/10\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d7e8b1b81291424d88d0bde70b3b21ee",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/5 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation Epoch: 1/10\n",
      "Val Loss: 0.13084974102675914\n",
      "Top 1 Acc: 0.9398496240601504\n",
      "\n",
      "Running Training Epoch 2/10\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c1fe04fab0584f509ceb2cba7a20e70f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/33 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training Epoch: 2/10\n",
      "Training Loss: 0.1433423740619963\n",
      "Top 1 Acc: 0.9497098646034816\n",
      "\n",
      "Running Validation Epoch 2/10\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "daf68b9458b84bb49a33f78bf0c5a68d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/5 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation Epoch: 2/10\n",
      "Val Loss: 0.06366700865328312\n",
      "Top 1 Acc: 0.9774436090225563\n",
      "\n",
      "Running Training Epoch 3/10\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3151d6a608db42a792372a3aa4467e38",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/33 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training Epoch: 3/10\n",
      "Training Loss: 0.04070456024033554\n",
      "Top 1 Acc: 0.9922630560928434\n",
      "\n",
      "Running Validation Epoch 3/10\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2ee08ae3a7144f75831f9e1cd3d7b454",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/5 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation Epoch: 3/10\n",
      "Val Loss: 0.06802571751177311\n",
      "Top 1 Acc: 0.9774436090225563\n",
      "\n",
      "Running Training Epoch 4/10\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "dc12cd7f00284ea58433ed0a5c826791",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/33 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training Epoch: 4/10\n",
      "Training Loss: 0.03408790949844953\n",
      "Top 1 Acc: 0.9903288201160542\n",
      "\n",
      "Running Validation Epoch 4/10\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "facab31d1719407d95198e46bd9444b6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/5 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation Epoch: 4/10\n",
      "Val Loss: 0.0302915308624506\n",
      "Top 1 Acc: 0.9924812030075187\n",
      "\n",
      "Running Training Epoch 5/10\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "59c280ab06744fbeb0234b55667c9fd4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/33 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training Epoch: 5/10\n",
      "Training Loss: 0.014188278910957954\n",
      "Top 1 Acc: 0.9980657640232108\n",
      "\n",
      "Running Validation Epoch 5/10\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "79553b09130d408f90e9825fcb16c1e5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/5 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation Epoch: 5/10\n",
      "Val Loss: 0.036852234788239\n",
      "Top 1 Acc: 0.9774436090225563\n",
      "\n",
      "Running Training Epoch 6/10\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "00eeaceb759d4bedade5095b8bccfc14",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/33 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training Epoch: 6/10\n",
      "Training Loss: 0.008212485654053815\n",
      "Top 1 Acc: 0.9990328820116054\n",
      "\n",
      "Running Validation Epoch 6/10\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "17012e571c1742d394678adda75e0a27",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/5 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation Epoch: 6/10\n",
      "Val Loss: 0.023899264284409583\n",
      "Top 1 Acc: 0.9924812030075187\n",
      "\n",
      "Running Training Epoch 7/10\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8423061cd4b3428d81985783be66ae24",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/33 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training Epoch: 7/10\n",
      "Training Loss: 0.003027547925188573\n",
      "Top 1 Acc: 1.0\n",
      "\n",
      "Running Validation Epoch 7/10\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f2eeb131d20f474585d960b1756fc6a8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/5 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation Epoch: 7/10\n",
      "Val Loss: 0.02565102996304631\n",
      "Top 1 Acc: 0.9924812030075187\n",
      "\n",
      "Running Training Epoch 8/10\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "94b20d80ad3b4c1d8536928b380a6306",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/33 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training Epoch: 8/10\n",
      "Training Loss: 0.003363958947981397\n",
      "Top 1 Acc: 1.0\n",
      "\n",
      "Running Validation Epoch 8/10\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "28472058de974ef48a845de629c800f9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/5 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation Epoch: 8/10\n",
      "Val Loss: 0.0232504392741248\n",
      "Top 1 Acc: 0.9849624060150376\n",
      "\n",
      "Running Training Epoch 9/10\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a7481a29e458478f9fd4113a025003dc",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/33 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training Epoch: 9/10\n",
      "Training Loss: 0.004655612831949397\n",
      "Top 1 Acc: 1.0\n",
      "\n",
      "Running Validation Epoch 9/10\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ff7d10297b3d4658a227df552c3a622d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/5 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation Epoch: 9/10\n",
      "Val Loss: 0.027894998691044746\n",
      "Top 1 Acc: 0.9849624060150376\n",
      "\n",
      "Running Training Epoch 10/10\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ad177de3d1384cb4a32a97a3fd5f5c46",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/33 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training Epoch: 10/10\n",
      "Training Loss: 0.01938260115937076\n",
      "Top 1 Acc: 0.9990328820116054\n",
      "\n",
      "Running Validation Epoch 10/10\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8637fb0cea6e45a5b47dcea94cf7cc71",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/5 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation Epoch: 10/10\n",
      "Val Loss: 0.027051440824288876\n",
      "Top 1 Acc: 0.9849624060150376\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# run transfer learning\n",
    "epoch = 0\n",
    "for epoch in range(manager.max_epochs):\n",
    "    # run training loop\n",
    "    epoch_name = f\"{epoch + 1}/{manager.max_epochs}\"\n",
    "    print(f\"Running Training Epoch {epoch_name}\")\n",
    "    train_loss, train_acc = run_model_one_epoch(model, train_loader, criterion, device, train=True, optimizer=optimizer)\n",
    "    print(f\"Training Epoch: {epoch_name}\\nTraining Loss: {train_loss}\\nTop 1 Acc: {train_acc}\\n\")\n",
    "\n",
    "    # run validation loop\n",
    "    print(f\"Running Validation Epoch {epoch_name}\")\n",
    "    val_loss, val_acc = run_model_one_epoch(model, val_loader, criterion, device)\n",
    "    print(f\"Validation Epoch: {epoch_name}\\nVal Loss: {val_loss}\\nTop 1 Acc: {val_acc}\\n\")\n",
    "\n",
    "manager.finalize(model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "11fa4cf2",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sparseml.pytorch.utils import ModuleExporter\n",
    "\n",
    "save_dir = \"dense_model\"\n",
    "exporter = ModuleExporter(model, output_dir=save_dir)\n",
    "exporter.export_pytorch(name=\"mobilenet-v2-dense-beans.pth\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "273ce478",
   "metadata": {},
   "source": [
    "## Step 4: Prune The Model\n",
    "\n",
    "With a model trained on Beans, we are now ready to apply the GMP algorithm to prune the model. The GMP algorithm is an interative pruning algorithm. At the end of each epoch, we identify the lowest magnitude weights (those closest to 0) and remove them from the network starting from an initial level of sparsity until a final level of sparsity. The remaining nonzero weights are then fine-tuned onto training dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "8447d074",
   "metadata": {},
   "outputs": [],
   "source": [
    "checkpoint = torch.load(\"./dense_model/training/mobilenet-v2-dense-beans.pth\")\n",
    "model = torchvision.models.mobilenet_v2()\n",
    "model.classifier[1] = torch.nn.Linear(model.classifier[1].in_features, NUM_LABELS)\n",
    "model.load_state_dict(checkpoint['state_dict'])\n",
    "model.to(device)\n",
    "\n",
    "# setup loss function and optimizer, LR will be overriden by sparseml\n",
    "criterion = CrossEntropyLoss()\n",
    "optimizer = Adam(model.parameters(), lr=8e-3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1b188d60",
   "metadata": {},
   "source": [
    "Next, we need to create a SparseML recipe which includes the GMP algorithm. The `!GlobalMagnitudePruningModifier` modifier instructs SparseML to apply the GMP algorithm at a global level (pruning the lowest magnitude weights across all layers).\n",
    "\n",
    "Firstly, we need to decide identify which parameters of the model to apply the GMP algorithm to. We can use the `get_prunable_layers` function to inspect:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "44578399",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "features.0.0\n",
      "features.1.conv.0.0\n",
      "features.1.conv.1\n",
      "features.2.conv.0.0\n",
      "features.2.conv.1.0\n",
      "features.2.conv.2\n",
      "features.3.conv.0.0\n",
      "features.3.conv.1.0\n",
      "features.3.conv.2\n",
      "features.4.conv.0.0\n",
      "features.4.conv.1.0\n",
      "features.4.conv.2\n",
      "features.5.conv.0.0\n",
      "features.5.conv.1.0\n",
      "features.5.conv.2\n",
      "features.6.conv.0.0\n",
      "features.6.conv.1.0\n",
      "features.6.conv.2\n",
      "features.7.conv.0.0\n",
      "features.7.conv.1.0\n",
      "features.7.conv.2\n",
      "features.8.conv.0.0\n",
      "features.8.conv.1.0\n",
      "features.8.conv.2\n",
      "features.9.conv.0.0\n",
      "features.9.conv.1.0\n",
      "features.9.conv.2\n",
      "features.10.conv.0.0\n",
      "features.10.conv.1.0\n",
      "features.10.conv.2\n",
      "features.11.conv.0.0\n",
      "features.11.conv.1.0\n",
      "features.11.conv.2\n",
      "features.12.conv.0.0\n",
      "features.12.conv.1.0\n",
      "features.12.conv.2\n",
      "features.13.conv.0.0\n",
      "features.13.conv.1.0\n",
      "features.13.conv.2\n",
      "features.14.conv.0.0\n",
      "features.14.conv.1.0\n",
      "features.14.conv.2\n",
      "features.15.conv.0.0\n",
      "features.15.conv.1.0\n",
      "features.15.conv.2\n",
      "features.16.conv.0.0\n",
      "features.16.conv.1.0\n",
      "features.16.conv.2\n",
      "features.17.conv.0.0\n",
      "features.17.conv.1.0\n",
      "features.17.conv.2\n",
      "features.18.0\n",
      "classifier.1\n"
     ]
    }
   ],
   "source": [
    "# print parameters\n",
    "for (name, layer) in get_prunable_layers(model):\n",
    "    print(f\"{name}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7249f54d",
   "metadata": {},
   "source": [
    "We will apply pruning to each of the convs and exclude the classifier layer (which is the final projection head). Fortunately, SparseML allows us to pass regexes to identify layers in the network, so we can use the following list to identify the relevant layers for pruning:\n",
    "    \n",
    "    - 'features.0.0.weight'\n",
    "    - 'features.18.0.weight'\n",
    "    - 're:features.*.conv.*.weight'\n",
    "    - 're:features.*.conv.*.*.weight'\n",
    "\n",
    "Here is what the recipe looks like:\n",
    "\n",
    "```yaml\n",
    "# Epoch hyperparams\n",
    "stabilization_epochs: 1.0\n",
    "pruning_epochs: 9.0\n",
    "finetuning_epochs: 5.0\n",
    "\n",
    "# Learning rate hyperparams\n",
    "init_lr: 0.0005\n",
    "final_lr: 0.00025\n",
    "\n",
    "# Pruning hyperparams\n",
    "init_sparsity: 0.05\n",
    "final_sparsity: 0.9\n",
    "\n",
    "# Stabalization Stage\n",
    "training_modifiers:\n",
    "  - !EpochRangeModifier\n",
    "    start_epoch: 0.0\n",
    "    end_epoch: eval(stabilization_epochs + pruning_epochs + finetuning_epochs)\n",
    "  \n",
    "  - !SetLearningRateModifier\n",
    "    start_epoch: 0.0\n",
    "    learning_rate: eval(init_lr)\n",
    "\n",
    "# Pruning Stage\n",
    "pruning_modifiers:\n",
    "  - !LearningRateFunctionModifier\n",
    "    init_lr: eval(init_lr)\n",
    "    final_lr: eval(final_lr)\n",
    "    lr_func: cosine\n",
    "    start_epoch: eval(stabilization_epochs)\n",
    "    end_epoch: eval(stabilization_epochs + pruning_epochs)\n",
    "    \n",
    "  - !GlobalMagnitudePruningModifier\n",
    "    init_sparsity: eval(init_sparsity)\n",
    "    final_sparsity: eval(final_sparsity)\n",
    "    start_epoch: eval(stabilization_epochs)\n",
    "    end_epoch: eval(stabilization_epochs + pruning_epochs)\n",
    "    update_frequency: 0.5\n",
    "    params:        \n",
    "        - 'features.0.0.weight'\n",
    "        - 'features.18.0.weight'\n",
    "        - 're:features.*.conv.*.weight'\n",
    "        - 're:features.*.conv.*.*.weight'\n",
    "    leave_enabled: True\n",
    "\n",
    "# Finetuning Stage\n",
    "finetuning_modifiers:\n",
    "  - !LearningRateFunctionModifier\n",
    "    init_lr: eval(init_lr)\n",
    "    final_lr: eval(final_lr)\n",
    "    lr_func: cosine\n",
    "    start_epoch: eval(stabilization_epochs + pruning_epochs)\n",
    "    end_epoch: eval(stabilization_epochs + pruning_epochs + finetuning_epochs)\n",
    "```\n",
    "\n",
    "This recipe specifies that we will run the GMP algorithm for 9 epochs after 1 warmup epoch. We start at an init_sparsity level of 5% and gradually increase sparsity to a final_sparsity level of 90% following a cubic curve. The pruning is applied in an unstructured manner, meaning that any weight can be pruned.\n",
    "\n",
    "Over the final 5 epochs, we will fine-tune the 90% pruned model further. Since we set leave_enabled=True the sparsity level will be maintained as the fine-tuning occurs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "ca5c28b2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# Epoch hyperparams\r\n",
      "stabilization_epochs: 1.0\r\n",
      "pruning_epochs: 9.0\r\n",
      "finetuning_epochs: 5.0\r\n",
      "\r\n",
      "# Learning rate hyperparams\r\n",
      "init_lr: 0.0005\r\n",
      "final_lr: 0.00025\r\n",
      "\r\n",
      "# Pruning hyperparams\r\n",
      "init_sparsity: 0.05\r\n",
      "final_sparsity: 0.9\r\n",
      "\r\n",
      "# Stabalization Stage\r\n",
      "training_modifiers:\r\n",
      "  - !EpochRangeModifier\r\n",
      "    start_epoch: 0.0\r\n",
      "    end_epoch: eval(stabilization_epochs + pruning_epochs + finetuning_epochs)\r\n",
      "  \r\n",
      "  - !SetLearningRateModifier\r\n",
      "    start_epoch: 0.0\r\n",
      "    learning_rate: eval(init_lr)\r\n",
      "\r\n",
      "# Pruning Stage\r\n",
      "pruning_modifiers:\r\n",
      "  - !LearningRateFunctionModifier\r\n",
      "    init_lr: eval(init_lr)\r\n",
      "    final_lr: eval(final_lr)\r\n",
      "    lr_func: cosine\r\n",
      "    start_epoch: eval(stabilization_epochs)\r\n",
      "    end_epoch: eval(stabilization_epochs + pruning_epochs)\r\n",
      "    \r\n",
      "  - !GlobalMagnitudePruningModifier\r\n",
      "    init_sparsity: eval(init_sparsity)\r\n",
      "    final_sparsity: eval(final_sparsity)\r\n",
      "    start_epoch: eval(stabilization_epochs)\r\n",
      "    end_epoch: eval(stabilization_epochs + pruning_epochs)\r\n",
      "    update_frequency: 0.5\r\n",
      "    params:        \r\n",
      "        - 'features.0.0.weight'\r\n",
      "        - 'features.18.0.weight'\r\n",
      "        - 're:features.*.conv.*.weight'\r\n",
      "        - 're:features.*.conv.*.*.weight'\r\n",
      "    leave_enabled: True\r\n",
      "\r\n",
      "# Finetuning Stage\r\n",
      "finetuning_modifiers:\r\n",
      "  - !LearningRateFunctionModifier\r\n",
      "    init_lr: eval(init_lr)\r\n",
      "    final_lr: eval(final_lr)\r\n",
      "    lr_func: cosine\r\n",
      "    start_epoch: eval(stabilization_epochs + pruning_epochs)\r\n",
      "    end_epoch: eval(stabilization_epochs + pruning_epochs + finetuning_epochs)"
     ]
    }
   ],
   "source": [
    "!cat ./recipes/mobilenetv2-beans-pruning-recipe.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "b55a7c74",
   "metadata": {},
   "outputs": [],
   "source": [
    "pruning_recipe_path = \"./recipes/mobilenetv2-beans-pruning-recipe.yaml\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "68646bc7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# create ScheduledModifierManager and Optimizer wrapper\n",
    "manager = ScheduledModifierManager.from_yaml(pruning_recipe_path)\n",
    "logger = TensorBoardLogger(log_path=\"./tensorboard_outputs\")\n",
    "optimizer = manager.modify(model, optimizer, loggers=[logger], steps_per_epoch=len(train_loader))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "37d4b6a4",
   "metadata": {},
   "source": [
    "Next, kick off the GMP training loop. \n",
    "\n",
    "As you can see, we use the wrapped `optimizer` and `model` in the same way as above. SparseML parsed the recipe and updated the `optimizer` with the logic of GMP algorithm from the recipe. This allows you to just the `optimizer` and `model` as usual, with all of the pruning-related logic specified by the declarative recipe interface.\n",
    "\n",
    "Our 90% pruned model reaches ~99% validation accuracy (vs ~98% for the dense model)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "601c8c21",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running Training Epoch 1/15\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3b037b49d42c41abadc2bbab546e5760",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/33 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training Epoch: 1/15\n",
      "Training Loss: 0.04981424145058334\n",
      "Top 1 Acc: 0.988394584139265\n",
      "\n",
      "Running Validation Epoch 1/15\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "aa75f114f7754ea4a4b7509826d769e3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/5 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation Epoch: 1/15\n",
      "Val Loss: 0.06650052629411221\n",
      "Top 1 Acc: 0.9774436090225563\n",
      "\n",
      "Running Training Epoch 2/15\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "50b86debcf3148a687d17c36c601c99d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/33 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training Epoch: 2/15\n",
      "Training Loss: 0.08335415836226082\n",
      "Top 1 Acc: 0.9777562862669246\n",
      "\n",
      "Running Validation Epoch 2/15\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e741553cb741416b98bfa577d529d575",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/5 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation Epoch: 2/15\n",
      "Val Loss: 0.06197752933949232\n",
      "Top 1 Acc: 0.9774436090225563\n",
      "\n",
      "Running Training Epoch 3/15\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0b86b60f094e4d1e9338aaa664cdf901",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/33 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training Epoch: 3/15\n",
      "Training Loss: 0.040097551427386476\n",
      "Top 1 Acc: 0.9854932301740812\n",
      "\n",
      "Running Validation Epoch 3/15\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c2899c110f954430a41ed5a8410f21a4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/5 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation Epoch: 3/15\n",
      "Val Loss: 0.05052837722469121\n",
      "Top 1 Acc: 0.9849624060150376\n",
      "\n",
      "Running Training Epoch 4/15\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6c7cc173588c4ade9a5315a16ac9eec8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/33 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training Epoch: 4/15\n",
      "Training Loss: 0.03676384532203277\n",
      "Top 1 Acc: 0.9893617021276596\n",
      "\n",
      "Running Validation Epoch 4/15\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "99aefcf3d0b2421a896fa2b9d03e4fc1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/5 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation Epoch: 4/15\n",
      "Val Loss: 0.08736732564866542\n",
      "Top 1 Acc: 0.9624060150375939\n",
      "\n",
      "Running Training Epoch 5/15\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "37d08c11516b44fbaf3cdb52b46b5144",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/33 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training Epoch: 5/15\n",
      "Training Loss: 0.03125915414244501\n",
      "Top 1 Acc: 0.9932301740812379\n",
      "\n",
      "Running Validation Epoch 5/15\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e2ff32388c1746588cc01452aa64e7bd",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/5 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation Epoch: 5/15\n",
      "Val Loss: 0.09458159506320954\n",
      "Top 1 Acc: 0.9924812030075187\n",
      "\n",
      "Running Training Epoch 6/15\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8ff4dcbc05de41d5b22850148996c74c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/33 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training Epoch: 6/15\n",
      "Training Loss: 0.0748459067421429\n",
      "Top 1 Acc: 0.9796905222437138\n",
      "\n",
      "Running Validation Epoch 6/15\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5d12d2c66f06462aa9cc276d9ee2ea38",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/5 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation Epoch: 6/15\n",
      "Val Loss: 0.20114150643348694\n",
      "Top 1 Acc: 0.9699248120300752\n",
      "\n",
      "Running Training Epoch 7/15\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4fab3bd349f44218a93177ca9d73481f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/33 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training Epoch: 7/15\n",
      "Training Loss: 0.10897209077621951\n",
      "Top 1 Acc: 0.965183752417795\n",
      "\n",
      "Running Validation Epoch 7/15\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4d54279b4f8d456eb485c6a749248603",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/5 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation Epoch: 7/15\n",
      "Val Loss: 0.3452451154589653\n",
      "Top 1 Acc: 0.8421052631578947\n",
      "\n",
      "Running Training Epoch 8/15\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "39115311d08a4bc6a94d8e266991cce7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/33 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training Epoch: 8/15\n",
      "Training Loss: 0.13403249407807985\n",
      "Top 1 Acc: 0.9506769825918762\n",
      "\n",
      "Running Validation Epoch 8/15\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a0cb864970ad4f4d928f59f34eaadeb4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/5 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation Epoch: 8/15\n",
      "Val Loss: 0.20368995219469072\n",
      "Top 1 Acc: 0.9323308270676691\n",
      "\n",
      "Running Training Epoch 9/15\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "655cc448f70746e7bc20bd6e834ae19f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/33 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training Epoch: 9/15\n",
      "Training Loss: 0.12121003064693826\n",
      "Top 1 Acc: 0.9671179883945842\n",
      "\n",
      "Running Validation Epoch 9/15\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2d55e9450fae497e96b86211ca33812e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/5 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation Epoch: 9/15\n",
      "Val Loss: 0.11063596755266189\n",
      "Top 1 Acc: 0.9774436090225563\n",
      "\n",
      "Running Training Epoch 10/15\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2744a665b1094c96a80cbe44a87207cb",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/33 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training Epoch: 10/15\n",
      "Training Loss: 0.09124855725376894\n",
      "Top 1 Acc: 0.9642166344294004\n",
      "\n",
      "Running Validation Epoch 10/15\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6779935332bb4ebdb701baedf3866911",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/5 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation Epoch: 10/15\n",
      "Val Loss: 0.0751135416328907\n",
      "Top 1 Acc: 0.9699248120300752\n",
      "\n",
      "Running Training Epoch 11/15\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4b84d196120844b9ba1254eda5a61c06",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/33 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training Epoch: 11/15\n",
      "Training Loss: 0.04337389501884128\n",
      "Top 1 Acc: 0.9912959381044487\n",
      "\n",
      "Running Validation Epoch 11/15\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "88d8406bbfcc4886a9415ace34de12a1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/5 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation Epoch: 11/15\n",
      "Val Loss: 0.034493994899094105\n",
      "Top 1 Acc: 0.9774436090225563\n",
      "\n",
      "Running Training Epoch 12/15\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "afec4141ee4d4e4080fb460168a09534",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/33 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training Epoch: 12/15\n",
      "Training Loss: 0.018291185736994852\n",
      "Top 1 Acc: 0.9980657640232108\n",
      "\n",
      "Running Validation Epoch 12/15\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f21726e565b14415975169049d1c5931",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/5 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation Epoch: 12/15\n",
      "Val Loss: 0.04321025339886546\n",
      "Top 1 Acc: 0.9774436090225563\n",
      "\n",
      "Running Training Epoch 13/15\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0eb6ef213d5f41758f2c89f31fb6aa01",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/33 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training Epoch: 13/15\n",
      "Training Loss: 0.0146882157899778\n",
      "Top 1 Acc: 0.9970986460348162\n",
      "\n",
      "Running Validation Epoch 13/15\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9aeef2ccf0dd4fb8941bafd967316189",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/5 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation Epoch: 13/15\n",
      "Val Loss: 0.03916811663657427\n",
      "Top 1 Acc: 0.9924812030075187\n",
      "\n",
      "Running Training Epoch 14/15\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "32f885475f8049b2ab560ffbc4c9390f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/33 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training Epoch: 14/15\n",
      "Training Loss: 0.014786866712242816\n",
      "Top 1 Acc: 0.9990328820116054\n",
      "\n",
      "Running Validation Epoch 14/15\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2a55281ea41c4263874838cd59440ef9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/5 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation Epoch: 14/15\n",
      "Val Loss: 0.040105257043614985\n",
      "Top 1 Acc: 0.9849624060150376\n",
      "\n",
      "Running Training Epoch 15/15\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b7bf416c8c5f4ee5893e170b01e1adcc",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/33 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training Epoch: 15/15\n",
      "Training Loss: 0.007484704270168688\n",
      "Top 1 Acc: 1.0\n",
      "\n",
      "Running Validation Epoch 15/15\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2e9f4636ad844a869db5541b37f9732a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/5 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation Epoch: 15/15\n",
      "Val Loss: 0.02973868220578879\n",
      "Top 1 Acc: 0.9774436090225563\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# run GMP algorithm\n",
    "epoch = 0\n",
    "for epoch in range(manager.max_epochs):\n",
    "    # run training loop\n",
    "    epoch_name = f\"{epoch + 1}/{manager.max_epochs}\"\n",
    "    print(f\"Running Training Epoch {epoch_name}\")\n",
    "    train_loss, train_acc = run_model_one_epoch(model, train_loader, criterion, device, train=True, optimizer=optimizer)\n",
    "    print(f\"Training Epoch: {epoch_name}\\nTraining Loss: {train_loss}\\nTop 1 Acc: {train_acc}\\n\")\n",
    "\n",
    "    # run validation loop\n",
    "    print(f\"Running Validation Epoch {epoch_name}\")\n",
    "    val_loss, val_acc = run_model_one_epoch(model, val_loader, criterion, device)\n",
    "    print(f\"Validation Epoch: {epoch_name}\\nVal Loss: {val_loss}\\nTop 1 Acc: {val_acc}\\n\")\n",
    "    \n",
    "    logger.log_scalar(\"Metrics/Loss (Train)\", train_loss, epoch)\n",
    "    logger.log_scalar(\"Metrics/Accuracy (Train)\", train_acc, epoch)\n",
    "    logger.log_scalar(\"Metrics/Loss (Validation)\", val_loss, epoch)\n",
    "    logger.log_scalar(\"Metrics/Accuracy (Validation)\", val_acc, epoch)\n",
    "    \n",
    "manager.finalize(model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "2bf48ed2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "features.0.0.weight: 0.4896\n",
      "features.1.conv.0.0.weight: 0.4722\n",
      "features.1.conv.1.weight: 0.5234\n",
      "features.2.conv.0.0.weight: 0.5501\n",
      "features.2.conv.1.0.weight: 0.2859\n",
      "features.2.conv.2.weight: 0.4813\n",
      "features.3.conv.0.0.weight: 0.6137\n",
      "features.3.conv.1.0.weight: 0.4977\n",
      "features.3.conv.2.weight: 0.6131\n",
      "features.4.conv.0.0.weight: 0.5214\n",
      "features.4.conv.1.0.weight: 0.2785\n",
      "features.4.conv.2.weight: 0.5184\n",
      "features.5.conv.0.0.weight: 0.6903\n",
      "features.5.conv.1.0.weight: 0.5804\n",
      "features.5.conv.2.weight: 0.7134\n",
      "features.6.conv.0.0.weight: 0.6929\n",
      "features.6.conv.1.0.weight: 0.6117\n",
      "features.6.conv.2.weight: 0.7383\n",
      "features.7.conv.0.0.weight: 0.5493\n",
      "features.7.conv.1.0.weight: 0.3588\n",
      "features.7.conv.2.weight: 0.6127\n",
      "features.8.conv.0.0.weight: 0.8385\n",
      "features.8.conv.1.0.weight: 0.7396\n",
      "features.8.conv.2.weight: 0.8687\n",
      "features.9.conv.0.0.weight: 0.8278\n",
      "features.9.conv.1.0.weight: 0.7376\n",
      "features.9.conv.2.weight: 0.8643\n",
      "features.10.conv.0.0.weight: 0.8293\n",
      "features.10.conv.1.0.weight: 0.7876\n",
      "features.10.conv.2.weight: 0.8721\n",
      "features.11.conv.0.0.weight: 0.7277\n",
      "features.11.conv.1.0.weight: 0.7995\n",
      "features.11.conv.2.weight: 0.7729\n",
      "features.12.conv.0.0.weight: 0.8823\n",
      "features.12.conv.1.0.weight: 0.7951\n",
      "features.12.conv.2.weight: 0.9053\n",
      "features.13.conv.0.0.weight: 0.8716\n",
      "features.13.conv.1.0.weight: 0.8135\n",
      "features.13.conv.2.weight: 0.9115\n",
      "features.14.conv.0.0.weight: 0.7872\n",
      "features.14.conv.1.0.weight: 0.6908\n",
      "features.14.conv.2.weight: 0.8274\n",
      "features.15.conv.0.0.weight: 0.9449\n",
      "features.15.conv.1.0.weight: 0.7853\n",
      "features.15.conv.2.weight: 0.9563\n",
      "features.16.conv.0.0.weight: 0.9238\n",
      "features.16.conv.1.0.weight: 0.7931\n",
      "features.16.conv.2.weight: 0.9572\n",
      "features.17.conv.0.0.weight: 0.8772\n",
      "features.17.conv.1.0.weight: 0.9105\n",
      "features.17.conv.2.weight: 0.9512\n",
      "features.18.0.weight: 0.9781\n",
      "classifier.1.weight: 0.0000\n"
     ]
    }
   ],
   "source": [
    "for (name, layer) in get_prunable_layers(model):\n",
    "    print(f\"{name}.weight: {tensor_sparsity(layer.weight).item():.4f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "00805b02",
   "metadata": {},
   "source": [
    "Finally, export your model to ONNX."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "48fb7c0e",
   "metadata": {},
   "outputs": [],
   "source": [
    "save_dir = \"experiment-0\"\n",
    "exporter = ModuleExporter(model, output_dir=save_dir)\n",
    "exporter.export_pytorch(name=\"mobilenet_v2-sparse-beans.pth\")\n",
    "exporter.export_onnx(torch.randn(1, 3, 224, 224), name=\"sparse-model.onnx\", convert_qat=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7960612c",
   "metadata": {},
   "source": [
    "## Wrap Up\n",
    "\n",
    "The resulting model is is 90% sparse and achieves validation accuracy of ~99% (vs the unoptimized dense model at ~99%) without much hyperparameter search.\n",
    "\n",
    "Key hyperparameter experiments you may want to run include:\n",
    "- Learning rate\n",
    "- Learning rate schedule\n",
    "- Sparsity level\n",
    "- Number of pruning epochs\n",
    "\n",
    "DeepSparse supports speedup from pruning and quantization. To reach maximum performance, check out our examples of quantizing a model!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}