{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.11.11","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":91249,"databundleVersionId":11294684,"sourceType":"competition"},{"sourceId":226382143,"sourceType":"kernelVersion"}],"dockerImageVersionId":31040,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## Overview\n\nIn this notebook, we will investigate the internal process of YOLO to get deeper understanding of them.\n\nWe will try to answer the followng questions:\n\n- How feature maps and anchors actually looks like?\n- What feature maps YOLO mostly relies on?\n\nWe can also showcases how to change default behavior of YOLO using a simple snippet.\n\n## Acknowledgements\n\nI used @andrewjdarley 's trained model for this visualization.\n\n- https://www.kaggle.com/code/andrewjdarley/train-yolo","metadata":{}},{"cell_type":"code","source":"!pip install ultralytics","metadata":{"trusted":true,"_kg_hide-output":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"from ultralytics import YOLO\n\n\nmodel = YOLO(\"/kaggle/input/train-yolo/yolo_weights/motor_detector/weights/best.pt\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-19T02:53:26.818109Z","iopub.execute_input":"2025-05-19T02:53:26.818315Z","iopub.status.idle":"2025-05-19T02:53:26.853563Z","shell.execute_reply.started":"2025-05-19T02:53:26.818293Z","shell.execute_reply":"2025-05-19T02:53:26.852492Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"from ultralytics.nn.tasks import DetectionModel\n\nDetectionModel(cfg=\"yolov8n.yaml\", ch=3, nc=1)\npass","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-19T02:53:26.854439Z","iopub.execute_input":"2025-05-19T02:53:26.854691Z","iopub.status.idle":"2025-05-19T02:53:27.032864Z","shell.execute_reply.started":"2025-05-19T02:53:26.854667Z","shell.execute_reply":"2025-05-19T02:53:27.031656Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Seeing the output of last layer (index 22), it shows `[15, 18, 21]` in the second column. \nThese are the indices of the input layers.\nThe output of these layers are called **anchors**.\nThese are the feature maps with different resolutions.\n\nAs we can see later, `yolov8n` model uses `[1/8, 1/16, 1/32]` resolution of feature maps as the inputs.","metadata":{}},{"cell_type":"markdown","source":"## Visualize Input Image","metadata":{}},{"cell_type":"code","source":"import polars as pl\n\nlabel_df = pl.read_csv(\"/kaggle/input/byu-locating-bacterial-flagellar-motors-2025/train_labels.csv\")\nlabel_df = label_df.filter(pl.col(\"Number of motors\") > 0)\nlabel_df.head(1)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-19T02:53:27.033745Z","iopub.execute_input":"2025-05-19T02:53:27.033992Z","iopub.status.idle":"2025-05-19T02:53:27.046549Z","shell.execute_reply.started":"2025-05-19T02:53:27.033967Z","shell.execute_reply":"2025-05-19T02:53:27.045128Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"row = label_df.to_dicts()[0]\ntomo_id = row[\"tomo_id\"]\nz0 = int(row[\"Motor axis 0\"])\ny0 = row[\"Motor axis 1\"]\nx0 = row[\"Motor axis 2\"]\nimage_path = (\n    f\"/kaggle/input/byu-locating-bacterial-flagellar-motors-2025/train/{tomo_id}/slice_{z0:04d}.jpg\"\n)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-19T02:53:27.047511Z","iopub.execute_input":"2025-05-19T02:53:27.047750Z","iopub.status.idle":"2025-05-19T02:53:27.060434Z","shell.execute_reply.started":"2025-05-19T02:53:27.047734Z","shell.execute_reply":"2025-05-19T02:53:27.059090Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"model.predictor = model._smart_load(\"predictor\")(overrides=dict(imgsz=960), _callbacks=model.callbacks)\nmodel.predictor.setup_model(model=model.model, verbose=True)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-19T03:13:45.962285Z","iopub.execute_input":"2025-05-19T03:13:45.962709Z","iopub.status.idle":"2025-05-19T03:13:45.979346Z","shell.execute_reply.started":"2025-05-19T03:13:45.962678Z","shell.execute_reply":"2025-05-19T03:13:45.978488Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"predictor = model.predictor\npredictor.setup_source(image_path)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-19T03:13:47.001962Z","iopub.execute_input":"2025-05-19T03:13:47.002348Z","iopub.status.idle":"2025-05-19T03:13:47.022935Z","shell.execute_reply.started":"2025-05-19T03:13:47.002326Z","shell.execute_reply":"2025-05-19T03:13:47.022085Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport numpy as np\nfrom matplotlib.patches import Rectangle\n\nfor predictor.batch in predictor.dataset:\n    original_img = model.predictor.batch[1][0]\n    transformed_img = (\n        model.predictor.preprocess(original_img[None, ...])\n        .squeeze(0)\n        .permute(1, 2, 0)\n        .detach()\n        .cpu()\n        .numpy()\n    )\n    print(original_img.shape)\n    print(transformed_img.shape)\n    \n    _, axes = plt.subplots(1, 2, figsize=(10, 5))\n    ax = axes[0]\n    ax.imshow(original_img, cmap=\"gray\")\n    H0, W0, _ = original_img.shape\n    ax.set_title(f\"Original Image ({H0}x{W0})\")\n    s = 24\n    ax.add_patch(\n        Rectangle(\n            (x0 - s / 2, y0 - s / 2),\n            s,\n            s,\n            linewidth=1,\n            edgecolor=\"r\",\n            facecolor=\"none\",\n            alpha=0.5,\n        )\n    )\n    \n    ax = axes[1]\n    ax.imshow(transformed_img, cmap=\"gray\")\n    H, W, _ = transformed_img.shape\n    ax.set_title(f\"Transformed Image ({H}x{W})\")\n    x = (x0 + 0.5) * W / W0 - 0.5\n    y = (y0 + 0.5) * H / H0 - 0.5\n    s = 24\n    ax.add_patch(\n        Rectangle(\n            (x - s / 2, y - s / 2),\n            s,\n            s,\n            linewidth=1,\n            edgecolor=\"r\",\n            facecolor=\"none\",\n            alpha=0.5,\n        )\n    )\n    plt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-19T03:13:47.800569Z","iopub.execute_input":"2025-05-19T03:13:47.800879Z","iopub.status.idle":"2025-05-19T03:13:48.362281Z","shell.execute_reply.started":"2025-05-19T03:13:47.800848Z","shell.execute_reply":"2025-05-19T03:13:48.361186Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"The above pictures shows the before and after preprocessing input tomogram slice.\nThey seems almost the same, but the resolution is adjusted to (960, 960), and we can also see they are padded in the shorter edges (the transform is called `Letterbox` transform which is the default preprocess of YOLO models).","metadata":{}},{"cell_type":"markdown","source":"## Visualize Feature Maps","metadata":{}},{"cell_type":"markdown","source":"Since YOLO does not support visualizing feature maps, we should use some tricks to change YOLO's default behaviour.\n\nWe can use the below snippet to dynamically replace the method of existing objects.\nThe below snippets will change the default behaviour of detection model **to output entire feature maps** (instead of just pooled embeddings).","metadata":{}},{"cell_type":"code","source":"from typing import Callable\n\n\ndef replace_method(obj, method_name: str, func: Callable):\n    \"\"\"\n    Replace a method of an object with a new function.\n\n    Args:\n        obj: The object whose method is to be replaced.\n        method_name (str): The name of the method to replace.\n        func (FunctionType): The new function to set as the method.\n    \"\"\"\n    if not hasattr(obj, method_name):\n        raise AttributeError(\n            f\"{obj.__class__.__name__} has no method '{method_name}' to replace.\"\n        )\n    bound = func.__get__(obj, obj.__class__)\n    setattr(obj, method_name, bound)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-19T03:13:56.729970Z","iopub.execute_input":"2025-05-19T03:13:56.730492Z","iopub.status.idle":"2025-05-19T03:13:56.737874Z","shell.execute_reply.started":"2025-05-19T03:13:56.730453Z","shell.execute_reply":"2025-05-19T03:13:56.736383Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def _predict_once(self, x, profile=False, visualize=False, embed=None):\n    \"\"\"\n    MODIFICATION:\n        - return entire feature map instead of pooled embedding when embed specified\n\n    License: AGPL-3.0\n    Source:\n        - https://github.com/ultralytics/ultralytics\n    \"\"\"\n    print(\"new predict once\")\n    y, dt, embeddings = [], [], []  # outputs\n    for m in self.model:\n        if m.f != -1:  # if not from previous layer\n            x = (\n                y[m.f]\n                if isinstance(m.f, int)\n                else [x if j == -1 else y[j] for j in m.f]\n            )  # from earlier layers\n        if profile:\n            self._profile_one_layer(m, x, dt)\n        x = m(x)  # run\n        y.append(x if m.i in self.save else None)  # save output\n        if embed and m.i in embed:\n            embeddings.append(x.detach())\n            if m.i == max(embed):\n                return embeddings\n    return x\n\n\nreplace_method(model.model, \"_predict_once\", _predict_once)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-19T03:13:56.916404Z","iopub.execute_input":"2025-05-19T03:13:56.916849Z","iopub.status.idle":"2025-05-19T03:13:56.925891Z","shell.execute_reply.started":"2025-05-19T03:13:56.916812Z","shell.execute_reply":"2025-05-19T03:13:56.924689Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Now we get prepared, let's process the sample image to get their feature maps.","metadata":{}},{"cell_type":"code","source":"embeddings = model(image_path, conf=0.005, imgsz=960, verbose=False, embed=[15, 18, 21])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-19T03:13:57.334369Z","iopub.execute_input":"2025-05-19T03:13:57.334828Z","iopub.status.idle":"2025-05-19T03:13:57.597165Z","shell.execute_reply.started":"2025-05-19T03:13:57.334791Z","shell.execute_reply":"2025-05-19T03:13:57.596097Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"for i, emb in enumerate(embeddings):\n    print(i, emb.shape)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-19T03:17:47.747909Z","iopub.execute_input":"2025-05-19T03:17:47.748418Z","iopub.status.idle":"2025-05-19T03:17:47.756297Z","shell.execute_reply.started":"2025-05-19T03:17:47.748387Z","shell.execute_reply":"2025-05-19T03:17:47.754307Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Visualize Feature Maps","metadata":{"_kg_hide-output":true}},{"cell_type":"markdown","source":"The below images are first 5 dimensions of each feature maps.\nWe can roughly see they capture some features in the input slice.","metadata":{}},{"cell_type":"code","source":"import numpy as np\nfrom skimage.transform import resize\n\nfor j, emb in enumerate(embeddings):\n    for i, fmap in enumerate(emb[0, :5]):\n        _, (ax0, ax1) = plt.subplots(1, 2, figsize=(8, 3))\n        ax0.imshow(transformed_img, extent=(0, transformed_img.shape[1], transformed_img.shape[0], 0))\n    \n        resized_fmap = resize(\n            fmap, (transformed_img.shape[0], transformed_img.shape[1]), anti_aliasing=True\n        )\n        ax1.imshow(transformed_img, extent=(0, transformed_img.shape[1], transformed_img.shape[0], 0))\n        g = ax1.imshow(resized_fmap, cmap=\"jet\", alpha=0.5, extent=(0, transformed_img.shape[1], transformed_img.shape[0], 0))\n        ax1.set_title(f\"Feature Map {j} (dim={i})\")\n        plt.colorbar(g, ax=ax1)\n        plt.show()\n    print(\"-\" * 80)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-19T03:24:24.139639Z","iopub.execute_input":"2025-05-19T03:24:24.140124Z","iopub.status.idle":"2025-05-19T03:24:32.040809Z","shell.execute_reply.started":"2025-05-19T03:24:24.140092Z","shell.execute_reply":"2025-05-19T03:24:32.039754Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Visualize Anchors","metadata":{}},{"cell_type":"code","source":"def calc_num_anchors(embedding):\n    _, _, h, w = embedding.shape\n    return h * w\n\nanchor_sizes = []\ntotal_anchors = 0\nfor i, emb in enumerate(embeddings):\n    num_anchors = calc_num_anchors(emb)\n    print(f\"Embedding {i}: {emb.shape}\")\n    print(f\"#anchors: {num_anchors}\")\n    total_anchors += num_anchors\n    anchor_sizes.append(num_anchors)\nprint(f\"Total #anchors: {total_anchors}\")\nprint(anchor_sizes)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-19T03:40:55.229204Z","iopub.execute_input":"2025-05-19T03:40:55.229730Z","iopub.status.idle":"2025-05-19T03:40:55.239229Z","shell.execute_reply.started":"2025-05-19T03:40:55.229697Z","shell.execute_reply":"2025-05-19T03:40:55.237842Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"The out put shows thre feature maps wich has 120x120, 60x60, 30x30 pixels, which corresponds with `[1/8, 1/16, 1/32]` for the original image size (640px).\nEach of these pixels in the feature maps (recall we call them **anchors**) predict the candidate boxes.\n\nMaybe we can get more intuitive image with plotting these anchors in 2D position:","metadata":{}},{"cell_type":"code","source":"from ultralytics.utils.tal import make_anchors\n\n\nhead = model.model.model[-1]\nanchors, strides = (x.transpose(0, 1) for x in make_anchors(embeddings, head.stride, 0.5))\nanchors.shape, strides.shape","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-19T03:13:57.673789Z","iopub.execute_input":"2025-05-19T03:13:57.674221Z","iopub.status.idle":"2025-05-19T03:13:57.688590Z","shell.execute_reply.started":"2025-05-19T03:13:57.674194Z","shell.execute_reply":"2025-05-19T03:13:57.686255Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"for i, emb in enumerate(embeddings):\n    anchor, stride = [\n        x.transpose(0, 1) for x in make_anchors([emb], head.stride[i : i + 1], 0.5)\n    ]\n    print(f\"Embedding {i}: {emb.shape}\")\n    print(f\"Anchor {i}: {anchor.shape}\")\n    print(f\"Stride {i}: {stride.shape}\")\n\n    _, ax  = plt.subplots(figsize=(8, 8))\n    ax.scatter(anchor[0].detach().cpu().numpy(), anchor[1].detach().cpu().numpy(), marker=\".\", alpha=0.5, s=1)\n    ax.set(\n        title=f\"Anchor Positions {i}\",\n        xlabel=\"X\",\n        ylabel=\"Y\",\n        aspect=\"equal\",\n    )\n    plt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-19T03:13:57.780260Z","iopub.execute_input":"2025-05-19T03:13:57.780727Z","iopub.status.idle":"2025-05-19T03:13:58.598309Z","shell.execute_reply.started":"2025-05-19T03:13:57.780694Z","shell.execute_reply":"2025-05-19T03:13:58.597061Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"The first feature map has more anchors which can capture more granular detail of the input image, and the last feature map can handle courser looks.","metadata":{}},{"cell_type":"markdown","source":"## Which Feature Maps are Mostly Used?","metadata":{}},{"cell_type":"markdown","source":"Next, we will investigate **which feature maps contributes much** for the prediction of mortors.\nTo do this, we should apply another replace trick:","metadata":{}},{"cell_type":"code","source":"import torch\n\n\ndef new_forward(self, x):\n    \"\"\"\n    MODIFICATION:\n        - stop before _inference\n\n    License: AGPL-3.0\n    Source:\n        - https://github.com/ultralytics/ultralytics\n    \"\"\"\n    print(\"new forward\")\n    if self.end2end:\n        x_detach = [xi.detach() for xi in x]\n        one2one = [\n            torch.cat(\n                (self.one2one_cv2[i](x_detach[i]), self.one2one_cv3[i](x_detach[i])), 1\n            )\n            for i in range(self.nl)\n        ]\n        return dict(one2many=[], one2one=one2one)\n\n    x_detach = [xi.detach() for xi in x]\n    one2one = [\n        torch.cat(\n            (self.cv2[i](x_detach[i]), self.cv3[i](x_detach[i])), 1\n        )\n        for i in range(self.nl)\n    ]\n    return dict(one2many=[], one2one=one2one)\n\n\nreplace_method(head, \"forward\", new_forward)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-19T03:13:58.599880Z","iopub.execute_input":"2025-05-19T03:13:58.600222Z","iopub.status.idle":"2025-05-19T03:13:58.609236Z","shell.execute_reply.started":"2025-05-19T03:13:58.600199Z","shell.execute_reply":"2025-05-19T03:13:58.607852Z"},"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import torch\nfrom ultralytics.utils.tal import make_anchors\n\n\ndef new_inference(self, x):\n    \"\"\"\n    MODIFICATION:\n        - return the result of dfl head\n\n    License: AGPL-3.0\n    Source:\n        - https://github.com/ultralytics/ultralytics\n    \"\"\"\n    # Inference path\n    shape = x[0].shape  # BCHW\n    x_cat = torch.cat([xi.view(shape[0], self.no, -1) for xi in x], 2)\n\n    if self.dynamic or shape != self.shape:\n        self.anchors, self.strides = (\n            x.transpose(0, 1) for x in make_anchors(x, self.stride, 0.5)\n        )\n        self.shape = shape\n    box, cls = x_cat.split((self.reg_max * 4, self.nc), 1)\n    dfl = self.dfl(box)\n    dbox = self.decode_bboxes(dfl, self.anchors.unsqueeze(0), xywh=False) * self.strides\n\n    return torch.cat((dbox, cls.sigmoid()), 1), dfl\n\n\nreplace_method(head, \"_inference\", new_inference)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-19T03:13:58.610178Z","iopub.execute_input":"2025-05-19T03:13:58.610444Z","iopub.status.idle":"2025-05-19T03:13:58.631350Z","shell.execute_reply.started":"2025-05-19T03:13:58.610424Z","shell.execute_reply":"2025-05-19T03:13:58.630190Z"},"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Visualize DFL Predictions of Each Feature Map","metadata":{}},{"cell_type":"markdown","source":"This replacement make YOLO returns the result of DFL predictions.\nThe **DFL** predicts how each edges of the bounding boxes are shifted from the center position of anchors.\nThey have four values `ltx, lty, rbx, rby` which corresponds with the `left, top, right, bottom` shifts respectively.","metadata":{}},{"cell_type":"code","source":"output = head(embeddings)\nresults, dfl = head._inference(output[\"one2one\"])\nprint(results.shape)\nprint(dfl.shape)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-19T03:14:02.988962Z","iopub.execute_input":"2025-05-19T03:14:02.989415Z","iopub.status.idle":"2025-05-19T03:14:03.088810Z","shell.execute_reply.started":"2025-05-19T03:14:02.989387Z","shell.execute_reply":"2025-05-19T03:14:03.086244Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"lt, rb = dfl.chunk(2, dim=1)\nlt = lt[0].detach().cpu().numpy()\nrb = rb[0].detach().cpu().numpy()\nwh = rb + lt\n\noffset = 0\nfor i, size in enumerate(anchor_sizes):\n    li = offset\n    ui = li + size\n    offset += size\n    s = head.stride[i].item()\n\n    _, ax = plt.subplots(figsize=(4, 3))\n    for label, value in zip(\n        [\"-lbx\", \"-lby\", \"+ubx\", \"+uby\"],\n        [-lt[0], -lt[1], rb[0], rb[1]],\n    ):\n        v = value[li:ui] * s\n\n        ax.hist(v, label=f\"{label}\", bins=100, alpha=0.5)\n    ax.set(\n        title=f\"Distribution of DFL {i}\",\n        xlabel=\"Value\",\n        ylabel=\"Frequency\",\n    )\n    ax.legend(bbox_to_anchor=(1.05, 1), loc=\"upper left\")\n    ax.axvline(-12, linestyle=\"--\", color=\"gray\", alpha=0.5, label=\"-12\")\n    ax.axvline(12, linestyle=\"--\", color=\"gray\", alpha=0.5, label=\"+12\")\n    plt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-19T03:14:03.128512Z","iopub.execute_input":"2025-05-19T03:14:03.129168Z","iopub.status.idle":"2025-05-19T03:14:04.461308Z","shell.execute_reply.started":"2025-05-19T03:14:03.128983Z","shell.execute_reply":"2025-05-19T03:14:04.460268Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"lt, rb = dfl.chunk(2, dim=1)\nlt = lt[0].detach().cpu().numpy()\nrb = rb[0].detach().cpu().numpy()\nwh = rb + lt\n\noffset = 0\nfor i, size in enumerate(anchor_sizes):\n    li = offset\n    ui = li + size\n    offset += size\n    s = head.stride[i].item()\n\n    _, ax = plt.subplots(figsize=(4, 3))\n    for label, value in zip([\"width\", \"height\"], [wh[0], wh[1]]):\n        v = value[li:ui] * s\n\n        ax.hist(v, label=f\"{label}\", bins=100, alpha=0.5)\n    ax.set(\n        title=f\"Distribution of DFL {i}\",\n        xlabel=\"Value\",\n        ylabel=\"Frequency\",\n    )\n    ax.legend(bbox_to_anchor=(1.05, 1), loc=\"upper left\")\n    plt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-19T03:14:04.462741Z","iopub.execute_input":"2025-05-19T03:14:04.463138Z","iopub.status.idle":"2025-05-19T03:14:05.661471Z","shell.execute_reply.started":"2025-05-19T03:14:04.463093Z","shell.execute_reply":"2025-05-19T03:14:05.660828Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Since the size of target bounding boxes are fixed at **24px**, we can see peaks around **20 pixels** in the predictions of first two feature maps.\nOn the other hand, we also see much larger peaks -- ~150px and ~200px respectively -- in the second and the third DFL predictions.\nMaybe these are came from the feature of pre-trained datasets (e.g. MS-COCO), which contains larger bounding boxes.\n\nNote that the DFL prediction of the last feature map doesn't have peaks at 12 pixels, **which indicates these feature map doesn't contribute much for the motor predictions**.\n\nTo see more clear evidence, let's look at the **confidence** of these anchors.","metadata":{}},{"cell_type":"markdown","source":"### Confidence Distribution of Each Feature Map","metadata":{}},{"cell_type":"markdown","source":"The below picture shows top-k distribution of confidence of each feature map.\nAs we anticipated, most of confident bounding boxes are came from the first and the second feature maps, and the third feature map has almost zero confidences. This indicates **the model rely more on the high-resolution feature maps to predict location of motors**.","metadata":{}},{"cell_type":"code","source":"_, ax = plt.subplots(figsize=(4, 3))\n\noffset = 0\nfor i, size in enumerate(anchor_sizes):\n    li = offset\n    ui = li + size\n    offset += size\n\n    conf = results[0, 4, li:ui].detach().cpu().numpy()\n    sorted_conf = np.sort(conf)[::-1]\n\n    ax.plot(\n        sorted_conf[:20],\n        label=f\"feature map {i}\",\n        alpha=0.5,\n    )\n    ax.set(\n        title=\"Confidence\",\n        xlabel=\"Top-K\",\n        ylabel=\"Confidence\",\n    )\n\nax.legend(bbox_to_anchor=(1.05, 1), loc=\"upper left\")\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-19T03:14:05.662289Z","iopub.execute_input":"2025-05-19T03:14:05.662512Z","iopub.status.idle":"2025-05-19T03:14:05.805091Z","shell.execute_reply.started":"2025-05-19T03:14:05.662493Z","shell.execute_reply":"2025-05-19T03:14:05.804315Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Final Result","metadata":{}},{"cell_type":"markdown","source":"The below picture shows the final result of YOLO's bounding box predictions.\nAs we can see, YOLO's prediction is highly \"decisive\" -- **they strongly concentrates on most confident few predictions rather than predicting various possible candidate boxes**. In other words, **they seems to weight precision more over recalls**. As such, it might struggle with tomogram which is very different from train dataset.","metadata":{}},{"cell_type":"code","source":"final_results = head.postprocess(results.permute(0, 2, 1), max_det=head.max_det, nc=head.nc)\nprint(final_results.shape)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"xyxys = final_results[0, :, :4].detach().cpu().numpy()\nconfs = final_results[0, :, 4].detach().cpu().numpy()","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"max_conf = confs.max()\n_, ax = plt.subplots(figsize=(8, 8))\nax.imshow(transformed_img, cmap=\"gray\")\nfor (x1, y1, x2, y2), c in zip(xyxys, confs):\n    ax.add_patch(\n        Rectangle(\n            (x1, y1),\n            x2 - x1,\n            y2 - y1,\n            linewidth=1,\n            edgecolor=\"r\",\n            facecolor=\"none\",\n            alpha=c**0.5 / max_conf**0.5,\n        )\n    )\nax.set(\n    title=\"Predicted Boxes\",\n    xlabel=\"X\",\n    ylabel=\"Y\",\n    aspect=\"equal\",\n)\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-19T03:28:00.715803Z","iopub.execute_input":"2025-05-19T03:28:00.716240Z","iopub.status.idle":"2025-05-19T03:28:01.321550Z","shell.execute_reply.started":"2025-05-19T03:28:00.716207Z","shell.execute_reply":"2025-05-19T03:28:01.320230Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Conclusion","metadata":{}},{"cell_type":"markdown","source":"In this notebook we investigated on the internal process of YOLO.\n\nThe key insights we can take from this notebook is:\n\n* YOLO mostly relies on high-resolution fearture maps than lower ones\n* YOLO seems to weights more on precision over recall","metadata":{}}]}