{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Removing failure cases of cropping bouding boxes\nIt seems like having a good cropping method in this competition is crucial to boost any solution performances. It removes unecessary informations and potential distractions for the model such as mountains in brackground, people, boats
\n\nIn this notebook, I start from the amazing work of @AWSAF in the notebook [Happywhale: BoundingBox [YOLOv5] š¦](https://www.kaggle.com/awsaf49/happywhale-boundingbox-yolov5/), and try to develop a way of keeping only tho ones that are correct.
\nThe work achieved in the above mentioned notebook is amazing, however their are a significant number of diverse failure cases which cannot be ignored.\n\n**EDIT**: (v4) I modified this notebook to take the detic cropped dataset as input\n\nIn this notebook I will use Pytorch Lightning ā”","metadata":{}},{"cell_type":"markdown","source":"# Failure cases ā\n## Overcropped images\n\n## Undercropped images\n\n## Images cropped on something else than a š³ or a š¬\n\n## \"Adversarial\" examples\n\n## Wtf are you doing examples\n","metadata":{}},{"cell_type":"markdown","source":"# Possibles methods š\nThere are several possibilities to achieve this goal.
\nThe first I thought about was to use OOD (Out Of Distribution) detection methods, because a wrong bounding box can be interpreted as an OOD sample.\nHowever, I'm not familiar enough with these methods so I'll try an homemade method inspired from what I know of these.
","metadata":{}},{"cell_type":"markdown","source":"# Method of this notebook šØāš\nI've trained a EfficientNetB0 classifier to recognize the species of the train images, and I plan to use it to detect failure cases by:\n1. Making a prediction on the raw images\n2. Making a prediction on the cropped images\n3. Comparing the predictions\n\nš” The idea is that if a bounding box is correct, then the prediction should be similar if not better (because the details on the animal are better detectable), while if it's not, the prediction should be less confident and or wrong.
\nAs we want to make predictions for the test set, we do not have the correct labels of the species so we will not use them. Instead, we will only look at the confidence scores of the answers and the entropy of the prediction.\n\n## Examples\nThe prediction should be somewhat similar if not better:\n\n\nThe prediction should be very different and very less confident:\n\n\n## Things to be careful about\nšØ As we will use the predictions we have to make sure that the samples don't come for the training distribution of the model. That's why I'll use 4 models trained using 4 folds.
\nFor the training data, I will make the predictions using the model that was not trained on their corresponding fold. For the test data, I'll take the average of the predictions of the 4 models.
\n\nšØ While I attempt to fix most of the failure cases, there are still that are left using this method. Indeed, when a failure case is detected, it will be replaced by the original image so no cropping will be performed. Also, this approach doesn't fix the images that were already not annotated by the original notebook.","metadata":{}},{"cell_type":"code","source":"!pip install -q bbox-utility # check https://github.com/awsaf49/bbox for source code","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-02-26T17:40:42.522094Z","iopub.execute_input":"2022-02-26T17:40:42.522483Z","iopub.status.idle":"2022-02-26T17:40:55.072935Z","shell.execute_reply.started":"2022-02-26T17:40:42.522394Z","shell.execute_reply":"2022-02-26T17:40:55.071814Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport os\nimport torch\nimport torchmetrics\nimport math\nimport copy\nimport cv2\nimport matplotlib.pyplot as plt\nimport time\nimport glob\nimport shutil\nimport albumentations\nfrom tqdm.notebook import tqdm\nfrom bbox.utils import yolo2voc, draw_bboxes\nfrom sklearn.model_selection import StratifiedKFold\nfrom torch.utils.data import Dataset\nfrom albumentations.pytorch.transforms import ToTensorV2\nfrom torchvision import transforms\nfrom torch import nn, Tensor\nfrom torch.nn import functional as F\nfrom pytorch_lightning import Callback, LightningModule, Trainer\nfrom pytorch_lightning.core.lightning import LightningModule","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-26T17:40:55.077277Z","iopub.execute_input":"2022-02-26T17:40:55.077553Z","iopub.status.idle":"2022-02-26T17:41:02.794764Z","shell.execute_reply.started":"2022-02-26T17:40:55.077521Z","shell.execute_reply":"2022-02-26T17:41:02.793683Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class CFG:\n # Seed to have deterministic results\n SEED = 0\n # Size of the images\n size = 224\n # inference batch size\n batch_size = 128 * 8#64\n # Number of workers to load the data\n num_workers = 2\n # Number of folds\n FOLDS = 4\n # Number of classes\n N_CLASSES = 26\n # Expected proportion of the data to filter\n FLAG_QUANTILE = 0.015\n # Minimal number of flag to be treated as out sample\n N_FLAGS = 2","metadata":{"_kg_hide-input":false,"execution":{"iopub.status.busy":"2022-02-26T17:41:02.797896Z","iopub.execute_input":"2022-02-26T17:41:02.7986Z","iopub.status.idle":"2022-02-26T17:41:02.808947Z","shell.execute_reply.started":"2022-02-26T17:41:02.798547Z","shell.execute_reply":"2022-02-26T17:41:02.804783Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def plot_images(batch, row=2, col=2, base_path=\"../input/w-d-224x224-fast-dataset/train_images/\"):\n \"\"\"\n Copied and adapted from https://www.kaggle.com/awsaf49/happywhale-data-distribution\n \"\"\"\n plt.figure(figsize=(col*3, row*3))\n for i in range(row*col):\n plt.subplot(row, col, i+1)\n img = cv2.imread(os.path.join(base_path, batch[\"image\"].iloc[i]))\n if img is None:\n continue\n img = img[:, :, ::-1]\n plt.imshow(img)\n if \"species\" in batch:\n plt.title(batch[\"species\"].iloc[i])\n plt.axis('off')\n plt.tight_layout()\n plt.show()\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-26T17:41:02.811703Z","iopub.execute_input":"2022-02-26T17:41:02.812057Z","iopub.status.idle":"2022-02-26T17:41:02.835112Z","shell.execute_reply.started":"2022-02-26T17:41:02.812008Z","shell.execute_reply":"2022-02-26T17:41:02.833541Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class WandDLoader(Dataset):\n def __init__(self, data, d_type, crop=False):\n self.crop = crop\n self.images = (data[\"image\"].str[:-3] + \"bmp\").values.tolist()\n \n transformations = albumentations.Compose([\n albumentations.Normalize(),\n ToTensorV2(p=1.0)\n ])\n\n def albumentations_transform(image, transform=transformations):\n if transform:\n image_np = np.array(image)\n augmented = transform(image=image_np)\n return augmented\n\n self.transforms = transforms.Compose([\n transforms.Lambda(albumentations_transform),\n ])\n path = \"../input/w-d-fast-224x224-cropped-dataset\" if crop else \"../input/w-d-224x224-fast-dataset\"\n self.base_path = os.path.join(path, f\"{d_type}_images\")\n \n def __getitem__(self, idx):\n image_name = self.images[idx]\n \n path = os.path.join(self.base_path, image_name)\n img = cv2.imread(path)\n \n if self.transforms is not None:\n img = self.transforms(img)[\"image\"]\n return img\n \n def __len__(self):\n return len(self.images)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-26T17:41:02.837604Z","iopub.execute_input":"2022-02-26T17:41:02.838025Z","iopub.status.idle":"2022-02-26T17:41:02.852111Z","shell.execute_reply.started":"2022-02-26T17:41:02.83795Z","shell.execute_reply":"2022-02-26T17:41:02.850816Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\"\"\"\n https://pytorch.org/vision/main/_modules/torchvision/models/efficientnet.html\n\"\"\"\n\nfrom typing import Any, Callable, Optional, List, Sequence\nfrom functools import partial\ntry:\n from torch.hub import load_state_dict_from_url # noqa: 401\nexcept ImportError:\n from torch.utils.model_zoo import load_url as load_state_dict_from_url # noqa: 401\n\ndef stochastic_depth(input: Tensor, p: float, mode: str, training: bool = True) -> Tensor:\n \"\"\"\n Implements the Stochastic Depth from `\"Deep Networks with Stochastic Depth\"\n `_ used for randomly dropping residual\n branches of residual architectures.\n Args:\n input (Tensor[N, ...]): The input tensor or arbitrary dimensions with the first one\n being its batch i.e. a batch with ``N`` rows.\n p (float): probability of the input to be zeroed.\n mode (str): ``\"batch\"`` or ``\"row\"``.\n ``\"batch\"`` randomly zeroes the entire input, ``\"row\"`` zeroes\n randomly selected rows from the batch.\n training: apply stochastic depth if is ``True``. Default: ``True``\n Returns:\n Tensor[N, ...]: The randomly zeroed tensor.\n \"\"\"\n if p < 0.0 or p > 1.0:\n raise ValueError(f\"drop probability has to be between 0 and 1, but got {p}\")\n if mode not in [\"batch\", \"row\"]:\n raise ValueError(f\"mode has to be either 'batch' or 'row', but got {mode}\")\n if not training or p == 0.0:\n return input\n\n survival_rate = 1.0 - p\n if mode == \"row\":\n size = [input.shape[0]] + [1] * (input.ndim - 1)\n else:\n size = [1] * input.ndim\n noise = torch.empty(size, dtype=input.dtype, device=input.device)\n noise = noise.bernoulli_(survival_rate)\n if survival_rate > 0.0:\n noise.div_(survival_rate)\n return input * noise\n\nclass StochasticDepth(nn.Module):\n \"\"\"\n See :func:`stochastic_depth`.\n \"\"\"\n\n def __init__(self, p: float, mode: str) -> None:\n super().__init__()\n self.p = p\n self.mode = mode\n\n def forward(self, input: Tensor) -> Tensor:\n return stochastic_depth(input, self.p, self.mode, self.training)\n\n def __repr__(self) -> str:\n tmpstr = self.__class__.__name__ + \"(\"\n tmpstr += \"p=\" + str(self.p)\n tmpstr += \", mode=\" + str(self.mode)\n tmpstr += \")\"\n return tmpstr\n\ndef _make_divisible(v: float, divisor: int, min_value: Optional[int] = None) -> int:\n \"\"\"\n This function is taken from the original tf repo.\n It ensures that all layers have a channel number that is divisible by 8\n It can be seen here:\n https://github.com/tensorflow/models/blob/master/research/slim/nets/mobilenet/mobilenet.py\n \"\"\"\n if min_value is None:\n min_value = divisor\n new_v = max(min_value, int(v + divisor / 2) // divisor * divisor)\n # Make sure that round down does not go down by more than 10%.\n if new_v < 0.9 * v:\n new_v += divisor\n return new_v\n\nclass ConvNormActivation(torch.nn.Sequential):\n \"\"\"\n Configurable block used for Convolution-Normalzation-Activation blocks.\n\n Args:\n in_channels (int): Number of channels in the input image\n out_channels (int): Number of channels produced by the Convolution-Normalzation-Activation block\n kernel_size: (int, optional): Size of the convolving kernel. Default: 3\n stride (int, optional): Stride of the convolution. Default: 1\n padding (int, tuple or str, optional): Padding added to all four sides of the input. Default: None, in wich case it will calculated as ``padding = (kernel_size - 1) // 2 * dilation``\n groups (int, optional): Number of blocked connections from input channels to output channels. Default: 1\n norm_layer (Callable[..., torch.nn.Module], optional): Norm layer that will be stacked on top of the convolutiuon layer. If ``None`` this layer wont be used. Default: ``torch.nn.BatchNorm2d``\n activation_layer (Callable[..., torch.nn.Module], optinal): Activation function which will be stacked on top of the normalization layer (if not None), otherwise on top of the conv layer. If ``None`` this layer wont be used. Default: ``torch.nn.ReLU``\n dilation (int): Spacing between kernel elements. Default: 1\n inplace (bool): Parameter for the activation layer, which can optionally do the operation in-place. Default ``True``\n bias (bool, optional): Whether to use bias in the convolution layer. By default, biases are included if ``norm_layer is None``.\n\n \"\"\"\n\n def __init__(\n self,\n in_channels: int,\n out_channels: int,\n kernel_size: int = 3,\n stride: int = 1,\n padding: Optional[int] = None,\n groups: int = 1,\n norm_layer: Optional[Callable[..., torch.nn.Module]] = torch.nn.BatchNorm2d,\n activation_layer: Optional[Callable[..., torch.nn.Module]] = torch.nn.ReLU,\n dilation: int = 1,\n inplace: Optional[bool] = True,\n bias: Optional[bool] = None,\n ) -> None:\n if padding is None:\n padding = (kernel_size - 1) // 2 * dilation\n if bias is None:\n bias = norm_layer is None\n layers = [\n torch.nn.Conv2d(\n in_channels,\n out_channels,\n kernel_size,\n stride,\n padding,\n dilation=dilation,\n groups=groups,\n bias=bias,\n )\n ]\n if norm_layer is not None:\n layers.append(norm_layer(out_channels))\n if activation_layer is not None:\n params = {} if inplace is None else {\"inplace\": inplace}\n layers.append(activation_layer(**params))\n super().__init__(*layers)\n self.out_channels = out_channels\n\nclass SqueezeExcitation(torch.nn.Module):\n \"\"\"\n This block implements the Squeeze-and-Excitation block from https://arxiv.org/abs/1709.01507 (see Fig. 1).\n Parameters ``activation``, and ``scale_activation`` correspond to ``delta`` and ``sigma`` in in eq. 3.\n\n Args:\n input_channels (int): Number of channels in the input image\n squeeze_channels (int): Number of squeeze channels\n activation (Callable[..., torch.nn.Module], optional): ``delta`` activation. Default: ``torch.nn.ReLU``\n scale_activation (Callable[..., torch.nn.Module]): ``sigma`` activation. Default: ``torch.nn.Sigmoid``\n \"\"\"\n\n def __init__(\n self,\n input_channels: int,\n squeeze_channels: int,\n activation: Callable[..., torch.nn.Module] = torch.nn.ReLU,\n scale_activation: Callable[..., torch.nn.Module] = torch.nn.Sigmoid,\n ) -> None:\n super().__init__()\n self.avgpool = torch.nn.AdaptiveAvgPool2d(1)\n self.fc1 = torch.nn.Conv2d(input_channels, squeeze_channels, 1)\n self.fc2 = torch.nn.Conv2d(squeeze_channels, input_channels, 1)\n self.activation = activation()\n self.scale_activation = scale_activation()\n\n def _scale(self, input: Tensor) -> Tensor:\n scale = self.avgpool(input)\n scale = self.fc1(scale)\n scale = self.activation(scale)\n scale = self.fc2(scale)\n return self.scale_activation(scale)\n\n def forward(self, input: Tensor) -> Tensor:\n scale = self._scale(input)\n return scale * input\n\n\nmodel_urls = {\n # Weights ported from https://github.com/rwightman/pytorch-image-models/\n \"efficientnet_b0\": \"https://download.pytorch.org/models/efficientnet_b0_rwightman-3dd342df.pth\",\n \"efficientnet_b1\": \"https://download.pytorch.org/models/efficientnet_b1_rwightman-533bc792.pth\",\n \"efficientnet_b2\": \"https://download.pytorch.org/models/efficientnet_b2_rwightman-bcdf34b7.pth\",\n \"efficientnet_b3\": \"https://download.pytorch.org/models/efficientnet_b3_rwightman-cf984f9c.pth\",\n \"efficientnet_b4\": \"https://download.pytorch.org/models/efficientnet_b4_rwightman-7eb33cd5.pth\",\n # Weights ported from https://github.com/lukemelas/EfficientNet-PyTorch/\n \"efficientnet_b5\": \"https://download.pytorch.org/models/efficientnet_b5_lukemelas-b6417697.pth\",\n \"efficientnet_b6\": \"https://download.pytorch.org/models/efficientnet_b6_lukemelas-c76e70fd.pth\",\n \"efficientnet_b7\": \"https://download.pytorch.org/models/efficientnet_b7_lukemelas-dcc49843.pth\",\n}\n\n\nclass MBConvConfig:\n # Stores information listed at Table 1 of the EfficientNet paper\n def __init__(\n self,\n expand_ratio: float,\n kernel: int,\n stride: int,\n input_channels: int,\n out_channels: int,\n num_layers: int,\n width_mult: float,\n depth_mult: float,\n ) -> None:\n self.expand_ratio = expand_ratio\n self.kernel = kernel\n self.stride = stride\n self.input_channels = self.adjust_channels(input_channels, width_mult)\n self.out_channels = self.adjust_channels(out_channels, width_mult)\n self.num_layers = self.adjust_depth(num_layers, depth_mult)\n\n def __repr__(self) -> str:\n s = self.__class__.__name__ + \"(\"\n s += \"expand_ratio={expand_ratio}\"\n s += \", kernel={kernel}\"\n s += \", stride={stride}\"\n s += \", input_channels={input_channels}\"\n s += \", out_channels={out_channels}\"\n s += \", num_layers={num_layers}\"\n s += \")\"\n return s.format(**self.__dict__)\n\n @staticmethod\n def adjust_channels(channels: int, width_mult: float, min_value: Optional[int] = None) -> int:\n return _make_divisible(channels * width_mult, 8, min_value)\n\n @staticmethod\n def adjust_depth(num_layers: int, depth_mult: float):\n return int(math.ceil(num_layers * depth_mult))\n\n\nclass MBConv(nn.Module):\n def __init__(\n self,\n cnf: MBConvConfig,\n stochastic_depth_prob: float,\n norm_layer: Callable[..., nn.Module],\n se_layer: Callable[..., nn.Module] = SqueezeExcitation,\n ) -> None:\n super().__init__()\n\n if not (1 <= cnf.stride <= 2):\n raise ValueError(\"illegal stride value\")\n\n self.use_res_connect = cnf.stride == 1 and cnf.input_channels == cnf.out_channels\n\n layers: List[nn.Module] = []\n activation_layer = nn.SiLU\n\n # expand\n expanded_channels = cnf.adjust_channels(cnf.input_channels, cnf.expand_ratio)\n if expanded_channels != cnf.input_channels:\n layers.append(\n ConvNormActivation(\n cnf.input_channels,\n expanded_channels,\n kernel_size=1,\n norm_layer=norm_layer,\n activation_layer=activation_layer,\n )\n )\n\n # depthwise\n layers.append(\n ConvNormActivation(\n expanded_channels,\n expanded_channels,\n kernel_size=cnf.kernel,\n stride=cnf.stride,\n groups=expanded_channels,\n norm_layer=norm_layer,\n activation_layer=activation_layer,\n )\n )\n\n # squeeze and excitation\n squeeze_channels = max(1, cnf.input_channels // 4)\n layers.append(se_layer(expanded_channels, squeeze_channels, activation=partial(nn.SiLU, inplace=True)))\n\n # project\n layers.append(\n ConvNormActivation(\n expanded_channels, cnf.out_channels, kernel_size=1, norm_layer=norm_layer, activation_layer=None\n )\n )\n\n self.block = nn.Sequential(*layers)\n self.stochastic_depth = StochasticDepth(stochastic_depth_prob, \"row\")\n self.out_channels = cnf.out_channels\n\n def forward(self, input: Tensor) -> Tensor:\n result = self.block(input)\n if self.use_res_connect:\n result = self.stochastic_depth(result)\n result += input\n return result\n\n\nclass EfficientNet(nn.Module):\n def __init__(\n self,\n inverted_residual_setting: List[MBConvConfig],\n dropout: float,\n stochastic_depth_prob: float = 0.2,\n num_classes: int = 1000,\n block: Optional[Callable[..., nn.Module]] = None,\n norm_layer: Optional[Callable[..., nn.Module]] = None,\n **kwargs: Any,\n ) -> None:\n \"\"\"\n EfficientNet main class\n\n Args:\n inverted_residual_setting (List[MBConvConfig]): Network structure\n dropout (float): The droupout probability\n stochastic_depth_prob (float): The stochastic depth probability\n num_classes (int): Number of classes\n block (Optional[Callable[..., nn.Module]]): Module specifying inverted residual building block for mobilenet\n norm_layer (Optional[Callable[..., nn.Module]]): Module specifying the normalization layer to use\n \"\"\"\n super().__init__()\n\n if not inverted_residual_setting:\n raise ValueError(\"The inverted_residual_setting should not be empty\")\n elif not (\n isinstance(inverted_residual_setting, Sequence)\n and all([isinstance(s, MBConvConfig) for s in inverted_residual_setting])\n ):\n raise TypeError(\"The inverted_residual_setting should be List[MBConvConfig]\")\n\n if block is None:\n block = MBConv\n\n if norm_layer is None:\n norm_layer = nn.BatchNorm2d\n\n layers: List[nn.Module] = []\n\n # building first layer\n firstconv_output_channels = inverted_residual_setting[0].input_channels\n layers.append(\n ConvNormActivation(\n 3, firstconv_output_channels, kernel_size=3, stride=2, norm_layer=norm_layer, activation_layer=nn.SiLU\n )\n )\n\n # building inverted residual blocks\n total_stage_blocks = sum(cnf.num_layers for cnf in inverted_residual_setting)\n stage_block_id = 0\n for cnf in inverted_residual_setting:\n stage: List[nn.Module] = []\n for _ in range(cnf.num_layers):\n # copy to avoid modifications. shallow copy is enough\n block_cnf = copy.copy(cnf)\n\n # overwrite info if not the first conv in the stage\n if stage:\n block_cnf.input_channels = block_cnf.out_channels\n block_cnf.stride = 1\n\n # adjust stochastic depth probability based on the depth of the stage block\n sd_prob = stochastic_depth_prob * float(stage_block_id) / total_stage_blocks\n\n stage.append(block(block_cnf, sd_prob, norm_layer))\n stage_block_id += 1\n\n layers.append(nn.Sequential(*stage))\n\n # building last several layers\n lastconv_input_channels = inverted_residual_setting[-1].out_channels\n lastconv_output_channels = 4 * lastconv_input_channels\n layers.append(\n ConvNormActivation(\n lastconv_input_channels,\n lastconv_output_channels,\n kernel_size=1,\n norm_layer=norm_layer,\n activation_layer=nn.SiLU,\n )\n )\n\n self.features = nn.Sequential(*layers)\n self.avgpool = nn.AdaptiveAvgPool2d(1)\n self.classifier = nn.Sequential(\n nn.Dropout(p=dropout, inplace=True),\n nn.Linear(lastconv_output_channels, num_classes),\n )\n\n for m in self.modules():\n if isinstance(m, nn.Conv2d):\n nn.init.kaiming_normal_(m.weight, mode=\"fan_out\")\n if m.bias is not None:\n nn.init.zeros_(m.bias)\n elif isinstance(m, (nn.BatchNorm2d, nn.GroupNorm)):\n nn.init.ones_(m.weight)\n nn.init.zeros_(m.bias)\n elif isinstance(m, nn.Linear):\n init_range = 1.0 / math.sqrt(m.out_features)\n nn.init.uniform_(m.weight, -init_range, init_range)\n nn.init.zeros_(m.bias)\n\n def _forward_impl(self, x: Tensor) -> Tensor:\n channels = self.features(x)\n\n x = self.avgpool(channels)\n features = torch.flatten(x, 1)\n\n x = self.classifier(features)\n\n return x, features\n\n def forward(self, x: Tensor) -> Tensor:\n return self._forward_impl(x)\n \n \ndef _efficientnet(\n arch: str,\n width_mult: float,\n depth_mult: float,\n dropout: float,\n pretrained: bool,\n progress: bool,\n **kwargs: Any,\n) -> EfficientNet:\n bneck_conf = partial(MBConvConfig, width_mult=width_mult, depth_mult=depth_mult)\n inverted_residual_setting = [\n bneck_conf(1, 3, 1, 32, 16, 1),\n bneck_conf(6, 3, 2, 16, 24, 2),\n bneck_conf(6, 5, 2, 24, 40, 2),\n bneck_conf(6, 3, 2, 40, 80, 3),\n bneck_conf(6, 5, 1, 80, 112, 3),\n bneck_conf(6, 5, 2, 112, 192, 4),\n bneck_conf(6, 3, 1, 192, 320, 1),\n ]\n model = EfficientNet(inverted_residual_setting, dropout, **kwargs)\n if pretrained:\n if model_urls.get(arch, None) is None:\n raise ValueError(f\"No checkpoint is available for model type {arch}\")\n state_dict = load_state_dict_from_url(model_urls[arch], progress=progress)\n model.load_state_dict(state_dict)\n return model\n\ndef efficientnet_b0(pretrained: bool = False, progress: bool = True, **kwargs: Any) -> EfficientNet:\n \"\"\"\n Constructs a EfficientNet B0 architecture from\n `\"EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks\" `_.\n\n Args:\n pretrained (bool): If True, returns a model pre-trained on ImageNet\n progress (bool): If True, displays a progress bar of the download to stderr\n \"\"\"\n return _efficientnet(\"efficientnet_b0\", 1.0, 1.0, 0.2, pretrained, progress, **kwargs)\n\n\ndef efficientnet_b1(pretrained: bool = False, progress: bool = True, **kwargs: Any) -> EfficientNet:\n \"\"\"\n Constructs a EfficientNet B1 architecture from\n `\"EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks\" `_.\n\n Args:\n pretrained (bool): If True, returns a model pre-trained on ImageNet\n progress (bool): If True, displays a progress bar of the download to stderr\n \"\"\"\n return _efficientnet(\"efficientnet_b1\", 1.0, 1.1, 0.2, pretrained, progress, **kwargs)\n\n\ndef efficientnet_b2(pretrained: bool = False, progress: bool = True, **kwargs: Any) -> EfficientNet:\n \"\"\"\n Constructs a EfficientNet B2 architecture from\n `\"EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks\" `_.\n\n Args:\n pretrained (bool): If True, returns a model pre-trained on ImageNet\n progress (bool): If True, displays a progress bar of the download to stderr\n \"\"\"\n return _efficientnet(\"efficientnet_b2\", 1.1, 1.2, 0.3, pretrained, progress, **kwargs)\n\n\ndef efficientnet_b3(pretrained: bool = False, progress: bool = True, **kwargs: Any) -> EfficientNet:\n \"\"\"\n Constructs a EfficientNet B3 architecture from\n `\"EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks\" `_.\n\n Args:\n pretrained (bool): If True, returns a model pre-trained on ImageNet\n progress (bool): If True, displays a progress bar of the download to stderr\n \"\"\"\n return _efficientnet(\"efficientnet_b3\", 1.2, 1.4, 0.3, pretrained, progress, **kwargs)\n\n\ndef efficientnet_b4(pretrained: bool = False, progress: bool = True, **kwargs: Any) -> EfficientNet:\n \"\"\"\n Constructs a EfficientNet B4 architecture from\n `\"EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks\" `_.\n\n Args:\n pretrained (bool): If True, returns a model pre-trained on ImageNet\n progress (bool): If True, displays a progress bar of the download to stderr\n \"\"\"\n return _efficientnet(\"efficientnet_b4\", 1.4, 1.8, 0.4, pretrained, progress, **kwargs)\n\n\ndef efficientnet_b5(pretrained: bool = False, progress: bool = True, **kwargs: Any) -> EfficientNet:\n \"\"\"\n Constructs a EfficientNet B5 architecture from\n `\"EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks\" `_.\n\n Args:\n pretrained (bool): If True, returns a model pre-trained on ImageNet\n progress (bool): If True, displays a progress bar of the download to stderr\n \"\"\"\n return _efficientnet(\n \"efficientnet_b5\",\n 1.6,\n 2.2,\n 0.4,\n pretrained,\n progress,\n norm_layer=partial(nn.BatchNorm2d, eps=0.001, momentum=0.01),\n **kwargs,\n )\n\n\n\ndef efficientnet_b6(pretrained: bool = False, progress: bool = True, **kwargs: Any) -> EfficientNet:\n \"\"\"\n Constructs a EfficientNet B6 architecture from\n `\"EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks\" `_.\n\n Args:\n pretrained (bool): If True, returns a model pre-trained on ImageNet\n progress (bool): If True, displays a progress bar of the download to stderr\n \"\"\"\n return _efficientnet(\n \"efficientnet_b6\",\n 1.8,\n 2.6,\n 0.5,\n pretrained,\n progress,\n norm_layer=partial(nn.BatchNorm2d, eps=0.001, momentum=0.01),\n **kwargs,\n )\n\n\n\ndef efficientnet_b7(pretrained: bool = False, progress: bool = True, **kwargs: Any) -> EfficientNet:\n \"\"\"\n Constructs a EfficientNet B7 architecture from\n `\"EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks\" `_.\n\n Args:\n pretrained (bool): If True, returns a model pre-trained on ImageNet\n progress (bool): If True, displays a progress bar of the download to stderr\n \"\"\"\n return _efficientnet(\n \"efficientnet_b7\",\n 2.0,\n 3.1,\n 0.5,\n pretrained,\n progress,\n norm_layer=partial(nn.BatchNorm2d, eps=0.001, momentum=0.01),\n **kwargs,\n )","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-26T17:41:02.854781Z","iopub.execute_input":"2022-02-26T17:41:02.855548Z","iopub.status.idle":"2022-02-26T17:41:02.94735Z","shell.execute_reply.started":"2022-02-26T17:41:02.855493Z","shell.execute_reply":"2022-02-26T17:41:02.946362Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class WDModel(LightningModule):\n def __init__(self, weights=torch.ones(CFG.N_CLASSES)):\n super().__init__()\n #self.save_hyperparameters()\n self.model = efficientnet_b0(True)\n self.classification_head = nn.Linear(1280, CFG.N_CLASSES)\n #self.criterion = LabelSmoothingCrossEntropy(0.0)\n self.criterion = nn.CrossEntropyLoss(weight=torch.tensor(weights))\n \n acc = torchmetrics.Accuracy()\n # use .clone so that each metric can maintain its own state\n self.train_acc = acc.clone()\n # assign all metrics as attributes of module so they are detected as children\n self.val_acc = acc.clone()\n \n self.validation_outputs = []\n \n def forward(self, x):\n _, x = self.model(x)\n x = self.classification_head(x)\n return x","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-26T17:41:02.948789Z","iopub.execute_input":"2022-02-26T17:41:02.94913Z","iopub.status.idle":"2022-02-26T17:41:02.975491Z","shell.execute_reply.started":"2022-02-26T17:41:02.949053Z","shell.execute_reply":"2022-02-26T17:41:02.974525Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_data = pd.read_csv(\"../input/w-d-classification/preds.csv\")\ntrain_data.index = train_data[\"image\"].str[:-3] + \"jpg\"\nsample_submission = pd.read_csv(\"../input/w-d-224x224-fast-dataset/sample_submission.csv\")\nsample_submission.index = sample_submission[\"image\"].str[:-3] + \"jpg\"","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-26T17:41:02.97735Z","iopub.execute_input":"2022-02-26T17:41:02.977706Z","iopub.status.idle":"2022-02-26T17:41:03.720918Z","shell.execute_reply.started":"2022-02-26T17:41:02.97766Z","shell.execute_reply":"2022-02-26T17:41:03.719945Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Inference š®\nFor each model, I want to make 4 loaders for:\n1. The raw training images\n2. The cropped training images\n3. The raw test images\n4. The cropped test images\n\n
\nThe raw training images have already a prediction coming from the training notebook, so all I have left to do for each model is:
\n\n1. Make predictions on the cropped training images that were not used for training\n2. Make predictions on the raw test images\n3. Make predictions on the cropped test images","metadata":{}},{"cell_type":"code","source":"test_dataset = WandDLoader(sample_submission, \"test\")\ntest_crop_dataset = WandDLoader(sample_submission, \"test\", True)\n\ntest_loader = torch.utils.data.DataLoader(\n test_dataset,\n batch_size=CFG.batch_size,\n num_workers=CFG.num_workers,\n pin_memory=True,\n shuffle=False\n)\ntest_crop_loader = torch.utils.data.DataLoader(\n test_crop_dataset,\n batch_size=CFG.batch_size,\n num_workers=CFG.num_workers,\n pin_memory=True,\n shuffle=False\n)\n\ntest_preds = []\ntest_crop_preds = []\ncropped_species_pred = np.zeros((len(train_data), CFG.N_CLASSES), dtype=float)\nfor i in range(CFG.FOLDS):\n base_dir = f\"../input/w-d-classification/model-wandd-species-fold{i}-val/\"\n file = sorted(os.listdir(base_dir))[0]\n path = os.path.join(base_dir, file)\n\n model = WDModel.load_from_checkpoint(path)\n trainer = Trainer(\n gpus=1\n )\n \n train_crop_dataset = WandDLoader(train_data[train_data[\"fold\"] == i], \"train\", True)\n train_crop_loader = torch.utils.data.DataLoader(\n train_crop_dataset,\n batch_size=CFG.batch_size,\n num_workers=CFG.num_workers,\n pin_memory=True,\n shuffle=False\n )\n \n preds = trainer.predict(model, dataloaders=train_crop_loader)\n preds = torch.cat(preds, dim=0)\n cropped_species_pred[train_data[\"fold\"] == i] = preds.numpy().tolist()\n \n preds = trainer.predict(model, dataloaders=test_loader)\n preds = torch.cat(preds, dim=0)\n test_preds.append(preds.numpy())\n \n preds = trainer.predict(model, dataloaders=test_crop_loader)\n preds = torch.cat(preds, dim=0)\n test_crop_preds.append(preds.numpy())\n \nsample_submission[\"species_pred\"] = np.mean(test_preds, axis=0).tolist()\nsample_submission[\"cropped_species_pred\"] = np.mean(test_crop_preds, axis=0).tolist()\ntrain_data[\"cropped_species_pred\"] = cropped_species_pred.tolist()","metadata":{"execution":{"iopub.status.busy":"2022-02-26T17:41:03.722414Z","iopub.execute_input":"2022-02-26T17:41:03.723575Z","iopub.status.idle":"2022-02-26T17:43:43.996439Z","shell.execute_reply.started":"2022-02-26T17:41:03.723538Z","shell.execute_reply":"2022-02-26T17:43:43.990638Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Apply softmax to logits\ntrain_data[\"species_pred\"] = train_data[\"species_pred\"].map(eval).map(np.array).map(lambda x: x - x.max()).map(np.exp).map(lambda x: x/x.sum())\ntrain_data[\"cropped_species_pred\"] = train_data[\"cropped_species_pred\"].map(np.array).map(lambda x: x - x.max()).map(np.exp).map(lambda x: x/x.sum())\nsample_submission[\"species_pred\"] = sample_submission[\"species_pred\"].map(np.array).map(lambda x: x - x.max()).map(np.exp).map(lambda x: x/x.sum())\nsample_submission[\"cropped_species_pred\"] = sample_submission[\"cropped_species_pred\"].map(np.array).map(lambda x: x - x.max()).map(np.exp).map(lambda x: x/x.sum())","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-26T17:43:44.001237Z","iopub.status.idle":"2022-02-26T17:43:44.005005Z","shell.execute_reply.started":"2022-02-26T17:43:44.004475Z","shell.execute_reply":"2022-02-26T17:43:44.00456Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def plot_flagged_samples(flagged_samples):\n print(\"Number of flagged samples:\", len(flagged_samples))\n print(\"Original image\")\n plot_images(flagged_samples, row=3, col=4, base_path=\"../input/w-d-224x224-fast-dataset/train_images\")\n print(\"Cropped\")\n plot_images(flagged_samples, row=3, col=4, base_path=\"../input/w-d-fast-224x224-cropped-dataset/train_images\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-26T17:43:44.010456Z","iopub.status.idle":"2022-02-26T17:43:44.012468Z","shell.execute_reply.started":"2022-02-26T17:43:44.012143Z","shell.execute_reply":"2022-02-26T17:43:44.012177Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Compute the metrics š\nFor each metric, we will compute the raw metric of the image for both original and cropped images, and the difference between the two.
\n\n## 1. Max confidence\nThe first metric we will have a look at will be the value of the maximum confidence. It's expected to be lower for OOD samples.
\n\n## 2. Entropy\nThe second metric we will use is the measure of the shannon entropy of the prediction distribution. To give some intuition, a large value of entropy means that the prediction is not confident towards any class while a low value of entropy means a big confidence towards one class.","metadata":{}},{"cell_type":"code","source":"\"\"\"\n Compute the metrics\n\"\"\"\n\ndef compute_max_confidence_metrics(data):\n data[\"pred_max_conf\"] = data[\"species_pred\"].map(np.max)\n data[\"cropped_pred_max_conf\"] = data[\"cropped_species_pred\"].map(np.max)\n data[\"pred_max_conf_delta\"] = data[\"pred_max_conf\"] - data[\"cropped_pred_max_conf\"]\n\ndef compute_entropy_metrics(data):\n data[\"pred_entropy\"] = data[\"species_pred\"].map(lambda x: -np.sum(x*np.log2(x)))\n data[\"cropped_pred_entropy\"] = data[\"cropped_species_pred\"].map(lambda x: -np.sum(x*np.log2(x)))\n data[\"pred_entropy_delta\"] = data[\"pred_entropy\"] - data[\"cropped_pred_entropy\"]\n \ncompute_max_confidence_metrics(train_data)\ncompute_max_confidence_metrics(sample_submission)\ncompute_entropy_metrics(train_data)\ncompute_entropy_metrics(sample_submission)","metadata":{"execution":{"iopub.status.busy":"2022-02-26T17:43:44.016591Z","iopub.status.idle":"2022-02-26T17:43:44.017787Z","shell.execute_reply.started":"2022-02-26T17:43:44.017132Z","shell.execute_reply":"2022-02-26T17:43:44.017164Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Save the results\ntrain_data.to_csv(\"train.csv\", index=False)\nsample_submission.to_csv(\"test.csv\", index=False)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-26T17:43:44.021514Z","iopub.status.idle":"2022-02-26T17:43:44.02341Z","shell.execute_reply.started":"2022-02-26T17:43:44.023072Z","shell.execute_reply":"2022-02-26T17:43:44.023119Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Examples of flagged samples š","metadata":{}},{"cell_type":"code","source":"n_flags = np.zeros(len(train_data))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-26T17:43:44.026749Z","iopub.status.idle":"2022-02-26T17:43:44.028644Z","shell.execute_reply.started":"2022-02-26T17:43:44.027463Z","shell.execute_reply":"2022-02-26T17:43:44.028258Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Flag based on the predicted class confidence","metadata":{}},{"cell_type":"code","source":"cropped_pred_max_conf = train_data[\"cropped_pred_max_conf\"].values\nplt.title(\"Distribution of the metric in the dataset\")\nplt.hist(cropped_pred_max_conf)\nplt.show()\nm = train_data[\"cropped_pred_max_conf\"] <= np.quantile(cropped_pred_max_conf, CFG.FLAG_QUANTILE)\nflagged_samples = train_data[m]\nn_flags[m] += 1\nplot_flagged_samples(flagged_samples)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-26T17:43:44.031434Z","iopub.status.idle":"2022-02-26T17:43:44.034089Z","shell.execute_reply.started":"2022-02-26T17:43:44.033046Z","shell.execute_reply":"2022-02-26T17:43:44.033079Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Flag based on the delta between the predicted class confidence","metadata":{}},{"cell_type":"code","source":"pred_max_conf_delta = train_data[\"pred_max_conf_delta\"].values\nplt.title(\"Distribution of the metric in the dataset\")\nplt.hist(pred_max_conf_delta)\nplt.show()\nm = train_data[\"pred_max_conf_delta\"] > np.quantile(pred_max_conf_delta, 1 - CFG.FLAG_QUANTILE)\nflagged_samples = train_data[m]\nn_flags[m] += 1\nplot_flagged_samples(flagged_samples)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-26T17:43:44.037551Z","iopub.status.idle":"2022-02-26T17:43:44.03958Z","shell.execute_reply.started":"2022-02-26T17:43:44.038424Z","shell.execute_reply":"2022-02-26T17:43:44.039261Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Flag based on the entropy of the prediction distribution","metadata":{"_kg_hide-input":true}},{"cell_type":"code","source":"cropped_pred_entropy = train_data[\"cropped_pred_entropy\"].values\nplt.title(\"Distribution of the metric in the dataset\")\nplt.hist(cropped_pred_entropy)\nplt.show()\nm = train_data[\"cropped_pred_entropy\"] > np.quantile(cropped_pred_entropy, 1 - CFG.FLAG_QUANTILE)\nflagged_samples = train_data[m]\nn_flags[m] += 1\nplot_flagged_samples(flagged_samples)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-26T17:43:44.042404Z","iopub.status.idle":"2022-02-26T17:43:44.044883Z","shell.execute_reply.started":"2022-02-26T17:43:44.043554Z","shell.execute_reply":"2022-02-26T17:43:44.043633Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Flag based on the delta between the entropy of the prediction distribution","metadata":{}},{"cell_type":"code","source":"pred_entropy_delta = train_data[\"pred_entropy_delta\"].values\nplt.title(\"Distribution of the metric in the dataset\")\nplt.hist(pred_entropy_delta)\nplt.show()\nm = pred_entropy_delta <= np.quantile(pred_entropy_delta, CFG.FLAG_QUANTILE)\nflagged_samples = train_data[m]\nn_flags[m] += 1\nplot_flagged_samples(flagged_samples)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-26T17:43:44.047317Z","iopub.status.idle":"2022-02-26T17:43:44.054438Z","shell.execute_reply.started":"2022-02-26T17:43:44.054128Z","shell.execute_reply":"2022-02-26T17:43:44.054162Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Ensembling the flags š©\nNow that we computed the different metrics, we can try to filter some false positive by removing only the samples that got flagged more than N times ","metadata":{}},{"cell_type":"code","source":"train_data[\"n_flags\"] = n_flags\nflagged_samples = train_data[train_data[\"n_flags\"] >= CFG.N_FLAGS]\nplot_flagged_samples(flagged_samples)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-26T17:43:44.05594Z","iopub.status.idle":"2022-02-26T17:43:44.056717Z","shell.execute_reply.started":"2022-02-26T17:43:44.05644Z","shell.execute_reply":"2022-02-26T17:43:44.056471Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Conclusion š¤·\nš¤ We can see that we still have some false positives with the proposed method of filtering, but hyperparameters can be tweaked depending on the usage. The result of this notebook could be used with very conservative parameters to create a dataset of valid bounding boxes on which we could train an other YOLOv5 model.
\nš Results could be enhanced by using other models or other OOD distribution metrics.
\n\nš If you found this notebook helpful please consider giving an upvote, and if you disagree with the content, I'll be pleased to dicsuss it with you in the coments.
\n\nš Happy Kaggling everyone !","metadata":{}}]}