{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"
\n

PyTorch Tutorial Series
Inference Series Part I

\n
","metadata":{}},{"cell_type":"markdown","source":"**VERSION 52: SETI - To check code on SETI, do go to that version or before.**\n\n**VERSION 53 and beyond-: Petfinder**","metadata":{}},{"cell_type":"markdown","source":"# NOTE: THERE ARE SOME CODE CHANGES IN INFERENCE ALL FOLDS, REFER TO MY SOURCE CODE AND CHANGE THE NARRATIVE ACCORDINGLY.\n# NOTE: THERE ARE SOME CODE CHANGES IN INFERENCE ALL FOLDS, REFER TO MY SOURCE CODE AND CHANGE THE NARRATIVE ACCORDINGLY.\n# NOTE: THERE ARE SOME CODE CHANGES IN INFERENCE ALL FOLDS, REFER TO MY SOURCE CODE AND CHANGE THE NARRATIVE ACCORDINGLY.","metadata":{}},{"cell_type":"markdown","source":"This notebook is a **GENERIC NOTEBOOK** talking about PyTorch's saving and loading, plus some details of inference I learnt during my learning journey. During each new competition I embark on, I will always improve it.\n\n---\n\nThis notebook is part I of the PyTorch Tutorial Inference Series. I will detail on how to save and load weights in PyTorch. I will split this notebook into two parts:\n\n1. First part: The real inference in action - for now, I will just add SETI as the main competition. However, I intend to share this notebook across multiple different competitions.\n\n2. The tutorial on PyTorch.\n\n---\n\nI have added a back to top button for each section for easy navigation.","metadata":{}},{"cell_type":"markdown","source":"References:\n\n1. [Using args and kwargs](https://note.nkmk.me/en/python-args-kwargs-usage/#:~:text=In%20Python%2C%20by%20adding%20*%20and,arguments)","metadata":{}},{"cell_type":"markdown","source":"\n\n
\n

Quick Navigation

\n\n \n* [Dependencies](#1)\n * [Import modules offline](#Import_modules_offline)\n * [Import other modules](#Import_other_modules)\n* [Configurations](#2)\n* [Utility](#30)\n * [Seeding](#3)\n * [Numpy to Latex](#31)\n* [Prepare Data](#Prepare_Data)\n* [Dataset](#Dataset)\n* [Augmentations](#Augmentations)\n* [Model Instantiation](#Model) \n* [Inference by Folds](#8)\n* [Post-Processing Techniques](#PP)\n * [Forward Ensembling]\n * [Training Image Embeddings]\n* [Saving and Loading Model Weights](#20) \n* [Dissecting Inference by Folds](#60)\n * [Problem Settings](#61)","metadata":{}},{"cell_type":"markdown","source":"\n\n

Dependencies

","metadata":{}},{"cell_type":"markdown","source":"### Import modules offline \n\nMost Kaggle competitions need you to inference in **offline** mode. \n\n- Firstly, download the `github` repo as a zip file.\n- Then, upload the zip file/folder as a Kaggle dataset and add the dataset to your Kaggle notebook.\n- Lastly, use `sys.path.append()` to put the path correctly in your environment. See example below.","metadata":{}},{"cell_type":"code","source":"import sys\n\ntimm_path = \"../input/timm-pytorch-image-models/pytorch-image-models-master\"\npytorch_gradcam_path = \"../input/reighns-gradcam/pytorch-grad-cam-master\"\npytorch_ttach_path = \"../input/reighns-ttach/ttach-master\"\n\nsys_paths = [timm_path, pytorch_gradcam_path, pytorch_ttach_path]\n\nfor paths in sys_paths:\n sys.path.append(paths)\n\nimport timm\n\nfrom pytorch_grad_cam import GradCAM\nfrom pytorch_grad_cam.utils.image import show_cam_on_image","metadata":{"_kg_hide-input":false,"_kg_hide-output":false,"execution":{"iopub.status.busy":"2021-12-14T12:29:29.265952Z","iopub.execute_input":"2021-12-14T12:29:29.266452Z","iopub.status.idle":"2021-12-14T12:29:32.78714Z","shell.execute_reply.started":"2021-12-14T12:29:29.266363Z","shell.execute_reply":"2021-12-14T12:29:32.786228Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Import other modules \n\nWe import the other modules normally since they are in Kaggle's environment already.","metadata":{}},{"cell_type":"code","source":"import collections\nimport gc\nimport math\nimport os\nimport random\nimport warnings\nfrom collections import defaultdict\nfrom dataclasses import asdict, dataclass, field\nfrom typing import *\n\nimport albumentations\nimport cv2\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\nimport timm\nimport torch\nimport torch.nn.functional as F\nfrom albumentations.pytorch.transforms import ToTensorV2\nfrom matplotlib import pyplot as plt\nfrom sklearn.model_selection import GroupKFold, KFold, StratifiedKFold\nfrom sklearn.preprocessing import LabelEncoder\nfrom torch import nn\nfrom torch.autograd import Variable\nfrom torch.optim.lr_scheduler import _LRScheduler\nfrom torch.optim.optimizer import Optimizer\nfrom torch.utils.data import DataLoader, Dataset\nfrom torchvision import models\nfrom tqdm.notebook import tqdm\n\nwarnings.filterwarnings(\"ignore\")\nimport collections\nimport gc\nimport logging\nimport os\nimport pathlib\nimport random\n# import torchsummary\nimport warnings\nfrom collections import defaultdict\nfrom dataclasses import asdict, dataclass, field\nfrom logging import INFO, FileHandler, Formatter, StreamHandler, getLogger\nfrom pathlib import Path\nfrom typing import *\nfrom typing import Any, Dict, List\n\nimport numpy as np\nimport pandas as pd\nimport timm\nimport torch\nfrom IPython.display import clear_output\nfrom torch.optim.lr_scheduler import (CosineAnnealingLR,\n CosineAnnealingWarmRestarts,\n ReduceLROnPlateau)\nfrom torch.utils.data import DataLoader, Dataset\n\n","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-12-14T12:29:32.78971Z","iopub.execute_input":"2021-12-14T12:29:32.790132Z","iopub.status.idle":"2021-12-14T12:29:34.651272Z","shell.execute_reply.started":"2021-12-14T12:29:32.79009Z","shell.execute_reply":"2021-12-14T12:29:34.650515Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Back to top","metadata":{}},{"cell_type":"markdown","source":"\n\n

Configurations

","metadata":{}},{"cell_type":"markdown","source":"### Config\n\nAt least for me, config is the most important part for my pipelines. It is also a good practice to have a configuration file to keep track of experiments.\n\nYou will almost always see Kagglers have a `config` class or a dict so that they can call the parameters. To go one step deeper, you can dump the dict into your favourite MLOps platform (read: wandb!).","metadata":{}},{"cell_type":"code","source":"from pathlib import Path\nimport pathlib\n\nclass config:\n warnings.filterwarnings(\"ignore\", category=UserWarning)\n \n DATA_DIR = Path(\"../input/petfinder-pawpularity-score\")\n OUTPUT_DIR = Path(\"./\")\n LOGS_DIR = Path(OUTPUT_DIR, \"logs\")\n MODEL_DIR = Path(OUTPUT_DIR, \"model\")\n OOF_DIR = Path(OUTPUT_DIR, \"oof\")\n LOGS_DIR.mkdir(parents=True, exist_ok=True)\n MODEL_DIR.mkdir(parents=True, exist_ok=True)\n OOF_DIR.mkdir(parents=True, exist_ok=True)","metadata":{"_kg_hide-input":false,"execution":{"iopub.status.busy":"2021-12-14T12:29:34.652291Z","iopub.execute_input":"2021-12-14T12:29:34.652549Z","iopub.status.idle":"2021-12-14T12:29:34.658805Z","shell.execute_reply.started":"2021-12-14T12:29:34.652516Z","shell.execute_reply":"2021-12-14T12:29:34.658133Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def init_logger(log_file: str = Path(config.LOGS_DIR, \"info.log\")) -> logging.Logger:\n \"\"\"Initialize logger and save to file.\n\n Consider having more log_file paths to save, eg: debug.log, error.log, etc.\n\n Args:\n log_file (str, optional): [description]. Defaults to Path(LOGS_DIR, \"info.log\").\n\n Returns:\n logging.Logger: [description]\n \"\"\"\n logger = getLogger(__name__)\n logger.setLevel(INFO)\n stream_handler = StreamHandler()\n stream_handler.setFormatter(\n Formatter(\"%(asctime)s: %(message)s\", \"%Y-%m-%d %H:%M:%S\")\n )\n file_handler = FileHandler(filename=log_file)\n file_handler.setFormatter(\n Formatter(\"%(asctime)s: %(message)s\", \"%Y-%m-%d %H:%M:%S\")\n )\n logger.addHandler(stream_handler)\n logger.addHandler(file_handler)\n\n return logger","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-12-14T12:29:34.660615Z","iopub.execute_input":"2021-12-14T12:29:34.6612Z","iopub.status.idle":"2021-12-14T12:29:34.674784Z","shell.execute_reply.started":"2021-12-14T12:29:34.661164Z","shell.execute_reply":"2021-12-14T12:29:34.673958Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"config.logger = init_logger()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-12-14T12:29:34.677337Z","iopub.execute_input":"2021-12-14T12:29:34.677782Z","iopub.status.idle":"2021-12-14T12:29:34.68399Z","shell.execute_reply.started":"2021-12-14T12:29:34.677689Z","shell.execute_reply":"2021-12-14T12:29:34.683243Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from dataclasses import dataclass, field, asdict\nimport pathlib\nfrom typing import Any, Dict, List\n\n\n@dataclass\nclass FilePaths:\n \"\"\"Class to keep track of the files.\"\"\"\n\n train_images: pathlib.Path = pathlib.Path(config.DATA_DIR, \"train\")\n test_images: pathlib.Path = pathlib.Path(config.DATA_DIR, \"test\")\n train_csv: pathlib.Path = pathlib.Path(config.DATA_DIR, \"train.csv\")\n test_csv: pathlib.Path = pathlib.Path(config.DATA_DIR, \"test.csv\")\n sub_csv: pathlib.Path = pathlib.Path(\n config.DATA_DIR, \"sample_submission.csv\"\n )\n folds_csv: pathlib.Path = pathlib.Path(\n config.OUTPUT_DIR, \"df_folds.csv\"\n )\n weight_path: pathlib.Path = pathlib.Path(config.MODEL_DIR)\n oof_csv: pathlib.Path = pathlib.Path(config.OOF_DIR)\n\n def to_dict(self) -> Dict[str, Any]:\n \"\"\"Convert to dictionary.\"\"\"\n return asdict(self)\n\n\n@dataclass\nclass DataLoaderParams:\n \"\"\"Class to keep track of the data loader parameters.\"\"\"\n\n train_loader: Dict[str, Any] = field(\n default_factory=lambda: {\n \"batch_size\": 32,\n \"num_workers\": 0,\n \"pin_memory\": False,\n \"drop_last\": True,\n \"shuffle\": True,\n \"collate_fn\": None,\n }\n )\n valid_loader: Dict[str, Any] = field(\n default_factory=lambda: {\n \"batch_size\": 32,\n \"num_workers\": 0,\n \"pin_memory\": False,\n \"drop_last\": False,\n \"shuffle\": False,\n \"collate_fn\": None,\n }\n )\n\n test_loader: Dict[str, Any] = field(\n default_factory=lambda: {\n \"batch_size\": 8,\n \"num_workers\": 0,\n \"pin_memory\": True,\n \"drop_last\": False,\n \"shuffle\": False,\n \"collate_fn\": None,\n }\n )\n\n def to_dict(self) -> Dict[str, Any]:\n \"\"\"Convert to dictionary.\"\"\"\n return asdict(self)\n \n def get_len_train_loader(self):\n total_rows = pd.read_csv(FilePaths().train_csv).shape[0] # get total number of rows/images\n total_rows_per_fold = total_rows / (MakeFolds().num_folds)\n total_rows_per_training = total_rows_per_fold * (MakeFolds().num_folds - 1) # if got 1000 images, 10 folds, then train on 9 folds = 1000/10 * (10-1) = 100 * 9 = 900\n len_of_train_loader = total_rows_per_training // self.train_loader[\"batch_size\"] # if 900 rows, bs is 16, then 900/16 = 56.25, but we drop last if dataloader, so become 56 steps. if not 57 steps.\n return int(len_of_train_loader)\n\n@dataclass\nclass MakeFolds:\n \"\"\"A class to keep track of cross-validation schema.\n seed (int): random seed for reproducibility.\n num_folds (int): number of folds.\n cv_schema (str): cross-validation schema.\n class_col_name (str): name of the target column.\n image_col_name (str): name of the image column.\n folds_csv (str): path to the folds csv.\n \"\"\"\n\n seed: int = 1992\n num_folds: int = 5\n cv_schema: str = \"StratifiedKFold\"\n class_col_name: str = \"Pawpularity\"\n image_col_name: str = \"Id\"\n folds_csv: pathlib.Path = FilePaths().folds_csv\n\n def to_dict(self) -> Dict[str, Any]:\n \"\"\"Convert to dictionary.\"\"\"\n return asdict(self)\n\n\n@dataclass\nclass AugmentationParams:\n \"\"\"Class to keep track of the augmentation parameters.\"\"\"\n\n mean: List[float] = field(default_factory=lambda: [0.485, 0.456, 0.406])\n std: List[float] = field(default_factory=lambda: [0.229, 0.224, 0.225])\n image_size: int = 224\n mixup: bool = True\n mixup_params: Dict[str, Any] = field(\n default_factory=lambda: {\"mixup_alpha\": 0.5, \"use_cuda\": True}\n )\n\n def to_dict(self) -> Dict[str, Any]:\n \"\"\"Convert to dictionary.\"\"\"\n return asdict(self)\n\n@dataclass\nclass CriterionParams:\n \"\"\"A class to track loss function parameters.\"\"\"\n\n train_criterion_name: str = \"BCEWithLogitsLoss\"\n valid_criterion_name: str = \"BCEWithLogitsLoss\"\n train_criterion_params: Dict[str, Any] = field(\n default_factory=lambda: {\n \"weight\": None,\n \"reduction\": \"mean\",\n \"pos_weight\": None,\n }\n )\n valid_criterion_params: Dict[str, Any] = field(\n default_factory=lambda: {\n \"weight\": None,\n \"reduction\": \"mean\",\n \"pos_weight\": None,\n }\n )\n@dataclass\nclass OptimizerParams:\n \"\"\"A class to track optimizer parameters.\n optimizer_name (str): name of the optimizer.\n lr (float): learning rate.\n weight_decay (float): weight decay.\n \"\"\"\n\n optimizer_name: str = \"AdamW\"\n optimizer_params: Dict[str, Any] = field(\n default_factory=lambda: {\n \"lr\": 5e-5,\n \"betas\": (0.9, 0.999),\n \"amsgrad\": False,\n \"weight_decay\": 1e-3,\n \"eps\": 1e-08,\n }\n )\n\n def to_dict(self) -> Dict[str, Any]:\n \"\"\"Convert to dictionary.\"\"\"\n return asdict(self)\n\n \n return torch.optim.lr_scheduler.OneCycleLR(optimizer, max_lr=3e-5, \n steps_per_epoch=len(train_loader), epochs=6)\n@dataclass\nclass SchedulerParams:\n \"\"\"A class to track Scheduler Params.\"\"\"\n\n scheduler_name: str = \"OneCycleLR\"\n if scheduler_name == \"OneCycleLR\":\n scheduler_params: Dict[str, Any] = field(\n default_factory=lambda: {\n \"max_lr\": 3e-5,\n \"steps_per_epoch\": DataLoaderParams().get_len_train_loader(),\n \"epochs\": 6,\n \"last_epoch\": -1,\n }\n )\n\n elif scheduler_name == \"CosineAnnealingWarmRestarts\":\n scheduler_params: Dict[str, Any] = field(\n default_factory=lambda: {\n \"T_0\": 10,\n \"T_mult\": 1,\n \"eta_min\": 1e-4,\n \"last_epoch\": -1,\n }\n )\n\n def to_dict(self) -> Dict[str, Any]:\n \"\"\"Convert to dictionary.\"\"\"\n return asdict(self)\n \n\n \n@dataclass\nclass ModelParams:\n \"\"\"A class to track model parameters.\n model_name (str): name of the model.\n pretrained (bool): If True, use pretrained model.\n input_channels (int): RGB image - 3 channels or Grayscale 1 channel\n output_dimension: Final output neuron.\n It is the number of classes in classification.\n Caution: If you use sigmoid layer for Binary, then it is 1.\n \"\"\"\n\n # model_name: str = \"tf_efficientnet_b4_ns\"\n model_name: str = \"swin_large_patch4_window7_224\"\n pretrained: bool = True\n input_channels: int = 3\n output_dimension: int = 1 \n\n def to_dict(self) -> Dict[str, Any]:\n \"\"\"Convert to dictionary.\"\"\"\n return asdict(self)\n \n def check_dimension(self):\n if CriterionParams().criterion_name == \"BCEWithLogitsLoss\":\n assert output_dimension == 1, \"This loss uses sigmoid layer, so expect output to be 1.\"\n\n\n\n@dataclass\nclass GlobalTrainParams:\n debug: bool = False\n debug_multipler: int = 16\n epochs: int = 6 # 6\n mixup: bool = AugmentationParams().mixup\n model_name: str = ModelParams().model_name\n num_classes: int = ModelParams().output_dimension\n classification_type: str = \"multiclass\"\n use_amp: bool = True\n \n def to_dict(self) -> Dict[str, Any]:\n \"\"\"Convert to dictionary.\"\"\"\n return asdict(self)\n \n \nFILES = FilePaths()\nFOLDS = MakeFolds()\nLOADER_PARAMS = DataLoaderParams()\nTRAIN_PARAMS = GlobalTrainParams()\nTRANSFORMS = AugmentationParams()\nCRITERION_PARAMS = CriterionParams()\nOPTIMIZER_PARAMS = OptimizerParams()\nSCHEDULER_PARAMS = SchedulerParams()\nMODEL_PARAMS = ModelParams()","metadata":{"_kg_hide-input":false,"execution":{"iopub.status.busy":"2021-12-14T12:29:34.685629Z","iopub.execute_input":"2021-12-14T12:29:34.685906Z","iopub.status.idle":"2021-12-14T12:29:34.891771Z","shell.execute_reply.started":"2021-12-14T12:29:34.685873Z","shell.execute_reply":"2021-12-14T12:29:34.891023Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Back to top","metadata":{}},{"cell_type":"markdown","source":"\n\n

Utils

","metadata":{}},{"cell_type":"markdown","source":"### Seeding and Reproducibility\n\nThis is very important, and I tend to have an OCD over it. Everytime I made changes to my code base, I will run the whole script on DEBUG mode just to check if the training loss changes. If my code change should not affect the training results, yet the loss changes, then it either means that my code is wrong, or a hidden pseudo number generator is called.","metadata":{}},{"cell_type":"code","source":"def seed_all(seed: int = 1992):\n \"\"\"Seed all random number generators.\"\"\"\n print(\"Using Seed Number {}\".format(seed))\n\n os.environ[\"PYTHONHASHSEED\"] = str(\n seed\n ) # set PYTHONHASHSEED env var at fixed value\n torch.manual_seed(seed)\n torch.cuda.manual_seed_all(seed)\n torch.cuda.manual_seed(seed) # pytorch (both CPU and CUDA)\n np.random.seed(seed) # for numpy pseudo-random generator\n # set fixed value for python built-in pseudo-random generator\n random.seed(seed)\n torch.backends.cudnn.deterministic = True\n torch.backends.cudnn.benchmark = True\n torch.backends.cudnn.enabled = True\n\n\ndef seed_worker(_worker_id):\n \"\"\"Seed a worker with the given ID.\"\"\"\n worker_seed = torch.initial_seed() % 2 ** 32\n np.random.seed(worker_seed)\n random.seed(worker_seed)\n\nseed_all()","metadata":{"_kg_hide-input":false,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-12-14T12:29:34.893271Z","iopub.execute_input":"2021-12-14T12:29:34.893539Z","iopub.status.idle":"2021-12-14T12:29:34.905384Z","shell.execute_reply.started":"2021-12-14T12:29:34.893506Z","shell.execute_reply":"2021-12-14T12:29:34.904372Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Back to top","metadata":{}},{"cell_type":"markdown","source":"### Converting Numpy Arrays into Latex Form","metadata":{}},{"cell_type":"code","source":"def bmatrix(a):\n \"\"\"Returns a LaTeX bmatrix\n\n :a: numpy array\n :returns: LaTeX bmatrix as a string\n \"\"\"\n if len(a.shape) > 2:\n raise ValueError(\"bmatrix can at most display two dimensions\")\n lines = (\n np.array2string(a, max_line_width=np.infty)\n .replace(\"[\", \"\")\n .replace(\"]\", \"\")\n .splitlines()\n )\n rv = [r\"\\begin{bmatrix}\"]\n rv += [\" \" + \" & \".join(l.split()) + r\"\\\\\" for l in lines]\n rv += [r\"\\end{bmatrix}\"]\n return \"\\n\".join(rv)","metadata":{"_kg_hide-input":false,"execution":{"iopub.status.busy":"2021-12-14T12:29:34.907103Z","iopub.execute_input":"2021-12-14T12:29:34.907651Z","iopub.status.idle":"2021-12-14T12:29:34.915131Z","shell.execute_reply.started":"2021-12-14T12:29:34.907612Z","shell.execute_reply":"2021-12-14T12:29:34.914261Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Back to top","metadata":{}},{"cell_type":"markdown","source":"\n\n

Prepare Data

","metadata":{}},{"cell_type":"markdown","source":"### Functions to prepare data, datasets and dataloaders\n\nWe won't go into details here as this is a inference-centric notebook.","metadata":{}},{"cell_type":"code","source":"import os\nfrom pathlib import Path\nfrom typing import Union\n\nimport pandas as pd\nimport torch\n\ndef return_filepath(image_id: str, folder: Path = FILES.train_images) -> str:\n \"\"\"Add a new column image_path to the train and test csv.\n We can call the images easily in __getitem__ in Dataset.\n We need to be careful if the image_id has extension already.\n In this case, there is no need to add the extension.\n Args:\n image_id (str): The unique image id: 1000015157.jpg\n folder (Path, optional): The train folder. Defaults to FILES().train_images.\n Returns:\n image_path (str): The path to the image: \"c:\\\\users\\\\reighns\\\\kaggle_projects\\\\cassava\\\\data\\\\train\\\\1000015157.jpg\"\n \"\"\"\n image_path = os.path.join(folder, f\"{image_id}.jpg\")\n return image_path\n\n\ndef prepare_data(\n image_col_name: str = FOLDS.image_col_name,\n) -> pd.DataFrame:\n \"\"\"Call a sequential number of steps to prepare the data.\n Args:\n image_col_name (str): The column name of the unique image id.\n In Cassava, it is \"image_id\".\n Returns:\n df_train (pd.DataFrame): The train dataframe.\n df_test (pd.DataFrame): The test dataframe.\n df_folds (pd.DataFrame): The folds dataframe with an additional column \"fold\".\n df_sub (pd.DataFrame): The submission dataframe.\n \"\"\"\n\n df_train = pd.read_csv(FILES.train_csv)\n df_test = pd.read_csv(FILES.test_csv)\n df_sub = pd.read_csv(FILES.sub_csv)\n\n df_train[\"image_path\"] = df_train[image_col_name].apply(\n lambda x: return_filepath(image_id=x, folder=FILES.train_images)\n )\n df_test[\"image_path\"] = df_test[image_col_name].apply(\n lambda x: return_filepath(x, folder=FILES.test_images)\n )\n \n ##################### CUSTOM Sturges' rule ####################################\n num_bins = int(np.floor(1+np.log2(len(df_train))))\n # Cut the target Pawpularity into `num_bins` using pd.cut\n df_train['bins'] = pd.cut(df_train['Pawpularity'], bins=num_bins, labels=False, ordered=True)\n df_train['bins'].hist()\n ################################################################################\n\n df_folds = make_folds(train_csv=df_train, cv_params=FOLDS)\n\n return df_train, df_test, df_folds, df_sub\n\n\ndef prepare_loaders(\n df_folds: pd.DataFrame,\n fold: int,\n) -> Union[torch.utils.data.DataLoader, torch.utils.data.DataLoader]:\n \"\"\"Prepare Data Loaders.\"\"\"\n\n if TRAIN_PARAMS.debug:\n df_train = df_folds[df_folds[\"fold\"] != fold].sample(\n LOADER_PARAMS.train_loader[\"batch_size\"]\n * TRAIN_PARAMS.debug_multipler, \n random_state = FOLDS.seed\n )\n df_valid = df_folds[df_folds[\"fold\"] == fold].sample(\n LOADER_PARAMS.train_loader[\"batch_size\"]\n * TRAIN_PARAMS.debug_multipler, \n random_state = FOLDS.seed\n )\n # TODO: https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.sample.html\n # TODO: Consider adding stratified sampling to avoid df_valid having 0 num samples of minority class, causing issues when computing roc.\n df_oof = df_valid.copy()\n else:\n df_train = df_folds[df_folds[\"fold\"] != fold].reset_index(drop=True)\n df_valid = df_folds[df_folds[\"fold\"] == fold].reset_index(drop=True)\n # Initiate OOF dataframe for this fold (same as df_valid).\n df_oof = df_valid.copy()\n\n dataset_train = CustomDataset(\n df_train, transforms=get_train_transforms(), mode=\"train\"\n )\n dataset_valid = CustomDataset(\n df_valid, transforms=get_valid_transforms(), mode=\"train\"\n )\n\n # Seeding workers for reproducibility.\n g = torch.Generator()\n g.manual_seed(0)\n\n train_loader = torch.utils.data.DataLoader(\n dataset_train,\n **LOADER_PARAMS.train_loader,\n worker_init_fn=seed_worker,\n generator=g,\n )\n valid_loader = torch.utils.data.DataLoader(\n dataset_valid,\n **LOADER_PARAMS.valid_loader,\n worker_init_fn=seed_worker,\n generator=g,\n )\n\n # TODO: consider decoupling the oof and loaders, and consider add test loader here for consistency.\n return train_loader, valid_loader, df_oof","metadata":{"_kg_hide-input":false,"execution":{"iopub.status.busy":"2021-12-14T12:29:34.918316Z","iopub.execute_input":"2021-12-14T12:29:34.918521Z","iopub.status.idle":"2021-12-14T12:29:34.935553Z","shell.execute_reply.started":"2021-12-14T12:29:34.918499Z","shell.execute_reply":"2021-12-14T12:29:34.934854Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Cross-Validation Strategy\n\nWe won't go into details here as this is a inference-centric notebook. But a snippet is here:\n\n```python\nif cv_params.cv_schema == \"StratifiedKFold\":\n df_folds = train_csv.copy()\n skf = StratifiedKFold(\n n_splits=cv_params.num_folds,\n shuffle=True,\n random_state=cv_params.seed,\n )\n df_folds.loc[val_idx, \"fold\"] = int(fold + 1)\n df_folds[\"fold\"] = df_folds[\"fold\"].astype(int)\n print(df_folds.groupby([\"fold\", cv_params.class_col_name]).size()) \n```","metadata":{}},{"cell_type":"code","source":"import pandas as pd\n\n# from IPython.display import display\nfrom sklearn.model_selection import GroupKFold, StratifiedKFold\n\n\ndef make_folds(\n train_csv: pd.DataFrame, cv_params: MakeFolds()\n) -> pd.DataFrame:\n \"\"\"Split the given dataframe into training folds.\"\"\"\n\n if cv_params.cv_schema == \"StratifiedKFold\":\n df_folds = train_csv.copy()\n skf = StratifiedKFold(\n n_splits=cv_params.num_folds,\n shuffle=True,\n random_state=cv_params.seed,\n )\n ######################## Custom Stratification ############\n for fold, (_train_idx, val_idx) in enumerate(\n skf.split(\n X=df_folds[cv_params.image_col_name],\n y=df_folds['bins'],\n )\n ):\n ##########################################################\n df_folds.loc[val_idx, \"fold\"] = int(fold + 1)\n df_folds[\"fold\"] = df_folds[\"fold\"].astype(int)\n print(df_folds.groupby([\"fold\", cv_params.class_col_name]).size())\n\n elif cv_params.cv_schema == \"GroupKfold\":\n df_folds = train_csv.copy()\n gkf = GroupKFold(n_splits=cv_params.num_folds)\n groups = df_folds[cv_params.group_kfold_split].values\n for fold, (_train_index, val_index) in enumerate(\n gkf.split(\n X=df_folds, y=df_folds[cv_params.class_col_name], groups=groups\n )\n ):\n df_folds.loc[val_index, \"fold\"] = int(fold + 1)\n df_folds[\"fold\"] = df_folds[\"fold\"].astype(int)\n\n print(df_folds.groupby([\"fold\", cv_params.class_col_name]).size())\n\n else:\n config.logger.error(\n f\"Unknown CV schema: {cv_params.cv_schema}, are you using custom split?\"\n )\n df_folds = train_csv.copy()\n print(df_folds.groupby([\"fold\", cv_params.class_col_name]).size())\n\n df_folds.to_csv(cv_params.folds_csv, index=False)\n\n return df_folds","metadata":{"_kg_hide-input":false,"execution":{"iopub.status.busy":"2021-12-14T12:29:34.936877Z","iopub.execute_input":"2021-12-14T12:29:34.937339Z","iopub.status.idle":"2021-12-14T12:29:34.954801Z","shell.execute_reply.started":"2021-12-14T12:29:34.937304Z","shell.execute_reply":"2021-12-14T12:29:34.953874Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train, df_test, df_folds, df_sub = prepare_data()\ndisplay(df_train.head())","metadata":{"execution":{"iopub.status.busy":"2021-12-14T12:29:34.956386Z","iopub.execute_input":"2021-12-14T12:29:34.956725Z","iopub.status.idle":"2021-12-14T12:29:35.370612Z","shell.execute_reply.started":"2021-12-14T12:29:34.956693Z","shell.execute_reply":"2021-12-14T12:29:35.369804Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"is_normalize_target = True\n\nif is_normalize_target:\n df_folds[\"Pawpularity\"] = df_folds[\"Pawpularity\"] / 100","metadata":{"execution":{"iopub.status.busy":"2021-12-14T12:29:35.372402Z","iopub.execute_input":"2021-12-14T12:29:35.3726Z","iopub.status.idle":"2021-12-14T12:29:35.379614Z","shell.execute_reply.started":"2021-12-14T12:29:35.372575Z","shell.execute_reply":"2021-12-14T12:29:35.37889Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Back to top","metadata":{}},{"cell_type":"markdown","source":"\n\n

Dataset

","metadata":{}},{"cell_type":"markdown","source":"`Dataset` may be unique to each competition. But in general, they have a fixed framework as follows. Here is a snippet of what a typical `dataset` class looks like! Note, you need to expand the cell to see the full code!\n\n```python\nclass CustomDataset(torch.utils.data.Dataset):\n def __len__(self) -> int:\n \"\"\"Return the length of the dataset.\"\"\"\n return len(self.df)\n \n def __getitem__(\n self, index: int\n ) -> Union[\n Dict[str, torch.FloatTensor],\n Dict[str, Union[torch.FloatTensor, torch.LongTensor]],\n ]:\n \"\"\"Implements the getitem method: https://www.geeksforgeeks.org/__getitem__-and-__setitem__-in-python/\n\n Be careful of Targets:\n BCEWithLogitsLoss expects a target.float()\n CrossEntropyLoss expects a target.long()\n\n Args:\n index (int): index of the dataset.\n\n Returns:\n Dict[str, torch.FloatTensor]:{\"X\": image_tensor}\n Dict[str, Union[torch.FloatTensor, torch.LongTensor]]: {\"y\": target_tensor} If BCEwithLogitsLoss then FloatTensor, else LongTensor\n \"\"\"\n image_path = self.image_path[index]\n image = cv2.imread(image_path)\n image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n\n # Get target for all modes except for test, if test, replace target with dummy ones to pass through return_dtype.\n target = self.targets[index] if self.mode != \"test\" else torch.ones(1)\n\n if self.transforms:\n image = self.transforms(image=image)[\"image\"]\n\n X, y, original_image = self.return_dtype(image, target, original_image)\n \n if self.mode in [\"train\", \"valid\"]:\n return {\"X\": X, \"y\": y}\n\n if self.mode == \"test\":\n return {\"X\": X}\n```","metadata":{}},{"cell_type":"code","source":"from typing import Dict, Union\nimport albumentations\n\nimport cv2\nimport pandas as pd\nimport torch\n\n\nclass CustomDataset(torch.utils.data.Dataset):\n \"\"\"Dataset class for the {insert competition/project name} dataset.\"\"\"\n\n def __init__(\n self,\n df: pd.DataFrame,\n transforms: albumentations.core.composition.Compose = None,\n mode: str = \"train\",\n ):\n \"\"\"Constructor for the dataset class.\n\n Args:\n df (pd.DataFrame): Dataframe for either train, valid or test.\n transforms (albumentations.core.composition.Compose): albumentations transforms to apply to the images.\n mode (str, optional): Defaults to \"train\". One of ['train', 'valid', 'test', 'gradcam']\n \"\"\"\n\n # \"image_path\" is hardcoded, as that is always defined in prepare_data.\n self.image_path = df[\"image_path\"].values\n self.image_ids = df[FOLDS.image_col_name].values\n self.df = df\n self.targets = (\n torch.from_numpy(df[FOLDS.class_col_name].values)\n if mode != \"test\"\n else None\n )\n\n self.transforms = transforms\n self.mode = mode\n\n if self.mode not in [\"train\", \"valid\", \"test\", \"gradcam\"]:\n raise ValueError(\n f\"Mode {self.mode} not in accepted list of modes {['train', 'valid', 'test', 'gradcam']}\"\n )\n\n def __len__(self) -> int:\n \"\"\"Return the length of the dataset.\"\"\"\n return len(self.df)\n\n @staticmethod\n def return_dtype(\n X: torch.Tensor, y: torch.Tensor, original_image: torch.Tensor\n ) -> torch.Tensor:\n \"\"\"Return the dtype of the dataset.\n Args:\n X (torch.Tensor): Image tensor.\n y (torch.Tensor): Target tensor.\n original_image (torch.Tensor): Original image tensor.\n Returns:\n X (torch.Tensor): Image tensor.\n y (torch.Tensor): Target tensor.\n original_image (torch.Tensor): Original image tensor.\n \"\"\"\n\n # TODO: To check if defining float32 here will affect how Mixed Precision works. If so, then change from float32 to float.\n if CRITERION_PARAMS.train_criterion_name == \"BCEWithLogitsLoss\":\n # Make changes to reshape rather than in Trainer.\n y = torch.as_tensor(y, dtype=torch.float32).view(-1, 1)\n else:\n y = torch.as_tensor(y, dtype=torch.long)\n\n X = torch.as_tensor(X, dtype=torch.float32)\n original_image = torch.as_tensor(original_image, dtype=torch.float32)\n\n return X, y, original_image\n \n\n def check_shape(self):\n \"\"\"Check the shape of the dataset.\n\n Add a tensor transpose if transformation is None since most images is HWC but ToTensorV2 transforms them to CHW.\"\"\"\n\n raise NotImplementedError\n\n def __getitem__(\n self, index: int\n ) -> Union[\n Dict[str, torch.FloatTensor],\n Dict[str, Union[torch.FloatTensor, torch.LongTensor]],\n ]:\n \"\"\"Implements the getitem method: https://www.geeksforgeeks.org/__getitem__-and-__setitem__-in-python/\n\n Be careful of Targets:\n BCEWithLogitsLoss expects a target.float()\n CrossEntropyLoss expects a target.long()\n\n Args:\n index (int): index of the dataset.\n\n Returns:\n Dict[str, torch.FloatTensor]:{\"X\": image_tensor}\n Dict[str, Union[torch.FloatTensor, torch.LongTensor]]: {\"y\": target_tensor} If BCEwithLogitsLoss then FloatTensor, else LongTensor\n \"\"\"\n image_path = self.image_path[index]\n image = cv2.imread(image_path)\n image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n\n # needed for gradcam.\n original_image = cv2.resize(\n image, (TRANSFORMS.image_size, TRANSFORMS.image_size)\n ).copy()\n\n # Get target for all modes except for test, if test, replace target with dummy ones to pass through return_dtype.\n target = self.targets[index] if self.mode != \"test\" else torch.ones(1)\n\n if self.transforms:\n image = self.transforms(image=image)[\"image\"]\n\n # TODO: Consider not returning original image if we don't need it, may cause more memory usage and speed issues?\n X, y, original_image = self.return_dtype(image, target, original_image)\n \n if self.mode in [\"train\", \"valid\"]:\n \n return {\"X\": X, \"y\": y}\n\n if self.mode == \"test\":\n return {\"X\": X}\n\n if self.mode == \"gradcam\":\n\n return {\n \"X\": X,\n \"y\": y,\n \"original_image\": original_image,\n \"image_id\": self.image_ids[index],\n }","metadata":{"_kg_hide-input":false,"execution":{"iopub.status.busy":"2021-12-14T12:29:35.381181Z","iopub.execute_input":"2021-12-14T12:29:35.381634Z","iopub.status.idle":"2021-12-14T12:29:35.400224Z","shell.execute_reply.started":"2021-12-14T12:29:35.381599Z","shell.execute_reply":"2021-12-14T12:29:35.399522Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Back to top","metadata":{}},{"cell_type":"markdown","source":"\n\n

Augmentations

","metadata":{}},{"cell_type":"markdown","source":"Notice that in `get_inference_transforms` I defined many TTAs, in which all of them were copied verbatim from `get_train_transforms`, except for `VerticalFlip`. Since I put very low probability for `VerticalFlip`, I reckon I will just not include it in TTA.\n\nOne other note is that in order to make TTA results deterministic, I will use p=1. for all TTAs. However, if you have `RandomResizedCrops`, then there will be some randomness.\n\nHere is a snippet of what the function looks like:\n\n```python\ndef get_inference_transforms(image_size: int = TRANSFORMS.image_size) -> Dict[str, albumentations.Compose]:\n \"\"\"Performs Augmentation on test dataset.\n Returns the transforms for inference in a dictionary which can hold TTA transforms.\n\n Args:\n image_size (int, optional): [description]. Defaults to AUG.image_size.\n\n Returns:\n Dict[str, albumentations.Compose]: [description]\n \"\"\"\n\n transforms_dict = {\n \"transforms_test\": albumentations.Compose(\n [\n albumentations.Resize(image_size, image_size),\n albumentations.Normalize(\n mean=TRANSFORMS.mean,\n std=TRANSFORMS.std,\n max_pixel_value=255.0,\n p=1.0,\n ),\n ToTensorV2(p=1.0),\n ]\n ),\n \"tta_flip\": albumentations.Compose(\n [\n albumentations.HorizontalFlip(p=1),\n albumentations.Resize(image_size, image_size),\n albumentations.Normalize(\n mean=TRANSFORMS.mean,\n std=TRANSFORMS.std,\n max_pixel_value=255.0,\n p=1.0,\n ),\n ToTensorV2(p=1.0),\n ]\n ),\n \"tta_rotate\": albumentations.Compose(\n [\n albumentations.Rotate(limit=180, p=1),\n albumentations.Resize(image_size, image_size),\n albumentations.Normalize(\n mean=TRANSFORMS.mean,\n std=TRANSFORMS.std,\n max_pixel_value=255.0,\n p=1.0,\n ),\n ToTensorV2(p=1.0),\n ]\n )\n }\n\n return transforms_dict\n```","metadata":{}},{"cell_type":"code","source":"from albumentations.pytorch.transforms import ToTensorV2\nimport albumentations\n\n\ndef get_train_transforms(image_size: int = TRANSFORMS.image_size):\n \"\"\"Performs Augmentation on training data.\n\n Args:\n image_size (int, optional): [description]. Defaults to AUG.image_size.\n\n Returns:\n [type]: [description]\n \"\"\"\n return albumentations.Compose(\n [ # albumentations.RandomResizedCrop(height=image_size, width=image_size),\n albumentations.HorizontalFlip(p=0.5),\n albumentations.VerticalFlip(p=0.1),\n albumentations.Rotate(limit=180, p=0.5),\n albumentations.ShiftScaleRotate(\n shift_limit=0.1, scale_limit=0.1, rotate_limit=45, p=0.5\n ),\n albumentations.HueSaturationValue(\n hue_shift_limit=0.2,\n sat_shift_limit=0.2,\n val_shift_limit=0.2,\n p=0.5,\n ),\n albumentations.RandomBrightnessContrast(\n brightness_limit=(-0.1, 0.1), contrast_limit=(-0.1, 0.1), p=0.5\n ),\n albumentations.Resize(image_size, image_size),\n albumentations.Normalize(\n mean=TRANSFORMS.mean,\n std=TRANSFORMS.std,\n max_pixel_value=255.0,\n p=1.0,\n ),\n ToTensorV2(p=1.0),\n ]\n )\n\n\ndef get_valid_transforms(image_size: int = TRANSFORMS.image_size):\n \"\"\"Performs Augmentation on validation data.\n\n Args:\n image_size (int, optional): [description]. Defaults to AUG.image_size.\n\n Returns:\n [type]: [description]\n \"\"\"\n return albumentations.Compose(\n [\n albumentations.Resize(image_size, image_size),\n albumentations.Normalize(\n mean=TRANSFORMS.mean,\n std=TRANSFORMS.std,\n max_pixel_value=255.0,\n p=1.0,\n ),\n ToTensorV2(p=1.0),\n ]\n )\n\n\ndef get_gradcam_transforms(image_size: int = TRANSFORMS.image_size):\n \"\"\"Performs Augmentation on gradcam data.\n\n Args:\n image_size (int, optional): [description]. Defaults to AUG.image_size.\n\n Returns:\n [type]: [description]\n \"\"\"\n return albumentations.Compose(\n [\n albumentations.Resize(image_size, image_size),\n albumentations.Normalize(\n mean=TRANSFORMS.mean,\n std=TRANSFORMS.std,\n max_pixel_value=255.0,\n p=1.0,\n ),\n ToTensorV2(p=1.0),\n ]\n )\n\n\ndef get_inference_transforms(image_size: int = TRANSFORMS.image_size) -> Dict[str, albumentations.Compose]:\n \"\"\"Performs Augmentation on test dataset.\n Returns the transforms for inference in a dictionary which can hold TTA transforms.\n\n Args:\n image_size (int, optional): [description]. Defaults to AUG.image_size.\n\n Returns:\n Dict[str, albumentations.Compose]: [description]\n \"\"\"\n\n transforms_dict = {\n \"transforms_test\": albumentations.Compose(\n [\n albumentations.Resize(image_size, image_size),\n albumentations.Normalize(\n mean=TRANSFORMS.mean,\n std=TRANSFORMS.std,\n max_pixel_value=255.0,\n p=1.0,\n ),\n ToTensorV2(p=1.0),\n ]\n ),\n \"tta_hflip\": albumentations.Compose(\n [\n albumentations.HorizontalFlip(p=1),\n albumentations.Resize(image_size, image_size),\n albumentations.Normalize(\n mean=TRANSFORMS.mean,\n std=TRANSFORMS.std,\n max_pixel_value=255.0,\n p=1.0,\n ),\n ToTensorV2(p=1.0),\n ]\n ),\n \n \"tta_rotate\": albumentations.Compose(\n [\n albumentations.Rotate(limit=180, p=1),\n albumentations.Resize(image_size, image_size),\n albumentations.Normalize(\n mean=TRANSFORMS.mean,\n std=TRANSFORMS.std,\n max_pixel_value=255.0,\n p=1.0,\n ),\n ToTensorV2(p=1.0),\n ]\n ),\n \"tta_shift_scale_rotate\": albumentations.Compose(\n [\n albumentations.ShiftScaleRotate(\n shift_limit=0.1, scale_limit=0.1, rotate_limit=45, p=1\n ),\n albumentations.Resize(image_size, image_size),\n albumentations.Normalize(\n mean=TRANSFORMS.mean,\n std=TRANSFORMS.std,\n max_pixel_value=255.0,\n p=1.0,\n ),\n ToTensorV2(p=1.0),\n ]\n ),\n \"tta_hue_saturation_value\": albumentations.Compose(\n [\n albumentations.HueSaturationValue(\n hue_shift_limit=0.2,\n sat_shift_limit=0.2,\n val_shift_limit=0.2,\n p=1,\n ),\n albumentations.Resize(image_size, image_size),\n albumentations.Normalize(\n mean=TRANSFORMS.mean,\n std=TRANSFORMS.std,\n max_pixel_value=255.0,\n p=1.0,\n ),\n ToTensorV2(p=1.0),\n ]\n ),\n \"tta_random_brightness_contrast\": albumentations.Compose(\n [\n albumentations.RandomBrightnessContrast(\n brightness_limit=(-0.1, 0.1), contrast_limit=(-0.1, 0.1), p=1\n ),\n albumentations.Resize(image_size, image_size),\n albumentations.Normalize(\n mean=TRANSFORMS.mean,\n std=TRANSFORMS.std,\n max_pixel_value=255.0,\n p=1.0,\n ),\n ToTensorV2(p=1.0),\n ]\n ),\n }\n\n return transforms_dict\n","metadata":{"_kg_hide-input":false,"execution":{"iopub.status.busy":"2021-12-14T12:29:35.40454Z","iopub.execute_input":"2021-12-14T12:29:35.404799Z","iopub.status.idle":"2021-12-14T12:29:35.428499Z","shell.execute_reply.started":"2021-12-14T12:29:35.404775Z","shell.execute_reply":"2021-12-14T12:29:35.427709Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Back to top","metadata":{}},{"cell_type":"markdown","source":"\n\n

Model Instantiation

","metadata":{}},{"cell_type":"markdown","source":"We are using custom activation in our custom `head`. A `head` is a jargon for the `classifier` layer in a CNN. In general, people finetune a pretrained model by simply removing the last classifier layer and replace it with the correct number of classes.\n\n---\n\nNote carefully here when you instantiate your model during **inference**. You need to do a few things:\n\n1. Copy the exact same model over! It might seem obvious, but when I was a beginner, I was only forking other people's work, and do not know you need the exact same model to inference!\n2. Set `pretrained=False`, technically, you do not need to since you are loading model from a checkpoint, but remember about Kaggle competition being mostly offline? If you set `pretrained=True`, then you need internet access.","metadata":{}},{"cell_type":"code","source":"sigmoid = torch.nn.Sigmoid()\n\nclass Swish(torch.autograd.Function):\n @staticmethod\n def forward(ctx, i):\n result = i * sigmoid(i)\n ctx.save_for_backward(i)\n return result\n\n @staticmethod\n def backward(ctx, grad_output):\n i = ctx.saved_variables[0]\n sigmoid_i = sigmoid(i)\n return grad_output * (sigmoid_i * (1 + i * (1 - sigmoid_i)))\n\n\nclass Swish_Module(torch.nn.Module):\n def forward(self, x):\n return Swish.apply(x)","metadata":{"_kg_hide-input":false,"execution":{"iopub.status.busy":"2021-12-14T12:29:35.430911Z","iopub.execute_input":"2021-12-14T12:29:35.431515Z","iopub.status.idle":"2021-12-14T12:29:35.439956Z","shell.execute_reply.started":"2021-12-14T12:29:35.431486Z","shell.execute_reply":"2021-12-14T12:29:35.439265Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Unfortunately, defining a `model` class may not necessarily be enough for reusability in every competition. However, a generic architecture as presented below should suffice for most.","metadata":{}},{"cell_type":"code","source":"class CustomNeuralNet(torch.nn.Module):\n def __init__(\n self,\n model_name: str = MODEL_PARAMS.model_name,\n out_features: int = MODEL_PARAMS.output_dimension,\n in_channels: int = MODEL_PARAMS.input_channels,\n pretrained: bool = MODEL_PARAMS.pretrained,\n ):\n \"\"\"Construct a new model.\n Args:\n model_name ([type], str): The name of the model to use. Defaults to MODEL_PARAMS.model_name.\n out_features ([type], int): The number of output features, this is usually the number of classes, but if you use sigmoid, then the output is 1. Defaults to MODEL_PARAMS.output_dimension.\n in_channels ([type], int): The number of input channels; RGB = 3, Grayscale = 1. Defaults to MODEL_PARAMS.input_channels.\n pretrained ([type], bool): If True, use pretrained model. Defaults to MODEL_PARAMS.pretrained.\n \"\"\"\n super().__init__()\n\n self.in_channels = in_channels\n self.pretrained = pretrained\n\n self.backbone = timm.create_model(\n model_name, pretrained=self.pretrained, in_chans=self.in_channels\n )\n config.logger.info(\n f\"\\nModel: {model_name} \\nPretrained: {pretrained} \\nIn Channels: {in_channels}\"\n )\n\n # removes head from backbone\n self.backbone.reset_classifier(num_classes=0, global_pool=\"avg\")\n\n # get the last layer's number of features in backbone (feature map)\n self.in_features = self.backbone.num_features\n self.out_features = out_features\n\n # Custom Head\n self.single_head_fc = torch.nn.Sequential(\n torch.nn.Linear(self.in_features, self.out_features),\n )\n\n self.architecture: Dict[str, Callable] = {\n \"backbone\": self.backbone,\n \"bottleneck\": None,\n \"head\": self.single_head_fc,\n }\n\n def extract_features(self, image: torch.FloatTensor) -> torch.FloatTensor:\n \"\"\"Extract the features mapping logits from the model.\n This is the output from the backbone of a CNN.\n Args:\n image (torch.FloatTensor): The input image.\n Returns:\n feature_logits (torch.FloatTensor): The features logits.\n \"\"\"\n feature_logits = self.architecture[\"backbone\"](image)\n return feature_logits\n\n def forward(self, image: torch.FloatTensor) -> torch.FloatTensor:\n \"\"\"The forward call of the model.\n Args:\n image (torch.FloatTensor): The input image.\n Returns:\n classifier_logits (torch.FloatTensor): The output logits of the classifier head.\n \"\"\"\n\n feature_logits = self.extract_features(image)\n classifier_logits = self.architecture[\"head\"](feature_logits)\n\n return classifier_logits\n\n def get_last_layer(self):\n # TODO: Implement this properly.\n \"\"\"Get the last layer information of TIMM Model.\n Returns:\n [type]: [description]\n \"\"\"\n last_layer_name = None\n for name, _param in self.model.named_modules():\n last_layer_name = name\n\n last_layer_attributes = last_layer_name.split(\".\") # + ['in_features']\n linear_layer = functools.reduce(\n getattr, last_layer_attributes, self.model\n )\n # reduce applies to a list recursively and reduce\n in_features = functools.reduce(\n getattr, last_layer_attributes, self.model\n ).in_features\n return last_layer_attributes, in_features, linear_layer\n","metadata":{"_kg_hide-input":false,"execution":{"iopub.status.busy":"2021-12-14T12:29:35.441241Z","iopub.execute_input":"2021-12-14T12:29:35.441488Z","iopub.status.idle":"2021-12-14T12:29:35.456697Z","shell.execute_reply.started":"2021-12-14T12:29:35.441455Z","shell.execute_reply":"2021-12-14T12:29:35.45587Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Back to top","metadata":{}},{"cell_type":"markdown","source":"\n\n

Out-of-Fold

","metadata":{}},{"cell_type":"markdown","source":"If you did not save OOF predictions during training, then the only way to recover back is to predict on each fold again! ","metadata":{}},{"cell_type":"markdown","source":"\n\n

Inference By Folds

","metadata":{}},{"cell_type":"markdown","source":"Inference by fold function has remained faithful to me throughout competitions, I only ever have to worry about changing `sigmoid` to `softmax` depending on the model's settings. One thing to note is that, I save a lot of things in the `model` `state_dict`. In particular, I save `oof` predictions in my `state_dict`. This makes me pulling out `oof` predictions easy when I accidentally didn't manage to save my `oof` during training, which happens often.\n\nGo down to my example section for better understanding how it works under the hood.","metadata":{}},{"cell_type":"code","source":"device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")","metadata":{"execution":{"iopub.status.busy":"2021-12-14T12:29:35.458773Z","iopub.execute_input":"2021-12-14T12:29:35.459769Z","iopub.status.idle":"2021-12-14T12:29:35.505482Z","shell.execute_reply.started":"2021-12-14T12:29:35.459731Z","shell.execute_reply":"2021-12-14T12:29:35.50478Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import glob\n\ndef inference_all_folds(\n model: CustomNeuralNet,\n state_dicts: List[collections.OrderedDict],\n test_loader: torch.utils.data.DataLoader,\n) -> np.ndarray:\n \"\"\"Inference the model on all K folds.\n\n Args:\n model (models.CustomNeuralNet): The model to be used for inference. Note that pretrained should be set to False.\n state_dicts (List[collections.OrderedDict]): The state dicts of the models. Generally, K Fold means K state dicts.\n test_loader (torch.utils.data.DataLoader): The dataloader for the test set.\n\n Returns:\n mean_preds (np.ndarray): The mean of the predictions of all folds.\n \"\"\"\n\n model.to(device)\n model.eval()\n\n with torch.no_grad():\n all_folds_preds = []\n\n for _fold_num, state in enumerate(state_dicts):\n if \"model_state_dict\" not in state:\n model.load_state_dict(state)\n else:\n model.load_state_dict(state[\"model_state_dict\"])\n\n current_fold_preds = []\n\n for data in tqdm(test_loader, position=0, leave=True):\n images = data[\"X\"].to(device, non_blocking=True)\n logits = model(images)\n test_prob = logits.sigmoid().cpu().numpy()\n test_prob = test_prob * 100 # Only for petfinder\n\n current_fold_preds.append(test_prob)\n\n current_fold_preds = np.concatenate(current_fold_preds, axis=0)\n all_folds_preds.append(current_fold_preds)\n mean_preds = np.mean(all_folds_preds, axis=0)\n return mean_preds\n\n\ndef inference(\n df_test: pd.DataFrame,\n model_dir: str,\n model: CustomNeuralNet = CustomNeuralNet(pretrained=False),\n df_sub: pd.DataFrame = None,\n transform_dict: Dict[str, albumentations.Compose] = get_inference_transforms()\n) -> Dict[str, np.ndarray]:\n \"\"\"Inference the model and perform TTA, if any.\n\n Dataset and Dataloader are constructed within this function because of TTA.\n\n Args:\n df_test (pd.DataFrame): The test dataframe.\n model_dir (str): model directory for the model.\n df_sub (pd.DataFrame, optional): The submission dataframe. Defaults to None.\n\n Returns:\n all_preds (Dict[str, np.ndarray]): {\"normal\": normal_preds, \"tta\": tta_preds}\n \"\"\"\n\n if df_sub is None:\n config.logger.info(\n \"No submission dataframe detected, setting df_sub to be df_test.\"\n )\n df_sub = df_test.copy()\n\n all_preds = {}\n\n # model = CustomNeuralNet(pretrained=False).to(device)\n\n # transform_dict = get_inference_transforms()\n\n weights = sorted(list([model_path for model_path in glob.glob(model_dir + \"/*.pt\")]))\n \n# weights = [\"../input/swin-transformer-mixup/model/swin_large_patch4_window7_224/swin_large_patch4_window7_224_best_valid_rmse_fold_1.pt\",\n# \"../input/swin-transformer-mixup/model/swin_large_patch4_window7_224/swin_large_patch4_window7_224_best_valid_rmse_fold_2.pt\",\n# \"../input/swin-transformer-mixup/model/swin_large_patch4_window7_224/swin_large_patch4_window7_224_best_valid_rmse_fold_3.pt\",\n# \"../input/swin-transformer-mixup/model/swin_large_patch4_window7_224/swin_large_patch4_window7_224_best_valid_rmse_fold_4.pt\",\n# \"../input/swin-transformer-mixup/model/swin_large_patch4_window7_224/swin_large_patch4_window7_224_best_valid_rmse_fold_5.pt\"\n# ]\n\n\n state_dicts = [torch.load(path)[\"model_state_dict\"] for path in weights]\n preds_list = []\n # Loop over each TTA transforms, if TTA is none, then loop once over normal inference_augs.\n for aug_name, aug_param in transform_dict.items():\n test_dataset = CustomDataset(\n df=df_test, transforms=aug_param, mode=\"test\"\n )\n test_loader = torch.utils.data.DataLoader(\n test_dataset, **LOADER_PARAMS.test_loader\n )\n predictions = inference_all_folds(\n model=model, state_dicts=state_dicts, test_loader=test_loader\n )\n\n all_preds[aug_name] = predictions\n\n ################# To change when necessary depending on the metrics needed for submission #################\n df_sub[FOLDS.class_col_name] = predictions\n\n df_sub[[FOLDS.image_col_name, FOLDS.class_col_name]].to_csv(\n f\"submission_{aug_name}.csv\", index=False\n )\n preds_list.append(predictions)\n print(df_sub.head())\n\n plt.figure(figsize=(12, 6))\n plt.hist(df_sub[FOLDS.class_col_name], bins=100)\n # For petfinder\n df_sub[FOLDS.class_col_name] = np.mean(np.stack(preds_list).squeeze(axis=2), axis=0)\n df_sub[[FOLDS.image_col_name, FOLDS.class_col_name]].to_csv(\n f\"submission.csv\", index=False\n )\n return all_preds, preds_list","metadata":{"execution":{"iopub.status.busy":"2021-12-14T12:30:19.142207Z","iopub.execute_input":"2021-12-14T12:30:19.142477Z","iopub.status.idle":"2021-12-14T12:30:22.729858Z","shell.execute_reply.started":"2021-12-14T12:30:19.142446Z","shell.execute_reply":"2021-12-14T12:30:22.728019Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"is_inference = False\n\nif is_inference:\n model_dir = \"../input/swin-transformer-mixup/model/swin_large_patch4_window7_224\"\n predictions, _ = inference(df_test, model_dir, df_sub)","metadata":{"execution":{"iopub.status.busy":"2021-12-14T12:29:38.941128Z","iopub.execute_input":"2021-12-14T12:29:38.941506Z","iopub.status.idle":"2021-12-14T12:29:38.94666Z","shell.execute_reply.started":"2021-12-14T12:29:38.941466Z","shell.execute_reply":"2021-12-14T12:29:38.945869Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Back to top","metadata":{}},{"cell_type":"markdown","source":"\n\n

Ensembling Techniques

","metadata":{}},{"cell_type":"markdown","source":"## Forward Ensembling\n\nWill add in soon!\n\nFor now, we do a simple weighted mean ensemble!\n\nIn practice, if we have 2 models, then we should have 2 callable config file, say a `yaml` file, and we can call them separately. In notebook, we just do the following.","metadata":{}},{"cell_type":"code","source":"tf_efficientnet_b4_ns_1llvj0qo = CustomNeuralNet(\n model_name=\"tf_efficientnet_b4_ns\",\n out_features=1,\n in_channels=3,\n pretrained=False,\n)\n\ntf_efficientnet_b4_ns_1llvj0qo_dir = \"../input/petfinder-tf-efficientnet-b4-ns-5-folds-1llvj0qo/model/tf_efficientnet_b4_ns\"\ntf_efficientnet_b4_ns_1llvj0qo_transform_dict = get_inference_transforms(\n image_size=448\n)\n\n_, tf_efficientnet_b4_ns_1llvj0qo_preds_list = inference(\n df_test,\n tf_efficientnet_b4_ns_1llvj0qo_dir,\n tf_efficientnet_b4_ns_1llvj0qo,\n df_sub,\n tf_efficientnet_b4_ns_1llvj0qo_transform_dict\n)\n","metadata":{"execution":{"iopub.status.busy":"2021-12-14T12:30:22.731596Z","iopub.execute_input":"2021-12-14T12:30:22.731845Z","iopub.status.idle":"2021-12-14T12:30:58.692136Z","shell.execute_reply.started":"2021-12-14T12:30:22.731812Z","shell.execute_reply":"2021-12-14T12:30:58.691449Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"swin_large_patch4_window7_224_246zq00g = CustomNeuralNet(\n model_name=\"swin_large_patch4_window7_224\",\n out_features=1,\n in_channels=3,\n pretrained=False,\n)\nswin_large_patch4_window7_224_246zq00g_dir = (\n \"../input/swin-transformer-mixup/model/swin_large_patch4_window7_224\"\n)\n\nswin_large_patch4_window7_224_246zq00g_transform_dict = (\n get_inference_transforms(image_size=224)\n)\n\n\n_, swin_large_patch4_window7_224_246zq00g_preds_list = inference(\n df_test,\n swin_large_patch4_window7_224_246zq00g_dir,\n swin_large_patch4_window7_224_246zq00g,\n df_sub,\n swin_large_patch4_window7_224_246zq00g_transform_dict\n)\n","metadata":{"execution":{"iopub.status.busy":"2021-12-14T12:26:06.808804Z","iopub.execute_input":"2021-12-14T12:26:06.809116Z","iopub.status.idle":"2021-12-14T12:27:12.647558Z","shell.execute_reply.started":"2021-12-14T12:26:06.809083Z","shell.execute_reply":"2021-12-14T12:27:12.64613Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_sub[FOLDS.class_col_name] = np.mean(np.stack(tf_efficientnet_b4_ns_1llvj0qo_preds_list).squeeze(axis=2), axis=0) * 0.33 + np.mean(np.stack(swin_large_patch4_window7_224_246zq00g_preds_list).squeeze(axis=2), axis=0) * 0.67\ndf_sub[[FOLDS.image_col_name, FOLDS.class_col_name]].to_csv(\n f\"submission.csv\", index=False\n )","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n\n

Post-Processing Techniques

","metadata":{}},{"cell_type":"markdown","source":"## Training Image Embeddings","metadata":{}},{"cell_type":"markdown","source":"### CatBoost","metadata":{}},{"cell_type":"code","source":"from catboost import CatBoostRegressor","metadata":{"execution":{"iopub.status.busy":"2021-12-12T06:09:06.007612Z","iopub.execute_input":"2021-12-12T06:09:06.007881Z","iopub.status.idle":"2021-12-12T06:09:06.134917Z","shell.execute_reply.started":"2021-12-12T06:09:06.007851Z","shell.execute_reply":"2021-12-12T06:09:06.134157Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"num_image_embeddings = 1536 # hardcoded the image embeds for swin transformer\nnum_image_embeddings = 1792 # tf_efficientnet_b4_ns","metadata":{"execution":{"iopub.status.busy":"2021-12-12T06:09:55.563663Z","iopub.execute_input":"2021-12-12T06:09:55.563931Z","iopub.status.idle":"2021-12-12T06:09:55.569132Z","shell.execute_reply.started":"2021-12-12T06:09:55.5639Z","shell.execute_reply":"2021-12-12T06:09:55.568331Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# weights = [\"../input/swin-transformer-mixup/model/swin_large_patch4_window7_224/swin_large_patch4_window7_224_best_valid_rmse_fold_1.pt\",\n# \"../input/swin-transformer-mixup/model/swin_large_patch4_window7_224/swin_large_patch4_window7_224_best_valid_rmse_fold_2.pt\",\n# \"../input/swin-transformer-mixup/model/swin_large_patch4_window7_224/swin_large_patch4_window7_224_best_valid_rmse_fold_3.pt\",\n# \"../input/swin-transformer-mixup/model/swin_large_patch4_window7_224/swin_large_patch4_window7_224_best_valid_rmse_fold_4.pt\",\n# \"../input/swin-transformer-mixup/model/swin_large_patch4_window7_224/swin_large_patch4_window7_224_best_valid_rmse_fold_5.pt\"\n# ]\n\nweights = [\"../input/petfinder-tf-efficientnet-b4-ns-5-folds-1llvj0qo/model/tf_efficientnet_b4_ns/tf_efficientnet_b4_ns_best_valid_rmse_fold_1.pt\",\n \"../input/petfinder-tf-efficientnet-b4-ns-5-folds-1llvj0qo/model/tf_efficientnet_b4_ns/tf_efficientnet_b4_ns_best_valid_rmse_fold_2.pt\",\n \"../input/petfinder-tf-efficientnet-b4-ns-5-folds-1llvj0qo/model/tf_efficientnet_b4_ns/tf_efficientnet_b4_ns_best_valid_rmse_fold_3.pt\",\n \"../input/petfinder-tf-efficientnet-b4-ns-5-folds-1llvj0qo/model/tf_efficientnet_b4_ns/tf_efficientnet_b4_ns_best_valid_rmse_fold_4.pt\",\n \"../input/petfinder-tf-efficientnet-b4-ns-5-folds-1llvj0qo/model/tf_efficientnet_b4_ns/tf_efficientnet_b4_ns_best_valid_rmse_fold_5.pt\",]\nbatch_size = 8","metadata":{"execution":{"iopub.status.busy":"2021-12-12T06:09:55.571062Z","iopub.execute_input":"2021-12-12T06:09:55.571521Z","iopub.status.idle":"2021-12-12T06:09:55.585572Z","shell.execute_reply.started":"2021-12-12T06:09:55.571483Z","shell.execute_reply":"2021-12-12T06:09:55.584857Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"inference_embeddings = False\n\nif inference_embeddings:\n\n # Utimately we want embeddings to be shape (n_samples, n_embeddings)\n # SO in my case if n_embeddings=1536, then if 100 samples, our final shape is (100, 1536)\n for index, fold in enumerate(range(1, FOLDS.num_folds+1)):\n print('#'*25)\n print('### FOLD',fold)\n print('#'*25)\n\n\n df_valid = df_folds[df_folds[\"fold\"] == fold].reset_index(drop=True)\n # Here hard code the embeddings first, as ultimately we want (n_samples, 1536)\n embeddings = torch.zeros(df_valid.shape[0], num_image_embeddings)\n\n model = CustomNeuralNet(pretrained=False).to(device)\n\n valid_transforms = get_inference_transforms()[\"transforms_test\"]\n\n state_dict = torch.load(weights[index])[\"model_state_dict\"]\n model.load_state_dict(state_dict)\n model.eval() # Think if u extract embeddings should set to eval too\n\n valid_dataset = CustomDataset(\n df=df_valid, transforms=valid_transforms, mode=\"train\"\n )\n valid_loader = torch.utils.data.DataLoader(\n valid_dataset, shuffle=False, batch_size = batch_size\n )\n row, col = 0, batch_size\n with torch.no_grad():\n for data in tqdm(valid_loader, position=0, leave=True):\n images = data[\"X\"].to(device)\n targets = data[\"y\"].to(device)\n # SWIN returns 1536 as embeddings, depends on how you take it it can be flattened or not yet\n curr_batch_embeds = model.extract_features(images)\n embeddings[row:col] = curr_batch_embeds\n row, col = row + batch_size, col + batch_size\n\n with open(f\"fold_{fold}_image_embeddings.npy\", 'wb') as f:\n np.save(f, embeddings.cpu().detach().numpy().astype('float32'))\n\n image_embeddings_paths = [\n \"fold_1_image_embeddings.npy\",\n \"fold_2_image_embeddings.npy\",\n \"fold_3_image_embeddings.npy\",\n \"fold_4_image_embeddings.npy\",\n \"fold_5_image_embeddings.npy\",\n ]\n\n image_embeddings_per_fold = [np.load(image_embeddings_path) for image_embeddings_path in image_embeddings_paths]\n\n print(image_embeddings_per_fold[0].shape)\n\n concat_all_image_embeddings = np.concatenate(image_embeddings_per_fold, axis=0) # fold 1-5 all \n\n # One important thing is you need the `df_folds` to be the same as you trained them in order to recover the image embeddings and its corresponding labels.\n # we want our oof np arrays to be of shape (num_folds, num_samples) where 1st row is the preds of the first fold etc \n # oof_trues, oof_preds = np.zeros(shape=(FOLDS.num_folds, df_folds.shape[0])), np.zeros(shape=(FOLDS.num_folds, df_folds.shape[0]))\n # print(oof_trues.shape)\n\n final_oof_preds, final_oof_trues = [], []\n\n df_folds['stratify_label'] = pd.qcut(df_folds['Pawpularity'], q = 30, labels = range(30))\n\n\n # skf = StratifiedKFold(n_splits = 6, shuffle = True, random_state = FOLDS.seed)\n # Y_strat = df_folds[\"stratify_label\"].values\n # Y_pawpularity = df_folds['Pawpularity'].values * 100\n\n # for idx, (train, val) in enumerate(skf.split(concat_all_image_embeddings, Y_strat)):\n # train_x, train_y = concat_all_image_embeddings[train], Y_pawpularity[train]\n # val_x, val_y = concat_all_image_embeddings[val], Y_pawpularity[val]\n # # Set CatBoost Parameters\n # cb_params = {'loss_function' : 'RMSE',\n # 'eval_metric' : 'RMSE',\n # 'iterations' : 1000,\n # 'grow_policy' : 'SymmetricTree',\n # 'depth' : 6,\n # 'l2_leaf_reg' : 2.0,\n # 'random_strength' : 1.0,\n # 'learning_rate' : 0.05,\n # 'task_type' : 'CPU',\n # 'devices' : '0',\n # 'verbose' : 0,\n # 'random_state': FOLDS.seed}\n\n # # Create and Fit CatBoost Model\n # cb_model = CatBoostRegressor(**cb_params)\n # cb_model.fit(train_x, train_y, eval_set = [(val_x, val_y)], early_stopping_rounds = 100, verbose = 250)\n\n\n import optuna\n from sklearn import metrics\n\n def objective(trial, train_x, train_y, valid_x, valid_y):\n\n param = {\n # \"objective\": trial.suggest_categorical(\"objective\", [\"Logloss\", \"CrossEntropy\"]),\n #\"colsample_bylevel\": trial.suggest_float(\"colsample_bylevel\", 0.01, 0.1),\n \"learning_rate\": trial.suggest_float(\"learning_rate\", 0.01, 0.05),\n \"depth\": trial.suggest_int(\"depth\", 1, 12),\n \"boosting_type\": trial.suggest_categorical(\"boosting_type\", [\"Ordered\", \"Plain\"]),\n \"l2_leaf_reg\": trial.suggest_discrete_uniform('l2_leaf_reg', 1.0, 5.5, 0.5),\n 'min_child_samples' : trial.suggest_categorical('min_child_samples', [1, 4, 8, 16, 32]),\n\n # \"bootstrap_type\": trial.suggest_categorical(\n # \"bootstrap_type\", [\"Bayesian\", \"Bernoulli\", \"MVS\"]\n # ),\n \"grow_policy\": \"SymmetricTree\",\n # \"used_ram_limit\": \"3gb\",\n \"task_type\": \"CPU\",\n \"devices\": \"0\",\n \"verbose\": 0,\n \"random_state\": FOLDS.seed\n }\n\n # if param[\"bootstrap_type\"] == \"Bayesian\":\n # param[\"bagging_temperature\"] = trial.suggest_float(\"bagging_temperature\", 0, 10)\n # elif param[\"bootstrap_type\"] == \"Bernoulli\":\n # param[\"subsample\"] = trial.suggest_float(\"subsample\", 0.1, 1)\n\n cb_model = CatBoostRegressor(**param)\n\n cb_model.fit(train_x, train_y, eval_set=[(valid_x, valid_y)], verbose=100, early_stopping_rounds=100)\n\n valid_preds = cb_model.predict(valid_x)\n\n rmse = metrics.mean_squared_error(valid_y.flatten(), valid_preds.flatten(), squared=False)\n return rmse\n\n\n\n # Utimately we want embeddings to be shape (n_samples, n_embeddings)\n # SO in my case if n_embeddings=1536, then if 100 samples, our final shape is (100, 1536)\n for index, fold in enumerate(range(1, FOLDS.num_folds+1)):\n print('#'*25)\n print('### FOLD',fold)\n print('#'*25)\n\n\n df_train = df_folds[df_folds[\"fold\"] != fold].reset_index(drop=True)\n df_valid = df_folds[df_folds[\"fold\"] == fold].reset_index(drop=True)\n oof_trues, oof_preds = np.zeros(shape=(df_valid.shape[0])), np.zeros(shape=(df_folds.shape[0]))\n train_x, train_y = np.concatenate(np.delete(image_embeddings_per_fold, index, axis=0), axis=0), df_train[\"Pawpularity\"].values * 100\n #train_x, train_y = image_embeddings_per_fold[2], df_folds[df_folds[\"fold\"] == 3].reset_index(drop=True)[\"Pawpularity\"].values * 100\n\n valid_x, valid_y = image_embeddings_per_fold[index], df_valid[\"Pawpularity\"].values * 100\n study = optuna.create_study(study_name=f'catboost-seed{FOLDS.seed}')\n # https://optuna.readthedocs.io/en/stable/faq.html\n study.optimize(lambda trial: objective(trial, train_x, train_y, valid_x, valid_y), n_trials=10000, n_jobs=-1, timeout=24000)\n break\n # # Set CatBoost Parameters\n # cb_params = {'loss_function' : 'RMSE',\n # 'eval_metric' : 'RMSE',\n # 'iterations' : 1000,\n # 'grow_policy' : 'SymmetricTree',\n # 'depth' : 6,\n # 'l2_leaf_reg' : 2.0,\n # 'random_strength' : 1.0,\n # 'learning_rate' : 0.05,\n # 'task_type' : 'CPU',\n # 'devices' : '0',\n # 'verbose' : 0,\n # 'random_state': FOLDS.seed}\n\n # # Create and Fit CatBoost Model\n # cb_model = CatBoostRegressor(**cb_params)\n # cb_model.fit(train_x, train_y, eval_set = [(valid_x, valid_y)], early_stopping_rounds = 1000, verbose = 250)\n\n # valid_preds = cb_model.predict(valid_x)\n # oof_trues[index] = valid_y # just assign it to the index, simple, oof_trues[0] -> first fold etc \n # oof_preds[index] = valid_preds\n\n\n # pickle.dump(cb_model, open(f\"catboost_fold_{fold}\", \"wb\"))\n\n\n # final_oof_trues.append(oof_trues)\n # final_oof_preds.append(oof_preds)\n # # Cleanup\n # del cb_model, valid_preds\n # del train_x, train_y\n # del valid_x, valid_y\n # gc.collect() \n\n final_oof_trues = final_oof_trues.flatten()\n final_oof_preds = final_oof_preds.flatten()\n\n\n metrics.mean_squared_error(final_oof_trues, final_oof_preds, squared=False)","metadata":{"execution":{"iopub.status.busy":"2021-12-14T12:17:18.505546Z","iopub.execute_input":"2021-12-14T12:17:18.50593Z","iopub.status.idle":"2021-12-14T12:17:18.597413Z","shell.execute_reply.started":"2021-12-14T12:17:18.505845Z","shell.execute_reply":"2021-12-14T12:17:18.596Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n\n

Dissecting Inference By Folds

","metadata":{}},{"cell_type":"markdown","source":"In Kaggle, the idea of K-Fold cross validation is rampant, and rightfully so. We take an extract from @cpmp below with original [discussion here](https://www.kaggle.com/c/ventilator-pressure-prediction/discussion/275883#1533680):\n\n> K-Folds cross validation was devised as a way to assess model performance using training data. A great paper on this from Sebastian Raschka is a must read https://arxiv.org/abs/1811.12808. You use k folds cv to tune you model, then retrain on all training data with best found tuning. However, once you have run K-Fold CV, you get K trained models. Kagglers quickly found that ensembling these models was giving good results at zero computation cost, rather than having to retrain a model on full data. It soon became a very common practice.\n\nThus, as a newbie last time, I was not at all familiar with the idea of inferencing five folds. I was more accustomed to the fact of say, training and **validation** with K-Folds to **obtain best hyperparameters** of a model, where the process is often called Model Selection, and then we use the best found hyperparameters $\\mathcal{C}$ to perform a full training on the whole training dataset (train+validation) $\\mathcal{X}$, and subsequently we use the parameters of the re-trained model as the final set of weights and inference on subsequent test set. **Note: hyperparameters and parameters are different! Do not get confused!**\n\nBelow we will break down what `inference_all_folds` does!","metadata":{}},{"cell_type":"markdown","source":"\n\n

Problem Settings

","metadata":{}},{"cell_type":"markdown","source":"### Classification Problem with Softmax Activation\n\nThe below problem illustrates a typical multiclass classification problem, which uses a `softmax` activation. The problem will mostly be the same if we use `sigmoid` activation as well.\n\n---\n\n- Batch Size: 2\n- Number of Test Images: 3 (img1, img2, img3)\n- Number of Folds/Weights: 5 (f1, f2, f3, f4, f5)\n- Number of classes: 5 (c1, c2, c3, c4, c5)\n\n---\n\n\n\nLet us walk through the typical inference pipeline in PyTorch step-by-step!","metadata":{}},{"cell_type":"markdown","source":"Back to top","metadata":{}},{"cell_type":"markdown","source":"### Initial steps for Inference\n\n1. We call the model first:\n\n```python\nmodel = CustomNeuralNet(pretrained=False).to(device)\n```\n\nNote that the emphasis is you need to use back the exact same model that you used for training during inference, and, for most cases, internet is off, so switch `pretrained` to `False`. Lastly, the usual drill for PyTorch, we need to put our model on to `device`, a flag to indicate whether our system is in GPU or CPU mode.\n\n---\n\n2. We then define the `datasets` as such:\n\n```python\ntest_dataset = CustomDataset(df=df_test, transforms=aug_param, mode=\"test\")\ntest_loader = torch.utils.data.DataLoader(test_dataset, **LOADER_PARAMS.test_loader)\n```\n\nHere we need to understand a few things:\n - `df_test`: Your test dataframe.\n - `mode`: Set to `test` during inference, this is on how you create your `dataset`, but in general, you can follow my template.\n - `transforms=aug_param`: As of now, just need to know that we should pass in the inference transforms, something very important here is, that you need to pass in the exact same `Normalization` parameters you used for training! In addition, it is also recommended to inference at the same image size you trained on; some of us may be adventurous, and use a bigger image size during inference. We will not go into TTA now, so hold your horses.\n```python\nalbumentations.Resize(image_size, image_size),\nalbumentations.Normalize(same as training)\n```\n\nLast but not least, equip your `dataset` with `DataLoader` and you are good to go! Note that the fancy looking notation `**LOADER_PARAMS.test_loader**` is just me loading off some pre-defined parameters for the DataLoader. If in doubt, print out `LOADER_PARAMS.test_loader` and you will know what I am saying.\n\n---\n\n3. Let us see what's next, we need to define and load up the weights from our trained models! Since we trained five folds, then there should be 5 sets of weights.\n\n```python\nweights = [model_path for model_path in glob.glob(model_dir + \"/*.pt\")] = [weight_for_fold_1, weight_for_fold_2, ..., weight_for_fold_5]\n```\n\nThe `weights` variable merely holds the `path` of the saved weights, we now need to **load** them as PyTorch's state dicts, where you can read up more below.\n\n```\nstate_dicts = [torch.load(path)[\"model_state_dict\"] for path in weights]\n```\n\nNow, one may wonder why there is `model_state_dict` after loading the path, this is unique to each individual, when you save a model in PyTorch, you can assign a key (name) to the ordered dict, and to identify it easily, I called it `model_state_dict`. You will also read more about it in details below.\n\n---\n\n### Inference All Folds\n\n4. We are ready to move on to my main function, named `inference_all_folds`.\n```python\ndef inference_all_folds(\n model: CustomNeuralNet,\n state_dicts: List[collections.OrderedDict],\n test_loader: torch.utils.data.DataLoader,\n) -> np.ndarray:\n```\nAs we see, my function takes in three arguments, all of which were defined accordingly from Step 1 to 3. So let's go!\n\n---\n\n5. We set `model` to `eval()` mode as we are in inference phase; then since we are inference phase, we say `torch.no_grad()` because we are no longer computing or storing gradients anymore. Usually the three buddies are as follows:\n\n```python\nmodel.to(device)\nmodel.eval()\nwith torch.no_grad():\n inference code...\n```\n\n### What happens in the loop?\n\nWe will dissect step by step on what happens in the inference loop, firstly, we have **3 images and our batch size is 2**, this means that our `DataLoader` needs 2 loops to finish inference. Now, this was designed on purpose to see how the loop works, if not, I can easily set the batch size to be 4, or 3 for that matter to let our inference be done in 1 single loop.\n\n#### First loop\n\nWe list what our first loop consists of, we will indicate the explaination below in line number. Note that the first loop is loop over `state_dicts`, in other words, the first loop is by folds, first loop -> first fold, and so on.\n\n```python\nwith torch.no_grad():\n all_folds_preds = []\n for _fold_num, state in enumerate(state_dicts):\n if \"model_state_dict\" not in state:\n model.load_state_dict(state)\n else:\n model.load_state_dict(state[\"model_state_dict\"])\n current_fold_preds = []\n```\n\n6. **Line 2:** Initiate with an empty list `all_folds_preds=[]`, where the final expectation of this `list` should contain 5 `numpy array` with each `array` having a shape of `3 by 5`, or `n by 5` where `n` represents the total number of test images. \n\n7. **Line 3-7:** Here is where we start looping through each fold's model's and make predictions; since there are 5 folds, this implies we have 5 models, and as mentioned earlier, `states_dicts` is a `list` holding 5 `ordered dict (information of each model's weights)` of each model. The subsequent `if-else` clause is to tell the model to load our state dict, intuitively, it means now our initiated model's weights (pretrained = False) are now replaced with the weights trained by us.\n\n8. **Line 8:** Initiate with another empty list: `current_fold_preds = []` where the end result of it is all the predictions of fold $i$ (F1,2,3,4,5) contained in this list in the following format:\n\n```python\ncurrent_fold_preds = [\n [[predictions_for_img1_for_fold_i], [predictions_for_img2_for_fold_i]],\n [predictions_for_img3_for_fold_i],\n]\n```\nNote that the shape of `[predictions_for_img1_for_fold_i]` may change depending on your activation function, and the number of classes, here we use softmax and has 5 classes. Something very interesting is that the `current_fold_preds` is a list of 2 differently shaped numpy arrays, the first entry is of shape `(2,5)`, and the second entry is an array of shape `(1,5)`. When stacked/concat together, we should have a numpy array of shape `(3,5)`. This is not surprising, since we have 3 test images, and hence 3 predictions, which is the rows, and 5 columns for each row is because of we have 5 classes, and each column is the \"probability/predictions\" for class $c_i$. Furthermore, the reason of them being split up is because we set the `batch_size` to be 2, so when you iterate through the `DataLoader`, the first for loop will append the first 2 images' predictions, and the second loop will append the remaining 1 image's predictions.\n \n#### Second Loop\n\nThe second loop, is looping over our `DataLoader`. Since we have 3 images and batch size of 2, we will loop over the loader twice in each fold. In essence, we loop through the `DataLoader/TestLoader` and predict for each batch of images by calling `model(images)`, which will in turn return you **logits** because that is how we defined it in our **Model**. Subsequently, we convert the **logits** into **softmax predictions**. The naming might be not suggestive enough, but both **logits and softmax_preds** are **tensor array and numpy array respectively**.\n\n```python\nfor data in tqdm(test_loader, position=0, leave=True):\n images = data[\"X\"].to(device, non_blocking=True)\n logits = model(images)\n test_prob = torch.nn.Softmax(dim=1)(input=logits).to('cpu').numpy() # logits.sigmoid().cpu().numpy()\n\n current_fold_preds.append(test_prob)\n```\n \n9. **Line 2-3:** We get our images and call our model on the set of images, just like how we did in our training pipeline.\n\n10. **Line 4:** This is just making predictions for each batch of images, note one needs to use the exact same activation as you did in training. Usually, in classification, its either `softmax` or `sigmoid`.\n\n11. **Line 5:** We append the predictions/probs to `current_fold_preds`, one can now connect this back with point 8.\n\n\n#### End of (Outer) Loop\n\n```python\ncurrent_fold_preds = np.concatenate(current_fold_preds, axis=0)\nall_folds_preds.append(current_fold_preds)\n```\n\n12. After finishing each inner loop, we then `concatenate` the `current_fold_preds` and `append` it to `all_folds_preds`. The reason for `concatenate` is to convert the `list` into `array` as follows:\n\n```python\n# After fold 1's inference, we have\ncurrent_fold_preds = [\n [[0.01, 0.02, 0.03, 0.9, 0.04], [0.02, 0.8, 0.05, 0.06, 0.07]],\n [0.01, 0.8, 0.03, 0.09, 0.07],\n]\n\ncurrent_fold_preds = np.concatenate(current_fold_preds, axis=0)\ncurrent_fold_preds = [\n [0.01, 0.02, 0.03, 0.9, 0.04],\n [0.02, 0.8, 0.05, 0.06, 0.07],\n [0.01, 0.8, 0.03, 0.09, 0.07],\n]\nall_folds_preds.append(current_fold_preds)\nall_folds_preds = [current_fold_preds (fold_1_preds)]\n```\n\nTherefore, after five folds, we have something like this:\n\n```python\nall_folds_preds = [fold_1_preds, fold_2_preds, ..., fold_5_preds] -> a list of 5 numpy arrays of shape 3x5\n```\n\n#### End of Both Loops\n\n13. At this junction, we are edging towards the end. We will do a summary as follows, along with the last line of code: `mean_preds = np.mean(all_folds_preds, axis=0)`.\n - Outer Loop: We loop through each model's (5 folds = 5 models) `states_dicts` and for each fold/model,\n - Inner Loop: We loop through the `DataLoader` to predict all the images in the test set.\n - End of Inner Loop: For each `end`, we have `current_fold_preds` for f1, f2, ...\n - End of Outer Loop: `all_folds_preds = [fold_1_preds, fold_2_preds, fold_3_preds, fold_4_preds, fold_5_preds]`\n - Finally, we average the `all_folds_preds` using `avg_preds = np.mean(all_folds_preds, axis=0)` to get our averaged predictions, it is worth mentioning that now `all_folds_preds`, if treated as a numpy array, has shape of (5, 3, 5) where dimension/axis 0 is the fold number, dimension/axis 1 and 2 are just the `current_fold_preds`. So it is simply a numpy array of 5 sets of 3x5 predictions. We there just need to average over dimension/axis 0 to get an average over folds! As simple as that!\n ```python\n all_folds_preds = np.array(\n [\n np.array(\n [\n [0.01, 0.02, 0.03, 0.9, 0.04],\n [0.02, 0.8, 0.05, 0.06, 0.07],\n [0.01, 0.8, 0.03, 0.09, 0.07],\n ]\n ),\n np.array(\n [\n [0.03, 0.05, 0.01, 0.88, 0.03],\n [0.01, 0.82, 0.02, 0.07, 0.08],\n [0.005, 0.81, 0.0205, 0.0555, 0.109],\n ]\n ),\n np.array(\n [\n [0.05, 0.03, 0.08, 0.83, 0.01],\n [0.05, 0.89, 0.01, 0.02, 0.03],\n [0.03, 0.78, 0.05, 0.02, 0.12],\n ]\n ),\n np.array(\n [\n [0.02, 0.01, 0.03, 0.92, 0.02],\n [0.05, 0.85, 0.01, 0.01, 0.08],\n [0.02, 0.88, 0.03, 0.05, 0.02],\n ]\n ),\n np.array(\n [\n [0.01, 0.02, 0.02, 0.93, 0.02],\n [0.03, 0.76, 0.06, 0.06, 0.09],\n [0.02, 0.83, 0.01, 0.07, 0.07],\n ]\n ),\n ]\n )\n # we use np.mean on axis=0 to calculate the average. axis=0 just means we take the mean of each row.\n mean_preds = np.mean(all_folds_preds, axis=0)\n ``` \n \n If you only pass in one fold/model, this inference function will still work.","metadata":{}},{"cell_type":"markdown","source":"- all_preds_array: each inner array is 2d and is of shape (3, 5) - 3 rows and 5 columns where 3 is the number of images, and 5 the number of predictions per image.\n\nI purposely made the `batch_size` to be 2 and total number of images to predict on to be 3. This is to tell you that your `test_loader` will take two loops to complete the predictions.\n\nThe below `all_preds_array` is a dummy predictions I made on the 3 images across 5 folds/models. Note that there are 5 inner arrays in this, as an example, we can take `all_preds_array[0]` to be the first fold predictions across all 3 images. This inner array is a 2d array of 3 by 5, where the first row, `all_preds_array[0][0]` is the 5 predictions outputed by `softmax` for image number 1. And to be more pedantic,`all_preds_array[0][0]` being 0.01 just means for image 1, fold 1 predicts the image's probability to be class 1 is 0.01, and `all_preds_array[0][1]` just mean for the same image 1, fold 1 predicts the image's probability to be class 2 is 0.02. Note that if you sum up each row, it must add up to 1 in this case, because we are using `softmax`, where the predictions for all 5 class must sum up to 1.\n\n---\n\nSo we have 5 folds, and let us put focus on just image 1 for simplicity. We then have perform inference 5 times on image 1 using the same model, holding everything else constant. \n\n- Image 1 Fold 1 Predictions: `[0.01, 0.02, 0.03, 0.9, 0.04]`\n- Image 1 Fold 2 Predictions: `[0.03, 0.05, 0.01, 0.88, 0.03]`\n- Image 1 Fold 3 Predictions: `[0.05, 0.03, 0.08, 0.83, 0.01]`\n- Image 1 Fold 4 Predictions: `[0.02, 0.01, 0.03, 0.92, 0.02]`\n- Image 1 Fold 5 Predictions: `[0.01, 0.02, 0.02, 0.93, 0.02]`\n\nWe then add all 5 folds up and **average** them. This is akin to performing a mean ensemble. In general, we average the predictions. This usually produces a more robust result (can you explain why?)\n\n---\n\n> So we have trained 5 models because we used 5 fold cross validation. For each of the 5 fold models that we have, we will load each fold model's weights and use them to make predictions on the unseen test images. As a result, we have 5 predictions for each test image. To be clear, if you have 1000 unseen test images, named $$i~~~ \\forall i \\in {1,2,...,1000}$$ then for each image $i$, there will be 5 predictions each for it, we can call them as such $$P(i_{j}) ~~~ \\forall j \\in {1,2,3,4,5}$$ where $j$ represents the number of folds. Thus, by convention, we take the **average/mean** of these 5 sets of predictions and take the average value/probability as the final prediction value. This score is then submitted to Kaggle and we get what we called the LB score, which should correlate to your CV/OOF score.\n\n","metadata":{}},{"cell_type":"markdown","source":"### Understanding with a concrete example\n\nUnhide to see a concrete example of how each fold and predictions are being averaged.","metadata":{}},{"cell_type":"code","source":"### Fold 1 ###\n### First Batch: 2 Predictions ###\nfold_1_pred_1 = np.array(\n [[0.01, 0.02, 0.03, 0.9, 0.04], [0.02, 0.8, 0.05, 0.06, 0.07]])\n\n### Second Batch: 1 Prediction ###\n\nfold_1_pred_2 = np.array([[0.01, 0.8, 0.03, 0.09, 0.07]])\n\n\n### Fold 2 ###\n### First Batch: 2 Predictions ###\n\nfold_2_pred_1 = np.array([[0.03, 0.05, 0.01, 0.88, 0.03], [\n 0.01, 0.82, 0.02, 0.07, 0.08]])\n\n### Second Batch: 1 Prediction ###\n\nfold_2_pred_2 = np.array([[0.005, 0.81, 0.0205, 0.0555, 0.109]])\n\n\n### Fold 3 ###\n### First Batch: 2 Predictions ###\n\nfold_3_pred_1 = np.array([[0.05, 0.03, 0.08, 0.83, 0.01], [\n 0.05, 0.89, 0.01, 0.02, 0.03]])\n\n### Second Batch: 1 Prediction ###\n\nfold_3_pred_2 = np.array([[0.03, 0.78, 0.05, 0.02, 0.12]])\n\n\n### Fold 4 ###\n### First Batch: 2 Predictions ###\n\nfold_4_pred_1 = np.array([[0.02, 0.01, 0.03, 0.92, 0.02], [\n 0.05, 0.85, 0.01, 0.01, 0.08]])\n\n### Second Batch: 1 Prediction ###\n\nfold_4_pred_2 = np.array([[0.02, 0.88, 0.03, 0.05, 0.02]])\n\n\n### Fold 5 ###\n### First Batch: 2 Predictions ###\n\nfold_5_pred_1 = np.array([[0.01, 0.02, 0.02, 0.93, 0.02], [\n 0.03, 0.76, 0.06, 0.06, 0.09]])\n\n### Second Batch: 1 Prediction ###\n\nfold_5_pred_2 = np.array([[0.02, 0.83, 0.01, 0.07, 0.07]])\n\n\n### This list should contain the predictions of all 5 folds ###\nall_folds_preds = []\n\n\nfold_1_preds = []\n\n### Fold 1 ###\n### All Batches: Total of 3 Predictions in the following format ###\n\nfold_1_preds = [fold_1_pred_1, fold_1_pred_2]\n#print('\\nfold_1_preds before np.concatenate\\n',fold_1_preds)\n\n### Concatenate because previous format is not good, we want it to be a numpy array ###\n\nfold_1_preds = np.concatenate(fold_1_preds, axis=0)\n\n# Something good to know, concatenate works exactly the same as such: #\n# The idea is you concatenate two list, over the axis 0 which is rows. #\n\nfold_1_preds_ = np.concatenate([fold_1_pred_1, fold_1_pred_2], axis=0)\n#print('fold_1_preds after np.concatenate\\n',fold_1_preds)\n#print('fold_1_preds after np.concatenate using different method\\n',fold_1_preds_)\n\nall_folds_preds.append(fold_1_preds)\nprint('All Folds Pred list after Fold 1 is \\n\\n{}\\n\\n'.format(all_folds_preds))\n###################################################################################\n\nfold_2_preds = []\n\n### Fold 2 ###\n### All Batches: Total of 3 Predictions in the following format ###\n\nfold_2_preds = [fold_2_pred_1, fold_2_pred_2]\nfold_2_preds = np.concatenate(fold_2_preds, axis=0)\n\nall_folds_preds.append(fold_2_preds)\nprint('All Folds Pred list after Fold 1+2 is \\n\\n{}\\n\\n'.format(all_folds_preds))\n\n\nfold_3_preds = []\n\n### Fold 3 ###\n### All Batches: Total of 3 Predictions in the following format ###\n\nfold_3_preds = [fold_3_pred_1, fold_3_pred_2]\nfold_3_preds = np.concatenate(fold_3_preds, axis=0)\nall_folds_preds.append(fold_3_preds)\nprint('All Folds Pred list after fold 1+2+3 is \\n\\n{}\\n\\n'.format(all_folds_preds))\n\n\nfold_4_preds = []\n\n### Fold 4 ###\n### All Batches: Total of 3 Predictions in the following format ###\n\nfold_4_preds = [fold_4_pred_1, fold_4_pred_2]\nfold_4_preds = np.concatenate(fold_4_preds, axis=0)\nall_folds_preds.append(fold_4_preds)\nprint('All Folds Pred list after fold 1+2+3+4 is \\n\\n{}\\n\\n'.format(all_folds_preds))\n\n\nfold_5_preds = []\n\n### Fold 5 ###\n### All Batches: Total of 3 Predictions in the following format ###\n\nfold_5_preds = [fold_5_pred_1, fold_5_pred_2]\nfold_5_preds\nfold_5_preds = np.concatenate(fold_5_preds, axis=0)\nfold_5_preds\nall_folds_preds.append(fold_5_preds)\nprint('All Folds Pred list after fold 1+2+3+4+5 is \\n\\n{}\\n\\n'.format(all_folds_preds))\n\n\n###### Finally, we take np.mean over all_folds_preds over row wise calculation ######\n###### Do not use concat here! ######\navg_pred_without_concat = np.mean(all_folds_preds, axis=0)\nprint('Average predictions is \\n\\n{}'.format(avg_pred_without_concat))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-12-12T06:17:36.620943Z","iopub.status.idle":"2021-12-12T06:17:36.621518Z","shell.execute_reply.started":"2021-12-12T06:17:36.621279Z","shell.execute_reply":"2021-12-12T06:17:36.621305Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Converting to matrix in latex form\n\nprint((bmatrix(fold_1_preds)+'\\n'))\nprint((bmatrix(fold_2_preds)+'\\n'))\nprint((bmatrix(fold_3_preds)+'\\n'))\nprint((bmatrix(fold_4_preds)+'\\n'))\nprint((bmatrix(fold_5_preds)+'\\n'))\n\nfive_folds_add = (fold_1_preds+fold_2_preds+fold_3_preds+fold_4_preds+fold_5_preds)\n# five_folds_add\nprint((bmatrix(five_folds_add)+'\\n'))\n\nfive_folds_add_avg = (fold_1_preds+fold_2_preds+fold_3_preds+fold_4_preds+fold_5_preds)/5\nfive_folds_add_avg\n# print((bmatrix(five_folds_add_avg)+'\\n'))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-12-12T06:17:36.622707Z","iopub.status.idle":"2021-12-12T06:17:36.623291Z","shell.execute_reply.started":"2021-12-12T06:17:36.623028Z","shell.execute_reply":"2021-12-12T06:17:36.623057Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Putting it all together","metadata":{}},{"cell_type":"markdown","source":"\nFinal results: `all_folds_preds = [fold_1_preds, fold_2_preds, fold_3_preds, fold_4_preds, fold_5_preds]`\n\nFinally, we average the `all_folds_preds` using `avg_preds = np.mean(all_folds_preds, axis=0)` to get our averaged predictions: \n\n [[0.024 0.026 0.034 0.892 0.024 ]\n [0.032 0.824 0.03 0.044 0.07 ]\n [0.017 0.82 0.0281 0.0571 0.0778]]\n\nWhich is similar to our matrix:\n \n$$\\text{Dividing/Averaging all 5 Fold's Predictions}\\begin{bmatrix}\n 0.024 & 0.026 & 0.034 & 0.892 & 0.024\\\\\n 0.032 & 0.824 & 0.03 & 0.044 & 0.07\\\\\n 0.017 & 0.82 & 0.0281 & 0.0571 & 0.0778\\\\\n\\end{bmatrix}$$\n\n\n","metadata":{}},{"cell_type":"markdown","source":"**Numpy Shape Rows vs Columns**\n\n\nThere are 3 images in the test set, our final predictions for all 3 images across all 5 folds are presented below, with `shape` to be `[3,5]` which means 3 rows and 5 columns, where $\\text{row}_i$ represents (1 by 5) vector containing 5 predictions in probabilities of how likely each class is; to put it even more explicit, if fold 1's prediction on the first image (call it $\\text{image}_1$) is $[0.01, 0.02, 0.03, 0.9 , 0.04]$, then it means $\\text{image}_1$ being class 1 is $1\\%$, class 2 is $2\\%$, class 3 is $3\\%$, class 4 is $90\\%$ and class 5, $4\\%$.\n\n\nOne with Linear Algebra background can envision a (3 by 5) 2-dimensional array akin to a (3 x 5) **Matrix**.\n\nSo we are clear, the 5 matrices below represents the predictions of each fold. \n\n\n$$\\text{Fold 1 Predictions}=\\begin{bmatrix}\n 0.01 & 0.02 & 0.03 & 0.9 & 0.04\\\\\n 0.02 & 0.8 & 0.05 & 0.06 & 0.07\\\\\n 0.01 & 0.8 & 0.03 & 0.09 & 0.07\\\\\n\\end{bmatrix}$$\n\n\n$$\\text{Fold 2 Predictions}\\begin{bmatrix}\n 0.03 & 0.05 & 0.01 & 0.88 & 0.03\\\\\n 0.01 & 0.82 & 0.02 & 0.07 & 0.08\\\\\n 0.005 & 0.81 & 0.0205 & 0.0555 & 0.109\\\\\n\\end{bmatrix}$$\n\n\n$$\\text{Fold 3 Predictions}=\\begin{bmatrix}\n 0.05 & 0.03 & 0.08 & 0.83 & 0.01\\\\\n 0.05 & 0.89 & 0.01 & 0.02 & 0.03\\\\\n 0.03 & 0.78 & 0.05 & 0.02 & 0.12\\\\\n\\end{bmatrix}$$\n\n$$\\text{Fold 4 Predictions}=\\begin{bmatrix}\n 0.02 & 0.01 & 0.03 & 0.92 & 0.02\\\\\n 0.05 & 0.85 & 0.01 & 0.01 & 0.08\\\\\n 0.02 & 0.88 & 0.03 & 0.05 & 0.02\\\\\n\\end{bmatrix}$$\n\n$$\\text{Fold 5 Predictions}\\begin{bmatrix}\n 0.01 & 0.02 & 0.02 & 0.93 & 0.02\\\\\n 0.03 & 0.76 & 0.06 & 0.06 & 0.09\\\\\n 0.02 & 0.83 & 0.01 & 0.07 & 0.07\\\\\n\\end{bmatrix}$$\n\n\n\nAll we are left to do is to add these 5 matrices, and divide by 5, as follows:\n\n$$\\text{Adding all 5 Fold's Predictions}\\begin{bmatrix}\n 0.12 & 0.13 & 0.17 & 4.46 & 0.12\\\\\n 0.16 & 4.12 & 0.15 & 0.22 & 0.35\\\\\n 0.085 & 4.1 & 0.1405 & 0.2855 & 0.389\\\\\n\\end{bmatrix}$$\n\n\nDividing by 5 (averaging):\n\n\n$$\\text{Dividing/Averaging all 5 Fold's Predictions}\\begin{bmatrix}\n 0.024 & 0.026 & 0.034 & 0.892 & 0.024\\\\\n 0.032 & 0.824 & 0.03 & 0.044 & 0.07\\\\\n 0.017 & 0.82 & 0.0281 & 0.0571 & 0.0778\\\\\n\\end{bmatrix}$$","metadata":{}},{"cell_type":"markdown","source":"## Submission Cautions\n\n- When **COMMITTING YOUR NOTEBOOK**, do not print out anything in the `inference` function to debug if not it might take too much ram/gpu. \n- Check if you are using the same activation function, i.e. softmax vs sigmoid.\n- Check in `inference` function if you are assigning the correct form of predictions to the test dataframe. In particular, refer to **SUBMISSION SAMPLE** to see what they want. If they want, say accuracy, then use `argmax()`, if just raw probabilities, then leave it as it is after you use sigmoid/softmax.","metadata":{}},{"cell_type":"markdown","source":"## Test-Time-Augmentation (TTA)","metadata":{}},{"cell_type":"markdown","source":"\n\n

Saving and Loading Models' Weights

","metadata":{}},{"cell_type":"markdown","source":"More often than not, we don't have resources to run the model day and night. Kaggle or Colab disconnects you once it reaches a certain number of hours. This bothers me when I just started out because my model has not **converged** yet.\n\nTherefore, it is important to be able to **checkpoint** or **save** our model, for two main reasons:\n\n**1. Save the model's weights and use it later to inference or make predictions.**\n\n**2. Save the model's weights as a checkpoint and use it later to resume training.**","metadata":{}},{"cell_type":"markdown","source":"I will be using example from both the PyTorch website and a book that I bought. Please find below for references:\n\n1. https://pytorch.org/tutorials/\n\n2. Deep Learning with PyTorch - Step by Step A Beginner's Guide - Daniel Voigt","metadata":{}},{"cell_type":"markdown","source":"Back to top","metadata":{}},{"cell_type":"markdown","source":"### Question\n\nNow, I am unsure if this has been answered, I remember my buddy telling me that anecdotally, one can \"reset\" the LR a little when one resumes training from a model's checkpoint.\n\nIt is a bit counter-intuitive to me, but say I trained a model for 16 epochs, with a custom scheduler, say OneCycleLr + Adam or something, then when you resume training, should we reset the initial learning rate? I would think resetting is counter-intuitive as the purpose of the learning rate is to tune it such that your model can slowly converge to the minima (global one if the function is convex).\n\nSo the question is:\n\nShould I reset the learning rate when resume training, if yes, reset to what?\n\nIf we should not reset, what is a good way to \"extract\" the last learning rate, as some scheduler depends on factors like epochs…\n\n\nUncle CPMP's advice will add in and credit later.\n","metadata":{}},{"cell_type":"code","source":"# def save_checkpoint(model, optimizer, scheduler, scaler, epoch, fold, seed, fname=fname):\n# checkpoint = {\n# 'model': model.state_dict(),\n# 'optimizer': optimizer.state_dict(),\n# 'scheduler': scheduler.state_dict(),\n# 'scaler': scaler.state_dict(),\n# 'epoch': epoch,\n# 'fold':fold,\n# 'seed':seed,\n# }\n# torch.save(checkpoint, '../checkpoints/%s/%s_%d_%d.pt' % (fname, fname, fold, seed))\n\n# def load_checkpoint(fold, seed, fname):\n# model = create_model().to(device)\n# optimizer = optimizer = torch.optim.Adam(model.parameters(), lr=MAX_LR)\n# scheduler = torch.optim.lr_scheduler.OneCycleLR(optimizer=optimizer, \n# pct_start=PCT_START, \n# div_factor=DIV_FACTOR \n# max_lr=MAX_LR, \n# epochs=EPOCHS, \n# steps_per_epoch=int(np.ceil(len(train_data_loader)/GRADIENT_ACCUMULATION)))\n# scaler = GradScaler()\n# checkpoint = torch.load('../checkpoints/%s/%s_%d_%d.pt' % (fname, fname, fold, seed))\n# model.load_state_dict(checkpoint['model'])\n# optimizer.load_state_dict(checkpoint['optimizer'])\n# scheduler.load_state_dict(checkpoint['scheduler'])\n# scaler.load_state_dict(checkpoint['scaler'])\n# return model, optimizer, scheduler, scaler, epoch","metadata":{"execution":{"iopub.status.busy":"2021-12-12T06:17:36.624457Z","iopub.status.idle":"2021-12-12T06:17:36.625038Z","shell.execute_reply.started":"2021-12-12T06:17:36.624783Z","shell.execute_reply":"2021-12-12T06:17:36.624809Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n\n## What is a state_dict? \n\nIn PyTorch, the learnable parameters (i.e. weights and biases) of an\n``torch.nn.Module`` model are contained in the model’s *parameters*\n(accessed with ``model.parameters()``). A *state_dict* is simply a\nPython dictionary object that maps each layer to its parameter tensor.\nNote that only layers with learnable parameters (convolutional layers,\nlinear layers, etc.) and registered buffers (batchnorm's running_mean)\nhave entries in the model’s *state_dict*. Optimizer\nobjects (``torch.optim``) also have a *state_dict*, which contains\ninformation about the optimizer’s state, as well as the hyperparameters\nused.\n\nBecause *state_dict* objects are Python dictionaries, they can be easily\nsaved, updated, altered, and restored, adding a great deal of modularity\nto PyTorch models and optimizers.","metadata":{}},{"cell_type":"markdown","source":"## Simple Example\n\n`in_channels` is the number of channels of the input to the convolutional layer. So, for example, in the case of the convolutional layer that applies to the image, `in_channels` refers to the number of channels of the image. In the case of an RGB image, `in_channels == 3` (red, green and blue); in the case of a gray image, `in_channels == 1`.\n\n`out_channels` is the number of feature maps, which is often equivalent to the number of kernels that you apply to the input. See [here](https://stats.stackexchange.com/a/292064/82135) for more info. \n\n`kernel_size` is just the size of the kernel, usually `3x3` or `5x5`.","metadata":{}},{"cell_type":"markdown","source":"Convolutional Layer : Consider a convolutional layer which takes “l” feature maps as the input and has “k” feature maps as output. The filter size is $n*m$.\n\nHere the input has l=32 feature maps as inputs, k=64 feature maps as outputs and filter size is n=3 and m=3. It is important to understand, that we don’t simply have a $3*3$ filter, but actually, we have $3*3*32$ filter, as our input has 32 dimensions. And as an output from first conv layer, we learn 64 different $3*3*32$ filters which total weights is $$n*m*k*l$$ Then there is a term called bias for each feature map. So, the total number of parameters are $$(n*m*l+1)*k$$\n\nThink of the convolutional layer as a $nxn$ matrix, and $nxn = n^2$ is the weights (a.k.a what values to take when you apply the convolution.","metadata":{}},{"cell_type":"code","source":"# Define model\nclass TheModelClass(nn.Module):\n def __init__(self):\n super(TheModelClass, self).__init__()\n \n self.conv1 = nn.Conv2d(in_channels=3, out_channels =6, kernel_size =5)\n self.pool = nn.MaxPool2d(2, 2)\n # here in channels = 6 because out channels of previous is 6.\n self.conv2 = nn.Conv2d(6, 16, 5)\n self.fc1 = nn.Linear(16 * 5 * 5, 120)\n self.fc2 = nn.Linear(120, 84)\n self.fc3 = nn.Linear(84, 10)\n\n def forward(self, x):\n x = self.pool(F.relu(self.conv1(x)))\n x = self.pool(F.relu(self.conv2(x)))\n x = x.view(-1, 16 * 5 * 5)\n x = F.relu(self.fc1(x))\n x = F.relu(self.fc2(x))\n x = self.fc3(x)\n return x\n\n# Initialize model\nmodel = TheModelClass()\n\n# Initialize optimizer\noptimizer = torch.optim.SGD(model.parameters(), lr=0.001, momentum=0.9)\n\n# Print model's state_dict\nprint(\"Model's state_dict:\")\nfor param_tensor in model.state_dict():\n print(param_tensor, \"\\t\", model.state_dict()[param_tensor].size())\n\n# Print optimizer's state_dict\nprint(\"Optimizer's state_dict:\")\nfor var_name in optimizer.state_dict():\n print(var_name, \"\\t\", optimizer.state_dict()[var_name])\n \n","metadata":{"execution":{"iopub.status.busy":"2021-12-12T06:17:36.626248Z","iopub.status.idle":"2021-12-12T06:17:36.626814Z","shell.execute_reply.started":"2021-12-12T06:17:36.626553Z","shell.execute_reply":"2021-12-12T06:17:36.626579Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"[Link to print parameters/weights for each layer](https://stackoverflow.com/questions/49201236/check-the-total-number-of-parameters-in-a-pytorch-model)\n\n[Calculation of parameters in CNN](https://medium.com/@iamvarman/how-to-calculate-the-number-of-parameters-in-the-cnn-5bd55364d7ca)\n\n[LeNet Architecture](https://engmrk.com/lenet-5-a-classic-cnn-architecture/#:~:text=The%20LeNet%2D5%20architecture%20consists,and%20finally%20a%20softmax%20classifier.)","metadata":{}},{"cell_type":"code","source":"from prettytable import PrettyTable\n\ndef count_parameters(model):\n table = PrettyTable([\"Modules\", \"Parameters\"])\n total_params = 0\n for name, parameter in model.named_parameters():\n if not parameter.requires_grad: continue\n param = parameter.numel()\n table.add_row([name, param])\n total_params+=param\n print(table)\n print(f\"Total Trainable Params: {total_params}\")\n return total_params\n \ncount_parameters(model)\n\n# indeed, 5x5x3x6 = 450, note the input here is 3 channels.","metadata":{"execution":{"iopub.status.busy":"2021-12-12T06:17:36.628042Z","iopub.status.idle":"2021-12-12T06:17:36.628599Z","shell.execute_reply.started":"2021-12-12T06:17:36.62837Z","shell.execute_reply":"2021-12-12T06:17:36.628397Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Loading and Saving\n\n**Save**\n\nIt is not a secret that one should use `torch.save` to save a model's weights in the following manner:\n\n`torch.save(model.state_dict(), PATH)`\n\n\n\n**Load**\n\nAnd similarly, one should use `torch.load` to load a model's weights.\n\n model = TheModelClass(*args, **kwargs)\n model.load_state_dict(torch.load(PATH))\n model.eval()\n \n \n**Advanced Save and Load**\n\nOne can also make what you save more sophisticated:\n\n torch.save({\n 'epoch': epoch,\n 'model_state_dict': model.state_dict(),\n 'optimizer_state_dict': optimizer.state_dict(),\n 'loss': loss,\n ...\n }, PATH)\n \n \nAnd to load them, we do it as follows:\n\n model = TheModelClass(*args, **kwargs)\n optimizer = TheOptimizerClass(*args, **kwargs)\n\n checkpoint = torch.load(PATH)\n model.load_state_dict(checkpoint['model_state_dict'])\n optimizer.load_state_dict(checkpoint['optimizer_state_dict'])\n epoch = checkpoint['epoch']\n loss = checkpoint['loss']\n\n model.eval()\n # - or -\n model.train()\n\n**Caution**\n\nNotice that the `load_state_dict()` function takes a dictionary object, **NOT a path to a saved object**. This means that you must deserialize the saved `state_dict` before you pass it to the `load_state_dict()` function. For example, you **CANNOT** load using `model.load_state_dict(PATH)`.","metadata":{}},{"cell_type":"markdown","source":"## **Loading and Saving on GPU**\n\n\n**Save:**\n\n torch.save(model.state_dict(), PATH)\n\n**Load:**\n\n\n\n device = torch.device(\"cuda\")\n model = TheModelClass(*args, **kwargs)\n model.load_state_dict(torch.load(PATH))\n model.to(device)\n \nWhen loading a model on a GPU that was trained and saved on GPU, simply\nconvert the initialized ``model`` to a CUDA optimized model using\n``model.to(torch.device('cuda'))``. Also, be sure to use the\n``.to(torch.device('cuda'))`` function on all model inputs to prepare\nthe data for the model. Note that calling ``my_tensor.to(device)``\nreturns a new copy of ``my_tensor`` on GPU. It does NOT overwrite\n``my_tensor``. Therefore, remember to manually overwrite tensors:\n``my_tensor = my_tensor.to(torch.device('cuda'))``. Make sure to call `input = input.to(device)` on any input tensors that you feed to the model, as you will see later. ","metadata":{}},{"cell_type":"markdown","source":"# Using it on a real example","metadata":{}},{"cell_type":"markdown","source":"**Reminder on what is a state_dict**\n\nIn PyTorch, the learnable parameters (i.e. weights and biases) of an\n``torch.nn.Module`` model are contained in the model’s *parameters*\n(accessed with ``model.parameters()``). A *state_dict* is simply a\nPython dictionary object that maps each layer to its parameter tensor.\nNote that only layers with learnable parameters (convolutional layers,\nlinear layers, etc.) and registered buffers (batchnorm's running_mean)\nhave entries in the model’s *state_dict*. Optimizer\nobjects (``torch.optim``) also have a *state_dict*, which contains\ninformation about the optimizer’s state, as well as the hyperparameters\nused.","metadata":{"trusted":true}},{"cell_type":"code","source":"# # First, we define the model, since we will be loading our own \"pretrained weights\", there is no need\n# # for one to set pretrained = True here, moreover, this competition is a No-Internet competition.\n\n# effnet_model = CustomEfficientNet(config, pretrained=False)\n\n\n# # The below code displays the parameter name, along with its size and tensors. \n# # Since the model is set to pretrained = False, the weights are initialized randomly (Xavier or something)\n# # However, if you set pretrained=True, the weights are fixed because it is already pretrained.\n\n# def count_parameters(model):\n# table = PrettyTable([\"Modules\", \"Size of Modules\", \"Number of Parameters\"])\n# total_params = 0\n# for param_name in model.state_dict():\n# param_tensor_size = model.state_dict()[param_name].size()\n# num_of_params = model.state_dict()[param_name].numel()\n# table.add_row([param_name, param_tensor_size, num_of_params])\n# total_params+=num_of_params\n# print(table)\n# print(f\"Total Trainable Params: {total_params}\")\n# return total_params\n \n# count_parameters(effnet_model)","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-12-12T06:17:36.629732Z","iopub.status.idle":"2021-12-12T06:17:36.630387Z","shell.execute_reply.started":"2021-12-12T06:17:36.630111Z","shell.execute_reply":"2021-12-12T06:17:36.630136Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Secondly, we use torch.load to load the pretrained weights that we trained on. Make sure the weights were\n# trained from the same EffficientNet, there was a time that I took a EfficientNetB3 weight and used it on\n# EfficientNetB0 model. It will not prompt you an error and disaster ensues.\n\n# state_dict_fold_1 = torch.load('../input/flowers/tf_efficientnet_b5_ns_fold_1_best_val.pt')","metadata":{"execution":{"iopub.status.busy":"2021-12-12T06:17:36.631518Z","iopub.status.idle":"2021-12-12T06:17:36.632101Z","shell.execute_reply.started":"2021-12-12T06:17:36.631837Z","shell.execute_reply":"2021-12-12T06:17:36.631864Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## CHECK EACH LAYER MATCHES IN SIZE.","metadata":{}},{"cell_type":"code","source":"# # Thirdly, state_dict_fold_1 is an OrderedDict, before we proceed, it is very important to call\n# # load_state_dict on the model! Because if not, we will get the same results as the previous \n# # count_parameters(effnet_model) when the model has yet to be trained!\n\n# from collections import OrderedDict\n\n# effnet_model.load_state_dict(state_dict_fold_1)\n# # note that after you load the weights using load_state_dict, the effnet_model.state_dict changes!\n\n# # Note that state_dict_fold_1 and effnet_model.state_dict() are equal!\n# # Check the keys of both dict matches\n# assert len(state_dict_fold_1) == len(effnet_model.state_dict())\n\n# count = 0\n# for param_name_1,param_name_2 in zip(state_dict_fold_1.keys(), effnet_model.state_dict().keys()):\n# if param_name_1 == param_name_2:\n# count+=1\n# assert count==len(state_dict_fold_1) == len(effnet_model.state_dict())\n\n# # ok the above code shows both have same param names, now to check if each tensor match:\n\n# tensor_match_count = 0\n# for param_name in state_dict_fold_1.keys() & effnet_model.state_dict().keys():\n# state_dict_fold_1_tensor = state_dict_fold_1[param_name].to(config.device)\n# effnet_model_state_dict = effnet_model.state_dict()[param_name].to(config.device)\n# if torch.all(torch.eq(state_dict_fold_1_tensor, effnet_model_state_dict)):\n# tensor_match_count+=1\n\n# assert tensor_match_count==count==len(state_dict_fold_1) == len(effnet_model.state_dict())\n# # so indeed all tensors matched.","metadata":{"execution":{"iopub.status.busy":"2021-12-12T06:17:36.633268Z","iopub.status.idle":"2021-12-12T06:17:36.633818Z","shell.execute_reply.started":"2021-12-12T06:17:36.633588Z","shell.execute_reply":"2021-12-12T06:17:36.633614Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"[References](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/175614)\n\n[Understanding TTA](https://stepup.ai/test_time_data_augmentation/)","metadata":{}},{"cell_type":"markdown","source":"### PyTorch Flashcard Questions\n\nWe create the `inference_by_fold` function; a step by step explanation based on my own test set is as follows:\n\n#### Qn 1: Why do we call `model.to(device)` in PyTorch?\n\n- We equip the `model` to `device`, telling PyTorch whether we are using GPU/CPU;\n\n#### Qn 2: Why do we set model to `eval()` mode in PyTorch during inference/validation?\n\n- Subsequently, set `model` to `eval()` mode as we are in inference phase; This is extremely important because if you DO NOT set it `eval` mode, then your model predictions during inference will be off. Why? Well I can spend the whole day explaining, but the simple idea is in your model, there are many **regularization** methods like `nn.Dropout()` or common `nn.BatchNorm1d(2d)` layers. Then during inference, if your model mode is `train`, then your predictions will experience DROPOUT as well. This will lead to different, and possibly worse predictions every time you inference.\n\n> What eval mode does to BatchNorm: During training, this layer keeps a running estimate of its computed mean and variance. The running sum is kept with a default momentum of 0.1. During evaluation, this running mean/variance is used for normalization.\n\n#### Qn 3: Why do we `@torch.no_grad` in PyTorch? \n\n- [Answer](https://stackoverflow.com/questions/63351268/torch-no-grad-affects-on-model-accuracy): Since we are in inference phase, we say `torch.no_grad()` because we are no longer computing or storing gradients anymore.","metadata":{}}]}