{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"\n# Image Classification with PyTorch and EfficientNetV2\n\n**Author:** Shingo Nakazawa ([@shnakazawa](https://twitter.com/shnakazawa))\n\n**Objective:** In this notebook, we will create a classification model using the [**EfficientNetV2 architecture**](https://arxiv.org/abs/2104.00298) with [**PyTorch**](https://pytorch.org/). ([A reference article (in Japanese)](https://zenn.dev/aidemy/articles/f851fb091dbb23))\n\nEfficientNetV2 models have achieved great performance on a wide range of computer vision tasks, including image classification, object detection, and semantic segmentation. We will use **a pretrained model provided by [timm](https://huggingface.co/docs/timm/index), a popular library for computer vision tasks**. This will allow us to take advantage of the knowledge and expertise that has gone into developing the models and apply them to our use case.\n\n\nPlease see also \n- [Object Detection with PyTorch and DETR](https://www.kaggle.com/code/shnakazawa/object-detection-with-pytorch-and-detr/notebook)\n- [Semantic Segmentation with PyTorch and U-Net](https://www.kaggle.com/code/shnakazawa/semantic-segmentation-with-pytorch-and-u-net)\n\n# Import modules","metadata":{"papermill":{"duration":0.008308,"end_time":"2023-03-07T07:10:11.815388","exception":false,"start_time":"2023-03-07T07:10:11.80708","status":"completed"},"tags":[]}},{"cell_type":"code","source":"import os\nimport numpy as np\nimport pandas as pd\nimport math\nimport time\nimport random\nimport gc\nimport cv2\nfrom pathlib import Path\nfrom tqdm.notebook import tqdm\nfrom sklearn.model_selection import StratifiedKFold\n\n# Image augmentation\nimport albumentations as A\nfrom albumentations.pytorch import ToTensorV2\n\n# Modeling\nimport torch\nimport torch.nn as nn\nimport torch.optim as optim\nfrom torch.utils.data import Dataset, DataLoader\nimport timm\n\n# Visualization\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n%matplotlib inline\n\nprint(f'PyTorch version {torch.__version__}')\nprint(f'Albumentations version {A.__version__}')","metadata":{"papermill":{"duration":4.505384,"end_time":"2023-03-07T07:10:16.340841","exception":false,"start_time":"2023-03-07T07:10:11.835457","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-10-15T18:04:25.714636Z","iopub.execute_input":"2023-10-15T18:04:25.715347Z","iopub.status.idle":"2023-10-15T18:04:29.094473Z","shell.execute_reply.started":"2023-10-15T18:04:25.715312Z","shell.execute_reply":"2023-10-15T18:04:29.093251Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Set Configs\n\nSeparatelly setting configs such as runnning objectives, file paths, hyperparameters, etc. is helpful in maintain a clear and organized workflow.\n\nWith clear configurations, it becomes easier to modify and fine-tune various aspects of the model. Moreover, it also helps in debugging the code, as each aspect can be tweaked and tested individually.","metadata":{"papermill":{"duration":0.006638,"end_time":"2023-03-07T07:10:16.354357","exception":false,"start_time":"2023-03-07T07:10:16.347719","status":"completed"},"tags":[]}},{"cell_type":"code","source":"RUN_EDA = True\nRUN_TRAINING = True\nTRAIN_ALL = False # If true, train with all data and output a single model. If False, run cross-validation and output multiple models.\nFOLD_NUM = 3 # For cross-validation\nEPOCHS = 5 # Training cycle\nRUN_INFERENCE = False\n\n# Directory setting\nDATA_DIR = '/kaggle/input/cassava-leaf-disease-classification/'\nMODEL_DIR = '/kaggle/working/'\nCSV_SAVE_DIR = '/kaggle/working/'\nIMG_SAVE_DIR = '/kaggle/working/' \n\n# PyTorch variables \n# Das hier sagt dir aus, welche Konfigurationen dein Netzwerk hat\nSEED = 42\nNUM_CLASSES = 3 # df.nunique()['label']n (waren ursprünglich 5 habe ich jetzt schon auf 3 runtergenommen, weil so viele braucht man da ja nicht)\nNUM_WORKERS = 2 #wie viele cpu kerne sind involviert, also das hat eigentlich keinen Einfluss auf das Ergebnis\nBATCH_SIZE = 8\n\nLR = 0.0001","metadata":{"papermill":{"duration":0.016255,"end_time":"2023-03-07T07:10:16.37999","exception":false,"start_time":"2023-03-07T07:10:16.363735","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-10-15T18:07:36.774363Z","iopub.execute_input":"2023-10-15T18:07:36.774754Z","iopub.status.idle":"2023-10-15T18:07:36.780908Z","shell.execute_reply.started":"2023-10-15T18:07:36.774719Z","shell.execute_reply":"2023-10-15T18:07:36.779662Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Define Helper Functions\n\nFunctions reused throughout the notebook should be defined at the top of the Notebook.\n\n- **Organization and Readability**: Having helper functions defined at the top of a notebook makes it easier to read and understand the overall structure of the code. It also makes it easier to find and modify the functions as needed.\n\n- **Debugging and Maintenance**: Defining helper functions separately allows for easier debugging and maintenance, as issues can be isolated to specific functions rather than having to comb through the entire notebook.\n\n- **Code Efficiency**: By defining helper functions separately, it's possible to reuse the same code for different tasks and avoid code duplication, which can lead to a more efficient and streamlined workflow.\n\n** Als nächstest werden nur ein Paar funktionen definert, die jetzt eher so mittel wichtig sind, aber schon auch praktisch sein können.\n","metadata":{"papermill":{"duration":0.006264,"end_time":"2023-03-07T07:10:16.394457","exception":false,"start_time":"2023-03-07T07:10:16.388193","status":"completed"},"tags":[]}},{"cell_type":"code","source":"def seed_everything(seed):\n    random.seed(seed)\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    np.random.seed(seed)\n    torch.manual_seed(seed)\n    torch.cuda.manual_seed(seed)\n    torch.backends.cudnn.deterministic = True\n    torch.backends.cudnn.benchmark = True\n    \n# Set seed\nseed_everything(SEED)\n\n\ndef show_gpu_memory(device):\n    print(f\"Allocated GPU memory: {torch.cuda.memory_allocated(device) / 1024 / 1024:.2f} MB\")\n    print(f\"Cached GPU memory: {torch.cuda.memory_cached(device) / 1024 / 1024:.2f} MB\")    \n\n    \ndef load_img(path):\n    img_bgr = cv2.imread(path)\n    img_rgb = img_bgr[:, :, ::-1]\n    return img_rgb\n\n\ndef create_gallery(array, ncols=3):\n    \"\"\"Display multiple images in a gallery style.\n    Source: https://www.amazon.co.jp/Data-Analysis-Machine-Learning-Kaggle-ebook/dp/B09F3STL34/\n    \n    Args:\n        array (numpy.ndarray): array of images.\n        ncols (int, optional): Num of columns. Defaults to 3.\n\n    Returns:\n        numpy.ndarray: One concatenated image.\n    \"\"\"    \n    nindex, height, width, intensity = array.shape\n    nrows = nindex//ncols\n    assert nindex == nrows * ncols\n    result = (array.reshape(nrows, ncols, height, width, intensity)\n        .swapaxes(1,2)\n        .reshape(height*nrows, width*ncols, intensity))\n    return result\n\n\ndef show_validation_score(train_acc_list, train_loss_list, valid_acc_list, valid_loss_list, save=False, save_dir=IMG_SAVE_DIR, save_name='classification_validation_score.png'):\n    fig = plt.figure(figsize=(15,15))\n    for i in range(FOLD_NUM):\n        train_acc = train_acc_list[i]\n        train_loss = train_loss_list[i]\n        valid_acc = valid_acc_list[i]\n        valid_loss = valid_loss_list[i]\n        \n        ax = fig.add_subplot(math.ceil(np.sqrt(FOLD_NUM))*2, math.ceil(np.sqrt(FOLD_NUM))*2, (i*2)+1, title=f'Fold {i+1}')\n        ax.plot(range(EPOCHS), train_acc, c='orange', label='train')\n        ax.plot(range(EPOCHS), valid_acc, c='blue', label='valid')\n        ax.set_xlabel('epoch')\n        ax.set_ylabel('accuracy')\n        ax.legend()\n\n        ax = fig.add_subplot(math.ceil(np.sqrt(FOLD_NUM))*2, math.ceil(np.sqrt(FOLD_NUM))*2, (i*2)+2, title=f'Fold {i+1}')\n        ax.plot(range(EPOCHS), train_loss, c='orange', label='train')\n        ax.plot(range(EPOCHS), valid_loss, c='blue', label='valid')\n        ax.set_xlabel('epoch')\n        ax.set_ylabel('loss')\n        ax.legend()\n    \n    plt.tight_layout()\n    if save:\n        os.makedirs(save_dir, exist_ok=True)\n        plt.savefig(save_dir+save_name)\n    else:\n        plt.show()","metadata":{"papermill":{"duration":0.027493,"end_time":"2023-03-07T07:10:16.428525","exception":false,"start_time":"2023-03-07T07:10:16.401032","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-10-15T18:09:44.253393Z","iopub.execute_input":"2023-10-15T18:09:44.253854Z","iopub.status.idle":"2023-10-15T18:09:44.289183Z","shell.execute_reply.started":"2023-10-15T18:09:44.253808Z","shell.execute_reply":"2023-10-15T18:09:44.288005Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Load Data","metadata":{"papermill":{"duration":0.006387,"end_time":"2023-03-07T07:10:16.441411","exception":false,"start_time":"2023-03-07T07:10:16.435024","status":"completed"},"tags":[]}},{"cell_type":"code","source":"df = pd.read_csv(DATA_DIR + 'train.csv') # ist jetzt eine vorgesetzte Variable, musst du dir noch auf deine Daten anpassen, deine Daten müssen da bei  Input mit rein\n# kannst du dir anschauen wie die gerade organisert sind und dann einfach so ähnlich organisieren. Wenn du drüberfährst kannst du auch direkt den Pfad kopieren.\n#besser schreibe auch:\nos.path.join(DATA_DIR,'train.csv')\n\n#csv bedeutet übirgens comma separated... das hat immer die Form\n#jedem Label wird eine Zahl zugeordnet\n# image_id, label\n#10001 , 0 \ndf.head()","metadata":{"papermill":{"duration":0.054464,"end_time":"2023-03-07T07:10:16.502318","exception":false,"start_time":"2023-03-07T07:10:16.447854","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-10-15T18:07:40.817412Z","iopub.execute_input":"2023-10-15T18:07:40.817786Z","iopub.status.idle":"2023-10-15T18:07:40.858373Z","shell.execute_reply.started":"2023-10-15T18:07:40.817753Z","shell.execute_reply":"2023-10-15T18:07:40.857348Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Exploratory Data Analysis (EDA)\n\nEDA for images is typically simpler than for tabular data.\n\n**Please set `RUN_EDA = True` in the `Set Config` section.**\n\n## Check the Distribution of Labels\n\nFirst, let's examine the images and their corresponding labels to see how many images are assigned to each label.","metadata":{"papermill":{"duration":0.006703,"end_time":"2023-03-07T07:10:16.515781","exception":false,"start_time":"2023-03-07T07:10:16.509078","status":"completed"},"tags":[]}},{"cell_type":"code","source":"# hier siehst du halt wie oft Ball und so weiter vorkommt in deinem Fall dann.\n# was auch immer Run_Eda genau ist, ist auf alle FÄlle oben als global variable mit definiert.\nif RUN_EDA:\n    print(df['label'].value_counts())\nelse:\n    print('RUN_EDA is False')","metadata":{"papermill":{"duration":0.018535,"end_time":"2023-03-07T07:10:16.541288","exception":false,"start_time":"2023-03-07T07:10:16.522753","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-10-15T18:07:56.940533Z","iopub.execute_input":"2023-10-15T18:07:56.941249Z","iopub.status.idle":"2023-10-15T18:07:56.953670Z","shell.execute_reply.started":"2023-10-15T18:07:56.941208Z","shell.execute_reply":"2023-10-15T18:07:56.952359Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Inspect Representative Images\n\nIt is also essential to visually inspect a sample of images. \n\nIf the images are noisy or weird, it may be necessary to apply some preprocessing steps before training a machine learning model.","metadata":{"papermill":{"duration":0.006549,"end_time":"2023-03-07T07:10:16.55469","exception":false,"start_time":"2023-03-07T07:10:16.548141","status":"completed"},"tags":[]}},{"cell_type":"code","source":"if RUN_EDA:\n    img_names = Path(DATA_DIR+'train_images/').glob('*.jpg')\n    img_list = []\n    for i, img_name in enumerate(img_names):\n        img_list.append(load_img(img_name.as_posix()))\n        print(f'{img_name.name}, label: {df[df[\"image_id\"]==img_name.name][\"label\"].to_numpy()[0]}')\n        if i == 5: \n            break\n    plt.figure(figsize=(10,10))\n    plt.imshow(create_gallery(np.array(img_list), ncols=3))\nelse:\n    print('RUN_EDA is False')","metadata":{"papermill":{"duration":1.906111,"end_time":"2023-03-07T07:10:18.467778","exception":false,"start_time":"2023-03-07T07:10:16.561667","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-10-15T18:09:47.780526Z","iopub.execute_input":"2023-10-15T18:09:47.781200Z","iopub.status.idle":"2023-10-15T18:09:48.849318Z","shell.execute_reply.started":"2023-10-15T18:09:47.781166Z","shell.execute_reply":"2023-10-15T18:09:48.848145Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Check Image Shape\n\nIt is also important to check the size and color of the images in the dataset. (The code below may take ~4 minutes.)\n\nCreating histograms can also be helpful.","metadata":{"papermill":{"duration":0.01328,"end_time":"2023-03-07T07:10:18.495701","exception":false,"start_time":"2023-03-07T07:10:18.482421","status":"completed"},"tags":[]}},{"cell_type":"code","source":"if RUN_EDA:\n    img_shape = set()\n    img_ext = set()\n    img_names = Path(DATA_DIR+'train_images/').glob('*')\n    pbar = tqdm(img_names, total=len(df))\n    for img_name in pbar:\n        img = load_img(img_name.as_posix())\n        img_shape.add(img.shape)\n        img_ext.add(img_name.suffix)\n    print(f'Image shapes are {img_shape}.')\n    print(f'Image extensions are {img_ext}.')\nelse:\n    print('RUN_EDA is False')","metadata":{"execution":{"iopub.execute_input":"2023-03-07T07:10:18.523653Z","iopub.status.busy":"2023-03-07T07:10:18.523332Z","iopub.status.idle":"2023-03-07T07:16:24.284168Z","shell.execute_reply":"2023-03-07T07:16:24.283105Z"},"papermill":{"duration":365.777485,"end_time":"2023-03-07T07:16:24.286329","exception":false,"start_time":"2023-03-07T07:10:18.508844","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We are confident that all the images in the dataset are 600 x 800 RGB jpg.","metadata":{"papermill":{"duration":0.013269,"end_time":"2023-03-07T07:16:24.313266","exception":false,"start_time":"2023-03-07T07:16:24.299997","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"## Plot histogram of pixel values\n\nCreating a histogram of pixel values can help identify any outlier images (such as those with all zeros). (The code belwo may take ~6 minutes.)","metadata":{"papermill":{"duration":0.012873,"end_time":"2023-03-07T07:16:24.339258","exception":false,"start_time":"2023-03-07T07:16:24.326385","status":"completed"},"tags":[]}},{"cell_type":"code","source":"if RUN_EDA:\n    img_names = Path(DATA_DIR+'train_images/').glob('*')\n    plt.figure(figsize=(10,10))\n    pbar = tqdm(img_names, total=len(df))\n    for img_name in pbar:\n        img = load_img(img_name.as_posix())\n        hist = cv2.calcHist([img],[0],None,[256],[0,256])\n        plt.plot(hist)\n    plt.show()\nelse:\n    print('RUN_EDA is False')","metadata":{"execution":{"iopub.execute_input":"2023-03-07T07:16:24.367858Z","iopub.status.busy":"2023-03-07T07:16:24.366986Z","iopub.status.idle":"2023-03-07T07:21:32.407861Z","shell.execute_reply":"2023-03-07T07:21:32.407012Z"},"papermill":{"duration":308.072446,"end_time":"2023-03-07T07:21:32.424894","exception":false,"start_time":"2023-03-07T07:16:24.352448","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Some images show weird peaks. It may be necessary to either remove or modify these images to ensure reliable results.\n\nWhile it is important to consider preprocessing steps to improve the accuracy of the model , for the purposes of simplicity, we will proceed to the next section without any preprocessing.","metadata":{"papermill":{"duration":0.013893,"end_time":"2023-03-07T07:21:32.452588","exception":false,"start_time":"2023-03-07T07:21:32.438695","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"# Define Model Components\n\nBefore training a model using PyTorch, the following steps need to be completed:\n\n1. Define Image Transformations and Augmentations\n2. Define the Dataset\n3. Define the DataLoader\n4. Define the Model\n\n## Define Image Transformations and Augmentations\n\nBefore creating the dataset, let's define the transformations and augmentations that will be applied to the images.\n\nHere, a package [Albumentations](https://albumentations.ai/) is used.\n","metadata":{"papermill":{"duration":0.013814,"end_time":"2023-03-07T07:21:32.480544","exception":false,"start_time":"2023-03-07T07:21:32.46673","status":"completed"},"tags":[]}},{"cell_type":"code","source":"# Image Augmentation\ndef transform_train():\n    transform = [\n        A.Resize(512,512,p=1),\n        A.HorizontalFlip(p=0.5),\n        A.ShiftScaleRotate(p=0.5),\n        A.CoarseDropout(p=0.5),\n        ToTensorV2(p=1.0)\n    ]\n    return A.Compose(transform)\n\n\n# Validation (and test) images should only be resized.\ndef transform_valid():\n    transform = [\n        A.Resize(512,512,p=1),\n        ToTensorV2(p=1.0)\n    ]\n    return A.Compose(transform)","metadata":{"execution":{"iopub.execute_input":"2023-03-07T07:21:32.510676Z","iopub.status.busy":"2023-03-07T07:21:32.509796Z","iopub.status.idle":"2023-03-07T07:21:32.51793Z","shell.execute_reply":"2023-03-07T07:21:32.517147Z"},"papermill":{"duration":0.025777,"end_time":"2023-03-07T07:21:32.520238","exception":false,"start_time":"2023-03-07T07:21:32.494461","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Define the Dataset\n\nUsing [`torch.datasets.ImageFolder`](https://pytorch.org/vision/main/generated/torchvision.datasets.ImageFolder.html) is an easier solution for preparing datasets. \n\nHowever, to better understand the process, we will define the datasets manually.","metadata":{"papermill":{"duration":0.01393,"end_time":"2023-03-07T07:21:32.549137","exception":false,"start_time":"2023-03-07T07:21:32.535207","status":"completed"},"tags":[]}},{"cell_type":"code","source":"# Dataset\nclass CassavaDataset(Dataset):\n    def __init__(self, df, data_root, transforms=None, give_label=True):\n        \"\"\"Performed only once when the Dataset object is instantiated.\n        give_label should be False for test data\n        \"\"\" \n        super().__init__()\n        self.df = df.reset_index(drop=True).copy()\n        self.data_root = data_root\n        self.transforms = transforms\n        self.give_label = give_label\n        \n        if give_label == True:\n            self.labels = self.df['label'].values\n\n    def __len__(self):\n        \"\"\"Function to return the number of records in the dataset\n        \"\"\" \n        return self.df.shape[0]\n    \n    def __getitem__(self, index):\n        \"\"\"Function to return samples corresponding to a given index from a dataset\n        \"\"\" \n        # get labels\n        if self.give_label:\n            target = self.labels[index]\n\n        # Load images\n        img  = load_img(f'{self.data_root}/{self.df.loc[index][\"image_id\"]}').astype(np.float32)\n        # img /= 255.0 # Normalization\n\n        # Transform images\n        if self.transforms:\n            img = self.transforms(image=img)['image']\n\n        if self.give_label == True:\n            return img, target\n        else:\n            return img","metadata":{"execution":{"iopub.execute_input":"2023-03-07T07:21:32.578652Z","iopub.status.busy":"2023-03-07T07:21:32.578353Z","iopub.status.idle":"2023-03-07T07:21:32.586648Z","shell.execute_reply":"2023-03-07T07:21:32.585673Z"},"papermill":{"duration":0.02549,"end_time":"2023-03-07T07:21:32.588711","exception":false,"start_time":"2023-03-07T07:21:32.563221","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Define the DataLoader","metadata":{"papermill":{"duration":0.013991,"end_time":"2023-03-07T07:21:32.61658","exception":false,"start_time":"2023-03-07T07:21:32.602589","status":"completed"},"tags":[]}},{"cell_type":"code","source":"# DataLoader\ndef create_dataloader(df, trn_idx, val_idx):\n    train_ = df.loc[trn_idx,:].reset_index(drop=True)\n    valid_ = df.loc[val_idx,:].reset_index(drop=True)\n\n    # Dataset\n    train_datasets = CassavaDataset(train_, DATA_DIR+'train_images/', transforms=transform_train())\n    valid_datasets = CassavaDataset(valid_, DATA_DIR+'train_images/', transforms=transform_valid())\n\n    # Data Loader\n    train_loader = DataLoader(train_datasets, batch_size=BATCH_SIZE, num_workers=NUM_WORKERS, shuffle=True, multiprocessing_context='fork')\n    valid_loader = DataLoader(valid_datasets, batch_size=BATCH_SIZE, num_workers=NUM_WORKERS, shuffle=False, multiprocessing_context='fork')\n\n    return train_loader, valid_loader","metadata":{"execution":{"iopub.execute_input":"2023-03-07T07:21:32.646857Z","iopub.status.busy":"2023-03-07T07:21:32.645971Z","iopub.status.idle":"2023-03-07T07:21:32.652833Z","shell.execute_reply":"2023-03-07T07:21:32.651938Z"},"papermill":{"duration":0.02415,"end_time":"2023-03-07T07:21:32.654947","exception":false,"start_time":"2023-03-07T07:21:32.630797","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Define the Model\n\nIn this notebook, we will use a pre-trained EfficientNetV2 model prepared by `timm`.","metadata":{"papermill":{"duration":0.014259,"end_time":"2023-03-07T07:21:32.683543","exception":false,"start_time":"2023-03-07T07:21:32.669284","status":"completed"},"tags":[]}},{"cell_type":"code","source":"class EfficientNet_V2(nn.Module):\n    def __init__(self, n_out):\n        super(EfficientNet_V2, self).__init__()\n        # Define model\n        self.effnet = timm.create_model('efficientnetv2_s', pretrained=True, num_classes=n_out)\n\n    def forward(self, x):\n        return self.effnet(x)","metadata":{"execution":{"iopub.execute_input":"2023-03-07T07:21:32.714369Z","iopub.status.busy":"2023-03-07T07:21:32.713472Z","iopub.status.idle":"2023-03-07T07:21:32.719573Z","shell.execute_reply":"2023-03-07T07:21:32.718596Z"},"papermill":{"duration":0.023832,"end_time":"2023-03-07T07:21:32.721645","exception":false,"start_time":"2023-03-07T07:21:32.697813","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Run Training\n\nTo create an effective deep learning model, it's crucial to determine the optimal architecture and hyperparameters. \n\nOne common approach for achieving this is to use **cross-validation**, a technique that involves dividing the available data into multiple subsets and using each subset in turn to train and validate the model. By **comparing the results obtained from different archigetcures and hyperparameters**, we can **identify the optimal combination** that maximizes the model's performance. To effectively track the model's performance, I recommend utilizing **MLOps tools like [MLFlow](https://mlflow.org/)**.\n\n**Once the optimal architecture and hyperparameters have been identified** through cross-validation, we can proceed to **train the model on all available data** using the fixed architecture and hyperparameters.\n\nIt's worth noting that there is some debate over whether a single model should be re-trained on all available data or if an ensemble of models created during each round of cross-validation should be used instead. The choice between these options will depend on the specific use case and available resources.\n\n- [StackExchange - How to choose a predictive model after k-fold cross-validation?](https://stats.stackexchange.com/questions/52274/how-to-choose-a-predictive-model-after-k-fold-cross-validation)\n- [StackOverflow - k-fold cross validation model selection method](https://stackoverflow.com/questions/46860325/k-fold-cross-validation-model-selection-method)\n\n## Cross-Validation\n\n**Before running the following cell, please set `RUN_TRAINING = True` and `TRAIN_ALL = False` in the `Set Config` section.**","metadata":{"papermill":{"duration":0.014237,"end_time":"2023-03-07T07:21:32.749885","exception":false,"start_time":"2023-03-07T07:21:32.735648","status":"completed"},"tags":[]}},{"cell_type":"code","source":"if RUN_TRAINING and (TRAIN_ALL == False):\n    # Set device\n    device = torch.device('cuda' if torch.cuda.is_available() else 'cpu') \n    print(f'Using {device} device')\n    \n    # Cross-validation\n    folds = StratifiedKFold(n_splits=FOLD_NUM, shuffle=True, random_state=SEED)\\\n            .split(np.arange(df.shape[0]), df['label'].to_numpy())\n\n    # For Visualization\n    train_acc_list = []\n    valid_acc_list = []\n    train_loss_list = []\n    valid_loss_list = []\n\n\n    for fold, (trn_idx, val_idx) in enumerate(folds):\n        print(f'==========Cross Validation Fold {fold+1}==========')\n        # Load Data\n        train_loader, valid_loader = create_dataloader(df, trn_idx, val_idx)\n\n        # Load model, loss function, and optimizing algorithm\n        model = EfficientNet_V2(NUM_CLASSES).to(device)\n        loss_fn = nn.CrossEntropyLoss().to(device)\n        optimizer = optim.Adam(model.parameters(), lr=LR)\n                \n        # For Visualization\n        train_accs = []\n        valid_accs = []\n        train_losses = []\n        valid_losses = []\n\n        # Start training\n        best_acc = 0\n        for epoch in range(EPOCHS):\n            time_start = time.time()\n            print(f'==========Epoch {epoch+1} Start Training==========')\n            model.train()\n            \n            epoch_loss = 0\n            epoch_accuracy = 0\n        \n            pbar = tqdm(enumerate(train_loader), total=len(train_loader))\n            for step, (img, label) in pbar:\n                img = img.to(device).float()\n                label = label.to(device).long()\n\n                output = model(img)\n                loss = loss_fn(output, label)\n\n                optimizer.zero_grad()\n                loss.backward()\n                optimizer.step()\n\n                acc = (output.argmax(dim=1) == label).float().mean()\n                epoch_accuracy += acc / len(train_loader)\n                epoch_loss += loss / len(train_loader)\n\n            print(f'==========Epoch {epoch+1} Start Validation==========')\n            with torch.no_grad():\n                epoch_val_accuracy = 0\n                epoch_val_loss = 0\n                val_labels = []\n                val_preds = []\n\n                pbar = tqdm(enumerate(valid_loader), total=len(valid_loader))\n                for step, (img, label) in pbar:\n                    img = img.to(device).float()\n                    label = label.to(device).long()\n\n                    val_output = model(img)\n                    val_loss = loss_fn(val_output, label)\n\n                    acc = (val_output.argmax(dim=1) == label).float().mean()\n                    epoch_val_accuracy += acc / len(valid_loader)\n                    epoch_val_loss += val_loss / len(valid_loader)\n\n                    val_labels += [label.detach().cpu().numpy()]\n                    val_preds += [torch.argmax(val_output, 1).detach().cpu().numpy()]\n                \n                val_labels = np.concatenate(val_labels)\n                val_preds = np.concatenate(val_preds)\n            \n            # print result from this epoch\n            exec_t = int((time.time() - time_start)/60)\n            print(\n                f'Epoch : {epoch+1} - loss : {epoch_loss:.4f} - acc: {epoch_accuracy:.4f} - val_loss : {epoch_val_loss:.4f} - val_acc: {epoch_val_accuracy:.4f} / Exec time {exec_t} min\\n'\n            )\n\n            # For visualization\n            train_accs.append(epoch_accuracy.cpu().numpy())\n            valid_accs.append(epoch_val_accuracy.cpu().numpy())\n            train_losses.append(epoch_loss.detach().cpu().numpy())\n            valid_losses.append(epoch_val_loss.detach().cpu().numpy())\n        \n        train_acc_list.append(train_accs)\n        valid_acc_list.append(valid_accs)\n        train_loss_list.append(train_losses)\n        valid_loss_list.append(valid_losses)\n        del model, optimizer, train_loader, valid_loader, train_accs, valid_accs, train_losses, valid_losses\n        gc.collect()\n        torch.cuda.empty_cache()\n\n    show_validation_score(train_acc_list, train_loss_list, valid_acc_list, valid_loss_list)\n\nelse:\n    print('Cross validation is not performed')\n","metadata":{"execution":{"iopub.execute_input":"2023-03-07T07:21:32.78044Z","iopub.status.busy":"2023-03-07T07:21:32.779552Z","iopub.status.idle":"2023-03-07T10:27:56.764997Z","shell.execute_reply":"2023-03-07T10:27:56.764005Z"},"papermill":{"duration":11184.00342,"end_time":"2023-03-07T10:27:56.767501","exception":false,"start_time":"2023-03-07T07:21:32.764081","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Loss is still decreasing. In practical terms, it is advisable to continue learning until the loss reaches a plateau. \n\nHere, we will go on the next step with this “incomplete” setting.","metadata":{"papermill":{"duration":0.023441,"end_time":"2023-03-07T10:27:56.815061","exception":false,"start_time":"2023-03-07T10:27:56.79162","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"## Training on All Data\n\nAfter identifying **the optimal number of epochs and hyperparameters** through cross-validation, the final step is to **use this information to train the model on all available data**.\n\nTo simplify the process and avoid errors, **combining the code for cross-validation and training on all data into a single script can be helpful**.  This approach allows for a more efficient workflow and simplifies the process of transitioning from cross-validation to training with all  data.\n\n**Before running the following cell, please set `RUN_TRAINING = True` and `TRAIN_ALL = True` in the `Set Config` section.**","metadata":{"papermill":{"duration":0.023014,"end_time":"2023-03-07T10:27:56.861374","exception":false,"start_time":"2023-03-07T10:27:56.83836","status":"completed"},"tags":[]}},{"cell_type":"code","source":"if RUN_TRAINING and TRAIN_ALL:\n    # Set device\n    device = torch.device('cuda' if torch.cuda.is_available() else 'cpu') \n    print(f'Using {device} device')\n\n    # Load Data\n    train_datasets = CassavaDataset(df, DATA_DIR+'train_images/', transforms=transform_train())\n    train_loader = DataLoader(train_datasets, batch_size=BATCH_SIZE, num_workers=NUM_WORKERS, shuffle=True, multiprocessing_context='fork')\n    \n\n    # Load model, loss function, and optimizing algorithm\n    model = EfficientNet_V2(NUM_CLASSES).to(device)\n    loss_fn = nn.CrossEntropyLoss().to(device)\n    optimizer = optim.Adam(model.parameters(), lr=LR)\n            \n    # Start training\n    for epoch in range(EPOCHS):\n        time_start = time.time()\n        print(f'==========Epoch {epoch+1} Start Training==========')\n        model.train()\n        \n        epoch_loss = 0\n        epoch_accuracy = 0\n    \n        pbar = tqdm(enumerate(train_loader), total=len(train_loader))\n        for step, (img, label) in pbar:\n            img = img.to(device).float()\n            label = label.to(device).long()\n\n            output = model(img)\n            loss = loss_fn(output, label)\n\n            optimizer.zero_grad()\n            loss.backward()\n            optimizer.step()\n\n            acc = (output.argmax(dim=1) == label).float().mean()\n            epoch_accuracy += acc / len(train_loader)\n            epoch_loss += loss / len(train_loader)\n                  \n        # print results from this epoch\n        exec_t = int((time.time() - time_start)/60)\n        print(\n            f'Epoch : {epoch+1} - loss : {epoch_loss:.4f} - acc: {epoch_accuracy:.4f} / Exec time {exec_t} min\\n'\n        )\n\n    print(f'Save model trained with all data')\n    os.makedirs(MODEL_DIR, exist_ok=True)\n    torch.save(model.state_dict(), MODEL_DIR+'classification.pth')       \n                 \n    del model, optimizer, train_loader\n    gc.collect()\n    torch.cuda.empty_cache()\n\nelse:\n    print('Training with all data is not performed')","metadata":{"execution":{"iopub.execute_input":"2023-03-07T10:27:56.909764Z","iopub.status.busy":"2023-03-07T10:27:56.909155Z","iopub.status.idle":"2023-03-07T10:27:56.9219Z","shell.execute_reply":"2023-03-07T10:27:56.920894Z"},"papermill":{"duration":0.039363,"end_time":"2023-03-07T10:27:56.923889","exception":false,"start_time":"2023-03-07T10:27:56.884526","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"By creating a final model using the parameters obtained from cross-validation, we could be confident that our deep learning model is optimized for the specific task at hand. \n\nHowever, it's important to keep in mind that deep learning models are not a one-size-fits-all solution and may **need to be adjusted or retrained as new data becomes available or the task requirements change**.\nThis is particularly important to consider when tackling your own projects.","metadata":{"papermill":{"duration":0.023179,"end_time":"2023-03-07T10:27:56.970223","exception":false,"start_time":"2023-03-07T10:27:56.947044","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"# Run Inference of Test Data\n\nIn this task, the same Dataset, Image transformation, and model as we used for validation. So no need to define new functions.\n\n**Before running the following cells, please set `RUN_INFERENCE = True` in the `Set Config` section.**","metadata":{"papermill":{"duration":0.02304,"end_time":"2023-03-07T10:27:57.016324","exception":false,"start_time":"2023-03-07T10:27:56.993284","status":"completed"},"tags":[]}},{"cell_type":"code","source":"if RUN_INFERENCE:\n    submission_df = pd.DataFrame()\n    submission_df['image_id'] = list(os.listdir(DATA_DIR+'test_images/'))\n\n    device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')  \n    print(f'Using {device} device')\n\n    # Load Data\n    test_datasets = CassavaDataset(submission_df, DATA_DIR+'test_images/', transforms=transform_valid(), give_label=False)\n\n    # Data Loader\n    test_loader = DataLoader(test_datasets, batch_size=BATCH_SIZE, num_workers=NUM_WORKERS, shuffle=False, multiprocessing_context='fork')\n\n    # Load model, loss function, and optimizing algorithm\n    model = EfficientNet_V2(NUM_CLASSES).to(device)\n    model.load_state_dict(torch.load(MODEL_DIR+'classification.pth'))\n\n    # Start Inference\n    print(f'==========Start Inference==========')\n    with torch.no_grad():\n        test_preds = []\n        pbar = tqdm(enumerate(test_loader), total=len(test_loader))\n        for step, img in pbar:\n            img = img.to(device).float()\n            test_output = model(img)\n            test_preds += [torch.argmax(test_output, 1).detach().cpu().numpy()]\n        test_preds = np.concatenate(test_preds)\n    submission_df['label'] = test_preds\n    submission_df.head()\n\nelse:\n    print('RUN_INFERENCE is False')","metadata":{"execution":{"iopub.execute_input":"2023-03-07T10:27:57.064869Z","iopub.status.busy":"2023-03-07T10:27:57.064008Z","iopub.status.idle":"2023-03-07T10:27:57.074217Z","shell.execute_reply":"2023-03-07T10:27:57.072912Z"},"papermill":{"duration":0.036616,"end_time":"2023-03-07T10:27:57.076333","exception":false,"start_time":"2023-03-07T10:27:57.039717","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Save inference result","metadata":{"papermill":{"duration":0.024656,"end_time":"2023-03-07T10:27:57.125055","exception":false,"start_time":"2023-03-07T10:27:57.100399","status":"completed"},"tags":[]}},{"cell_type":"code","source":"if RUN_INFERENCE:\n    submission_df.to_csv('submission.csv', index=False)\n    \nelse:\n    print('RUN_INFERENCE is False')","metadata":{"execution":{"iopub.execute_input":"2023-03-07T10:27:57.175209Z","iopub.status.busy":"2023-03-07T10:27:57.174462Z","iopub.status.idle":"2023-03-07T10:27:57.180167Z","shell.execute_reply":"2023-03-07T10:27:57.179157Z"},"papermill":{"duration":0.033558,"end_time":"2023-03-07T10:27:57.182579","exception":false,"start_time":"2023-03-07T10:27:57.149021","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# To Improve the Model\n\nOur classification model is now on the starting point. There are many techniques we can use to improve its accuracy.\n\nSome potential approaches to consider include:\n\n- **Image preprocessing**\n    - Dimensionality reduction, feature extraction, removal of unnecessary background\n    - Correction or removal of mislabeled images\n    - Review of augmentation\n        - Increase the patterns generated\n        - **Check if any strange images are generated as a result of augmentation**\n- **Handling imbalanced data**\n    - Up-sampling/down-sampling \n    - Changing the loss function\n        - [This loss function](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/173733) was commonly used in this competition\n- Increase the image size beyon the current 224 x 224 pixels\n- **Model optimization**\n    - Switching to larger architectures such as 'efficientnetv2_l'\n    - Applying different architectures\n    - Increasing input size\n    - Tuning hyperparameters\n- **Feedback from prediction results**\n    - **Creating a confusion matrix** from validation results\n    - Identifying mislabeled data and labels that are prone to errors\n- **Ensemble of multiple models**\n- Checking for bugs\n- etc...\n\nBy evaluating the performance of each approach carefully, we can identify the best approach to improve the accuracy of our model for our specific task and dataset.\n\nI hope this notebook has been helpful in building a classification model, and I wish you the best of luck in your modeling life!\n\n# Acknowledgement\n\nI thank to [Matchan from YouTube Channel - Engawa AI Research Institute](https://www.youtube.com/channel/UCRwO-ewBHhNiC4qBEppi_JQ) and Mr. Masaaki Aiba for critical reviews and valuable discussion.\n\n# References\n\n- [EfficientNetV2: Smaller Models and Faster Training](https://arxiv.org/abs/2104.00298) \n- [PyTorch](https://pytorch.org/)\n\t- [PyTorch - SAVING AND LOADING MODELS](https://pytorch.org/tutorials/beginner/saving_loading_models.html)\n- [Albumentations](https://albumentations.ai/)\n- [huggingface/timm](https://huggingface.co/docs/timm/index)\n- [sklearn.model_selection.StratifiedKFold](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.StratifiedKFold.html)\n- [Kaggle Cassava Leaf Disease Classification competition](https://www.kaggle.com/c/cassava-leaf-disease-classification)\n- [Pytorch Efficientnet Baseline [Train] AMP+Aug](https://www.kaggle.com/code/khyeh0719/pytorch-efficientnet-baseline-train-amp-aug)\n- [Pytorch Efficientnet Baseline [Inference] TTA](https://www.kaggle.com/code/khyeh0719/pytorch-efficientnet-baseline-inference-tta)\n- [StackExchange - How to choose a predictive model after k-fold cross-validation?](https://stats.stackexchange.com/questions/52274/how-to-choose-a-predictive-model-after-k-fold-cross-validation)\n- [StackOverflow - k-fold cross validation model selection method](https://stackoverflow.com/questions/46860325/k-fold-cross-validation-model-selection-method)","metadata":{"papermill":{"duration":0.024602,"end_time":"2023-03-07T10:27:57.231429","exception":false,"start_time":"2023-03-07T10:27:57.206827","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"","metadata":{}}]}