{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# 🐳 Happywhale - Whale and Dolphin Identification on Kaggle","metadata":{"id":"voi5Q1b4tlPv"}},{"cell_type":"markdown","source":"**Goal: reach 0.50 and release code on public**\n\n**[📈 WandB Experiment Record for this Notebook](https://wandb.ai/snoop2head/HappyWhale)**\n\n### Done\n- ~~Notate top k 5 accuracy metrics~~\n- ~~Integrate wandb~~\n- ~~Compare Crossentropy and FocalLoss -> FocalLoss turned out to be better~~\n- ~~Stack grayscale images from [height, width] to [h, w, 3] image~~\n- ~~stratified Kfold for training~~\n- ~~stratified Kfold for Out of Fold inference~~","metadata":{}},{"cell_type":"markdown","source":"### To-Do\n**Train**\n- Apply Arcface Head (or Loss)\n- Check how AMP model.half() affects performance and time saving\n- **Contrastive Learning using negative batching from the same species**\n- Find optimal learning rate using efficientnetB0\n\n**Inference**\n- Inference Method with KNN\n- Set threshold for new_individual in inference","metadata":{}},{"cell_type":"markdown","source":"### Dependencies Installation","metadata":{"id":"dw4kDbaIWAEV"}},{"cell_type":"code","source":"from IPython.display import clear_output\n!pip install albumentations==0.4.6\n!pip install timm\n!pip install adamp\n!pip install wandb\nclear_output()","metadata":{"id":"0QZ3KD036xgA"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Define Train / Validation Configuration\n\n**You may restart runtime from here**\n- Factory reset runtime deletes files attached to colab and disconnects from machine\n- Resetting runtime only removes variables from the instance but keeps connection to the server device","metadata":{"id":"fYFTKxsl4bUs"}},{"cell_type":"code","source":"import os\nimport wandb\nwandb.login()\nCFG = wandb.config # wandb.config provides functionality of easydict.EasyDict\n\nCFG.DEBUG = False\n\nprint(f\"YOU ARE WORKING ON DEBUG = {CFG.DEBUG} MODE\")","metadata":{"id":"lM7CJxPF4WdC","outputId":"de7b165f-005d-45bc-ed15-33304809cd0c"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"CFG.learning_rate = 3e-4 # Efficientnet Learning rate\nCFG.weight_decay = 3e-6 # default weight decay ratio is 0.1(https://github.com/clovaai/AdamP), but using 0.01 for small dataset \n# CFG.learning_rate = 1e-5 # swin transformers learning rate for finetuning: In ImageNet-1K fine-tuning, we train the models for 30 epochs with a batch size of 1024, a constant learning rate of 10−5, \n# CFG.weight_decay = 1e-8 # and a weightdecay of 10−8.\nCFG.gamma = 0.5 # gamma for focal loss\nCFG.image_resolution = 512\nCFG.input_resolution = 512\nCFG.rgb_mean = [0.43818492, 0.49098103, 0.54812671]\nCFG.rgb_sd = [0.16021039, 0.16227327, 0.16930125]\nCFG.train_batch_size = 22\nCFG.val_batch_size = 16\nCFG.num_epochs = 9\nCFG.split_ratio = 0.0\nCFG.num_folds = 5\nCFG.logging_steps = 900\n\n# overwrite configuration when debug mode\nif CFG.DEBUG:\n  CFG.image_resolution = 512\n  CFG.input_resolution = 384\n  # EfficientNetB0 & 224 resolution & batch_size 400 = 40GB on GPU -> 224 resolution is fast but not high enough to get resolution\n  # EfficientNetB0 & 384 resolution & batch_size 128 = 38.4GB on GPU\n  CFG.train_batch_size = 128 # bigger the batch, faster the iteration.\n  CFG.val_batch_size = 64 #\n  CFG.num_epochs = 10\n  \nprint(f\"YOU ARE WORKING ON DEBUG = {CFG.DEBUG} MODE\")","metadata":{"id":"t1aGUQtY4aIb","outputId":"0fec89a4-7e64-463a-ca77-9d521838edb0"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Get dataset and Define Paths","metadata":{"id":"CMjqrjkXoE1v"}},{"cell_type":"code","source":"# get current working directory\nimport os\nos.environ[\"CUDA_LAUNCH_BLOCKING\"] = \"1\"\nPROJECT_PATH = os.getcwd()\nPROJECT_PATH","metadata":{"id":"c4UaFP_J6aWI","outputId":"3513a7f7-18d0-458c-d04a-98d083a3da50"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ROOT_DIR = os.path.join(\n    os.getcwd(), \n    f\"{CFG.image_resolution}x{CFG.image_resolution}_resized_dataset\"\n)\n# TRAIN_DIR = os.path.join(ROOT_DIR, \"train_images\")\nTRAIN_DIR = ROOT_DIR\nos.path.exists(ROOT_DIR)","metadata":{"id":"SH6Rp3CT6c9J","outputId":"9a5d394d-8d08-456a-d6e4-309ad4838eea"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if not os.path.exists(ROOT_DIR):\n    from google.colab import drive\n    drive.mount('/content/drive')","metadata":{"id":"n9RrMy6BoKP3"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import shutil\nif not os.path.exists(ROOT_DIR):\n    # get train dataset from google drive\n    shutil.copy(f\"/content/drive/MyDrive/HappyWhale/data/{CFG.image_resolution}x{CFG.image_resolution}_resized_dataset.zip\", \"./\")\n\n    # get test dataset\n    shutil.copy(f\"/content/drive/MyDrive/HappyWhale/data/{CFG.image_resolution}x{CFG.image_resolution}_resized_test_dataset.zip\", \"./\")\n\n    # copy csv files\n    shutil.copy(\"/content/drive/MyDrive/HappyWhale/data/sample_submission.csv\", f\"./{CFG.image_resolution}x{CFG.image_resolution}_resized_dataset\")\n    shutil.copy(\"/content/drive/MyDrive/HappyWhale/data/train.csv\", f\"./{CFG.image_resolution}x{CFG.image_resolution}_resized_dataset\")","metadata":{"id":"90FS6FJUoPhs"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# unzip train dataset\nimport os\nfrom IPython.display import clear_output\n\nif not os.path.exists(ROOT_DIR):\n    TRAIN_ZIP_FILE_PATH = f\"/content/{CFG.image_resolution}x{CFG.image_resolution}_resized_dataset.zip\"\n    TEST_ZIP_FILE_PATH = f\"/content/{CFG.image_resolution}x{CFG.image_resolution}_resized_test_dataset.zip\"\n    !unzip $TRAIN_ZIP_FILE_PATH\n    !unzip $TEST_ZIP_FILE_PATH\n    os.remove(TRAIN_ZIP_FILE_PATH)\n    os.remove(TEST_ZIP_FILE_PATH)\n    clear_output()","metadata":{"id":"QarGxqb33bu7"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ROOT_DIR","metadata":{"id":"UZiWvgiaM6Yk","outputId":"d68f8fe1-1510-4000-e676-2684383acf92"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if os.path.exists(ROOT_DIR):\n    from google.colab import drive\n    drive.flush_and_unmount()","metadata":{"id":"6PlH7q1zooFE","outputId":"1acd6b71-3c4c-476a-ac96-38ae117cb113"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Read training labels","metadata":{"id":"L5taGIty6szd"}},{"cell_type":"code","source":"import random\nimport torch\nimport numpy as np\n\ndef seed_everything(seed) :\n    torch.manual_seed(seed)\n    torch.cuda.manual_seed(seed)\n    torch.cuda.manual_seed_all(seed) # if use multi-GPU\n    torch.backends.cudnn.deterministic = True\n    torch.backends.cudnn.benchmark = False\n    np.random.seed(seed)\n    random.seed(seed)\nseed_everything(42)","metadata":{"id":"YemdO_s_Jqh8"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\npath_label = os.path.join(ROOT_DIR, \"train.csv\")\nlabel = pd.read_csv(path_label)\nlabel.head()","metadata":{"execution":{"iopub.execute_input":"2022-02-08T01:01:04.380581Z","iopub.status.busy":"2022-02-08T01:01:04.380294Z","iopub.status.idle":"2022-02-08T01:01:04.464329Z","shell.execute_reply":"2022-02-08T01:01:04.46311Z","shell.execute_reply.started":"2022-02-08T01:01:04.380538Z"},"id":"wtGpm3GroE1y","outputId":"71e6f316-8ab2-46f4-cd76-bd990ba3b02d","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_train_file_path(id):\n    return f\"{TRAIN_DIR}/{id}\"\n\ndf = pd.read_csv(path_label)\ndf['file_path'] = df['image'].apply(get_train_file_path)\ndf.head()","metadata":{"execution":{"iopub.execute_input":"2022-02-08T01:02:25.602825Z","iopub.status.busy":"2022-02-08T01:02:25.601804Z","iopub.status.idle":"2022-02-08T01:02:25.717128Z","shell.execute_reply":"2022-02-08T01:02:25.716516Z","shell.execute_reply.started":"2022-02-08T01:02:25.602744Z"},"id":"i9tWCvuToE1z","outputId":"439a3160-c846-49a2-fe0e-013341edfb97","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"species = df.species.unique().tolist()\nprint('Number of species in the dataset:', len(species))","metadata":{"execution":{"iopub.execute_input":"2022-02-08T00:47:03.013814Z","iopub.status.busy":"2022-02-08T00:47:03.01346Z","iopub.status.idle":"2022-02-08T00:47:03.03054Z","shell.execute_reply":"2022-02-08T00:47:03.029852Z","shell.execute_reply.started":"2022-02-08T00:47:03.013781Z"},"id":"sxpQhjgUoE1z","outputId":"e05b8f51-81e7-46bc-8ef6-b05ebf9a74cf","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"individual_ids = df.individual_id.unique().tolist()\nprint('Number of Mr and Mrs Dolphins/Whales in the dataset:', len(individual_ids))","metadata":{"id":"ZdUTfL1_RE2L","outputId":"aacef77d-b9dd-4777-f9b8-398483613969"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"path_submission = os.path.join(ROOT_DIR, \"sample_submission.csv\")\ndf_sample_submission = pd.read_csv(path_submission)\ndf_sample_submission.head()","metadata":{"execution":{"iopub.execute_input":"2022-02-08T00:53:29.544135Z","iopub.status.busy":"2022-02-08T00:53:29.543556Z","iopub.status.idle":"2022-02-08T00:53:29.593168Z","shell.execute_reply":"2022-02-08T00:53:29.592166Z","shell.execute_reply.started":"2022-02-08T00:53:29.544099Z"},"id":"FTXj3W4doE11","outputId":"2255eb3f-eb86-4009-d296-10aa10d9f56a","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Encoding invidiaul_ids(string object) to integer labels with Labelencoder","metadata":{"id":"WE6eW5AOoE11"}},{"cell_type":"code","source":"from sklearn.preprocessing import LabelEncoder\n\n# encode object string label into integer label mapping\n# https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.LabelEncoder.html\nle_species = LabelEncoder()\nle_species.fit(df.species)\ndf.species = le_species.transform(df.species)\ndf.head()","metadata":{"id":"S0LL6u6joE11","outputId":"41513a8b-cb95-405e-b71a-628acfec8364"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.preprocessing import LabelEncoder\n\nle_individual_id = LabelEncoder()\nle_individual_id.fit(df.individual_id)\ndf.individual_id = le_individual_id.transform(df.individual_id)\ndf.head()","metadata":{"id":"2i8YH8d1Re0k","outputId":"e8859a09-fff9-4c1c-d873-e4fcdb3570cf"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# visualize species value counts distribution with histogram\nprint(df.individual_id.value_counts().values[:100])","metadata":{"id":"haK6PxxaoE12","outputId":"81836b83-eecf-4ce6-902e-f711440198b6"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# visualize species value counts distribution with histogram\nprint(df.individual_id.value_counts().values[-100:])","metadata":{"id":"l46R53FUFkbp","outputId":"95b7c094-f0d6-4104-f0af-5c0c3532ad16"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import joblib\n\nwith open(\"le_individual_id.pkl\", \"wb\") as fp:\n    joblib.dump(le_individual_id, fp)","metadata":{"id":"Faz_1fgJoE12"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Set Dataset Class","metadata":{"id":"FkSgVPU6oE13"}},{"cell_type":"code","source":"import torch\nfrom torch.utils.data import Dataset, DataLoader\nfrom PIL import Image\n\nclass HappyWhaleDataset(Dataset):\n    def __init__(self, df, transforms=None):\n        self.df = df\n        self.file_names = df.file_path.values\n        self.species = df.species.values\n        self.labels = df.individual_id.values\n        self.transforms = transforms\n    \n    def __getitem__(self, index):\n        image_path = self.file_names[index]\n        image = Image.open(image_path)\n        image = np.array(image)\n        # take care of grayscale image by adding channel\n        if len(image.shape) == 2: \n          image = np.dstack((image,)*3)\n\n        label = self.labels[index]\n        \n        if self.transforms:\n            image_transform = self.transform(image=image)['image']\n            return image_transform, label\n        else:\n            return image, label\n\n    def __len__(self):\n        return len(self.df)\n    \n    def set_transform(self, transform):\n        self.transform = transform","metadata":{"id":"9PBpp2-HoE13"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from albumentations import *\nfrom albumentations.pytorch import ToTensorV2\n\n\ndef get_transforms(\n    need=('train', 'val'), \n    img_size=(CFG.input_resolution, CFG.input_resolution), \n    mean= [0.43818492, 0.49098103, 0.54812671],\n    std= [0.16021039, 0.16227327,0.16930125]\n    ):\n    # https://vfdev-5-albumentations.readthedocs.io/en/docs_pytorch_fix/api/augmentations.html\n    transformations = {}\n    if 'train' in need:  \n        transformations['train'] = Compose([\n            Resize(img_size[0], img_size[1], p=1.0),\n            # CenterCrop(height = CFG.input_resolution, width = CFG.input_resolution), # add centercrop\n            \n            # shape augmentation\n            HorizontalFlip(p=0.5),\n            ShiftScaleRotate(p=0.5), ## NEED TO CHECK WHETHER THIS IS GOOD OR NOT. IF PERFORMANCE DROPS EVEN AFTER CHANGING THE MODEL TO EFFB5, THIS MIGHT BE THE REASON\n\n            # pixel level augmentation\n            HueSaturationValue(hue_shift_limit=0.2, sat_shift_limit=0.2, val_shift_limit=0.2, p=0.2),\n            RandomBrightnessContrast(brightness_limit=(-0.1, 0.1), contrast_limit=(-0.1, 0.1), p=0.2),\n            # GaussNoise(p=0.5),\n\n            # normalizing and tensorizing\n            Normalize(mean=mean, std=std, max_pixel_value=255.0, p=1.0),\n            ToTensorV2(p=1.0),\n        ], p=1.0)\n    \n    if 'val' in need:\n        transformations['val'] = Compose([\n            Resize(img_size[0], img_size[1]),\n            # CenterCrop(height = CFG.input_resolution, width = CFG.input_resolution), # add centercrop\n            Normalize(mean=mean, std=std, max_pixel_value=255.0, p=1.0),\n            ToTensorV2(p=1.0),\n        ], p=1.0)\n    \n    # minimal augmentation for the transformation\n    if CFG.DEBUG == True:\n        transformations['train'] = transformations['val']\n    \n    return transformations","metadata":{"id":"RgJlZ_6MoE14"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if CFG.DEBUG == True:\n  transform = get_transforms(\n      img_size=(384, 384), \n      mean=CFG.rgb_mean, \n      std=CFG.rgb_sd\n  )\nelse:\n  transform = get_transforms(\n      img_size = (CFG.input_resolution, CFG.input_resolution),\n      mean=CFG.rgb_mean, \n      std=CFG.rgb_sd\n  )","metadata":{"id":"8ETFKEdJoE17"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from torch.utils.data import random_split\n\nif CFG.num_folds:\n  print(f\"Stratified K-Fold Training Scheme {CFG.num_folds}\")\n  train_dataset = HappyWhaleDataset(df, transforms=transform)\n  val_dataset = HappyWhaleDataset(df, transforms=transform)\n\n  train_dataset.set_transform(transform['train'])\n  val_dataset.set_transform(transform['val'])\n\nelif CFG.num_folds==0 and CFG.split_ratio != 0:\n  print(f\"Random Split Scheme {CFG.split_ratio}\")\n  dataset = HappyWhaleDataset(df, transforms=transform)\n  \n  # split train and validation dataset\n  n_val = int(len(dataset) * CFG.split_ratio)\n  n_train = len(dataset) - n_val\n  train_dataset, val_dataset = random_split(dataset, [n_train, n_val])\n\n  # after random split assign augmentation method\n  train_dataset.dataset.set_transform(transform['train'])\n  val_dataset.dataset.set_transform(transform['val'])\n\nelif CFG.num_folds==0 and CFG.split_ratio == 0:\n  print(f\"Train Dataset only scheme\")\n  train_dataset = HappyWhaleDataset(df, transforms=transform)\n  train_dataset.set_transform(transform['train'])\n\nCFG.transformations = transform['train'] # record transformation on config","metadata":{"id":"ExDHfxJIGGL1","outputId":"af3d9ae2-f389-4edf-cfcc-3bcaff20cc39"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# swintransformers batch size of 8, img size of 386 x 386 exceeds 16GB\n# swintransformers batch size of 24, img size of 386 x 386 equals to 40GB\n# swintransformers batch size of 42, img size of 224 x 224 equals to 20GB\n# swintransformers batch size of 64, img size of 224 x 224 equals to 28GB","metadata":{"id":"km4CghPUoE17"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Custom Transfer Learning","metadata":{"id":"gXi9nKeEoE17"}},{"cell_type":"code","source":"# device designation\nif torch.cuda.is_available():    \n    print(f'There are {torch.cuda.device_count()} GPU(s) available.')\n    print('GPU Name:', torch.cuda.get_device_name(0))\n    device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')\nelse:\n    print('No GPU, using CPU.')\n    device = torch.device(\"cpu\")\n\n# if cpu then num_workers are 0 else num_workers = 2\nNUM_WORKERS = 2 if torch.cuda.is_available() else 0","metadata":{"id":"VwrvLKw0oE17","outputId":"a440be02-82e3-4d74-9dfc-1878800c391b"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Use Pretrained models as backbone model\n- [swintransformers](https://github.com/rwightman/pytorch-image-models/blob/ef72ad417709b5ba6404d85d3adafd830d507b2a/timm/models/swin_transformer.py#L47-L89)\n- [convnext](https://github.com/rwightman/pytorch-image-models/blob/738a9cd63554104635351ced21d6f5808c1b6072/timm/models/convnext.py#L40-L71)\n- [efficinetnet](https://github.com/rwightman/pytorch-image-models/blob/83b40c5a58b1fc43d053de537ef3201362cc4753/timm/models/efficientnet.py#L190-L201)","metadata":{"id":"CYo5KO_hjAiq"}},{"cell_type":"code","source":"# import resnet and set model\nfrom torch import nn\nfrom torchvision import models\nimport timm\n\nindividual_ids = df.individual_id.unique().tolist()\nprint('Number of Mr.Whales in the dataset:', len(individual_ids))\n\nclass SwinTransformersModel(nn.Module):\n    def __init__(self, num_classes: int = len(individual_ids)):\n        super(SwinTransformersModel, self).__init__()        \n        # https://github.com/rwightman/pytorch-image-models/blob/ef72ad417709b5ba6404d85d3adafd830d507b2a/timm/models/swin_transformer.py#L47-L89\n        # model_architecture = \"swin_large_patch4_window7_224\" # pretrained with classifier output with 1000 classes\n        # model_architecture = \"swin_large_patch4_window7_224_in22k\" # pretrained with classifier output with 22000 classes\n        model_architecture = \"swin_large_patch4_window12_384_in22k\" # pretrained model with bigger resolution\n        self.model = timm.create_model(model_architecture, pretrained=True)\n        # self.backbone = timm.create_model(model_architecture, pretrained=True)\n        # self.backbone.classifier.out_features = num_classes\n        num_input_features = self.model.head.in_features # pretrained model's default fully connected Linear Layer\n        self.model.head = nn.Linear(in_features=num_input_features, out_features=num_classes, bias=True)  # replacing output with class number\n\n    def forward(self, x: torch.Tensor) -> torch.Tensor:\n        x = self.model(x)\n        return x\n\nclass ConvNextModel(nn.Module):\n    def __init__(self, num_classes: int = len(individual_ids)):\n        super(ConvNextModel, self).__init__()\n        \n        model_architecture = \"convnext_large_384_in22ft1k\"\n        self.backbone = timm.create_model(model_architecture, pretrained=True)\n        # self.backbone.classifier.out_features = num_classes\n        n_features = self.backbone.classifier.in_features\n        self.backbone.fc = nn.Linear(in_features=n_features, out_features=num_classes, bias=True)\n        \n    def forward(self, x: torch.Tensor) -> torch.Tensor:\n        x = self.backbone(x)\n        return x\n\nclass EfficientNetB0Model(nn.Module):\n    # https://github.com/lukemelas/EfficientNet-PyTorch\n    # https://github.com/rwightman/pytorch-image-models/blob/83b40c5a58b1fc43d053de537ef3201362cc4753/timm/models/efficientnet.py#L190-L201\n    # inputsize: https://github.com/lukemelas/EfficientNet-PyTorch/blob/7e8b0d312162f335785fb5dcfa1df29a75a1783a/efficientnet_pytorch/utils.py#L457-L479\n    def __init__(self, num_classes: int = len(individual_ids)):\n        super(EfficientNetB0Model, self).__init__()\n\n        self.backbone = models.efficientnet_b0(pretrained=True)\n        self.backbone.classifier = nn.Sequential(\n            nn.Dropout(p=0.2, inplace=True),\n            nn.Linear(in_features=1280, out_features=num_classes, bias=True),\n        )\n        \n    def forward(self, x: torch.Tensor) -> torch.Tensor:\n        x = self.backbone(x)\n        return x\n\nclass EfficientNetB5Model(nn.Module):\n    # https://github.com/lukemelas/EfficientNet-PyTorch\n    # https://github.com/rwightman/pytorch-image-models/blob/83b40c5a58b1fc43d053de537ef3201362cc4753/timm/models/efficientnet.py#L190-L201\n    # inputsize: https://github.com/lukemelas/EfficientNet-PyTorch/blob/7e8b0d312162f335785fb5dcfa1df29a75a1783a/efficientnet_pytorch/utils.py#L457-L479\n    def __init__(self, num_classes: int = len(individual_ids)):\n        super(EfficientNetB5Model, self).__init__()\n\n        self.backbone = models.efficientnet_b5(pretrained=True)\n        self.backbone.classifier = nn.Sequential(\n            nn.Dropout(p=0.4, inplace=True),\n            nn.Linear(in_features=2048, out_features=num_classes, bias=True),\n        )\n        \n    def forward(self, x: torch.Tensor) -> torch.Tensor:\n        x = self.backbone(x)\n        return x","metadata":{"id":"cZ3tkxFyoE18","outputId":"786f63b5-f74b-4afb-d42e-369a5b0d81b9"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Loss function Optimizer and scheduler","metadata":{"id":"zsaertvOaS__"}},{"cell_type":"code","source":"import torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nfrom torch.autograd import Variable\n\n# https://github.com/clcarwin/focal_loss_pytorch/blob/master/focalloss.py\nclass FocalLoss(nn.Module):\n    def __init__(self, gamma=0, alpha=None, size_average=True):\n        super(FocalLoss, self).__init__()\n        self.gamma = gamma\n        self.alpha = alpha\n        if isinstance(alpha,(float,int)): self.alpha = torch.Tensor([alpha,1-alpha])\n        if isinstance(alpha,list): self.alpha = torch.Tensor(alpha)\n        self.size_average = size_average\n\n    def forward(self, input, target):\n        if input.dim()>2:\n            input = input.view(input.size(0),input.size(1),-1)  # N,C,H,W => N,C,H*W\n            input = input.transpose(1,2)    # N,C,H*W => N,H*W,C\n            input = input.contiguous().view(-1,input.size(2))   # N,H*W,C => N*H*W,C\n        target = target.view(-1,1)\n\n        logpt = F.log_softmax(input)\n        logpt = logpt.gather(1,target)\n        logpt = logpt.view(-1)\n        pt = Variable(logpt.data.exp())\n\n        if self.alpha is not None:\n            if self.alpha.type()!=input.data.type():\n                self.alpha = self.alpha.type_as(input.data)\n            at = self.alpha.gather(0,target.data.view(-1))\n            logpt = logpt * Variable(at)\n\n        loss = -1 * (1-pt)**self.gamma * logpt\n        if self.size_average: return loss.mean()\n        else: return loss.sum()","metadata":{"id":"7Fk-LmMGhmYd"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# criterion = nn.CrossEntropyLoss()\ncriterion = FocalLoss(gamma=CFG.gamma)","metadata":{"id":"pmpGH9Rgalbl"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Train and Validation Functions","metadata":{"id":"vAwxKjrmqXyh"}},{"cell_type":"code","source":"# Metrics Class definition\n# Reference: https://github.com/pytorch/examples/blob/00ea159a99f5cb3f3301a9bf0baa1a5089c7e217/imagenet/main.py#L361-L450\n\nfrom enum import Enum\n\nclass Summary(Enum):\n    NONE = 0\n    AVERAGE = 1\n    SUM = 2\n    COUNT = 3\n\nclass AverageMeter(object):\n    \"\"\"Computes and stores the average and current value\"\"\"\n    def __init__(self, name, fmt=':f', summary_type=Summary.AVERAGE):\n        self.name = name\n        self.fmt = fmt\n        self.summary_type = summary_type\n        self.reset()\n\n    def reset(self):\n        self.val = 0\n        self.avg = 0\n        self.sum = 0\n        self.count = 0\n\n    def update(self, val, n=1):\n        self.val = val\n        self.sum += val * n\n        self.count += n\n        self.avg = self.sum / self.count\n\n    def __str__(self):\n        fmtstr = '{name} {val' + self.fmt + '} ({avg' + self.fmt + '})'\n        return fmtstr.format(**self.__dict__)\n    \n    def summary(self):\n        fmtstr = ''\n        if self.summary_type is Summary.NONE:\n            fmtstr = ''\n        elif self.summary_type is Summary.AVERAGE:\n            fmtstr = '{name} {avg:.3f}'\n        elif self.summary_type is Summary.SUM:\n            fmtstr = '{name} {sum:.3f}'\n        elif self.summary_type is Summary.COUNT:\n            fmtstr = '{name} {count:.3f}'\n        else:\n            raise ValueError('invalid summary type %r' % self.summary_type)\n        \n        return fmtstr.format(**self.__dict__)\n\nclass ProgressMeter(object):\n    def __init__(self, num_batches, meters, prefix=\"\"):\n        self.batch_fmtstr = self._get_batch_fmtstr(num_batches)\n        self.meters = meters\n        self.prefix = prefix\n\n    def display(self, batch):\n        entries = [self.prefix + self.batch_fmtstr.format(batch)]\n        entries += [str(meter) for meter in self.meters]\n        print('\\t'.join(entries))\n        \n    def display_summary(self):\n        entries = [\" *\"]\n        entries += [meter.summary() for meter in self.meters]\n        print(' '.join(entries))\n\n    def _get_batch_fmtstr(self, num_batches):\n        num_digits = len(str(num_batches // 1))\n        fmt = '{:' + str(num_digits) + 'd}'\n        return '[' + fmt + '/' + fmt.format(num_batches) + ']'\n\ndef accuracy(output, target, topk=(1,)):\n    \"\"\"Computes the accuracy over the k top predictions for the specified values of k\"\"\"\n    with torch.no_grad():\n        maxk = max(topk)\n        batch_size = target.size(0)\n\n        _, pred = output.topk(maxk, 1, True, True)\n        pred = pred.t()\n        correct = pred.eq(target.view(1, -1).expand_as(pred))\n\n        res = []\n        for k in topk:\n            correct_k = correct[:k].reshape(-1).float().sum(0, keepdim=True)\n            res.append(correct_k.mul_(100.0 / batch_size))\n        return res","metadata":{"id":"uIClVya3oE19"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# validation function\nfrom tqdm.notebook import tqdm\n\ndef validate(val_loader, model, criterion, device):\n    # Reference: https://github.com/pytorch/examples/blob/00ea159a99f5cb3f3301a9bf0baa1a5089c7e217/imagenet/main.py#L313-L353\n    batch_time = AverageMeter('Time', ':6.3f', Summary.NONE)\n    losses = AverageMeter('Loss', ':.4f', Summary.AVERAGE)\n    top1 = AverageMeter('Acc@1', ':6.2f', Summary.AVERAGE)\n    top5 = AverageMeter('Acc@5', ':6.2f', Summary.AVERAGE)\n    progress = ProgressMeter(\n        len(val_loader),\n        [batch_time, losses, top1, top5],\n        prefix='Test: ')\n\n    # switch to evaluate mode\n    model.eval()\n\n    with torch.no_grad():\n        end = time.time()\n        for i, (images, labels) in enumerate(tqdm(val_loader)):\n            images = images.to(device)\n            labels = labels.to(device)\n\n            # compute output\n            output = model(images)\n            loss = criterion(output, labels)\n\n            # measure accuracy and record loss\n            acc1, acc5 = accuracy(output, labels, topk=(1, 5))\n            losses.update(loss.item(), images.size(0))\n            top1.update(acc1[0], images.size(0))\n            top5.update(acc5[0], images.size(0))\n\n            # measure elapsed time\n            batch_time.update(time.time() - end)\n            end = time.time()\n\n        progress.display_summary()\n\n    return losses.avg, top1.avg, top5.avg","metadata":{"id":"0jny9I2_mlI4"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import time\nfrom tqdm.notebook import tqdm\n\ndef train(model, epochs, train_loader, valid_loader, optimizer, save_path, scheduler = None):\n    best_valid_acc = 0\n    best_valid_loss = 10\n    \n    for epoch in range(epochs):\n        # Train Code Reference: https://github.com/pytorch/examples/blob/00ea159a99f5cb3f3301a9bf0baa1a5089c7e217/imagenet/main.py#L266-L310\n        batch_time = AverageMeter('Time', ':6.3f')\n        data_time = AverageMeter('Data', ':6.3f')\n        losses = AverageMeter('Loss', ':.4f')\n        top1 = AverageMeter('Acc@1', ':6.2f')\n        top5 = AverageMeter('Acc@5', ':6.2f')\n        progress = ProgressMeter(\n            len(train_loader),\n            [batch_time, data_time, losses, top1, top5],\n            prefix=\"Epoch: [{}]\".format(epoch))\n\n        end = time.time()\n        for iter, (images, labels) in enumerate(tqdm(train_loader)):\n            # initialize gradients\n            optimizer.zero_grad()\n\n            # assign images and labels to the device\n            # images, labels = images.type(torch.FloatTensor).to(device), labels.to(device)\n            images, labels = images.to(device), labels.to(device)\n\n            # switch to train mode\n            model.train()\n            \n            # compute output\n            output = model(images)\n            loss = criterion(output, labels)\n\n            # measure accuracy and record loss\n            acc1, acc5 = accuracy(output, labels, topk=(1, 5))\n            losses.update(loss.item(), images.size(0))\n            top1.update(acc1[0], images.size(0))\n            top5.update(acc5[0], images.size(0))\n\n            # compute gradient and do SGD step\n            optimizer.zero_grad()\n            loss.backward()\n            optimizer.step()\n            if scheduler:\n              scheduler.step()\n            \n            # measure elapsed time\n            batch_time.update(time.time() - end)\n            end = time.time()\n\n            if iter % CFG.logging_steps == 0:\n                progress.display(iter) # display train status\n                wandb.log(\n                    {\n                      'epoch': epoch + 1,\n                      'train/loss':losses.avg, \n                      'train/top5_accuracy':top5.avg, \n                      'train/top1_accuracy':top1.avg,\n                      'learning_rate':optimizer.param_groups[0]['lr'],\n                    }\n                )\n        \n        # Validate on each epoch\n        print(\"Epoch Finished... Validating\")\n        valid_loss, valid_acc1, valid_acc5 = validate(valid_loader, model, criterion, device)\n        wandb.log(\n                    {\n                      'valid/loss':valid_loss, \n                      'valid/top5_accuracy':valid_acc5, \n                      'valid/top1_accuracy':valid_acc1,\n                    }\n        )\n        if valid_loss < best_valid_loss:\n            print(\"New valid model for val loss! saving the model...\")\n            torch.save(model.state_dict(),save_path + f\"{epoch:03}_loss_{valid_loss:4.2}.ckpt\")\n            best_valid_loss = valid_loss\n            wandb.log({'best_valid_loss':best_valid_loss})\n        \n        if valid_acc5 > best_valid_acc:\n            print(\"New valid model for val accuracy! saving the model...\")\n            torch.save(model.state_dict(),save_path + f\"{epoch:03}_loss_{valid_loss:4.2}.ckpt\")\n            best_valid_acc = valid_acc5\n            wandb.log({'best_valid_top5_acc':best_valid_acc})","metadata":{"id":"nhibBF1aoE19"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Training Iteration (K-Fold)","metadata":{"id":"Rd0_1sXhZZTP"}},{"cell_type":"code","source":"# K-Fold training iteration\nfrom torch.optim import Adam, AdamW\nfrom adamp import AdamP, SGDP\n\nfrom sklearn.model_selection import StratifiedKFold\nfrom torch.utils.data.dataset import Subset\n\nif CFG.num_folds != 0:\n  SAVE_PATH = \"./\"\n  run_name = \"EfficientNetB5Model-5-fold\"\n  wandb.init(project=\"HappyWhale\", name=run_name)\n\n  stf = StratifiedKFold(n_splits = CFG.num_folds, shuffle = True, random_state = seed_everything(42))\n\n  for fold_num, (train_idx, valid_idx) in enumerate(stf.split(df, list(df['individual_id']))):\n    print(f\"#################### Fold: {fold_num + 1} ######################\")\n\n    # make subset\n    train_set = Subset(train_dataset, train_idx)\n    val_set = Subset(val_dataset, valid_idx)\n\n    # make dataloader out of subset\n    train_loader = DataLoader(\n      train_set,\n      batch_size=CFG.train_batch_size,\n      num_workers=NUM_WORKERS,\n      shuffle=True\n    )\n    valid_loader = DataLoader(\n        val_set,\n        batch_size=CFG.val_batch_size,\n        num_workers=NUM_WORKERS,\n        shuffle=False\n    )\n\n    # designate model\n    if CFG.DEBUG == True:\n      model = EfficientNetB0Model(num_classes=len(individual_ids))\n      model = model.to(device)\n    else:\n      model = EfficientNetB5Model(num_classes=len(individual_ids))\n      model = model.to(device)\n    \n    model_name = model.__class__.__name__\n    print(f\"Training with {model_name}\")\n\n    # get optimizer and scheduler\n    optimizer = AdamP(model.parameters(), lr=CFG.learning_rate, betas=(0.9, 0.999), weight_decay= CFG.weight_decay)\n\n    # scheduler comparison: https://www.kaggle.com/isbhargav/guide-to-pytorch-learning-rate-scheduling\n    # scheduler = torch.optim.lr_scheduler.OneCycleLR(\n    #     optimizer, \n    #     max_lr=CFG.learning_rate, \n    #     steps_per_epoch=len(train_loader),\n    #     epochs=CFG.num_epochs,\n    #     anneal_strategy='linear'\n    # )\n\n    # conduct training\n    train(model, CFG.num_epochs, train_loader, valid_loader, optimizer, SAVE_PATH, scheduler=None)\n\n    # Prevent Out Of Memory error\n    model.cpu()\n    del model\n    torch.cuda.empty_cache()","metadata":{"id":"4PiiMSBRoE19","outputId":"97360cf6-a9ae-4116-e600-85f2b68f32eb"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Training Iteration (Not K-Fold)","metadata":{"id":"lXYGhX75YtNb"}},{"cell_type":"code","source":"# DataLoader\nimport numpy as np \n\nif CFG.num_folds==0 and CFG.split_ratio != 0:\n    train_loader = DataLoader(\n        train_dataset,\n        batch_size=CFG.train_batch_size,\n        num_workers=NUM_WORKERS,\n        shuffle=True\n    )\n\n    valid_loader = DataLoader(\n            val_dataset,\n            batch_size=CFG.val_batch_size,\n            num_workers=NUM_WORKERS,\n            shuffle=False\n    )\n\nelif CFG.num_folds==0 and CFG.split_ratio == 0:\n    train_loader = DataLoader(\n      train_dataset,\n      batch_size=CFG.train_batch_size,\n      num_workers=NUM_WORKERS,\n      shuffle=True\n    )\n\nimages, labels = next(iter(train_loader))\nprint(f'images shape: {images.shape}')\nprint(f'labels shape: {labels.shape}')","metadata":{"id":"iCxczOQbYUSr"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Training Iteration when not using K-Fold\nif CFG.num_folds==0:\n    \n    SAVE_PATH = \"./\"\n    run_name = \"EfficientNetB5Model-90to10-split\"\n    wandb.init(project=\"HappyWhale\", name=run_name)\n    \n    # designate model\n    if CFG.DEBUG == True:\n      model = EfficientNetB0Model(num_classes=len(individual_ids))\n      model = model.to(device)\n    else:\n      model = EfficientNetB5Model(num_classes=len(individual_ids))\n      model = model.to(device)\n    \n    model_name = model.__class__.__name__\n    print(f\"Training with {model_name}\")\n\n    # get optimizer: AdamP = AdamW > Adam > SGDP > SGD https://github.com/clovaai/AdamP. \n    optimizer = AdamP(model.parameters(), lr=CFG.learning_rate, betas=(0.9, 0.999), weight_decay= CFG.weight_decay)\n\n    # scheduler: https://www.kaggle.com/isbhargav/guide-to-pytorch-learning-rate-scheduling\n    # scheduler = torch.optim.lr_scheduler.OneCycleLR(\n    #     optimizer, \n    #     max_lr=CFG.learning_rate, \n    #     steps_per_epoch=len(train_loader),\n    #     epochs=CFG.num_epochs,\n    #     anneal_strategy='linear'\n    # )\n\n    # conduct training\n    train(model, CFG.num_epochs, train_loader, valid_loader, optimizer, SAVE_PATH, scheduler=None)","metadata":{"id":"zOIzkkZJY9Ib"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!nvidia-smi","metadata":{"id":"ojuJT9BF3hjw"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model.cpu()\ndel model\ntorch.cuda.empty_cache() ","metadata":{"id":"0zqeZvG60xhs"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Inference: Out of Fold Ensemble","metadata":{"id":"-0YQOUSi0ijK"}},{"cell_type":"code","source":"# label decoder read from the le.pkl dump\n\nimport joblib\nwith open(\"le_individual_id.pkl\", \"rb\") as fp:\n    le_individual_id = joblib.load(fp)","metadata":{"id":"8_vSvxsQ5YOQ"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"TEST_DIR = os.path.join(PROJECT_PATH, f\"{CFG.image_resolution}x{CFG.image_resolution}_resized_test_dataset\")\n\ndef get_test_file_path(id):\n    return f\"{TEST_DIR}/{id}\"\n\npath_submission = os.path.join(ROOT_DIR, \"sample_submission.csv\")\ndf_sample_submission = pd.read_csv(path_submission)\ndf_sample_submission['file_path'] = df_sample_submission['image'].apply(get_test_file_path)\ndf_sample_submission.head()","metadata":{"id":"NNqzPFm6um2s"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_sample_submission.shape","metadata":{"id":"ycaOa9ZA4brJ"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_sample_submission.describe()","metadata":{"id":"aGPG9zwaAhIt"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import torch\nfrom torch.utils.data import Dataset, DataLoader\nfrom PIL import Image\n\nclass HappyWhaleTestDataset(Dataset):\n    def __init__(self, df, transforms=None):\n        self.df = df\n        self.image_name = df.image.values\n        self.file_names = df.file_path.values\n        self.transforms = transforms\n    \n    def __getitem__(self, index):\n        image_name = self.image_name[index]\n        image_path = self.file_names[index]\n        image = Image.open(image_path)\n        image = np.array(image)\n        \n        if len(image.shape) == 2: \n            image = np.dstack((image,)*3)\n        \n        if self.transforms:\n            image_transform = self.transform(image=image)['image']\n            return image_transform, image_name, image_path\n        else:\n            return image, image_name, image_path\n\n    def __len__(self):\n        return len(self.df)\n    \n    def set_transform(self, transform):\n        self.transform = transform","metadata":{"id":"a21dGuGA0lB1"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_dataset = HappyWhaleTestDataset(df_sample_submission, transforms=transform)\ntest_dataset.set_transform(transform['val'])\ntest_loader = DataLoader(test_dataset, batch_size=CFG.val_batch_size, shuffle=False, num_workers=NUM_WORKERS)","metadata":{"id":"sip_qezO0m-e"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# inference compatible both for k-fold and which are not\nfrom tqdm import tqdm\nfrom torch import topk\n\noof_model_paths = [\n              \"/content/008_loss_ 7.5.ckpt\"\n]\n\nif len(oof_model_paths) == 1:\n  model_path = oof_model_paths[0]\n  ckpt_name = model_path.split(\"/\")[-1]\n  ckpt_name_without_type = ckpt_name.split(\".\")[0]\n  model_loss = ckpt_name.split(\".\")[1]\n  ckpt_name = ckpt_name_without_type + model_loss\n\noof_pred = [] # out of fold prediction list\nfor model_path in oof_model_paths:\n  if CFG.DEBUG == True:\n    model = EfficientNetB0Model(num_classes=len(individual_ids))\n  else:\n    model = EfficientNetB5Model(num_classes=len(individual_ids))\n  model_name = model.__class__.__name__\n  model.load_state_dict(torch.load(model_path, map_location=device)) # load state dict defaults to load on device \n  model.to(device)\n  model.eval()\n  \n  output_pred = []\n  for images, image_name, path in tqdm(test_loader):\n    with torch.no_grad():\n      images = images.type(torch.FloatTensor).to(device)\n      outputs = model(images)\n      output_pred.extend(outputs.cpu().detach().numpy())\n  # change logit to prbability\n  output_proba = F.softmax(torch.Tensor(output_pred), dim=1) \n  oof_pred.append(np.array(output_proba)[:,np.newaxis])\n\n  # Prevent OOM error\n  model.cpu()\n  del model\n  torch.cuda.empty_cache()\n\n# mean logits of fold predictions\noof_pred = np.mean(oof_pred, axis=0)\n# designate oof_pred as torch tensor\noof_pred = torch.Tensor(oof_pred)\n\nall_predictions = list(topk(oof_pred, 5))[1].cpu().numpy() # indices of highest topk\nall_probabilities = list(topk(oof_pred, 5))[0].cpu().numpy() # get top 5 predictions' probability","metadata":{"id":"ekNPAmcKAiCp"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def flatten(t):\n    return [item for sublist in t for item in sublist]\n\n# flatten batched items into array\nflattened_labels = flatten(all_predictions)\nflattened_probabilities = flatten(all_probabilities)\n\n# decode integer prediction labels into string labels\ndecoded_labels = []\nfor item in flattened_labels:\n    top_5_label = le_individual_id.inverse_transform(item)\n    str_top_5_label = np.array2string(top_5_label, separator=',')\n    str_top_5_label_without_ln = str_top_5_label.replace(\"\\n\", \"\")\n    decoded_labels.append(str_top_5_label_without_ln)","metadata":{"id":"OsYFUESnFxpZ"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from ast import literal_eval\n\npd.options.display.max_colwidth = 100 # display max length for pandas\ndf_empty = pd.DataFrame({})\ndf_empty[\"image\"] = df_sample_submission.image\n\n# make rows for predictions\ndf_empty['predictions'] = decoded_labels\ndf_empty['predictions'] = df_empty['predictions'].apply(literal_eval) # str obj val -> list obj val\n\ndf_empty['probabilities'] = flattened_probabilities\ndf_empty['sum_probability'] = df_empty['probabilities'].apply(lambda x: np.sum(x))\ndf_empty['probabilities_sd'] = df_empty['probabilities'].apply(lambda x: np.std(x))\ndf_empty.head()","metadata":{"id":"a6yZwPZg04RZ"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# get sum of top 5 probability: lower it is, lower the model's confidence \n# (side note) sum of top 5 probability is not equal to 1\nquantile_sum = df_empty['sum_probability'].quantile(0.25)\n\n# get the quantile standard deviation of probabilities: lower the standard deviation the difficult classification was\nquantile_sd = df_empty['probabilities_sd'].quantile(0.25)\n\nprint(\"sum:\", quantile_sum, \"standard deviation:\", quantile_sd)","metadata":{"id":"vo2jNqJBbRLd"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for index, row in df_empty.iterrows():\n    # if model's 1st pick isn't confident and difficult the classification it is, assign one label as new individual\n    if row['sum_probability'] < quantile_sum:\n        row['predictions'][-1] = \"new_individual\"\ndf_empty.head()","metadata":{"id":"MDFWXDYCdKPB"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_submission = df_empty[['image','predictions']]\ndf_submission['predictions'] = df_submission['predictions'].apply(lambda x: ' '.join(x))\ndf_submission['predictions'] = df_submission['predictions'].apply(lambda x: x.strip())\ndf_submission.head()","metadata":{"id":"3Be7Rn90dNcF"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"file_name = f\"{model_name}_submission_{ckpt_name}_{CFG.num_folds}-folds_{CFG.split_ratio}-Split.csv\"\ndf_submission.to_csv(file_name, index=False)","metadata":{"id":"XNN56VAQR6cM"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### References\n- [📄 Kaggle Competition Description](https://www.kaggle.com/c/happy-whale-and-dolphin/overview/evaluation)\n- [EFFNET B6 WHALE COMP 0.605](https://www.kaggle.com/manojprabhaakr/effnet-b6-whale-comp)\n- [Accuracy 0.586 code with efficientnet b6, KFold](https://www.kaggle.com/aikhmelnytskyy/happywhale-arcface-baseline-eff-net-kfold5-0-586)\n- [HappyWhale ArcFace Baseline (TPU) 0.522](https://www.kaggle.com/ks2019/happywhale-arcface-baseline-tpu)\n- [😊🐳&🐬 - EDA and Baseline Solution 0.402](https://www.kaggle.com/dschettler8845/eda-and-baseline-solution#model_baseline)\n\n- EDA: https://www.kaggle.com/bsridatta/happywhale\n- Pytorch + VGG16: https://www.kaggle.com/palash97/happywhale-pytorch-vgg16\n- Timm + EfficientNetB0 + ArcFace: https://www.kaggle.com/debarshichanda/pytorch-arcface-gem-pooling-starter\n","metadata":{}}]}