{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"
\n

HappyWhale Siamese Starter

\n
","metadata":{}},{"cell_type":"markdown","source":"![](https://media.istockphoto.com/illustrations/the-whale-is-blowing-illustration-id164494826?k=20&m=164494826&s=612x612&w=0&h=SGm8bwFqE7-h_ekqaXOVfIUIpKN8aW2AAMcFSbvpwYg=)","metadata":{}},{"cell_type":"markdown","source":"In this kernel my idea is to create a Siamese type network and optimize it using CosineEmbeddingLoss\n\nThis is work in progress and will be improved over time. Current challenges are that the probability of 2 examples of same individual_id is very less","metadata":{}},{"cell_type":"markdown","source":"#

Install Required Libraries

","metadata":{}},{"cell_type":"code","source":"!pip install git+https://github.com/rwightman/pytorch-image-models\n!pip install --upgrade wandb","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-02-02T04:38:06.971294Z","iopub.execute_input":"2022-02-02T04:38:06.971721Z","iopub.status.idle":"2022-02-02T04:38:39.691911Z","shell.execute_reply.started":"2022-02-02T04:38:06.971614Z","shell.execute_reply":"2022-02-02T04:38:39.690862Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#

Import Required Libraries 📚

","metadata":{}},{"cell_type":"code","source":"import os\nimport gc\nimport cv2\nimport copy\nimport time\nimport random\n\n# For data manipulation\nimport numpy as np\nimport pandas as pd\n\n# Pytorch Imports\nimport torch\nimport torch.nn as nn\nimport torch.optim as optim\nfrom torch.optim import lr_scheduler\nfrom torch.utils.data import Dataset, DataLoader\nfrom torch.cuda import amp\n\n# Utils\nimport joblib\nfrom tqdm import tqdm\nfrom collections import defaultdict\n\n# Sklearn Imports\nfrom sklearn.model_selection import GroupKFold\n\n# For Image Models\nimport timm\n\n# Albumentations for augmentations\nimport albumentations as A\nfrom albumentations.pytorch import ToTensorV2\n\n# For colored terminal text\nfrom colorama import Fore, Back, Style\nb_ = Fore.BLUE\nsr_ = Style.RESET_ALL\n\nimport warnings\nwarnings.filterwarnings(\"ignore\")\n\n# For descriptive error messages\nos.environ['CUDA_LAUNCH_BLOCKING'] = \"1\"","metadata":{"execution":{"iopub.status.busy":"2022-02-02T04:38:39.694790Z","iopub.execute_input":"2022-02-02T04:38:39.695372Z","iopub.status.idle":"2022-02-02T04:38:44.747940Z","shell.execute_reply.started":"2022-02-02T04:38:39.695296Z","shell.execute_reply":"2022-02-02T04:38:44.746826Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\"Weights\n\n Weights & Biases (W&B) is a set of machine learning tools that helps you build better models faster. Kaggle competitions require fast-paced model development and evaluation. There are a lot of components: exploring the training data, training different models, combining trained models in different combinations (ensembling), and so on.\n\n> ⏳ Lots of components = Lots of places to go wrong = Lots of time spent debugging\n\nW&B can be useful for Kaggle competition with it's lightweight and interoperable tools:\n\n* Quickly track experiments,
\n* Version and iterate on datasets,
\n* Evaluate model performance,
\n* Reproduce models,
\n* Visualize results and spot regressions,
\n* Share findings with colleagues.\n\nTo learn more about Weights and Biases check out this kernel.","metadata":{}},{"cell_type":"code","source":"import wandb\n\ntry:\n from kaggle_secrets import UserSecretsClient\n user_secrets = UserSecretsClient()\n api_key = user_secrets.get_secret(\"wandb_api\")\n wandb.login(key=api_key)\n anony = None\nexcept:\n anony = \"must\"\n print('If you want to use your W&B account, go to Add-ons -> Secrets and provide your W&B access token. Use the Label name as wandb_api. \\nGet your W&B access token from here: https://wandb.ai/authorize')","metadata":{"execution":{"iopub.status.busy":"2022-02-02T04:38:44.750317Z","iopub.execute_input":"2022-02-02T04:38:44.750700Z","iopub.status.idle":"2022-02-02T04:38:46.501787Z","shell.execute_reply.started":"2022-02-02T04:38:44.750652Z","shell.execute_reply":"2022-02-02T04:38:46.500817Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#

Training Configuration ⚙️

","metadata":{}},{"cell_type":"code","source":"CONFIG = {\"seed\": 2021,\n \"epochs\": 3,\n \"img_size\": 256,\n \"model_name\": \"tf_efficientnet_b0\",\n \"embedding_size\": 256,\n \"train_batch_size\": 32,\n \"valid_batch_size\": 64,\n \"learning_rate\": 1e-4,\n \"scheduler\": 'CosineAnnealingLR',\n \"min_lr\": 1e-6,\n \"T_max\": 500,\n \"weight_decay\": 1e-6,\n \"n_fold\": 5,\n \"margin\": 0,\n \"n_accumulate\": 1,\n \"device\": torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n }","metadata":{"execution":{"iopub.status.busy":"2022-02-02T04:38:46.502994Z","iopub.execute_input":"2022-02-02T04:38:46.503231Z","iopub.status.idle":"2022-02-02T04:38:46.559600Z","shell.execute_reply.started":"2022-02-02T04:38:46.503201Z","shell.execute_reply":"2022-02-02T04:38:46.558289Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#

Set Seed for Reproducibility

","metadata":{}},{"cell_type":"code","source":"def set_seed(seed=42):\n '''Sets the seed of the entire notebook so results are the same every time we run.\n This is for REPRODUCIBILITY.'''\n np.random.seed(seed)\n torch.manual_seed(seed)\n torch.cuda.manual_seed(seed)\n # When running on the CuDNN backend, two further options must be set\n torch.backends.cudnn.deterministic = True\n torch.backends.cudnn.benchmark = False\n # Set a fixed value for the hash seed\n os.environ['PYTHONHASHSEED'] = str(seed)\n \nset_seed(CONFIG['seed'])","metadata":{"execution":{"iopub.status.busy":"2022-02-02T04:38:46.563809Z","iopub.execute_input":"2022-02-02T04:38:46.564280Z","iopub.status.idle":"2022-02-02T04:38:46.575984Z","shell.execute_reply.started":"2022-02-02T04:38:46.564244Z","shell.execute_reply":"2022-02-02T04:38:46.574884Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ROOT_DIR = '../input/happy-whale-and-dolphin'\nTRAIN_DIR = '../input/happy-whale-and-dolphin/train_images'\nTEST_DIR = '../input/happy-whale-and-dolphin/test_images'","metadata":{"execution":{"iopub.status.busy":"2022-02-02T04:38:46.578300Z","iopub.execute_input":"2022-02-02T04:38:46.578710Z","iopub.status.idle":"2022-02-02T04:38:46.586259Z","shell.execute_reply.started":"2022-02-02T04:38:46.578657Z","shell.execute_reply":"2022-02-02T04:38:46.585328Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_train_file_path(id):\n return f\"{TRAIN_DIR}/{id}\"","metadata":{"execution":{"iopub.status.busy":"2022-02-02T04:38:46.588287Z","iopub.execute_input":"2022-02-02T04:38:46.588677Z","iopub.status.idle":"2022-02-02T04:38:46.596619Z","shell.execute_reply.started":"2022-02-02T04:38:46.588632Z","shell.execute_reply":"2022-02-02T04:38:46.595680Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#

Read the Data 📖

","metadata":{}},{"cell_type":"code","source":"df = pd.read_csv(f\"{ROOT_DIR}/train.csv\")\ndf['file_path'] = df['image'].apply(get_train_file_path)","metadata":{"execution":{"iopub.status.busy":"2022-02-02T04:38:46.598534Z","iopub.execute_input":"2022-02-02T04:38:46.599370Z","iopub.status.idle":"2022-02-02T04:38:46.727656Z","shell.execute_reply.started":"2022-02-02T04:38:46.599294Z","shell.execute_reply":"2022-02-02T04:38:46.726694Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.head()","metadata":{"execution":{"iopub.status.busy":"2022-02-02T04:38:46.730266Z","iopub.execute_input":"2022-02-02T04:38:46.731133Z","iopub.status.idle":"2022-02-02T04:38:46.766486Z","shell.execute_reply.started":"2022-02-02T04:38:46.731086Z","shell.execute_reply":"2022-02-02T04:38:46.765561Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#

Create Folds

","metadata":{}},{"cell_type":"code","source":"gkf = GroupKFold(n_splits=CONFIG['n_fold'])\n\nfor fold, ( _, val_) in enumerate(gkf.split(X=df, y=df.individual_id, groups=df.individual_id)):\n df.loc[val_ , \"kfold\"] = fold","metadata":{"execution":{"iopub.status.busy":"2022-02-02T04:38:46.768067Z","iopub.execute_input":"2022-02-02T04:38:46.770096Z","iopub.status.idle":"2022-02-02T04:38:46.922014Z","shell.execute_reply.started":"2022-02-02T04:38:46.770044Z","shell.execute_reply":"2022-02-02T04:38:46.921016Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.kfold.value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-02-02T04:38:46.923617Z","iopub.execute_input":"2022-02-02T04:38:46.923962Z","iopub.status.idle":"2022-02-02T04:38:46.940113Z","shell.execute_reply.started":"2022-02-02T04:38:46.923919Z","shell.execute_reply":"2022-02-02T04:38:46.938882Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#

Dataset Class

","metadata":{}},{"cell_type":"code","source":"class HappyWhaleDataset(Dataset):\n def __init__(self, df, transforms=None):\n self.df = df\n self.groups = df.groupby('individual_id').groups\n self.keys = list(self.groups.keys())\n self.transforms = transforms\n \n def __len__(self):\n return len(self.groups)\n \n def __getitem__(self, index): \n image_indices_1 = self.groups[self.keys[index]]\n image_path_1 = self.df.iloc[image_indices_1, :].sample(n=1)['file_path'].values[0]\n image_1 = cv2.cvtColor(cv2.imread(image_path_1), cv2.COLOR_BGR2RGB)\n individual_id_1 = self.df.iloc[image_indices_1, :]['individual_id'].values[0]\n \n image_index_2 = self.df.sample(n=1).index\n image_path_2 = self.df.iloc[image_index_2, :]['file_path'].values[0]\n image_2 = cv2.cvtColor(cv2.imread(image_path_2), cv2.COLOR_BGR2RGB)\n individual_id_2 = self.df.iloc[image_index_2, :]['individual_id'].values[0]\n \n target = 1 if individual_id_1 == individual_id_2 else -1\n \n if self.transforms:\n image_1 = self.transforms(image=image_1)[\"image\"]\n image_2 = self.transforms(image=image_2)[\"image\"]\n \n return {\n 'image1': image_1,\n 'image2': image_2,\n 'target': torch.tensor(target, dtype=torch.long)\n }","metadata":{"execution":{"iopub.status.busy":"2022-02-02T04:38:46.942504Z","iopub.execute_input":"2022-02-02T04:38:46.942847Z","iopub.status.idle":"2022-02-02T04:38:46.955586Z","shell.execute_reply.started":"2022-02-02T04:38:46.942799Z","shell.execute_reply":"2022-02-02T04:38:46.954482Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#

Augmentations

","metadata":{}},{"cell_type":"code","source":"data_transforms = {\n \"train\": A.Compose([\n A.Resize(CONFIG['img_size'], CONFIG['img_size']),\n A.HorizontalFlip(p=0.5),\n A.VerticalFlip(p=0.5),\n A.Normalize(\n mean=[0.485, 0.456, 0.406], \n std=[0.229, 0.224, 0.225], \n max_pixel_value=255.0, \n p=1.0\n ),\n ToTensorV2()], p=1.),\n \n \"valid\": A.Compose([\n A.Resize(CONFIG['img_size'], CONFIG['img_size']),\n A.Normalize(\n mean=[0.485, 0.456, 0.406], \n std=[0.229, 0.224, 0.225], \n max_pixel_value=255.0, \n p=1.0\n ),\n ToTensorV2()], p=1.)\n}","metadata":{"execution":{"iopub.status.busy":"2022-02-02T04:38:46.957862Z","iopub.execute_input":"2022-02-02T04:38:46.958570Z","iopub.status.idle":"2022-02-02T04:38:46.969838Z","shell.execute_reply.started":"2022-02-02T04:38:46.958523Z","shell.execute_reply":"2022-02-02T04:38:46.968691Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#

Create Model

","metadata":{}},{"cell_type":"code","source":"class HappyWhaleModel(nn.Module):\n def __init__(self, model_name, pretrained=True):\n super(HappyWhaleModel, self).__init__()\n self.model = timm.create_model(model_name, pretrained=pretrained, num_classes=0)\n self.fc = nn.LazyLinear(CONFIG['embedding_size'])\n self.dropout = nn.Dropout(p=0.3)\n\n def forward(self, images):\n features = self.model(images)\n features = self.dropout(features)\n output = self.fc(features)\n return output\n \nmodel = HappyWhaleModel(CONFIG['model_name'])\nmodel.to(CONFIG['device']);","metadata":{"execution":{"iopub.status.busy":"2022-02-02T04:38:46.975854Z","iopub.execute_input":"2022-02-02T04:38:46.976795Z","iopub.status.idle":"2022-02-02T04:38:52.632928Z","shell.execute_reply.started":"2022-02-02T04:38:46.976733Z","shell.execute_reply":"2022-02-02T04:38:52.631826Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# dummy run to initialize the layers \nimg = torch.randn(1, 3, CONFIG['img_size'], CONFIG['img_size']).to(CONFIG['device'])\nmodel(img);","metadata":{"execution":{"iopub.status.busy":"2022-02-02T04:38:52.634548Z","iopub.execute_input":"2022-02-02T04:38:52.634896Z","iopub.status.idle":"2022-02-02T04:38:58.867268Z","shell.execute_reply.started":"2022-02-02T04:38:52.634830Z","shell.execute_reply":"2022-02-02T04:38:58.866222Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#

Loss Function

\n\n![](https://i.imgur.com/Qxd5t7Y.jpg)\n\nCheck the official documentation here","metadata":{}},{"cell_type":"code","source":"def criterion(outputs1, outputs2, targets):\n return nn.CosineEmbeddingLoss(margin=CONFIG['margin'])(outputs1, outputs2, targets)","metadata":{"execution":{"iopub.status.busy":"2022-02-02T04:38:58.869085Z","iopub.execute_input":"2022-02-02T04:38:58.869409Z","iopub.status.idle":"2022-02-02T04:38:58.875580Z","shell.execute_reply.started":"2022-02-02T04:38:58.869362Z","shell.execute_reply":"2022-02-02T04:38:58.874443Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#

Training Function

","metadata":{}},{"cell_type":"code","source":"def train_one_epoch(model, optimizer, scheduler, dataloader, device, epoch):\n model.train()\n \n dataset_size = 0\n running_loss = 0.0\n \n bar = tqdm(enumerate(dataloader), total=len(dataloader))\n for step, data in bar:\n images1 = data['image1'].to(device, dtype=torch.float)\n images2 = data['image2'].to(device, dtype=torch.float)\n targets = data['target'].to(device, dtype=torch.long)\n \n batch_size = images1.size(0)\n\n outputs1 = model(images1)\n outputs2 = model(images2)\n loss = criterion(outputs1, outputs2, targets)\n loss = loss / CONFIG['n_accumulate']\n loss.backward()\n \n if (step + 1) % CONFIG['n_accumulate'] == 0:\n optimizer.step()\n\n # zero the parameter gradients\n optimizer.zero_grad()\n\n if scheduler is not None:\n scheduler.step()\n \n running_loss += (loss.item() * batch_size)\n dataset_size += batch_size\n \n epoch_loss = running_loss / dataset_size\n \n bar.set_postfix(Epoch=epoch, Train_Loss=epoch_loss,\n LR=optimizer.param_groups[0]['lr'])\n gc.collect()\n \n return epoch_loss","metadata":{"execution":{"iopub.status.busy":"2022-02-02T04:38:58.877906Z","iopub.execute_input":"2022-02-02T04:38:58.878658Z","iopub.status.idle":"2022-02-02T04:38:58.893316Z","shell.execute_reply.started":"2022-02-02T04:38:58.878607Z","shell.execute_reply":"2022-02-02T04:38:58.892213Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#

Validation Function

","metadata":{}},{"cell_type":"code","source":"@torch.inference_mode()\ndef valid_one_epoch(model, dataloader, device, epoch):\n model.eval()\n \n dataset_size = 0\n running_loss = 0.0\n \n bar = tqdm(enumerate(dataloader), total=len(dataloader))\n for step, data in bar: \n images1 = data['image1'].to(device, dtype=torch.float)\n images2 = data['image2'].to(device, dtype=torch.float)\n targets = data['target'].to(device, dtype=torch.long)\n \n batch_size = images1.size(0)\n\n outputs1 = model(images1)\n outputs2 = model(images2)\n loss = criterion(outputs1, outputs2, targets)\n \n running_loss += (loss.item() * batch_size)\n dataset_size += batch_size\n \n epoch_loss = running_loss / dataset_size\n \n bar.set_postfix(Epoch=epoch, Valid_Loss=epoch_loss,\n LR=optimizer.param_groups[0]['lr']) \n \n gc.collect()\n \n return epoch_loss","metadata":{"execution":{"iopub.status.busy":"2022-02-02T04:38:58.895568Z","iopub.execute_input":"2022-02-02T04:38:58.896328Z","iopub.status.idle":"2022-02-02T04:38:58.908657Z","shell.execute_reply.started":"2022-02-02T04:38:58.896267Z","shell.execute_reply":"2022-02-02T04:38:58.907746Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#

Run Training

","metadata":{}},{"cell_type":"code","source":"def run_training(model, optimizer, scheduler, device, num_epochs):\n # To automatically log gradients\n wandb.watch(model, log_freq=100)\n \n if torch.cuda.is_available():\n print(\"[INFO] Using GPU: {}\\n\".format(torch.cuda.get_device_name()))\n \n start = time.time()\n best_model_wts = copy.deepcopy(model.state_dict())\n best_epoch_loss = np.inf\n history = defaultdict(list)\n \n for epoch in range(1, num_epochs + 1): \n gc.collect()\n train_epoch_loss = train_one_epoch(model, optimizer, scheduler, \n dataloader=train_loader, \n device=CONFIG['device'], epoch=epoch)\n \n val_epoch_loss = valid_one_epoch(model, valid_loader, device=CONFIG['device'], \n epoch=epoch)\n \n history['Train Loss'].append(train_epoch_loss)\n history['Valid Loss'].append(val_epoch_loss)\n \n # Log the metrics\n wandb.log({\"Train Loss\": train_epoch_loss})\n wandb.log({\"Valid Loss\": val_epoch_loss})\n \n # deep copy the model\n if val_epoch_loss <= best_epoch_loss:\n print(f\"{b_}Validation Loss Improved ({best_epoch_loss} ---> {val_epoch_loss})\")\n best_epoch_loss = val_epoch_loss\n run.summary[\"Best Loss\"] = best_epoch_loss\n best_model_wts = copy.deepcopy(model.state_dict())\n PATH = \"Loss{:.4f}_epoch{:.0f}.bin\".format(best_epoch_loss, epoch)\n torch.save(model.state_dict(), PATH)\n # Save a model file from the current directory\n print(f\"Model Saved{sr_}\")\n \n print()\n \n end = time.time()\n time_elapsed = end - start\n print('Training complete in {:.0f}h {:.0f}m {:.0f}s'.format(\n time_elapsed // 3600, (time_elapsed % 3600) // 60, (time_elapsed % 3600) % 60))\n print(\"Best Loss: {:.4f}\".format(best_epoch_loss))\n \n # load best model weights\n model.load_state_dict(best_model_wts)\n \n return model, history","metadata":{"execution":{"iopub.status.busy":"2022-02-02T04:38:58.911087Z","iopub.execute_input":"2022-02-02T04:38:58.911528Z","iopub.status.idle":"2022-02-02T04:38:58.928636Z","shell.execute_reply.started":"2022-02-02T04:38:58.911483Z","shell.execute_reply":"2022-02-02T04:38:58.927578Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def fetch_scheduler(optimizer):\n if CONFIG['scheduler'] == 'CosineAnnealingLR':\n scheduler = lr_scheduler.CosineAnnealingLR(optimizer,T_max=CONFIG['T_max'], \n eta_min=CONFIG['min_lr'])\n elif CONFIG['scheduler'] == 'CosineAnnealingWarmRestarts':\n scheduler = lr_scheduler.CosineAnnealingWarmRestarts(optimizer,T_0=CONFIG['T_0'], \n eta_min=CONFIG['min_lr'])\n elif CONFIG['scheduler'] == None:\n return None\n \n return scheduler","metadata":{"execution":{"iopub.status.busy":"2022-02-02T04:38:58.930694Z","iopub.execute_input":"2022-02-02T04:38:58.931300Z","iopub.status.idle":"2022-02-02T04:38:58.943675Z","shell.execute_reply.started":"2022-02-02T04:38:58.931172Z","shell.execute_reply":"2022-02-02T04:38:58.942643Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def prepare_loaders(df, fold):\n df_train = df[df.kfold != fold].reset_index(drop=True)\n df_valid = df[df.kfold == fold].reset_index(drop=True)\n \n train_dataset = HappyWhaleDataset(df_train, transforms=data_transforms[\"train\"])\n valid_dataset = HappyWhaleDataset(df_valid, transforms=data_transforms[\"valid\"])\n\n train_loader = DataLoader(train_dataset, batch_size=CONFIG['train_batch_size'], \n num_workers=2, shuffle=True, pin_memory=True, drop_last=True)\n valid_loader = DataLoader(valid_dataset, batch_size=CONFIG['valid_batch_size'], \n num_workers=2, shuffle=False, pin_memory=True)\n \n return train_loader, valid_loader","metadata":{"execution":{"iopub.status.busy":"2022-02-02T04:38:58.945173Z","iopub.execute_input":"2022-02-02T04:38:58.945561Z","iopub.status.idle":"2022-02-02T04:38:58.956664Z","shell.execute_reply.started":"2022-02-02T04:38:58.945516Z","shell.execute_reply":"2022-02-02T04:38:58.953477Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Prepare Dataloaders","metadata":{}},{"cell_type":"code","source":"train_loader, valid_loader = prepare_loaders(df, fold=0)","metadata":{"execution":{"iopub.status.busy":"2022-02-02T04:38:58.958329Z","iopub.execute_input":"2022-02-02T04:38:58.958981Z","iopub.status.idle":"2022-02-02T04:38:59.409299Z","shell.execute_reply.started":"2022-02-02T04:38:58.958934Z","shell.execute_reply":"2022-02-02T04:38:59.408337Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Define Optimizer and Scheduler","metadata":{}},{"cell_type":"code","source":"optimizer = optim.Adam(model.parameters(), lr=CONFIG['learning_rate'], \n weight_decay=CONFIG['weight_decay'])\nscheduler = fetch_scheduler(optimizer)","metadata":{"execution":{"iopub.status.busy":"2022-02-02T04:38:59.411061Z","iopub.execute_input":"2022-02-02T04:38:59.411410Z","iopub.status.idle":"2022-02-02T04:38:59.418644Z","shell.execute_reply.started":"2022-02-02T04:38:59.411363Z","shell.execute_reply":"2022-02-02T04:38:59.417412Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Start Training","metadata":{}},{"cell_type":"code","source":"run = wandb.init(project='HappyWhale', \n config=CONFIG,\n job_type='Train',\n tags=['cos-loss', 'effnet-b0', '256'],\n anonymous='must')","metadata":{"execution":{"iopub.status.busy":"2022-02-02T04:38:59.420526Z","iopub.execute_input":"2022-02-02T04:38:59.421158Z","iopub.status.idle":"2022-02-02T04:39:06.591888Z","shell.execute_reply.started":"2022-02-02T04:38:59.421109Z","shell.execute_reply":"2022-02-02T04:39:06.590723Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model, history = run_training(model, optimizer, scheduler, \n device=CONFIG['device'],\n num_epochs=CONFIG['epochs'])","metadata":{"execution":{"iopub.status.busy":"2022-02-02T04:39:06.596826Z","iopub.execute_input":"2022-02-02T04:39:06.597220Z","iopub.status.idle":"2022-02-02T06:15:37.513108Z","shell.execute_reply.started":"2022-02-02T04:39:06.597160Z","shell.execute_reply":"2022-02-02T06:15:37.510937Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"run.finish()","metadata":{"execution":{"iopub.status.busy":"2022-02-02T06:15:37.519659Z","iopub.execute_input":"2022-02-02T06:15:37.520557Z","iopub.status.idle":"2022-02-02T06:15:44.174871Z","shell.execute_reply.started":"2022-02-02T06:15:37.520511Z","shell.execute_reply":"2022-02-02T06:15:44.173784Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#

Visualizations

\nView the Complete Dashboard Here ⮕","metadata":{}},{"cell_type":"markdown","source":"![](https://i.imgur.com/biqEsOT.jpg)","metadata":{}},{"cell_type":"markdown","source":"![Upvote!](https://img.shields.io/badge/Upvote-If%20you%20like%20my%20work-07b3c8?style=for-the-badge&logo=kaggle)","metadata":{}}]}