{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Overview\n\nThis notebook provides a baseline stacking framework with TPU and GPU models. I did not include my best weights here as there is no free lunch in this world. An **important point** to note is that the TPU weights are trained with TensorFlow wheres the GPU ones are with PyTorch!\n\nThe stacking methods shown here are:\n- mean\n- median\n- min-max mean\n- min-max median\n- pushout-median\n\nMy public GPU and TPU weights for EfficientNetb0 to b7 can be found here:https://www.kaggle.com/khoongweihao/alaska2-efficientnet-trained-model-weights. The weights will be updated periodically to reflect my progress in the competition.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Acknowledgements\n\n- Many thanks to Siddhartha for publishing his 'Alaska2 CNN Multiclass Classifier' notebook at https://www.kaggle.com/meaninglesslives/alaska2-cnn-multiclass-classifier\n- The inference pipeline and training for EfficientNetb0 with GPU in this notebook was done with Siddhartha's notebook above\n- TPU-trained EfficientNetb0 to b7 weights are from my private notebooks, built-upon xhlulu's notebook at https://www.kaggle.com/xhlulu/alaska2-efficientnet-on-tpus\n- For training/inference on TPU-trained models in TensorFlow, you may refer to xhlulu's notebook above\n- For TPU blending, you may refer to my other notebook at https://www.kaggle.com/khoongweihao/alaska2-blending-efficientnets-on-tpus","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Some Tips\n\n- The weights for GPU have a 'lb' suffix in their filename. This reflects the LB scores I obtained for the weights respectively\n- TPU LB scores are not included\n- How I obtained higher LB scores with Siddhartha's notebook above:\n    - modified training parameters like epochs, learning rate, lr_scheduler params, etc\n    - due to GPU runtime limit, weights were repeatedly loaded and 'transfer-learnt', while varying training parameters","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Load Libraries","execution_count":null},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":true,"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":true},"cell_type":"code","source":"!pip install -q efficientnet_pytorch\nfrom efficientnet_pytorch import EfficientNet\nfrom albumentations.pytorch import ToTensor\nfrom albumentations import (\n    Compose, HorizontalFlip, CLAHE, HueSaturationValue,\n    RandomBrightness, RandomContrast, RandomGamma, OneOf, Resize,\n    ToFloat, ShiftScaleRotate, GridDistortion, ElasticTransform, JpegCompression, HueSaturationValue,\n    RGBShift, RandomBrightness, RandomContrast, Blur, MotionBlur, MedianBlur, GaussNoise, CenterCrop,\n    IAAAdditiveGaussianNoise, GaussNoise, OpticalDistortion, RandomSizedCrop, VerticalFlip\n)\nimport os\nimport torch\nimport pandas as pd\nimport numpy as np\nimport random\nimport torch.nn as nn\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nfrom glob import glob\nimport torchvision\nfrom torch.utils.data import Dataset\nimport time\nfrom tqdm.notebook import tqdm\n# from tqdm import tqdm\nfrom sklearn import metrics\nimport cv2\nimport gc\nimport torch.nn.functional as F","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Seed everything","execution_count":null},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"seed = 42\nprint(f'setting everything to seed {seed}')\nrandom.seed(seed)\nos.environ['PYTHONHASHSEED'] = str(seed)\nnp.random.seed(seed)\ntorch.manual_seed(seed)\ntorch.cuda.manual_seed(seed)\ntorch.backends.cudnn.deterministic = True","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Create dataset for training and Validation","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"data_dir = '../input/alaska2-image-steganalysis'\nfolder_names = ['JMiPOD/', 'JUNIWARD/', 'UERD/']\nclass_names = ['Normal', 'JMiPOD_75', 'JMiPOD_90', 'JMiPOD_95', \n               'JUNIWARD_75', 'JUNIWARD_90', 'JUNIWARD_95',\n                'UERD_75', 'UERD_90', 'UERD_95']\nclass_labels = { name: i for i, name in enumerate(class_names)}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df = pd.read_csv('../input/alaska2trainvalsplit/alaska2_train_df.csv')\nval_df = pd.read_csv('../input/alaska2trainvalsplit/alaska2_val_df.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Pytorch Dataset","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"class Alaska2Dataset(Dataset):\n\n    def __init__(self, df, augmentations=None):\n\n        self.data = df\n        self.augment = augmentations\n\n    def __len__(self):\n        return len(self.data)\n\n    def __getitem__(self, idx):\n        fn, label = self.data.loc[idx]\n        im = cv2.imread(fn)[:, :, ::-1]\n        if self.augment:\n            # Apply transformations\n            im = self.augment(image=im)\n        return im, label\n\n\nimg_size = 512\nAUGMENTATIONS_TRAIN = Compose([\n    VerticalFlip(p=0.5),\n    HorizontalFlip(p=0.5),\n    ToFloat(max_value=255),\n    ToTensor()\n], p=1)\n\n\nAUGMENTATIONS_TEST = Compose([\n    ToFloat(max_value=255),\n    ToTensor()\n], p=1)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# CNN Model for multiclass classification","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"class Net(nn.Module):\n    def __init__(self, num_classes):\n        super().__init__()\n        self.model = EfficientNet.from_pretrained('efficientnet-b0')\n        # 1280 is the number of neurons in last layer. is diff for diff. architecture\n        self.dense_output = nn.Linear(1280, num_classes)\n\n    def forward(self, x):\n        feat = self.model.extract_features(x)\n        feat = F.avg_pool2d(feat, feat.size()[2:]).reshape(-1, 1280)\n        return self.dense_output(feat)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!ls ../input/alaska2-efficientnet-trained-model-weights","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Model Weights","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## All Model Weights","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"models = os.listdir('../input/alaska2-efficientnet-trained-model-weights/')\nmodels ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## TPU-Trained Model Weights\n\nI'm not stacking my TPU-trained model weights here as their LB scores are between 0.750 and 0.790+. Feel free to include your own weights here to stack with GPU ones.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"tpu_models = [fn for fn in models if 'model_effnet' in fn]\ntpu_models","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## GPU-Trained Model Weights","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"gpu_models =  [fn for fn in models if 'efficientnetb0_lb' in fn]\ngpu_models","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Load TPU Models","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# we will not load these here as the TPU weights provided do not help in the stacking","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Load GPU Models","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"loaded_gpu_models = []\nfor fn in gpu_models:\n    device = 'cuda'\n    model = Net(num_classes=len(class_labels)).to(device)\n    model.load_state_dict(torch.load(('../input/alaska2-efficientnet-trained-model-weights/' + fn)))\n    loaded_gpu_models.append(model)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Create Inference Dataset","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# # Inference\nclass Alaska2TestDataset(Dataset):\n\n    def __init__(self, df, augmentations=None):\n\n        self.data = df\n        self.augment = augmentations\n\n    def __len__(self):\n        return len(self.data)\n\n    def __getitem__(self, idx):\n        fn = self.data.loc[idx][0]\n        im = cv2.imread(fn)[:, :, ::-1]\n\n        if self.augment:\n            # Apply transformations\n            im = self.augment(image=im)\n\n        return im\n\n\ntest_filenames = sorted(glob(f\"{data_dir}/Test/*.jpg\"))\ntest_df = pd.DataFrame({'ImageFileName': list(\n    test_filenames)}, columns=['ImageFileName'])\n\nbatch_size = 16\nnum_workers = 4\ntest_dataset = Alaska2TestDataset(test_df, augmentations=AUGMENTATIONS_TEST)\ntest_loader = torch.utils.data.DataLoader(test_dataset,\n                                          batch_size=batch_size,\n                                          num_workers=num_workers,\n                                          shuffle=False,\n                                          drop_last=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Inference on Each Model","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df['Id'] = test_df['ImageFileName'].apply(lambda x: x.split(os.sep)[-1])\n\nfor k, model in enumerate(loaded_gpu_models):\n    model.eval()\n\n    preds = []\n    tk0 = tqdm(test_loader)\n    with torch.no_grad():\n        for i, im in enumerate(tk0):\n            inputs = im[\"image\"].to(device)\n            # flip vertical\n            im = inputs.flip(2)\n            outputs = model(im)\n            # fliplr\n            im = inputs.flip(3)\n            outputs = (0.25*outputs + 0.25*model(im))\n            outputs = (outputs + 0.5*model(inputs))        \n            preds.extend(F.softmax(outputs, 1).cpu().numpy())\n\n    preds = np.array(preds)\n    labels = preds.argmax(1)\n    new_preds = np.zeros((len(preds),))\n    new_preds[labels != 0] = preds[labels != 0, 1:].sum(1)\n    new_preds[labels == 0] = 1 - preds[labels == 0, 0]\n    \n    test_df['Label' + str(k)] = new_preds\n\ntest_df = test_df.drop('ImageFileName', axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Check Correlations","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"ncol = test_df.shape[1]\ntest_df.iloc[:,1:ncol].corr()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"corr = test_df.iloc[:,1:10].corr()\nmask = np.zeros_like(corr, dtype=np.bool)\nmask[np.triu_indices_from(mask)] = True\n\n# Set up the matplotlib figure\nf, ax = plt.subplots(figsize=(11, 9))\n\n# Generate a custom diverging colormap\ncmap = sns.diverging_palette(220, 10, as_cmap=True)\n\n# Draw the heatmap with the mask and correct aspect ratio\nsns.heatmap(corr, mask=mask, cmap=cmap, vmax=.3, center=0, square=True, linewidths=.5, cbar_kws={\"shrink\": .5})","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Stacking (Mean/Median/Minmax/Pushout)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df['max'] = test_df.iloc[:, 1:ncol].max(axis=1)\ntest_df['min'] = test_df.iloc[:, 1:ncol].min(axis=1)\ntest_df['mean'] = test_df.iloc[:, 1:ncol].mean(axis=1)\ntest_df['median'] = test_df.iloc[:, 1:ncol].median(axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"cutoff_lo = 0.3\ncutoff_hi = 0.7","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df['Label'] = test_df['mean']\ntest_df[['Id', 'Label']].to_csv('stack_mean.csv', index=False, float_format='%.6f')\ntest_df['Label'] = test_df['median']\ntest_df[['Id', 'Label']].to_csv('stack_median.csv', index=False, float_format='%.6f')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df['Label'] = np.where(np.all(test_df.iloc[:,1:ncol] > cutoff_lo, axis=1), 1, \n                                    np.where(np.all(test_df.iloc[:,1:ncol] < cutoff_hi, axis=1),\n                                             0, test_df['median']))\ntest_df[['Id', 'Label']].to_csv('stack_pushout_median.csv', index=False, float_format='%.6f')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df['Label'] = np.where(np.all(test_df.iloc[:,1:ncol] > cutoff_lo, axis=1), \n                                    test_df['max'], \n                                    np.where(np.all(test_df.iloc[:,1:ncol] < cutoff_hi, axis=1),\n                                             test_df['min'], \n                                             test_df['mean']))\ntest_df[['Id', 'Label']].to_csv('stack_minmax_mean.csv', index=False, float_format='%.6f')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df['Label'] = np.where(np.all(test_df.iloc[:,1:ncol] > cutoff_lo, axis=1), \n                                    test_df['max'], \n                                    np.where(np.all(test_df.iloc[:,1:ncol] < cutoff_hi, axis=1),\n                                             test_df['min'], \n                                             test_df['median']))\ntest_df[['Id', 'Label']].to_csv('stack_minmax_median.csv',  index=False, float_format='%.6f')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## This notebook will be updated periodically with better private/public submissions!\n## Hope this will help you out and happy Kaggling! :)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Quod erat demonstrandum (Q.E.D.)","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}