{"cells":[{"metadata":{},"cell_type":"markdown","source":"**To encourage Learning and Sharing, PLEASE SUPPORT AND UPVOTE**\n<br/>\n**Note:** Any Improvement and suggestions would be highly appreciated.\n\n## densenet201 outperforms all others\n\n|Version | base_arch | Additional | KFold-Mean | Kfold-Std |\n| --- | --- | --- | --- | --- |\n| v6 | resnet152 | kappa, Floss, 5 Fold, 7 epoch, (1e-06,1e-03) | 0.8489601 | 0.010163763|\n| v5 | densenet201 | kappa, Floss, 5 Fold, 7 epoch, (1e-06,1e-03) | **0.879603**| 0.01543674|\n| v4 | senet154 | kappa, Floss, 5 Fold, 7 epoch, (1e-06,1e-03) | 0.8435775 | 0.002381802|\n| v3 | se_resnext101_32x4d | kappa, Floss, 5 Fold, 7 epoch, (1e-06,1e-03) | 0.8356689 | 0.0125074405 |\n| v9 | resnet34 | kappa, Floss, 5 Fold, 7 epoch, (1e-06,1e-03) | 0.8190452 | 0.011472594 |\n| v10 | resnet101 | kappa, Floss, 5 Fold, 7 epoch, (1e-06,1e-03) | 0.8445773 | 0.0 |\n| v11 | resnet50 | kappa, Floss, 5 Fold, 7 epoch, (1e-06,1e-03) | 0.83830297 | 0.0 |\n| v12 | densenet169 | kappa, Floss, TTA, 5 Fold, 7 epoch, (1e-06,1e-03) | Failed | Failed |\n| v13 | densenet201 | kappa, Floss, Train-Valid-Test, augment, 5 Fold, 12 epoch, 8,(1e-06,1e-03) 4,(1e-06,1e-03) | Failed | Failed |\n| v14 | densenet201 | kappa, Floss, Train-Valid-Test, augment, oversampling, 5 Fold, 12 epoch, 8,(1e-06,1e-03) 4,(1e-06,1e-03) | Failed | Failed |\n| v15 | densenet201 | kappa, Floss, Train-Valid-Test, augment, 2015 data, 5 Fold, 12 epoch, 8,(1e-06,1e-03) 4,(1e-06,1e-03) | Failed | Failed |\n| v16 | densenet201 | kappa, Floss, Train-Valid-Test, augment, 2015 data, 5 Fold, 12 epoch, 8,(1e-06,1e-03) 4,(1e-06,1e-03) | Failed | Failed |\n| v17 | densenet201 | kappa, Floss, Train-Valid-Test, augment, 2015 data, 5 Fold, 7 epoch, 5,(1e-06,1e-03) 2,(1e-06,1e-03) | Processing | Processing |\n| v18-(rev13) | densenet201 | kappa, Floss, Train-Valid-Test, augment, 5 Fold, 7 epoch, 5,(1e-06,1e-03) 2,(1e-06,1e-03) | Processing | Processing |\n| v19-(rev14) | densenet201 | kappa, Floss, Train-Valid-Test, augment, oversampling,5 Fold, 7 epoch, 5,(1e-06,1e-03) 2,(1e-06,1e-03) | Current | Current |\n\n\n\n **References:**\n- Thanks @ilovescience for this wonderful kernel, https://www.kaggle.com/tanlikesmath/intro-aptos-diabetic-retinopathy-eda-starter. This can be used for training once you have selcted base_arch\n- Thanks @jhoward for https://course.fast.ai/videos/\n- Thanks @ilovescience for the old resized data - https://www.kaggle.com/tanlikesmath/diabetic-retinopathy-resized\n\n**To avoid overfit - Make sure your model has**\n- train_loss to val_loss difference as close as possible (low)\n- Low error_rate \n- High Kappa Score > .90\n- Kfold Verification - Hold out sample check (Same class Distribution as Training Set)\n\n**Improvements could be:**\n- Try kfold on other architectures anf later create weighted ensemble\n- Try - https://www.kaggle.com/ratthachat/aptos-updated-preprocessing-ben-s-cropping/comments from @rathhachat\n- Try Oversampling\n\n**Handling Data Imbalance:**\n- Focal Loss: To address this problem, the focal loss introduces the focusing parameter γ to down-weight the loss assigned to easily classified examples. This effect increases as value of γ increases and makes the network focus more on hard examples.\n- Kappa metric : Kappa is an important measure on classifier performance, especially on imbalanced data set. It measures how much better the classifier is, compared to guessing with the target distribution.\n\n**Prediction Boosting:**\n\n- Test-time augmentation, or TTA for short, is an application of data augmentation to the test dataset.Specifically, it involves creating multiple augmented copies of each image in the test set, having the model make a prediction for each, then returning an ensemble of those predictions.\n\n**Versions:**\n- Self Forked -v1: Cross Validation - 5 Fold, 7 epoch, (1e-06,1e-03) whole train\n- v1: Cross Validation - restnet34, kappa, Floss, 5 Fold, 7 epoch, (1e-06,1e-03) whole train - Cancelled\n- v2: Cross Validation - densenet201, kappa, Floss, 5 Fold, 7 epoch, (1e-06,1e-03) whole train - Cancelled\n- v3: Cross Validation - se_resnext101_32x4d, kappa, Floss, 5 Fold, 7 epoch, (1e-06,1e-03) whole train\n- v4: Cross Validation - senet154, kappa, Floss, 5 Fold, 7 epoch, (1e-06,1e-03) whole train\n- v5: Cross Validation - densenet201, kappa, Floss, 5 Fold, 7 epoch, (1e-06,1e-03) whole train\n- v6: Cross Validation - resnet152, kappa, Floss, 5 Fold, 7 epoch, (1e-06,1e-03) whole train\n- v9: Cross Validation - resnet34, kappa, Floss, 5 Fold, 7 epoch, (1e-06,1e-03) whole train\n- v10: Cross Validation - resnet101, kappa, Floss, 5 Fold, 7 epoch, (1e-06,1e-03) whole train\n- v11: Cross Validation - resnet50, kappa, Floss, 5 Fold, 7 epoch, (1e-06,1e-03) whole train\n- v12: Cross Validation - densenet169, kappa, Floss, TTA, 5 Fold, 12 epoch, 8, (1e-03,1e-02) --> 4 (1e-06,1e-03) whole train\n- v13: Cross Validation - densenet201, kappa, Floss, Train-Valid-Test, augment, 5 Fold, 12 epoch, 8,(1e-06,1e-03) 4,(1e-06,1e-03)\n- v14: Cross Validation - densenet201, kappa, Floss, Train-Valid-Test, augment, oversampling, 5 Fold, 12 epoch, 8,(1e-06,1e-03) 4,(1e-06,1e-03)\n- v15: Cross Validation - densenet201, kappa, Floss, Train-Valid-Test, augment, 2015 data, 5 Fold, 12 epoch, 8,(1e-06,1e-03) 4,(1e-06,1e-03)\n- v16: Cross Validation - densenet201, kappa, Floss, Train-Valid-Test, augment, 2015 data, 5 Fold, 12 epoch, 8,(1e-06,1e-03) 4,(1e-06,1e-03) - fixed .jpeg suffix\n- v17: Cross Validation - densenet201, kappa, Floss, Train-Valid-Test, augment, 2015 data, 5 Fold, 7 epoch, 5,(1e-06,1e-03) 2,(1e-06,1e-03) - fixed .jpeg suffix\n- v18(re-v13): Cross Validation - densenet201, kappa, Floss, Train-Valid-Test, augment, 5 Fold,  7 epoch, 5,(1e-06,1e-03) 2,(1e-06,1e-03)\n- v19(re-v14): Cross Validation - densenet201, kappa, Floss, Train-Valid-Test, oversampling, augment, 5 Fold,  7 epoch, 5,(1e-06,1e-03) 2,(1e-06,1e-03)"},{"metadata":{},"cell_type":"markdown","source":"## Step 1: Install Cadene pretrained-models.pytorch"},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install \"../input/pretrainedmodels/pretrainedmodels/pretrained-models.pytorch-master\"\n#from fastai.vision.models import cadene_models","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#these are the 2 pretrainedmodels we used , for the purpose of the example we only used  se_resnext101_32x4d\nimport pretrainedmodels\n\ndef model_f(pretrained=True, **kwargs):\n    return pretrainedmodels.senet154(num_classes=1000,pretrained='imagenet')\n\n#def model_f(pretrained=True,**kwargs):\n#    return pretrainedmodels.se_resnext101_32x4d(num_classes=1000,pretrained='imagenet')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Step 2: Load Pre-requisites"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nimport os, gc\nprint(os.listdir(\"../input\"))\n# Any results you write to the current directory are saved as output.\n\n%reload_ext autoreload\n%autoreload 2\n%matplotlib inline\nfrom pathlib import Path\nimport matplotlib.pyplot as plt\n\nimport fastai\nfrom fastai.vision import *\nfrom fastai.metrics import error_rate\n\ndef seed_everything(seed):\n    random.seed(seed)\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    np.random.seed(seed)\n    torch.manual_seed(seed)\n    torch.cuda.manual_seed(seed)\n    torch.backends.cudnn.deterministic = True\n\nSEED = 2019\nseed_everything(SEED)\nprint('Make sure cudnn is enabled:', torch.backends.cudnn.enabled)\nfastai.__version__","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Step 3: Quick EDA - Data Imbalanced\nTraining samples are imbalanced where class 0 being 49.2 % of all training samples."},{"metadata":{"trusted":true},"cell_type":"code","source":"PATH = Path('../input/aptos2019-blindness-detection/')\ntrain = pd.read_csv(PATH/'train.csv') #, nrows=500\nprint (\"Number of training samples: \", train.shape)\n#test = pd.read_csv(PATH/'test.csv')\n#print (\"Number of test samples: \", test.shape)\ndef ret_percentage(column):\n    return round(column.value_counts(normalize=True) * 100,2)\nprint(\"Original Percentage Class Dist:\\n\", ret_percentage(train[\"diagnosis\"]))\n#train['diagnosis'].hist(figsize = (10, 5))\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Holdout test set from training samples\nfrom sklearn.model_selection import StratifiedShuffleSplit\n\nsplit = StratifiedShuffleSplit(n_splits=1, test_size=0.15, random_state= SEED)\nfor train_index, test_index in split.split(train[\"id_code\"], train[\"diagnosis\"]):\n    df_train = train.iloc[train_index]\n    df_test = train.iloc[test_index]\n    \nprint(\"New Train Sample Size\", df_train.shape)\nprint(\"New Train Class Percentage Dist:\\n\", ret_percentage(df_train[\"diagnosis\"]))\n\nprint(\"New Test Sample Size\",df_test.shape)\nprint(\"New Test Class Percentage Dist:\\n\", ret_percentage(df_test[\"diagnosis\"]))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Step 4: Initialize model parameters\n* Initialize batch processing size\n* Copy base_arch (For example, resnet34) to fastai default load directory\n* Declare Loss Function\n* Evaluation metric - quadratic KappaScore\n* Oversampling"},{"metadata":{"trusted":true},"cell_type":"code","source":"## Initialize batch processing size\nbs = 16  #64\n# bs = 16   # uncomment this line if you run out of memory even after clicking Kernel->Restart\nsz = 256 #224 #Image size\nn_folds = 5\nmodel_name = \"densenet201\"\n\n# Copy model (use resnet50) to fastai load directory\n!mkdir -p /tmp/.cache/torch/checkpoints/\n#!cp \"../input/resnet34/resnet34.pth\" /tmp/.cache/torch/checkpoints/resnet34-333f7ec4.pth\n#!cp \"../input/resnet50/resnet50.pth\" /tmp/.cache/torch/checkpoints/resnet50-19c8e357.pth\n#!cp \"../input/resnet101/resnet101.pth\" /tmp/.cache/torch/checkpoints/resnet101-5d3b4d8f.pth\n#!cp \"../input/resnet152/resnet152.pth\" /tmp/.cache/torch/checkpoints/resnet152-b121ed2d.pth\n!cp \"../input/pytorch-model-zoo/densenet201-5750cbb1e.pth\" /tmp/.cache/torch/checkpoints/densenet201-c1103571.pth\n#!cp \"../input/pytorch-model-zoo/densenet169-f470b90a4.pth\" /tmp/.cache/torch/checkpoints/densenet169-b2777c0a.pth\n#!cp \"../input/pytorch-model-zoo/senet154-c7b49a05.pth\" /tmp/.cache/torch/checkpoints/senet154-c7b49a05.pth\n#!cp \"../input/pytorch-model-zoo/se_resnext101_32x4d-3b2fe3d8.pth\" /tmp/.cache/torch/checkpoints/se_resnext101_32x4d-3b2fe3d8.pth\n#!cp \"../input/seresnext50/se_resnext50_32x4d-a260b3a4.pth\" /tmp/.cache/torch/checkpoints/se_resnext50_32x4d-a260b3a4.pth\n\n#Loss Function\nclass FocalLoss(nn.Module):\n    def __init__(self, alpha=1., gamma=1.):\n        super().__init__()\n        self.alpha = alpha\n        self.gamma = gamma\n\n    def forward(self, inputs, targets, **kwargs):\n        CE_loss = nn.CrossEntropyLoss(reduction='none')(inputs, targets)\n        pt = torch.exp(-CE_loss)\n        F_loss = self.alpha * ((1-pt)**self.gamma) * CE_loss\n        return F_loss.mean()\n\nloss_func = FocalLoss(gamma=1.)\n    \n#Evaluation metric - quadratic KappaScore\nkappa = KappaScore()\nkappa.weights = \"quadratic\"\n\n#Oversampling\n\nfrom torch.utils.data.sampler import WeightedRandomSampler\n\nclass OverSamplingCallback(LearnerCallback):\n    def __init__(self,learn:Learner, weights:torch.Tensor=None):\n        super().__init__(learn)\n        labels = self.learn.data.train_dl.dataset.y.items.astype(int)\n        _,counts = np.unique(labels, return_counts=True)\n        counts = 1. / counts\n        self.weights = (weights if weights is not None else torch.DoubleTensor(counts[labels]))\n\n    def on_train_begin(self, **kwargs):\n        self.learn.data.train_dl.dl.batch_sampler = BatchSampler(\n            WeightedRandomSampler(self.weights, len(self.learn.data.train_dl.dataset)),\n            self.learn.data.train_dl.batch_size, False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Step 5: Model Evaluation - StratifiedKFold"},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.model_selection import StratifiedKFold\nfrom tqdm import tqdm\n\nskf = StratifiedKFold(n_splits=n_folds, shuffle=True, random_state = SEED)\n\ntfms = get_transforms(do_flip=True,flip_vert=True,max_rotate=360,max_warp=0,max_zoom=1.1,max_lighting=0.1,p_lighting=0.5)\nkp_score = []\nerr_rate = []\nlosses = []\nloss_func = FocalLoss(gamma=1.)\nlearn = \"\"\ndata_fold = \"\"\npredictions = torch.from_numpy(np.zeros((len(df_test))))\n\nfor fold, (train_index, val_index) in tqdm(enumerate(skf.split(df_train[\"id_code\"], df_train[\"diagnosis\"]))):\n    del learn, data_fold\n    gc.collect()\n    filename = model_name + \"fold_\" + str(fold)+\".pkl\"\n    print(\"Fold:\", filename)\n    print(\"TRAIN:\", train_index, \"VALIDATE:\", val_index)\n    data_fold = (ImageList.from_df(df_train, PATH, folder='train_images', cols=\"id_code\",suffix='.png')\n        .split_by_idxs(train_index, val_index)\n        .label_from_df(cols='diagnosis')\n        .transform(tfms,size=sz,resize_method=ResizeMethod.SQUISH,padding_mode='zeros') #Data transform\n        .databunch(bs=bs)).normalize(imagenet_stats)\n    learn = cnn_learner(data_fold, base_arch=models.densenet201 , metrics=[error_rate, kappa], \n                        loss_func=loss_func, path=Path('/kaggle/working'),callback_fns=[partial(OverSamplingCallback)])\n    learn.fit_one_cycle(5, max_lr=slice(1e-03,1e-02))\n    learn.unfreeze()\n    learn.fit_one_cycle(2, max_lr=slice(1e-06,1e-04))    \n    learn.export(filename) #smaller pkl export\n    learn.recorder.plot_losses()\n    learn.recorder.plot_metrics()\n    loss, err, kp = learn.validate()\n    kp_score.append(kp.numpy())\n    err_rate.append(err.numpy())\n    losses.append(loss)\n    learn.data.add_test(ImageList.from_df(df_test ,PATH ,folder='train_images',suffix='.png'))\n    preds, _ = learn.TTA(ds_type=DatasetType.Test)\n    predictions = predictions + preds.argmax(dim=-1).double()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"predictions = torch.round(predictions/n_folds)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_test['diagnosis_pred'] = pd.Series(predictions.numpy().astype(int), index=df_test.index)\n#df_test = df_test.assign(diagnosis_pred=column_series)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_test.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from scipy.spatial.distance import cosine\n\nprint(\"New Test Set Correlation:\", df_test['diagnosis'].corr(df_test['diagnosis_pred']))\nprint(\"New Test Set Cosine Similarity:\", 1 - cosine(df_test[\"diagnosis\"], df_test[\"diagnosis_pred\"]))\ndf_test.to_csv('submission.csv',index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"KFold kp_score Mean: \",np.mean(kp_score))\nprint(\"KFold kp_score std: \",np.std(kp_score))\nprint(\"KFold err_rate Mean: \",np.mean(err_rate))\nprint(\"KFold err_rate std: \",np.std(err_rate))\nlosses = np.asarray(losses)\nprint(\"Validation Loss Mean: \",np.mean(losses))\nprint(\"KFold losses std: \",np.std(losses))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Step 6: Results - Confusion Metric"},{"metadata":{"trusted":true},"cell_type":"code","source":"interp = ClassificationInterpretation.from_learner(learn)\n#losses,idxs = interp.top_losses()\n#len(data.valid_ds)==len(losses)==len(idxs)\n#interp.plot_top_losses(9, figsize=(15,11))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"interp.plot_confusion_matrix(figsize=(6,12), dpi=60)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"interp.most_confused(min_val=2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"os.listdir(\"/kaggle/working\")","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}