{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Ohoh! Don't turn into a smoothie! \n\n## The shake-up is likely to come! :-)\n\nTake a look at some EDA findings to observe differences in train and test data. Be careful to not overfit too badly to the training data and keep in mind the train/test differences. ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"\n![](\"https://cdn.pixabay.com/photo/2017/04/23/09/44/smoothies-2253423_1280.jpg\")
\n\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Table of contents\n\n1. [Important findings](#findings)\n2. [Prepare to start](#prepare) \n3. [What is given by the meta data?](#meta_data) \n * [Missing values](#missing_vals) \n * [Image names](#images_names) \n * [Patient id counts](#patient_ids)\n * [Overlapping patients in train/test](#overlapping_patients)\n * [Gender counts](#gender_counts)\n * [Age distributions](#age_distributions)\n * [Image location](#image_location)\n4. [Feature-feature interactions](#interactions)\n * [Ages per patient](#ages_per_patient)\n * [Age and gender](#age_gender)\n * [Age, gender and cancer](#age_gender_cancer)\n * [Individual patient information](#patient_information)\n5. [How do train and test set images differ?](#train_test_images_eda)\n * [File structure and dicom images](#file_structure)\n * [Train and test image EDA](#images_eda) \n6. [Building up the model](#modelling)\n * [Validation strategy](#validation)\n * [Dataset](#datasetloader)\n * [Augmentations](#augmentations)\n * [Loss and evaluation](#loss)\n * [Model structure](#model)\n * [Predict on whatever you like](#predict)\n * [Training loop](#training_loop)\n * [Searching for an optimal learning rate](#learning_rate_search)\n7. [Experimental zone](#experimental_zone)\n * [Creating a hold-out dataset](#hold_out)\n * [Settings](#settings)\n * [Using Bojans resized images to speed up](#resized_images)\n * [Searching for learning rate boundaries](#lr_bounds)\n * [Running a model](#running)\n * [Exploring predictions and weaknesses](#result_analysis)\n * [Submission](#submission)\n8. [Final conclusion](#conclusion)\n ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Important findings \n\n* **We can clearly observe groups of images with similar statistics that depend on the image shapes!!!**\n* **There is one group of the test images that is missing in train(1080 rows 1920 columns)! This would mean a complete new type of images that may lead to notable differences in CV scores and LB!**\n* For most of the patients there were only a few images recorded (range from 1 to 20).\n* 5% of the patients show more than 45 images. There is an extreme outlier in the test data with roughly 250 images!\n* We have more males than females in both train and test data. For the test set this imbalance is even higher!\n* We can observe **more older patients in test than in train**! The age is normally distributed in train but shows multiple modes in test.\n* The distributions of image locations look very similar for train and test.\n* We have highly imbalanced target classes!\n* Multiple images does not mean that there are multiple ages involved! \n* We can observe a high surplus of males in the ages 45 to 50 and 70, 75 in train and test but in test we can find **even more males of high age > 75**.\n* We have more malignant cases of higher age than benign cases.\n* 62 % of the malignant cases belong to males and only 38 % to females! **We have to be very careful!!! As we have a surpus of males with ages above 70 and 75 it's unclear if the sex is really an important feature for having melanoma or not.** It could also be that the age is most important and that we only have more malignant cases for males due to their higher age! ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Prepare to start ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install efficientnet_pytorch torchtoolbox","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt\nimport pylab as pl\nfrom IPython import display\nimport seaborn as sns\nsns.set()\n\nimport re\n\nimport pydicom\nimport random\n\nimport torch\nimport torch.nn.functional as F\nfrom torch.utils.data import Dataset, DataLoader\nfrom torchvision import transforms, models\nfrom efficientnet_pytorch import EfficientNet\n\nfrom scipy.special import softmax\n\nfrom sklearn.model_selection import train_test_split, StratifiedKFold\nfrom sklearn.utils.class_weight import compute_class_weight\nfrom sklearn.metrics import roc_auc_score, auc\n\nfrom skimage.io import imread\nfrom PIL import Image\n\nimport plotly.offline as py\npy.init_notebook_mode(connected=True)\nimport plotly.graph_objs as go\nfrom plotly.subplots import make_subplots\n\nimport os\nimport copy\n\nfrom albumentations import Compose, RandomCrop, Normalize,HorizontalFlip, Resize\nfrom albumentations import VerticalFlip, RGBShift, RandomBrightness\nfrom albumentations.core.transforms_interface import ImageOnlyTransform\nfrom albumentations.pytorch import ToTensor\n\nfrom tqdm.notebook import tqdm\n\nos.listdir(\"../input/\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"basepath = \"../input/siim-isic-melanoma-classification/\"\nmodelspath = \"../input/pytorch-pretrained-image-models/\"\nimagestatspath = \"../input/siimisic-melanoma-classification-image-stats/\"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"os.listdir(basepath)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_info = pd.read_csv(basepath + \"train.csv\")\ntrain_info.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_info = pd.read_csv(basepath + \"test.csv\")\ntest_info.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Our test set misses three columns: diagnosis, benign_malignant & target.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_info.shape[0] / test_info.shape[0]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Three times more entries in train than in test.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# What is given by the meta data? ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Missing values ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"missing_vals_train = train_info.isnull().sum() / train_info.shape[0]\nmissing_vals_train[missing_vals_train > 0].sort_values(ascending=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"missing_vals_test = test_info.isnull().sum() / test_info.shape[0]\nmissing_vals_test[missing_vals_test > 0].sort_values(ascending=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The anatomy shows most missing values. ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Image names ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_info.image_name.value_counts().max()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_info.image_name.value_counts().max()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Ok, great, all names are unique.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Patient id counts ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_info.patient_id.value_counts().max()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_info.patient_id.value_counts().max()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In contrast we can find multiple images for one patient!","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"patient_counts_train = train_info.patient_id.value_counts()\npatient_counts_test = test_info.patient_id.value_counts()\n\nfig, ax = plt.subplots(2,2,figsize=(20,12))\n\nsns.distplot(patient_counts_train, ax=ax[0,0], color=\"orangered\", kde=True);\nax[0,0].set_xlabel(\"Counts\")\nax[0,0].set_ylabel(\"Frequency\")\nax[0,0].set_title(\"Patient id value counts in train\");\n\nsns.distplot(patient_counts_test, ax=ax[0,1], color=\"lightseagreen\", kde=True);\nax[0,1].set_xlabel(\"Counts\")\nax[0,1].set_ylabel(\"Frequency\")\nax[0,1].set_title(\"Patient id value counts in test\");\n\nsns.boxplot(patient_counts_train, ax=ax[1,0], color=\"orangered\");\nax[1,0].set_xlim(0, 250)\nsns.boxplot(patient_counts_test, ax=ax[1,1], color=\"lightseagreen\");\nax[1,1].set_xlim(0, 250);","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"np.quantile(patient_counts_train, 0.75) - np.quantile(patient_counts_train, 0.25)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"np.quantile(patient_counts_train, 0.5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(np.quantile(patient_counts_train, 0.95))\nprint(np.quantile(patient_counts_test, 0.95))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Insights\n\n* For most of the patients we only have a few images ranging fom 1 to roughly 20.\n* More than 45 images per patient is very seldom! \n* Nonetheless we have patients with more than 100 images.\n* There is one heavy outlier patient in the test set with close to 250 images.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"200/test_info.shape[0] * 100","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This outlier patient holds ~1.8 % of the test data! Ohoh! :-O","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Overlapping patients in train/test ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_patient_ids = set(train_info.patient_id.unique())\ntest_patient_ids = set(test_info.patient_id.unique())\n\ntrain_patient_ids.intersection(test_patient_ids)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Ok, that's great! There seem to be no patients in train that can be found in test as well. We can't be sure as we don't know how the naming assignment process was designed. We might better check the images themselves as well!","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Gender counts ","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1,2,figsize=(20,5))\nsns.countplot(train_info.sex, palette=\"Reds_r\", ax=ax[0]);\nax[0].set_xlabel(\"\")\nax[0].set_title(\"Gender counts\");\n\nsns.countplot(test_info.sex, palette=\"Blues_r\", ax=ax[1]);\nax[1].set_xlabel(\"\")\nax[1].set_title(\"Gender counts\");","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Insights\n\n* We observe more males than females in both train and test data.\n* The surplus of males is even higher in test than in train!","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Age distributions ","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1,2,figsize=(20,5))\n\nsns.countplot(train_info.age_approx, color=\"orangered\", ax=ax[0]);\nlabels = ax[0].get_xticklabels();\nax[0].set_xticklabels(labels, rotation=90);\nax[0].set_xlabel(\"\");\nax[0].set_title(\"Age distribution in train\");\n\nsns.countplot(test_info.age_approx, color=\"lightseagreen\", ax=ax[1]);\nlabels = ax[1].get_xticklabels();\nax[1].set_xticklabels(labels, rotation=90);\nax[1].set_xlabel(\"\");\nax[1].set_title(\"Age distribution in test\");","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Insights\n\n* The age distribution in train looks almost normally distributed.\n* In contrast, the age distribution in test shows multiple modes and interesting peaks at the ageof 55 and 70!\n* We can observe more older patients in test than in train! This kind of imbalance can be important for our model performance if the age is an important feature. ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Image location ","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1,2,figsize=(20,5))\n\nimage_locations_train = train_info.anatom_site_general_challenge.value_counts().sort_values(ascending=False)\nimage_locations_test = test_info.anatom_site_general_challenge.value_counts().sort_values(ascending=False)\n\nsns.barplot(x=image_locations_train.index.values, y=image_locations_train.values, ax=ax[0], color=\"orangered\");\nax[0].set_xlabel(\"\");\nlabels = ax[0].get_xticklabels();\nax[0].set_xticklabels(labels, rotation=90);\nax[0].set_title(\"Image locations in train\");\n\nsns.barplot(x=image_locations_test.index.values, y=image_locations_test.values, ax=ax[1], color=\"lightseagreen\");\nax[1].set_xlabel(\"\");\nlabels = ax[1].get_xticklabels();\nax[1].set_xticklabels(labels, rotation=90);\nax[1].set_title(\"Image locations in test\");","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Insights\n\n* The distributions of image locations in train and test look very similar. \n* Most images are related to the torso or to the lower extremity.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Target distribution ","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1,2, figsize=(20,5))\n\nsns.countplot(x=train_info.diagnosis, orient=\"v\", ax=ax[0], color=\"Orangered\")\nax[0].set_xlabel(\"\")\nlabels = ax[0].get_xticklabels();\nax[0].set_xticklabels(labels, rotation=90);\nax[0].set_title(\"Diagnosis\");\n\nsns.countplot(train_info.benign_malignant, ax=ax[1], palette=\"Reds_r\");\nax[1].set_xlabel(\"\")\nax[1].set_title(\"Type\");","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Insights\n\n* The diagnosis is often unknown and for those known we observe a very high imbalance. Most likely we can't expect much from this additional target feature.\n* The target is highly imbalanced and we have to find proper strategies to deal with this kind of target distribution during learning.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_info.groupby(\"benign_malignant\").target.nunique()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The benign_malignant column is the same as the target.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Feature-feature interactions ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Ages per patient ","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"patient_ages_table_train = train_info.groupby([\"patient_id\", \"age_approx\"]).size() / train_info.groupby(\"patient_id\").size()\npatient_ages_table_train = patient_ages_table_train.unstack().transpose()\npatient_ages_table_test = test_info.groupby([\"patient_id\", \"age_approx\"]).size() / test_info.groupby(\"patient_id\").size()\npatient_ages_table_test = patient_ages_table_test.unstack().transpose()\n\npatient_with_known_ages_train = train_info[train_info.patient_id.isin(patient_ages_table_train.columns.values)]\n\nsorted_patients_train = patient_with_known_ages_train.patient_id.value_counts().index.values\npatient_with_known_ages_test = test_info[test_info.patient_id.isin(patient_ages_table_test.columns.values)]\nsorted_patients_test = patient_with_known_ages_test.patient_id.value_counts().index.values\n\nfig, ax = plt.subplots(2,1, figsize=(20,20))\nsns.heatmap(patient_ages_table_train[sorted_patients_train], cmap=\"Reds\", ax=ax[0], cbar=False);\nax[0].set_title(\"Image coverage in % per patient and age in train data\");\nsns.heatmap(patient_ages_table_test[sorted_patients_test], cmap=\"Blues\", ax=ax[1], cbar=False);\nax[1].set_title(\"Image coverage in % per patient and age in test data\");\nax[0].set_xlabel(\"\")\nax[1].set_xlabel(\"\");","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Insights\n\n* Be careful with interpreting these heatmaps: \n * **The patients are soreted by value_counts**. Patients with the most number of images are given on the left and those with only a few or a single images are on the righthand side.\n * The color represents how much percentage of the images for one patient is covered by a given age. \n * For example the most left patient in test is the one with almost 250 images. The related images are not spread over a wide range of different ages and are very concentrated at an old age. (Dark blue color at the age of 70). \n* We can conclude that more images does not mean that there are multiple ages involved! \n* It's possible that multiple images are spread over a wide range of ages but it's also possible that multiple images are concentrated at one age.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Age and gender ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Looking at the sex per patient (excluding the multiple counts due to multiple images) we can observe that we still have more males than females.","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(2,2,figsize=(20,15))\n\nsns.boxplot(train_info.sex, train_info.age_approx, ax=ax[0,0], palette=\"Reds_r\");\nax[0,0].set_title(\"Age per gender in train\");\n\nsns.boxplot(test_info.sex, test_info.age_approx, ax=ax[0,1], palette=\"Blues_r\");\nax[0,1].set_title(\"Age per gender in test\");\n\nsns.countplot(train_info.age_approx, hue=train_info.sex, ax=ax[1,0], palette=\"Reds_r\");\nsns.countplot(test_info.age_approx, hue=test_info.sex, ax=ax[1,1], palette=\"Blues_r\");","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Insights\nThere are some significant differences in train and test regarding the gender per age level:\n\n* At the ages between 25 and 35 we have much more females than males in train but a balanced count in test!\n* We can observe a high surplus of males in the ages 45 to 50 and 70, 75 in train and test but in test we can find even more males of high age > 75.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Age, gender and cancer ","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"sex_and_cancer_map = train_info.groupby(\n [\"benign_malignant\", \"sex\"]\n).size().unstack(level=0) / train_info.groupby(\"benign_malignant\").size() * 100\n\ncancer_sex_map = train_info.groupby(\n [\"benign_malignant\", \"sex\"]\n).size().unstack(level=1) / train_info.groupby(\"sex\").size() * 100\n\n\nfig, ax = plt.subplots(1,3,figsize=(20,5))\n\nsns.boxplot(train_info.benign_malignant, train_info.age_approx, ax=ax[0], palette=\"Greens\");\nax[0].set_title(\"Age and cancer\");\nax[0].set_xlabel(\"\");\n\nsns.heatmap(sex_and_cancer_map, annot=True, cmap=\"Greens\", cbar=False, ax=ax[1])\nax[1].set_xlabel(\"\")\nax[1].set_ylabel(\"\");\n\nsns.heatmap(cancer_sex_map, annot=True, cmap=\"Greens\", cbar=False, ax=ax[2])\nax[2].set_xlabel(\"\")\nax[2].set_ylabel(\"\");","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Insights\n\n* We have more malignant cases of higher age than benign cases.\n* 62 % of the malignant cases belong to males and only 38 % to females.\n* Roughly 2 % of the males in the train dataset show malignant cases, but only 1.4 % of the females.\n\nWe have to be very careful!!! As we have a surpus of males with ages above 70 and 75 it's unclear if the sex is really an important feature for having melanoma or not. It could also be that the age is most important and that we only have more malignant cases for males due to their higher age! ","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(2,2,figsize=(20,15))\n\nsns.countplot(train_info[train_info.benign_malignant==\"benign\"].age_approx, hue=train_info.sex, palette=\"Purples_r\", ax=ax[0,0])\nax[0,0].set_title(\"Benign cases in train\");\n\nsns.countplot(train_info[train_info.benign_malignant==\"malignant\"].age_approx, hue=train_info.sex, palette=\"Oranges_r\", ax=ax[0,1])\nax[0,1].set_title(\"Malignant cases in train\");\n\nsns.violinplot(train_info.sex, train_info.age_approx, hue=train_info.benign_malignant, split=True, ax=ax[1,0], palette=\"Greens_r\");\nsns.violinplot(train_info.benign_malignant, train_info.age_approx, hue=train_info.sex, split=True, ax=ax[1,1], palette=\"RdPu\");","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Insights\n\n* For the benign cases we can see that there is still a surplus of males in the ages of 45 and 70, but the other ones look quite good and balanced.\n* **In contrast we can find a high gender imbalance for a wide range of ages for the malignant cases!** That's really interesting and the features age and gender as well as their interaction with cancer are definitely some to play with during modelling.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Individual patient information \n\nLet's collect some information for each patient:\n\n* the number of recorded images\n* the gender\n* the min, max age and the age span\n* the number of benign & malignant cases\n* the minimum and maximum age of a patient with malignant cases","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"patient_gender_train = train_info.groupby(\"patient_id\").sex.unique().apply(lambda l: l[0])\npatient_gender_test = test_info.groupby(\"patient_id\").sex.unique().apply(lambda l: l[0])\n\ntrain_patients = pd.DataFrame(index=patient_gender_train.index.values, data=patient_gender_train.values, columns=[\"sex\"])\ntest_patients = pd.DataFrame(index=patient_gender_test.index.values, data=patient_gender_test.values, columns=[\"sex\"])\n\ntrain_patients.loc[:, \"num_images\"] = train_info.groupby(\"patient_id\").size()\ntest_patients.loc[:, \"num_images\"] = test_info.groupby(\"patient_id\").size()\n\ntrain_patients.loc[:, \"min_age\"] = train_info.groupby(\"patient_id\").age_approx.min()\ntrain_patients.loc[:, \"max_age\"] = train_info.groupby(\"patient_id\").age_approx.max()\ntest_patients.loc[:, \"min_age\"] = test_info.groupby(\"patient_id\").age_approx.min()\ntest_patients.loc[:, \"max_age\"] = test_info.groupby(\"patient_id\").age_approx.max()\n\ntrain_patients.loc[:, \"age_span\"] = train_patients[\"max_age\"] - train_patients[\"min_age\"]\ntest_patients.loc[:, \"age_span\"] = test_patients[\"max_age\"] - test_patients[\"min_age\"]\n\ntrain_patients.loc[:, \"benign_cases\"] = train_info.groupby([\"patient_id\", \"benign_malignant\"]).size().loc[:, \"benign\"]\ntrain_patients.loc[:, \"malignant_cases\"] = train_info.groupby([\"patient_id\", \"benign_malignant\"]).size().loc[:, \"malignant\"]\ntrain_patients[\"min_age_malignant\"] = train_info.groupby([\"patient_id\", \"benign_malignant\"]).age_approx.min().loc[:, \"malignant\"]\ntrain_patients[\"max_age_malignant\"] = train_info.groupby([\"patient_id\", \"benign_malignant\"]).age_approx.max().loc[:, \"malignant\"]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_patients.sort_values(by=\"malignant_cases\", ascending=False).head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig, ax = plt.subplots(2,2,figsize=(20,12))\nsns.countplot(train_patients.sex, ax=ax[0,0], palette=\"Reds\")\nax[0,0].set_title(\"Gender counts with unique patient ids in train\")\nsns.countplot(test_patients.sex, ax=ax[0,1], palette=\"Blues\");\nax[0,1].set_title(\"Gender counts with unique patient ids in test\");\n\ntrain_age_span_perc = train_patients.age_span.value_counts() / train_patients.shape[0] * 100\ntest_age_span_perc = test_patients.age_span.value_counts() / test_patients.shape[0] * 100\n\nsns.barplot(train_age_span_perc.index, train_age_span_perc.values, ax=ax[1,0], color=\"Orangered\");\nsns.barplot(test_age_span_perc.index, test_age_span_perc.values, ax=ax[1,1], color=\"Lightseagreen\");\nax[1,0].set_title(\"Patients age span in train\")\nax[1,1].set_title(\"Patients age span in test\")\nfor n in range(2):\n ax[1,n].set_ylabel(\"% in data\")\n ax[1,n].set_xlabel(\"age span\");","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Insights\n\n* Even on the patient id level we have more males than females in both train and test data.\n* The age span has more cases of 5 years in train than in test and less example with no age differences at all (age span of 0).","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# How do train and test set images differ? ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## File structure and dicom images \nLet's take a look at the file structure:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"example_files = os.listdir(basepath + \"train/\")[0:2]\nexample_files","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Ok, dicom images and the image names can be found in the train and test info (meta data) as well:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_info.head(2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Perhaps we can do both: exploring the images and building up datasets and dataloaders for modelling. Let's start with the dataset and the corresponding dataframes. All we need it the imagepath and for the training data the target:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_info[\"dcm_path\"] = basepath + \"train/\" + train_info.image_name + \".dcm\"\ntest_info[\"dcm_path\"] = basepath + \"test/\" + test_info.image_name + \".dcm\"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(train_info.dcm_path[0])\nprint(test_info.dcm_path[0])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's load an example:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"example_dcm = pydicom.dcmread(train_info.dcm_path[2])\nexample_dcm","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Probably the most interesting information is given by Rows, Columns and Pixel Data.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"image = example_dcm.pixel_array\nprint(image.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As reading the dicom image file is really slow, let's use the jpeg files:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_info[\"image_path\"] = basepath + \"jpeg/train/\" + train_info.image_name + \".jpg\"\ntest_info[\"image_path\"] = basepath + \"jpeg/test/\" + test_info.image_name + \".jpg\"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Train and test image EDA \n\n**Caution** Everything after this part is under construction! ;-)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"I have created a dataset that covers some simple image statistics for train and test set images. It's not complete at the moment and I will update it soon, but we can already do some EDA using it.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"os.listdir(imagestatspath)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_image_stats = pd.read_csv(imagestatspath + \"test_image_stats.csv\")\ntest_image_stats.head(1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_image_stats_1 = pd.read_csv(imagestatspath + \"train_image_stats_10000.csv\")\ntrain_image_stats_2 = pd.read_csv(imagestatspath + \"train_image_stats_20000.csv\")\ntrain_image_stats_3 = pd.read_csv(imagestatspath + \"train_image_stats_toend.csv\")\ntrain_image_stats_4 = train_image_stats_1.append(train_image_stats_2)\ntrain_image_stats = train_image_stats_4.append(train_image_stats_3)\ntrain_image_stats.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Test image statistics\n\nTo get started I have taken the mean, std and skewness of each test image and performed a 3D-scatterplot. To understand wheather the result also depends on the image shape, I have colored the points with the value of columns each image has and added a text description to each point that holds the row value.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_test = True","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"If you set plot_test to False the following scatter plot will show statistics of all training examples instead:","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"if plot_test:\n N = test_image_stats.shape[0]\n selected_data = test_image_stats\n my_title = \"Test image statistics\"\nelse:\n N = train_image_stats.shape[0]\n selected_data = train_image_stats\n my_title = \"Train image statistics\"\n\ntrace1 = go.Scatter3d(\n x=selected_data.img_mean.values[0:N], \n y=selected_data.img_std.values[0:N],\n z=selected_data.img_skew.values[0:N],\n mode='markers',\n text=selected_data[\"rows\"].values[0:N],\n marker=dict(\n color=selected_data[\"columns\"].values[0:N],\n colorscale = \"Jet\",\n colorbar=dict(thickness=10, title=\"image columns\", len=0.8),\n opacity=0.4,\n size=2\n )\n)\n\nfigure_data = [trace1]\nlayout = go.Layout(\n title = my_title,\n scene = dict(\n xaxis = dict(title=\"Image mean\"),\n yaxis = dict(title=\"Image standard deviation\"),\n zaxis = dict(title=\"Image skewness\"),\n ),\n margin=dict(\n l=0,\n r=0,\n b=0,\n t=0\n ),\n showlegend=True\n)\n\nfig = go.Figure(data=figure_data, layout=layout)\npy.iplot(fig, filename='simple-3d-scatter')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Insights\n\nUhh, that's a bit surprising... \n\n* One example - there are a lot of images with 6000 columns (red cluster) that show a high image mean close to 160, a wide range of standard deviations but a small range of skewness compared to other images with different column size.\n* There is also a very interesting kind of outlier image shape with 4288 columns that show extreme negative image skewnesses but have narrow image means and stds. \n* Looking at the training examples it seems that one group of the test images is missing (1080 rows 1920 columns)! This would mean a complete new type of images that may lead to notable differences in CV scores and LB!\n\n\nTo conclude - Performing a bit more EDA on image shapes and their relationships with image statistics and differences in train and test sets may be of great help to understand this dataset better and to choose proper modelling structures. ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"test_image_stats.groupby([\"rows\", \"columns\"]).size().sort_values(ascending=False).iloc[0:10] / test_image_stats.shape[0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_image_stats.groupby([\"rows\", \"columns\"]).size().sort_values(ascending=False).iloc[0:10] / train_image_stats.shape[0]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There is indeed a big group of test images (1716 images with 1080 rows 1920 columns) that is not presented in train (at least not with similar % of the data)!","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"examples1 = {\"rows\": 1080, \"columns\": 1920}\nexamples2 = {\"rows\": 4000, \"columns\": 6000}","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"selection1 = np.random.choice(test_image_stats[\n (test_image_stats[\"rows\"]==examples1[\"rows\"]) & (test_image_stats[\"columns\"]==examples1[\"columns\"])\n].path.values, size=8, replace=False)\n\nfig, ax = plt.subplots(2,4,figsize=(20,8))\n\nfor n in range(2):\n for m in range(4):\n path = selection1[m + n*4]\n dcm_file = pydicom.dcmread(path)\n image = dcm_file.pixel_array\n ax[n,m].imshow(image)\n ax[n,m].grid(False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This is the myterious unknown group of test images that holds 15% of the test data! Keep them in mind. ;-)","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"selection2 = np.random.choice(test_image_stats[\n (test_image_stats[\"rows\"]==examples2[\"rows\"]) & (test_image_stats[\"columns\"]==examples2[\"columns\"])\n].path.values, size=8, replace=False)\n\nfig, ax = plt.subplots(2,4,figsize=(20,6))\n\nfor n in range(2):\n for m in range(4):\n path = selection2[m + n*4]\n dcm_file = pydicom.dcmread(path)\n image = dcm_file.pixel_array\n ax[n,m].imshow(image)\n ax[n,m].grid(False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Ohoh! :-O It's really astonishing how well these images can be grouped given the image shapes! Browsing through the shapes above you can cleary observe these kind of groups. ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Building up the model ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Dataset ","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"class MelanomaDataset(Dataset):\n \n def __init__(self, df, transform=None):\n self.transform = transform\n self.df = df\n \n def __getitem__(self, idx):\n path = self.df.iloc[idx][\"image_path\"]\n image = Image.open(path)\n \n if self.transform:\n image = self.transform(image)\n \n if \"target\" in self.df.columns.values:\n target = self.df.iloc[idx][\"target\"]\n return {\"image\": image,\n \"target\": target}\n else:\n return {\"image\": image}\n \n def __len__(self):\n return len(self.df)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"class ResizedNpyMelanomaDataset(Dataset):\n \n def __init__(self, npy_file, indices_to_select, df=None, transform=None):\n self.transform = transform\n self.npy_file = npy_file\n self.df = df\n self.indices_to_select = indices_to_select\n \n def __getitem__(self, n):\n idx = self.indices_to_select[n]\n \n image = Image.fromarray(self.npy_file[idx])\n if self.transform:\n image = self.transform(image)\n \n target = self.df.loc[idx].target\n \n return {\"image\": image,\n \"target\": target}\n \n def __len__(self):\n return len(self.indices_to_select)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":false,"trusted":true,"_kg_hide-output":false},"cell_type":"code","source":"class AlbuMelanomaDataset(Dataset):\n \n def __init__(self, df, transform=None):\n self.transform = transform\n self.df = df\n \n def __getitem__(self, idx):\n path = self.df.iloc[idx][\"image_path\"]\n image = cv2.imread(path)\n image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n \n if self.transform:\n augmented = self.transform(image=image)\n image = augmented['image']\n \n if \"target\" in self.df.columns.values:\n target = self.df.iloc[idx][\"target\"]\n return {\"image\": image,\n \"target\": target}\n else:\n return {\"image\": image}\n \n def __len__(self):\n return len(self.df)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Augmentations ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Many thanks to Roman for his notebook [Melanoma. Pytorch starter. EfficientNet](https://www.kaggle.com/nroman/melanoma-pytorch-starter-efficientnet). I liked the special augmentations very much and will use them here as well:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def random_microscope(img):\n circle = cv2.circle((np.ones(img.shape) * 255).astype(np.uint8), # image placeholder\n (img.shape[0]//2, img.shape[1]//2), # center point of circle\n random.randint(img.shape[0]//2 - 3, img.shape[0]//2 + 15), # radius\n (0, 0, 0), # color\n -1)\n\n mask = circle - 255\n img = np.multiply(img, mask)\n return img","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":false,"trusted":true},"cell_type":"code","source":"import cv2\n\nclass Microscope:\n \"\"\"\n Cutting out the edges around the center circle of the image\n Imitating a picture, taken through the microscope\n\n Args:\n p (float): probability of applying an augmentation\n \"\"\"\n\n def __init__(self, p: float = 0.5):\n self.p = p\n \n def __call__(self, img):\n \"\"\"\n Args:\n img (PIL Image): Image to apply transformation to.\n\n Returns:\n PIL Image: Image with transformation.\n \"\"\"\n img = np.asarray(img)\n if random.random() < p:\n img = random_microscope(img)\n img = Image.fromarray(np.uint8(img))\n return img\n \n def __repr__(self):\n return f'{self.__class__.__name__}(p={self.p})'\n\n \nclass AlbuMicroscope(ImageOnlyTransform):\n \n def __init__(self, always_apply=False, p=0.5):\n super(AlbuMicroscope, self).__init__(always_apply, p)\n \n def apply(self, img, **params):\n return random_microscope(img)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def transform_fun(resize_shape, key=\"train\", plot=False):\n train_sequence = [transforms.Resize((resize_shape, resize_shape)),\n transforms.RandomHorizontalFlip(),\n transforms.RandomVerticalFlip(),\n Microscope(p=0.6)]\n dev_sequence = [transforms.Resize((resize_shape, resize_shape))]\n if plot==False:\n train_sequence.extend([\n transforms.ToTensor(),\n transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])])\n dev_sequence.extend([\n transforms.ToTensor(),\n transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])])\n \n data_transforms = {'train': transforms.Compose(train_sequence),\n 'dev': transforms.Compose(dev_sequence),\n 'test_tta': transforms.Compose(train_sequence),\n 'test': transforms.Compose(dev_sequence)}\n return data_transforms[key]","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":false,"trusted":true},"cell_type":"code","source":"def albu_transform_fun(resize_shape=None, key=\"train\", plot=False):\n train_sequence = [\n Resize(resize_shape, resize_shape),\n RandomCrop(224,224),\n VerticalFlip(),\n HorizontalFlip(),\n RGBShift(r_shift_limit=40),\n RandomBrightness(0.1),\n AlbuMicroscope(p=0.6)]\n dev_sequence = [Resize(224, 224)]\n \n if plot==False:\n train_sequence.extend([\n Normalize(mean=[0.485, 0.456, 0.406],\n std=[0.229, 0.224, 0.225],),\n ToTensor()])\n dev_sequence.extend([Normalize(mean=[0.485, 0.456, 0.406],\n std=[0.229, 0.224, 0.225],),\n ToTensor()])\n \n data_transforms = {'train': Compose(train_sequence),\n 'dev': Compose(dev_sequence),\n 'test_tta': Compose(train_sequence),\n 'test': Compose(dev_sequence)}\n return data_transforms[key]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's take a look at some example images and their augmented couterparts in train:","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"N = 10\n\nfig, ax = plt.subplots(2,N,figsize=(20,5))\n\nselection = np.random.choice(train_info.index.values, size=N, replace=False)\n\nfor n in range(N):\n \n org_image = cv2.imread(train_info.loc[selection[n]].image_path)\n org_image = cv2.cvtColor(org_image, cv2.COLOR_BGR2RGB)\n label = train_info.loc[selection[n]].target\n augmented = albu_transform_fun(resize_shape=256, key=\"train\", plot=True)(**{\"image\":org_image, \"label\": label})\n ax[0,n].imshow(org_image)\n ax[1,n].imshow(augmented[\"image\"])\n ax[1,n].axis(\"off\")\n ax[0,n].axis(\"off\")\n ax[0,n].set_title(\"Original\")\n ax[1,n].set_title(\"Augmented\");","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Loss and evaluation ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Cross entropy loss\n\n\n$$L_{bce} = - \\sum_{n}^{N} \\sum_{k}^{2} t_{n,k} \\cdot \\log(y_{n,k}) = \\sum_{n}^{N} \\cdot l_{bce}$$\n\n$$l_{bce} = - \\sum_{k}^{2} t_{n,k} \\cdot \\log(y_{n,k}) $$","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":false},"cell_type":"code","source":"def get_ce_loss(): \n criterion = torch.nn.CrossEntropyLoss()\n return criterion","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Weighted cross entropy loss\n\n\n$$L_{bce} = - \\sum_{n}^{N} \\sum_{k}^{2} \\alpha_{k} \\cdot t_{n,k} \\cdot \\log(y_{n,k}) $$","execution_count":null},{"metadata":{"_kg_hide-input":false,"trusted":true},"cell_type":"code","source":"def get_wce_loss(train_targets):\n weights = compute_class_weight(y=train_targets,\n class_weight=\"balanced\",\n classes=np.unique(train_targets)) \n class_weights = torch.FloatTensor(weights)\n if device.type==\"cuda\":\n class_weights = class_weights.cuda()\n criterion = torch.nn.CrossEntropyLoss(weight=class_weights)\n return criterion","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Focal loss","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"$$L_{focal} = - \\sum_{n}^{N} \\sum_{k}^{2} \\alpha_{k} \\cdot t_{n,k} \\cdot (1-y_{n,k})^{\\gamma} \\cdot \\log(y_{n,k})$$","execution_count":null},{"metadata":{"_kg_hide-input":false,"trusted":true},"cell_type":"code","source":"class MulticlassFocalLoss(torch.nn.Module):\n \n def __init__(self, train_targets=None, gamma=2):\n super(MulticlassFocalLoss, self).__init__()\n self.gamma = gamma\n if train_targets is None:\n self.class_weights = None\n else:\n weights = compute_class_weight(y=train_targets,\n class_weight=\"balanced\",\n classes=np.unique(train_targets)) \n self.class_weights = torch.FloatTensor(weights)\n if device.type==\"cuda\":\n self.class_weights = self.class_weights.cuda()\n \n def forward(self, input, target):\n if self.class_weights is None:\n ce_loss = F.cross_entropy(input, target, reduction='none')\n else:\n ce_loss = F.cross_entropy(input, target, reduction='none', weight=self.class_weights)\n pt = torch.exp(-ce_loss)\n loss = (1-pt)**self.gamma * ce_loss\n return torch.mean(loss)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Model structure \n\nCaution: I haven't implemented some way for densenet so far.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"os.listdir(modelspath)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def get_model(kind=\"resnet34\"):\n if kind == \"resnet34\":\n model = models.resnet34(pretrained=False)\n model.load_state_dict(torch.load(modelspath + \"resnet34.pth\"))\n elif kind == \"resnet50\":\n model = models.resnet50(pretrained=False)\n model.load_state_dict(torch.load(modelspath + \"resnet50.pth\"))\n elif kind == \"densenet121\":\n model = models.densenet121(pretrained=False)\n model.load_state_dict(torch.load(modelspath + \"densenet121.pth\"))\n elif kind == \"densenet201\":\n model = models.densenet201(pretrained=False)\n model.load_state_dict(torch.load(modelspath + \"densenet201.pth\"))\n elif kind == \"efficientnet_b1\":\n model = EfficientNet.from_pretrained('efficientnet-b1')\n else:\n model = models.resnet34(pretrained=False)\n model.load_state_dict(torch.load(modelspath + \"resnet34.pth\"))\n return model ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":false},"cell_type":"code","source":"def init_weights(m):\n if type(m) == torch.nn.Linear:\n torch.nn.init.xavier_uniform_(m.weight)\n m.bias.data.fill_(0.01)\n\ndef build_model(the_model):\n model = get_model(the_model)\n \n if \"efficientnet\" in the_model:\n num_features = model._fc.in_features\n else:\n num_features = model.fc.in_features\n \n basic_modules = torch.nn.Sequential(torch.nn.Linear(num_features, 128),\n torch.nn.ReLU(),\n torch.nn.BatchNorm1d(128),\n torch.nn.Dropout(0.2),\n\n torch.nn.Linear(128, num_classes))\n \n if \"efficientnet\" in the_model:\n model._fc = basic_modules\n else:\n model.fc = basic_modules\n \n \n return model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Predict on whatever you like ","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":false},"cell_type":"code","source":"# make sure that counter*batch_size is the same as len(dataset)\ndef predict(fold_results, dataloader, TTA=1):\n device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n avg_preds = np.zeros(len(dataloader.dataset))\n avg_probas = np.zeros((len(dataloader.dataset),2))\n \n for fold_num in fold_results.keys():\n \n results = fold_results[fold_num]\n model = results.model\n \n dataloader_iterator = tqdm(dataloader, total=int(len(dataloader)))\n \n for t in range(TTA):\n print(\"TTA phase {}\".format(t))\n for counter, data in enumerate(dataloader_iterator): \n image_input = data[\"image\"]\n image_input = image_input.to(device, dtype=torch.float)\n\n pred_probas = model(image_input)\n _, preds = torch.max(pred_probas, 1)\n\n avg_preds[\n (counter*dataloader.batch_size):(dataloader.batch_size*(counter+1))\n ] += preds.cpu().detach().numpy()/(len(fold_results)*TTA)\n avg_probas[\n (counter*dataloader.batch_size):(dataloader.batch_size*(counter+1))\n ] += softmax(pred_probas.cpu().detach().numpy(), axis=1)/(len(fold_results)*TTA)\n \n return avg_preds, avg_probas","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Searching for an optimal learning rate ","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":false},"cell_type":"code","source":"def get_scheduler(optimiser, min_lr, max_lr, stepsize):\n # suggested_stepsize = 2*num_iterations_within_epoch\n stepsize_up = np.int(stepsize/2)\n scheduler = torch.optim.lr_scheduler.CyclicLR(optimizer=optimiser,\n base_lr=min_lr,\n max_lr=max_lr,\n step_size_up=stepsize_up,\n step_size_down=stepsize_up,\n mode=\"triangular\")\n return scheduler\n ","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def get_lr_search_scheduler(optimiser, min_lr, max_lr, max_iterations):\n # max_iterations should be the number of steps within num_epochs_*epoch_iterations\n # this way the learning rate increases linearily within the period num_epochs*epoch_iterations \n scheduler = torch.optim.lr_scheduler.CyclicLR(optimizer=optimiser, \n base_lr=min_lr,\n max_lr=max_lr,\n step_size_up=max_iterations,\n step_size_down=max_iterations,\n mode=\"triangular\")\n \n return scheduler","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Training loop ","execution_count":null},{"metadata":{"_kg_hide-input":false,"trusted":true},"cell_type":"code","source":"from scipy.special import expit\n\ndef run_training(criterion,\n num_epochs,\n dataloaders_dict,\n fold_num,\n patience,\n results,\n find_lr):\n \n if find_lr:\n phases = [\"train\"]\n else:\n phases = [\"train\", \"dev\"]\n \n best_auc = 0\n patience_counter = 0\n epsilon = 1e-7\n \n for epoch in range(num_epochs):\n \n for phase in phases:\n \n dataloader = dataloaders_dict[phase]\n dataloader_iterator = tqdm(dataloader, total=int(len(dataloader)))\n \n if phase==\"train\":\n results.model.train()\n else:\n results.model.eval()\n \n all_preds = np.zeros(len(dataloader)*dataloader.batch_size)\n all_targets = np.zeros(len(dataloader)*dataloader.batch_size) \n running_loss = 0.0\n running_true_positives = 0\n running_false_positives = 0\n running_false_negatives = 0\n \n \n for counter, data in enumerate(dataloader_iterator):\n image_input = data[\"image\"]\n target_input = data[\"target\"]\n \n image_input = image_input.to(device, dtype=torch.float)\n target_input = target_input.to(device, dtype=torch.long)\n \n results.optimiser.zero_grad()\n \n raw_output = results.model(image_input) \n \n _, preds = torch.max(raw_output,1)\n \n running_true_positives += (preds*target_input).sum().cpu().detach().numpy()\n running_false_positives += ((1-target_input)*preds).sum().cpu().detach().numpy()\n running_false_negatives += (target_input*(1-preds)).sum().cpu().detach().numpy()\n\n precision = running_true_positives / (running_true_positives + running_false_positives + epsilon)\n recall = running_true_positives / (running_true_positives + running_false_negatives + epsilon)\n \n f1_score = 2*precision*recall / (precision+recall+epsilon) \n \n \n results.results[phase].learning_rates.append(optimiser.state_dict()[\"param_groups\"][0][\"lr\"])\n results.results[phase].precision.append(precision)\n results.results[phase].recall.append(recall)\n results.results[phase].f1_scores.append(f1_score)\n \n batch_size = dataloader.batch_size\n all_targets[(counter*batch_size):((counter+1)*batch_size)] = target_input.cpu().detach().numpy()\n all_preds[(counter*batch_size):((counter+1)*batch_size)] = preds.cpu().detach().numpy()\n \n loss = criterion(raw_output, target_input)\n # redo the average over mini_batch\n running_loss += (loss.item() * batch_size)\n \n # save averaged loss over processed number of batches:\n processed_loss = running_loss / ((counter+1) * batch_size)\n results.results[phase].losses.append(processed_loss)\n \n if phase == 'train':\n loss.backward()\n results.optimiser.step()\n if results.scheduler is not None:\n results.scheduler.step()\n \n epoch_auc_score = roc_auc_score(all_targets, all_preds)\n results.results[phase].epoch_scores.append(epoch_auc_score)\n \n \n # average over all samples to obtain the epoch loss\n epoch_loss = running_loss / len(dataloader.dataset)\n results.results[phase].epoch_losses.append(epoch_loss)\n \n print(\"fold: {}, epoch: {}, phase: {}, e-loss: {}, e-auc: {}\".format(\n fold_num, epoch, phase, epoch_loss, epoch_auc_score))\n \n if not find_lr:\n if phase == \"dev\":\n if epoch_auc_score >= best_auc:\n best_auc = epoch_auc_score\n best_model_wts = copy.deepcopy(results.model.state_dict())\n best_model_optimiser = copy.deepcopy(results.optimiser.state_dict())\n best_scheduler = copy.deepcopy(results.scheduler.state_dict())\n best_epoch = epoch\n best_loss = processed_loss\n else:\n patience_counter += 1\n if patience_counter == patience:\n print(\"Model hasn't improved for {} epochs. Training finished.\".format(patience))\n break\n \n # load best model weights\n if not find_lr:\n results.model.load_state_dict(best_model_wts)\n results.optimiser.load_state_dict(best_model_optimiser)\n results.scheduler.load_state_dict(best_scheduler)\n results.best_epoch = best_epoch\n results.best_loss = best_loss\n return results","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"class ResultsBean:\n \n def __init__(self):\n \n self.precision = []\n self.recall = []\n self.f1_scores = []\n self.losses = []\n self.learning_rates = []\n self.epoch_losses = []\n self.epoch_scores = []\n\nclass Results:\n \n def __init__(self, fold_num, model=None, optimiser=None, scheduler=None, model_kind=\"resnet34\"):\n self.model = model\n self.model_kind = model_kind\n self.optimiser = optimiser\n self.scheduler = scheduler\n self.best_epoch = 0\n self.best_loss = 0\n \n self.fold_num = fold_num\n self.train_results = ResultsBean()\n self.dev_results = ResultsBean()\n self.results = {\"train\": self.train_results,\n \"dev\": self.dev_results}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":false},"cell_type":"code","source":"def train(model,\n model_kind,\n criterion,\n optimiser,\n num_epochs,\n dataloaders_dict,\n fold_num,\n scheduler,\n patience,\n find_lr=False):\n \n single_results = Results(fold_num=fold_num,\n model=model,\n optimiser=optimiser,\n scheduler=scheduler,\n model_kind=model_kind)\n \n \n single_results = run_training(criterion,\n num_epochs,\n dataloaders_dict,\n fold_num,\n patience,\n single_results, \n find_lr=find_lr)\n \n return single_results","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def save_as_csv(series, name, path):\n df = pd.DataFrame(index=np.arange(len(series)), data=series, columns=[name])\n output_path = path + name + \".csv\"\n df.to_csv(output_path, index=False)\n\ndef save_results(results, foldername):\n for fold in results.keys():\n \n base_dir = foldername + \"/fold_\" + str(fold) + \"/\"\n if not os.path.exists(base_dir):\n os.makedirs(base_dir)\n \n # save the model for inference\n model = results[fold].model\n model_kind = results[fold].model_kind\n #model_path = base_dir + model_kind + \".pth\"\n #torch.save(model.state_dict(), model_path)\n \n # save checkpoint for inference and retraining:\n checkpoint_path = base_dir + model_kind + \".tar\"\n torch.save({\n 'epoch': results[fold].best_epoch,\n 'loss': results[fold].best_loss,\n 'model_state_dict': results[fold].model.state_dict(),\n 'optimizer_state_dict': results[fold].optimiser.state_dict(),\n 'scheduler_state_dict': results[fold].scheduler.state_dict()}, checkpoint_path)\n \n for phase in [\"train\", \"dev\"]:\n losses = results[fold].results[phase].losses\n epoch_losses = results[fold].results[phase].epoch_losses\n epoch_scores = results[fold].results[phase].epoch_scores\n lr_rates = results[fold].results[phase].learning_rates\n f1_scores = results[fold].results[phase].f1_scores\n precision = results[fold].results[phase].precision\n recall = results[fold].results[phase].recall\n \n save_as_csv(losses, phase + \"_losses\", base_dir)\n save_as_csv(epoch_losses, phase + \"_epoch_losses\", base_dir)\n save_as_csv(epoch_scores, phase + \"_epoch_scores\", base_dir)\n save_as_csv(lr_rates, phase + \"_lr_rates\", base_dir)\n save_as_csv(f1_scores, phase + \"_f1_scores\", base_dir)\n save_as_csv(precision, phase + \"_precision\", base_dir)\n save_as_csv(recall, phase + \"_recall\", base_dir)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def load_checkpoint(model_kind,\n checkpoint_path,\n for_inference,\n single_results,\n lr,\n num_epochs,\n min_lr, max_lr, len_train):\n device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n \n checkpoint = torch.load(checkpoint_path)\n \n single_results.model = build_model(model_kind)\n single_results.model.load_state_dict(checkpoint[\"model_state_dict\"])\n single_results.model.to(device)\n \n if \"efficientnet\" in model_kind:\n single_results.optimiser = torch.optim.SGD(single_results.model._fc.parameters(), lr=lr)\n else:\n single_results.optimiser = torch.optim.SGD(single_results.model.fc.parameters(), lr=lr)\n single_results.optimiser.load_state_dict(checkpoint[\"optimizer_state_dict\"])\n \n max_iterations = num_epochs * len_train\n single_results.scheduler = get_lr_search_scheduler(single_results.optimiser, min_lr, max_lr, max_iterations)\n single_results.scheduler.load_state_dict(checkpoint[\"scheduler_state_dict\"])\n \n single_results.best_epoch = checkpoint[\"epoch\"]\n single_results.best_loss = checkpoint[\"loss\"]\n \n # set into inference state\n if for_inference:\n single_results.model.eval()\n else:\n single_results.model.train()\n \n return single_results","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def load_results(save_folder, total_folds, model_kind, lr, num_epochs, len_train, min_lr, max_lr, for_inference=True):\n results = {}\n \n for fold in range(total_folds): \n single_results = Results(fold)\n \n base_path = save_folder + \"/fold_\" + str(fold) + \"/\"\n checkpoint_path = base_path + model_kind + \".tar\"\n single_results = load_checkpoint(model_kind,\n checkpoint_path,\n for_inference,\n single_results,\n lr,\n num_epochs,\n min_lr, max_lr, len_train)\n \n for phase in [\"train\", \"dev\"]:\n single_results.results[phase].losses = pd.read_csv(base_path + phase + \"_losses.csv\")\n single_results.results[phase].epoch_losses = pd.read_csv(base_path + phase + \"_epoch_losses.csv\")\n single_results.results[phase].epoch_scores = pd.read_csv(base_path + phase + \"_epoch_scores.csv\")\n single_results.results[phase].learning_rates = pd.read_csv(base_path + phase + \"_lr_rates.csv\")\n single_results.results[phase].f1_scores = pd.read_csv(base_path + phase + \"_f1_scores.csv\")\n single_results.results[phase].precision = pd.read_csv(base_path + phase + \"_precision.csv\")\n single_results.results[phase].recall = pd.read_csv(base_path + phase + \"_recall.csv\")\n \n results[fold] = single_results\n return results","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Experimental zone ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Creating a hold-out dataset \n\nWe have observed a big missing group of test images that is not present in the training data. With ~15 % this should have a big impact on the score-differences between CV-scores and the leaderboard. To find strategies to overcome this problem we could exclude a small common group of the training data and use their images within a hold-out dataset. Furthermore we should also use a part of the remaining images to cover all other training groups in the hold-out as well. ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_image_stats.head(1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_image_stats.groupby(\n [\"rows\", \"columns\"]).size().sort_values(ascending=False).iloc[0:10] / train_image_stats.shape[0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"relevant_groups = train_image_stats.groupby(\n [\"rows\", \"columns\"]).size().sort_values(ascending=False).iloc[0:10].index.values","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Ok, let's pick the group of 480 rows and 640 columns or the group with 3456 rows and 5184 columns. They are big enough to simulate what will happen when they are present in the hold-out data but not in train. To get a feeling how different their image information may look like from the other groups, we could plot the median of image means and stds of all groups and our candidates:","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"group_stats = pd.DataFrame(train_image_stats.groupby([\"rows\", \"columns\"]).img_mean.median().loc[relevant_groups])\ngroup_stats[\"img_std\"] = train_image_stats.groupby([\"rows\", \"columns\"]).img_std.median().loc[relevant_groups]","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(20,5))\nplt.scatter(group_stats.img_mean, group_stats.img_std, label=\"remaining train groups\");\nplt.scatter(group_stats.loc[(480, 640)].img_mean, group_stats.loc[(480, 640)].img_std,\n c=\"lime\", label=\"candidate 480, 640\")\nplt.scatter(group_stats.loc[(3456, 5184)].img_mean, group_stats.loc[(3456, 5184)].img_std,\n c=\"deeppink\", label=\"candidate 3456, 5184\");\nplt.title(\"Image statistics groups\");\nplt.legend()\nplt.xlabel(\"Median of group image means\")\nplt.ylabel(\"Std of group image means\");","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Ok, both groups look somehow far away from the others. Let's take the smaller one: 3456, 5184.","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"selected_hold_out_group = train_image_stats.loc[\n (train_image_stats[\"rows\"]==3456) & (train_image_stats[\"columns\"]==5184)\n].path.values\n\nhold_out_df = train_info.loc[train_info.dcm_path.isin(selected_hold_out_group)].copy()\nreduced_train_df = train_info.loc[train_info.dcm_path.isin(selected_hold_out_group)==False].copy()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"hold_out_df.shape[0] / reduced_train_df.shape[0]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Ok, now we have created a hold out dataset that only consists of one type of image group. We need to fill it up with further training samples of all other groups. To be similar to the test set we should try to reach a 33% split. ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"test_info.shape[0] / train_info.shape[0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"reduced_train_df, add_to_hold_out_df = train_test_split(\n reduced_train_df, test_size=0.163, stratify=reduced_train_df.target.values)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"hold_out_df = hold_out_df.append(add_to_hold_out_df)\nprint(hold_out_df.shape[0] / reduced_train_df.shape[0])\nprint(hold_out_df.shape[0], reduced_train_df.shape[0])","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1,2,figsize=(20,5))\n\nh_target_perc = hold_out_df.target.value_counts() / hold_out_df.shape[0] * 100\nrt_target_perc = reduced_train_df.target.value_counts() / reduced_train_df.shape[0] * 100 \n\nsns.barplot(h_target_perc.index, h_target_perc.values, ax=ax[0], palette=\"Oranges_r\")\nsns.barplot(rt_target_perc.index, rt_target_perc.values, ax=ax[1], palette=\"Purples_r\");\n\nax[0].set_title(\"Target distribution of \\n hold-out\");\nax[1].set_title(\"Target distribution of \\n reduced train\");\nfor n in range(2):\n ax[n].set_ylabel(\"% in data\")\n ax[n].set_xlabel(\"Target\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Both have very similar target distributions now:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"h_target_perc","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"rt_target_perc","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Settings ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Let's now chose a model structure:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\ndevice","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"torch.manual_seed(0)\nnp.random.seed(0)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's start with resnet:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"num_classes = 2\n\nmy_model = \"resnet50\"\nTRAIN_BATCH_SIZE = 64\nLR = 0.01","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Using Bojans resized original images to speed up ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"And let's pick a resize shape of Bojan Tunguz resized images:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"os.listdir(\"../input/siimisic-melanoma-resized-images\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This way loading the images will be much faster than doing this on the fly.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"RESIZE_SHAPE = 256","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#train_npy = np.load(\"../input/siimisic-melanoma-resized-images/x_train_\" + str(RESIZE_SHAPE) + \".npy\", mmap_mode=\"r\")\n#train_npy.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#x_test = np.load(\"../input/siimisic-melanoma-resized-images/x_test_\" + str(RESIZE_SHAPE) + \".npy\", mmap_mode=\"r\")\n#x_test.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As we are using a hold-out dataset to simulate what happens when there is a image group in test that is missed in train, we need to selected the proper indices:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"hold_out_indices = hold_out_df.index.values\nreduced_train_indices = reduced_train_df.index.values","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#hold_out_dataset_1 = ResizedNpyMelanomaDataset(train_npy, hold_out_indices, df=hold_out_df,\n# transform=transform_fun(RESIZE_SHAPE, key=\"dev\", plot=True))\n#hold_out_dataset_2 = MelanomaDataset(hold_out_df, transform=transform_fun(RESIZE_SHAPE, key=\"dev\", plot=True))\n\n","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"#idx = 10\n\n#hold_out_example_1 = hold_out_dataset_1.__getitem__(idx)\n#hold_out_example_2 = hold_out_dataset_2.__getitem__(idx)\n\n#fig, ax = plt.subplots(1,4,figsize=(20,5))\n#ax[0].imshow(hold_out_example_1[\"image\"])\n#ax[0].axis(\"off\")\n#ax[0].set_title(hold_out_example_1[\"target\"]);\n#sns.distplot(hold_out_example_1[\"image\"], ax=ax[1])\n#ax[2].imshow(hold_out_example_2[\"image\"])\n#ax[2].axis(\"off\")\n#ax[2].set_title(hold_out_example_2[\"target\"]);\n#sns.distplot(hold_out_example_1[\"image\"], ax=ax[3])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Doing the resize-preprocessing in advance is definitely speeding up the computation! If you are using more images than the original data you should consider to do so as well.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_indices = train_info.index.values","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Searching for learning rate boundaries \n\nOur first task would be to find proper learning rate boundaries to use the cyclical learning rate approach. I'm following the triangular method described in [Cyclical Learning Rates for Training Neural Networks](https://arxiv.org/pdf/1506.01186.pdf).\n\n\n*More explanations are following soon*\n\n\n#### But how to find the best min and max learning rates?\n\n* Within a predefined number of epochs the learning rates is increased linearily between to boundary values of your choise: min_lr and max_lr. \n* While training I'm currently measuring the running f1_score on train data to observe how this increasing rate changes the quality of predictions.\n* I'm not using accuracy score as our target distribution is highly imbalanced and it's easy to get high accuracy scores by only predicting lots of zeros. ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"If you like to search for optimal min and max learning rates, just choose your values and set find_lr=True. The results of the search will be saved in the save_folder specified. If you like you can add your result as a dataset and specify their path in load_folder. This way you can visualize them in the plot that follows the search.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"find_lr = True\nmin_lr = 0.001\nmax_lr = 1\nNUM_EPOCHS = 3\nsave_folder = \"learning_rate_search\"\nload_folder = \"../input/melanomaclassificationsmoothiestarter/learning_rate_search\"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"At the moment I'm using this [external data ](https://www.kaggle.com/nroman/melanoma-external-malignant-256) as I found it difficult to train without more positive cases. Consequently the idea of predicting on a hold-out group should be done on external data. I haven't done this yet and I probably won't find enough time to redo everything on external data. This is up to you! ;-)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"external_data_path = \"../input/melanoma-external-malignant-256/\"\nexternal_train = pd.read_csv(external_data_path + \"/train_concat.csv\")\nexternal_train[\"image_path\"] = external_data_path + \"train/train/\" + external_train.image_name + \".jpg\"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's take a look at the different target distributions:","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1,2,figsize=(20,5))\nsns.countplot(train_info.target, ax=ax[0], palette=\"Reds_r\")\nsns.countplot(external_train.target, ax=ax[1], palette=\"Reds_r\")\nax[1].set_title(\"Target imbalance in external train\")\nax[0].set_title(\"Target imbalance in original train\");","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Ok, let's try to find good learning rate boundaries:","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"if find_lr:\n \n results = {}\n \n #train_idx, dev_idx = train_test_split(train_indices,\n #stratify=train_info.target.values,\n #test_size=0.3,\n #random_state=0)\n \n #train_dataset = ResizedNpyMelanomaDataset(npy_file=train_npy,\n # indices_to_select=train_idx, \n # df=train_info,\n # transform=transform_fun(RESIZE_SHAPE, \"train\"))\n #dev_dataset = ResizedNpyMelanomaDataset(npy_file=train_npy,\n # indices_to_select=dev_idx, \n # df=train_info,\n # transform=transform_fun(RESIZE_SHAPE, \"dev\"))\n \n \n train_df, dev_df = train_test_split(external_train,\n stratify=external_train.target.values,\n test_size=0.3,\n random_state=0)\n \n train_dataset = AlbuMelanomaDataset(train_df, albu_transform_fun(RESIZE_SHAPE, key=\"train\"))\n dev_dataset = AlbuMelanomaDataset(dev_df, albu_transform_fun(key=\"dev\"))\n \n \n train_dataloader = DataLoader(train_dataset, batch_size=TRAIN_BATCH_SIZE, shuffle=True, drop_last=True)\n dev_dataloader = DataLoader(dev_dataset, batch_size=TRAIN_BATCH_SIZE, shuffle=False, drop_last=True)\n dataloaders_dict = {\"train\": train_dataloader, \"dev\": dev_dataloader}\n \n model = build_model(my_model)\n model.apply(init_weights)\n model = model.to(device)\n \n # if you are using the external data\n criterion = MulticlassFocalLoss(gamma=2)\n # if you are using the resized data:\n #criterion = MulticlassFocalLoss(train_info.iloc[train_idx].target.values)\n #criterion = get_wce_loss(train_df.target.values)\n \n if \"efficientnet\" in my_model:\n optimiser = torch.optim.SGD(model._fc.parameters(), lr=LR)\n else:\n optimiser = torch.optim.SGD(model.fc.parameters(), lr=LR)\n \n max_iterations = NUM_EPOCHS * len(train_dataloader)\n scheduler = get_lr_search_scheduler(optimiser, min_lr, max_lr, max_iterations)\n \n single_results = train(model=model,\n model_kind=my_model,\n criterion=criterion,\n optimiser=optimiser,\n num_epochs=NUM_EPOCHS,\n dataloaders_dict=dataloaders_dict,\n fold_num=0,\n scheduler=scheduler, \n patience=1,\n find_lr=find_lr)\n \n results = {0: single_results}\n save_results(results, save_folder)\n \n# prepare for retraining and/or inference:\nelse:\n train_df, dev_df = train_test_split(external_train,\n stratify=external_train.target.values,\n test_size=0.3,\n random_state=0)\n \n train_dataset = AlbuMelanomaDataset(train_df, albu_transform_fun(RESIZE_SHAPE, key=\"train\"))\n dev_dataset = AlbuMelanomaDataset(dev_df, albu_transform_fun(key=\"dev\"))\n \n train_dataloader = DataLoader(train_dataset, batch_size=TRAIN_BATCH_SIZE, shuffle=True, drop_last=True)\n dev_dataloader = DataLoader(dev_dataset, batch_size=TRAIN_BATCH_SIZE, shuffle=False, drop_last=True)\n dataloaders_dict = {\"train\": train_dataloader, \"dev\": dev_dataloader}\n \n criterion = MulticlassFocalLoss(gamma=2)\n \n results = load_results(load_folder,\n total_folds=1,\n model_kind=my_model,\n lr=LR,\n num_epochs=NUM_EPOCHS,\n len_train=len(train_dataloader),\n min_lr=min_lr,\n max_lr=max_lr,\n for_inference=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Personally I find it a bit easier to use weighted cross entropy loss but perhaps with tuning the hyperparameters properly the focal loss could be a good choice as well. Try to start with $\\gamma=0$ as the focal loss would turn into weighted cross entropy loss in this case. I'm also working on a better way to set $\\alpha$. The class weights are not working well and there seems to be a way to set them based on the effective number of samples ([look at this paper](https://arxiv.org/abs/1901.05555)). I would like to try it out. ","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(3,2,figsize=(20,18))\n\nrates = results[0].results[\"train\"].learning_rates\nf1_score = results[0].results[\"train\"].f1_scores\nprecision = results[0].results[\"train\"].precision\nrecall = results[0].results[\"train\"].recall\nlosses = results[0].results[\"train\"].losses\nepoch_losses = results[0].results[\"train\"].epoch_losses\n\nax[0,0].plot(rates, f1_score, '.-', c=\"maroon\", label=\"f1-score\")\nax[0,0].plot(rates, precision, '.-', c=\"salmon\", label=\"precision\")\nax[0,0].plot(rates, recall, '.-', c=\"lightsalmon\", label=\"recall\")\n\n\nax[0,0].legend();\nax[0,0].set_xlabel(\"Learning rate\")\nax[0,0].set_ylabel(\"Score values\")\nax[0,0].set_title(\"Evaluation scores for learning rate search within {} epochs\".format(NUM_EPOCHS));\n\nax[1,1].plot(rates, precision, '.-', c=\"salmon\", label=\"precision\")\nax[1,1].set_title(\"Precision\")\nax[1,1].set_xlabel(\"learning rates\")\nax[1,1].set_ylabel(\"precision\")\n\nax[1,0].plot(rates, recall, '.-', c=\"lightsalmon\", label=\"recall\")\nax[1,0].set_title(\"Recall\")\nax[1,0].set_xlabel(\"learning rates\")\nax[1,0].set_ylabel(\"recall\")\n\nax[0,1].plot(rates, f1_score, '.-', c=\"maroon\", label=\"f1-score\")\nax[0,1].set_title(\"F1-score\")\nax[0,1].set_xlabel(\"learning rates\")\nax[0,1].set_ylabel(\"f1-score\")\n\nax[2,0].plot(rates, losses, 'o-', c=\"deepskyblue\")\nax[2,0].set_title(\"Loss change with rates\")\nax[2,0].set_ylabel(\"loss\")\nax[2,0].set_xlabel(\"Learning rates\")\n\nax[2,1].set_title(\"Learning rate increase\")\nax[2,1].plot(rates, 'o', c=\"mediumseagreen\");\nax[2,1].set_ylabel(\"learning rate\")\nax[2,1].set_xlabel(\"Iteration step\");","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Insights\n\n* You may like to play with the smallest learning rate first. During my experiments I found that how small the first one is definitely influences the success of these curves and how much you can increase the max learning rate.\n* One can also see that the loss may go up and down a bit even though the scores are still increasing!","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Running a model \n\nLet's now check wheater everything works as expected:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"check_workflow = False\nsave_folder = \"check_workflow\"\nload_folder = \"../input/melanomaclassificationsmoothiestarter/check_workflow\"\nNUM_EPOCHS = 10\nLR = 0.01\nmin_lr = 0.0001\nmax_lr = 0.25\nfind_lr=False","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"if check_workflow:\n \n results = {}\n \n train_df, dev_df = train_test_split(external_train,\n stratify=external_train.target.values,\n test_size=0.3,\n random_state=0)\n \n train_dataset = AlbuMelanomaDataset(train_df, albu_transform_fun(RESIZE_SHAPE, key=\"train\"))\n dev_dataset = AlbuMelanomaDataset(dev_df, albu_transform_fun(key=\"dev\"))\n \n train_dataloader = DataLoader(train_dataset, batch_size=TRAIN_BATCH_SIZE, shuffle=True, drop_last=True)\n dev_dataloader = DataLoader(dev_dataset, batch_size=TRAIN_BATCH_SIZE, shuffle=False, drop_last=True)\n dataloaders_dict = {\"train\": train_dataloader, \"dev\": dev_dataloader}\n \n model = build_model(my_model)\n model.apply(init_weights)\n model = model.to(device)\n \n criterion = MulticlassFocalLoss(gamma=2)\n #criterion = get_wce_loss(train_df.target.values)\n if \"efficientnet\" in my_model:\n optimiser = torch.optim.SGD(model._fc.parameters(), lr=LR)\n else:\n optimiser = torch.optim.SGD(model.fc.parameters(), lr=LR)\n \n stepsize = 2*len(train_dataloader)\n scheduler = get_scheduler(optimiser, min_lr, max_lr, stepsize)\n \n single_results = train(model=model,\n model_kind=my_model,\n criterion=criterion,\n optimiser=optimiser,\n num_epochs=NUM_EPOCHS,\n dataloaders_dict=dataloaders_dict,\n fold_num=0,\n scheduler=scheduler, \n patience=1,\n find_lr=find_lr)\n \n results = {0: single_results}\n save_results(results, save_folder)\n\nelse:\n \n train_df, dev_df = train_test_split(external_train,\n stratify=external_train.target.values,\n test_size=0.3,\n random_state=0)\n \n train_dataset = AlbuMelanomaDataset(train_df, albu_transform_fun(RESIZE_SHAPE, key=\"train\"))\n dev_dataset = AlbuMelanomaDataset(dev_df, albu_transform_fun(key=\"dev\"))\n \n train_dataloader = DataLoader(train_dataset, batch_size=TRAIN_BATCH_SIZE, shuffle=True, drop_last=True)\n dev_dataloader = DataLoader(dev_dataset, batch_size=TRAIN_BATCH_SIZE, shuffle=False, drop_last=True)\n dataloaders_dict = {\"train\": train_dataloader, \"dev\": dev_dataloader}\n \n criterion = MulticlassFocalLoss(gamma=2)\n \n results = load_results(load_folder,\n total_folds=1,\n model_kind=my_model,\n lr=LR,\n num_epochs=NUM_EPOCHS,\n len_train=len(train_dataloader),\n min_lr=min_lr,\n max_lr=max_lr,\n for_inference=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"save_results(results, save_folder)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(3,2,figsize=(20,15))\n\nrates = results[0].results[\"train\"].learning_rates\nf1_score = results[0].results[\"train\"].f1_scores\nprecision = results[0].results[\"train\"].precision\nrecall = results[0].results[\"train\"].recall\n\ntrain_losses = results[0].results[\"train\"].losses\ndev_losses = results[0].results[\"dev\"].losses\n\ntrain_epoch_losses = results[0].results[\"train\"].epoch_losses\ndev_epoch_losses = results[0].results[\"dev\"].epoch_losses\ntrain_epoch_auc = results[0].results[\"train\"].epoch_scores\ndev_epoch_auc = results[0].results[\"dev\"].epoch_scores\n\nax[0,0].plot(f1_score, '.-', c=\"maroon\", label=\"f1-score\")\nax[0,0].plot(precision, '.-', c=\"salmon\", label=\"precision\")\nax[0,0].plot(recall, '.-', c=\"lightsalmon\", label=\"recall\")\n\nax[0,0].legend();\nax[0,0].set_xlabel(\"Learning rate\")\nax[0,0].set_ylabel(\"Score values\")\nax[0,0].set_title(\"Evaluation scores for learning rate search within {} epochs\".format(NUM_EPOCHS));\n\nax[0,1].plot(rates)\nax[0,1].set_title(\"Learning rates\")\n\nax[1,0].plot(train_losses, label=\"train\")\n\nax[1,1].plot(dev_losses, label=\"dev\");\nax[1,1].legend()\nax[1,1].set_title(\"Losses\")\n\nax[2,0].plot(train_epoch_losses, label=\"train\")\nax[2,0].plot(dev_epoch_losses, label=\"dev\")\nax[2,0].set_title(\"Epoch losses\")\n\nax[2,1].plot(train_epoch_auc)\nax[2,1].plot(dev_epoch_auc)\nax[2,1].set_title(\"Epoch AUC\");","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Running models with StratifiedKFold","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"run_kfold = False\nn_splits = 3\nsave_folder = \"kfold_workflow\"\nload_folder = \"../input/melanomaclassificationsmoothiestarter/kfold_workflow\"\nNUM_EPOCHS = 10\nLR = 0.01\nmin_lr = 0.0001\nmax_lr = 0.25\nfind_lr=False","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"skf = StratifiedKFold(n_splits=5, random_state=0)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"if run_kfold:\n \n results = {}\n \n n_fold = 0\n for train_idx, dev_idx in skf.split(external_train, external_train.target.values):\n train_df = external_train.iloc[train_idx]\n dev_df = external_train.iloc[dev_idx]\n \n \n train_dataset = AlbuMelanomaDataset(train_df, albu_transform_fun(RESIZE_SHAPE, key=\"train\"))\n dev_dataset = AlbuMelanomaDataset(dev_df, albu_transform_fun(key=\"dev\"))\n \n train_dataloader = DataLoader(train_dataset, batch_size=TRAIN_BATCH_SIZE, shuffle=True, drop_last=True)\n dev_dataloader = DataLoader(dev_dataset, batch_size=TRAIN_BATCH_SIZE, shuffle=False, drop_last=True)\n dataloaders_dict = {\"train\": train_dataloader, \"dev\": dev_dataloader}\n\n model = build_model(my_model)\n model.apply(init_weights)\n model = model.to(device)\n \n criterion = MulticlassFocalLoss(gamma=2)\n #criterion = get_wce_loss(train_df.target.values)\n if \"efficientnet\" in my_model:\n optimiser = torch.optim.SGD(model._fc.parameters(), lr=LR)\n else:\n optimiser = torch.optim.SGD(model.fc.parameters(), lr=LR)\n \n stepsize = 2*len(train_dataloader)\n scheduler = get_scheduler(optimiser, min_lr, max_lr, stepsize)\n \n single_results = train(model=model,\n model_kind=my_model,\n criterion=criterion,\n optimiser=optimiser,\n num_epochs=NUM_EPOCHS,\n dataloaders_dict=dataloaders_dict,\n fold_num=0,\n scheduler=scheduler, \n patience=1,\n find_lr=find_lr)\n \n results = {n_fold: single_results}\n n_fold += 1\n \n save_results(results, save_folder)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Exploring predictions and weaknesses ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Let's use the last dev dataset to yield some insights about predictions and weaknesses of our model:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"max_size = 120\n\nfor m in range(max_size+1):\n to_try = max_size - m\n if dev_df.shape[0] % to_try == 0:\n break\n \nDEV_BATCH_SIZE = to_try\nto_try","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Confusion matrix","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.metrics import confusion_matrix\n\ndef get_confusion_matrix(y_true, y_pred):\n transdict = {1: \"malignant\", 0: \"benign\"}\n y_t = np.array([transdict[x] for x in y_true])\n y_p = np.array([transdict[x] for x in y_pred])\n \n labels = [\"benign\", \"malignant\"]\n index_labels = [\"actual benign\", \"actual malignant\"]\n col_labels = [\"predicted benign\", \"predicted malignant\"]\n confusion = confusion_matrix(y_t, y_p, labels=labels)\n confusion_df = pd.DataFrame(confusion, index=index_labels, columns=col_labels)\n for n in range(2):\n confusion_df.iloc[n] = confusion_df.iloc[n] / confusion_df.sum(axis=1).iloc[n]\n return confusion_df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dev_dataset = AlbuMelanomaDataset(dev_df, albu_transform_fun(key=\"dev\"))\ndev_dataloader = DataLoader(dev_dataset, batch_size=DEV_BATCH_SIZE, shuffle=False, drop_last=False)\npreds, probas = predict(results, dev_dataloader)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"confusion = get_confusion_matrix(dev_df.target.values, preds)\nplt.figure(figsize=(6,6))\nsns.heatmap(confusion, cbar=False, annot=True, fmt=\"g\", square=True, cmap=\"Reds\");","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Submission ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"external_test_path = \"../input/melanoma-external-malignant-256/test/test/\"\ntest_info[\"image_path\"] = external_test_path + test_info.image_name +\".jpg\"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"TEST_BATCH_SIZE=68\ntest_dataset = AlbuMelanomaDataset(test_info, albu_transform_fun(RESIZE_SHAPE, \"test\"))\ntest_dataloader = DataLoader(test_dataset, batch_size=TEST_BATCH_SIZE, shuffle=False, drop_last=False)\npreds, probas = predict(results, test_dataloader)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission = pd.read_csv(basepath + \"sample_submission.csv\")\nsubmission.target = probas[:,1]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.distplot(submission.target)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission.to_csv(\"submission.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Conclusion ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## TODO","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"1. Add efficientnet as option (almost done)\n2. Make retraining for best models possible (almost done)\n * save all state dicts for model, optimizer, scheduler (done)\n * show inference (done) and retraining of the model (todo)\n * show updated losses and scores after retraining (todo)\n3. Add stratified k-fold as validation scheme + oof to csv\n8. Add more explanations \n9. Write a conclusion","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}