{"cells":[{"metadata":{},"cell_type":"markdown","source":"# About the Competition","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## SIIM-ISIC Melanoma Classification\n### Identify melanoma in lesion images","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Skin cancer is the most prevalent type of cancer. Melanoma, specifically, is responsible for 75% of skin cancer deaths, despite being the least common skin cancer. The American Cancer Society estimates over 100,000 new melanoma cases will be diagnosed in 2020. It's also expected that almost 7,000 people will die from the disease. As with other cancers, early and accurate detection—potentially aided by data science—can make treatment more effective.\n\nCurrently, dermatologists evaluate every one of a patient's moles to identify outlier lesions or “ugly ducklings” that are most likely to be melanoma. Existing AI approaches have not adequately considered this clinical frame of reference. Dermatologists could enhance their diagnostic accuracy if detection algorithms take into account “contextual” images within the same patient to determine which images represent a melanoma. If successful, classifiers would be more accurate and could better support dermatological clinic work.\n\nAs the leading healthcare organization for informatics in medical imaging, the Society for Imaging Informatics in Medicine (SIIM)'s mission is to advance medical imaging informatics through education, research, and innovation in a multi-disciplinary community. SIIM is joined by the International Skin Imaging Collaboration (ISIC), an international effort to improve melanoma diagnosis. The ISIC Archive contains the largest publicly available collection of quality-controlled dermoscopic images of skin lesions.\n\nIn this competition, you’ll identify melanoma in images of skin lesions. In particular, you’ll use images within the same patient and determine which are likely to represent a melanoma. Using patient-level contextual information may help the development of image analysis tools, which could better support clinical dermatologists.\n\nMelanoma is a deadly disease, but if caught early, most melanomas can be cured with minor surgery. Image analysis tools that automate the diagnosis of melanoma will improve dermatologists' diagnostic accuracy. Better detection of melanoma has the opportunity to positively impact millions of people.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Important findings <a class=\"anchor\" id=\"findings\"></a>\n\n* **We can clearly observe groups of images with similar statistics that depend on the image shapes!!!**\n* **There is one group of the test images that is missing in train(1080 rows 1920 columns)! This would mean a complete new type of images that may lead to notable differences in CV scores and LB!**\n* For most of the patients there were only a few images recorded (range from 1 to 20).\n* 5% of the patients show more than 45 images. There is an extreme outlier in the test data with roughly 250 images!\n* We have more males than females in both train and test data. For the test set this imbalance is even higher!\n* We can observe **more older patients in test than in train**! The age is normally distributed in train but shows multiple modes in test.\n* The distributions of image locations look very similar for train and test.\n* We have highly imbalanced target classes!\n* Multiple images does not mean that there are multiple ages involved! \n* We can observe a high surplus of males in the ages 45 to 50 and 70, 75 in train and test but in test we can find **even more males of high age > 75**.\n* We have more malignant cases of higher age than benign cases.\n* 62 % of the malignant cases belong to males and only 38 % to females! **We have to be very careful!!! As we have a surpus of males with ages above 70 and 75 it's unclear if the sex is really an important feature for having melanoma or not.** It could also be that the age is most important and that we only have more malignant cases for males due to their higher age! ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Upload Necessary Library","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt\nimport pylab as pl\nfrom IPython import display\nimport seaborn as sns\nsns.set()\n\nimport re\n\nimport pydicom\nimport random\n\nimport torch\nimport torch.nn.functional as F\nfrom torch.utils.data import Dataset, DataLoader\nfrom torchvision import transforms, models\n\n#import pytorch_lightning as pl\nfrom scipy.special import softmax\n\nfrom sklearn.model_selection import train_test_split, StratifiedKFold\nfrom sklearn.utils.class_weight import compute_class_weight\nfrom sklearn.metrics import roc_auc_score, auc\n\nfrom skimage.io import imread\nfrom PIL import Image\n\nimport plotly.offline as py\npy.init_notebook_mode(connected=True)\nimport plotly.graph_objs as go\nfrom plotly.subplots import make_subplots\n\nimport os\nimport copy\n\nfrom albumentations import Compose, RandomCrop, Normalize,HorizontalFlip, Resize\nfrom albumentations import VerticalFlip, RGBShift, RandomBrightness\nfrom albumentations.core.transforms_interface import ImageOnlyTransform\nfrom albumentations.pytorch import ToTensor\n\nfrom tqdm.notebook import tqdm\n\nos.listdir(\"../input/\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"basepath = \"../input/siim-isic-melanoma-classification/\"\nmodelspath = \"../input/pytorch-pretrained-image-models/\"\nimagestatspath = \"../input/siimisic-melanoma-classification-image-stats/\"\nfoldspath = \"../input/irndropout/\"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"os.listdir(basepath)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_info = pd.read_csv(basepath + \"train.csv\")\ntrain_info.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_info = pd.read_csv(basepath + \"test.csv\")\ntest_info.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Our test set misses three columns: diagnosis, benign_malignant & target.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_info.shape[0] / test_info.shape[0]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Three times more entries in train than in test.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# What is given by the meta data? ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Missing values","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"missing_vals_train = train_info.isnull().sum() / train_info.shape[0]\nmissing_vals_train[missing_vals_train > 0].sort_values(ascending=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"missing_vals_test = test_info.isnull().sum() / test_info.shape[0]\nmissing_vals_test[missing_vals_test > 0].sort_values(ascending=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The anatomy shows most missing values. ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Image names","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_info.image_name.value_counts().max()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_info.image_name.value_counts().max()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Patient id counts","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_info.patient_id.value_counts().max()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_info.patient_id.value_counts().max()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In contrast we can find multiple images for one patient!","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"patient_counts_train = train_info.patient_id.value_counts()\npatient_counts_test = test_info.patient_id.value_counts()\n\nfig, ax = plt.subplots(2,2,figsize=(20,12))\n\nsns.distplot(patient_counts_train, ax=ax[0,0], color=\"orangered\", kde=True);\nax[0,0].set_xlabel(\"Counts\")\nax[0,0].set_ylabel(\"Frequency\")\nax[0,0].set_title(\"Patient id value counts in train\");\n\nsns.distplot(patient_counts_test, ax=ax[0,1], color=\"lightseagreen\", kde=True);\nax[0,1].set_xlabel(\"Counts\")\nax[0,1].set_ylabel(\"Frequency\")\nax[0,1].set_title(\"Patient id value counts in test\");\n\nsns.boxplot(patient_counts_train, ax=ax[1,0], color=\"orangered\");\nax[1,0].set_xlim(0, 250)\nsns.boxplot(patient_counts_test, ax=ax[1,1], color=\"lightseagreen\");\nax[1,1].set_xlim(0, 250);","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"np.quantile(patient_counts_train, 0.75) - np.quantile(patient_counts_train, 0.25)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"np.quantile(patient_counts_train, 0.5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(np.quantile(patient_counts_train, 0.95))\nprint(np.quantile(patient_counts_test, 0.95))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Insights\n\n* For most of the patients we only have a few images ranging fom 1 to roughly 20.\n* More than 45 images per patient is very seldom! \n* Nonetheless we have patients with more than 100 images.\n* There is one heavy outlier patient in the test set with close to 250 images.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"200/test_info.shape[0] * 100","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Overlapping patients in train/test","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_patient_ids = set(train_info.patient_id.unique())\ntest_patient_ids = set(test_info.patient_id.unique())\n\ntrain_patient_ids.intersection(test_patient_ids)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Ok, that's great! There seem to be no patients in train that can be found in test as well. We can't be sure as we don't know how the naming assignment process was designed. We might better check the images themselves as well!","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Gender counts","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1,2,figsize=(20,5))\nsns.countplot(train_info.sex, palette=\"Reds_r\", ax=ax[0]);\nax[0].set_xlabel(\"\")\nax[0].set_title(\"Gender counts\");\n\nsns.countplot(test_info.sex, palette=\"Blues_r\", ax=ax[1]);\nax[1].set_xlabel(\"\")\nax[1].set_title(\"Gender counts\");","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Insights\n\n* We observe more males than females in both train and test data.\n* The surplus of males is even higher in test than in train!","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Age distributions","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1,2,figsize=(20,5))\n\nsns.countplot(train_info.age_approx, color=\"orangered\", ax=ax[0]);\nlabels = ax[0].get_xticklabels();\nax[0].set_xticklabels(labels, rotation=90);\nax[0].set_xlabel(\"\");\nax[0].set_title(\"Age distribution in train\");\n\nsns.countplot(test_info.age_approx, color=\"lightseagreen\", ax=ax[1]);\nlabels = ax[1].get_xticklabels();\nax[1].set_xticklabels(labels, rotation=90);\nax[1].set_xlabel(\"\");\nax[1].set_title(\"Age distribution in test\");","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Image Recognition","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1,2,figsize=(20,5))\n\nimage_locations_train = train_info.anatom_site_general_challenge.value_counts().sort_values(ascending=False)\nimage_locations_test = test_info.anatom_site_general_challenge.value_counts().sort_values(ascending=False)\n\nsns.barplot(x=image_locations_train.index.values, y=image_locations_train.values, ax=ax[0], color=\"orangered\");\nax[0].set_xlabel(\"\");\nlabels = ax[0].get_xticklabels();\nax[0].set_xticklabels(labels, rotation=90);\nax[0].set_title(\"Image locations in train\");\n\nsns.barplot(x=image_locations_test.index.values, y=image_locations_test.values, ax=ax[1], color=\"lightseagreen\");\nax[1].set_xlabel(\"\");\nlabels = ax[1].get_xticklabels();\nax[1].set_xticklabels(labels, rotation=90);\nax[1].set_title(\"Image locations in test\");","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Target distribution\nfig, ax = plt.subplots(1,2, figsize=(20,5))\n\nsns.countplot(x=train_info.diagnosis, orient=\"v\", ax=ax[0], color=\"Orangered\")\nax[0].set_xlabel(\"\")\nlabels = ax[0].get_xticklabels();\nax[0].set_xticklabels(labels, rotation=90);\nax[0].set_title(\"Diagnosis\");\n\nsns.countplot(train_info.benign_malignant, ax=ax[1], palette=\"Reds_r\");\nax[1].set_xlabel(\"\")\nax[1].set_title(\"Type\");","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Training Data Grouped by benign_malignant\ntrain_info.groupby(\"benign_malignant\").target.nunique()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Feature Selection","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"patient_ages_table_train = train_info.groupby([\"patient_id\", \"age_approx\"]).size() / train_info.groupby(\"patient_id\").size()\npatient_ages_table_train = patient_ages_table_train.unstack().transpose()\npatient_ages_table_test = test_info.groupby([\"patient_id\", \"age_approx\"]).size() / test_info.groupby(\"patient_id\").size()\npatient_ages_table_test = patient_ages_table_test.unstack().transpose()\n\npatient_with_known_ages_train = train_info[train_info.patient_id.isin(patient_ages_table_train.columns.values)]\n\nsorted_patients_train = patient_with_known_ages_train.patient_id.value_counts().index.values\npatient_with_known_ages_test = test_info[test_info.patient_id.isin(patient_ages_table_test.columns.values)]\nsorted_patients_test = patient_with_known_ages_test.patient_id.value_counts().index.values\n\nfig, ax = plt.subplots(2,1, figsize=(20,20))\nsns.heatmap(patient_ages_table_train[sorted_patients_train], cmap=\"Reds\", ax=ax[0], cbar=False);\nax[0].set_title(\"Image coverage in % per patient and age in train data\");\nsns.heatmap(patient_ages_table_test[sorted_patients_test], cmap=\"Blues\", ax=ax[1], cbar=False);\nax[1].set_title(\"Image coverage in % per patient and age in test data\");\nax[0].set_xlabel(\"\")\nax[1].set_xlabel(\"\");","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Data Visualization Age Vs. Gender","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(2,2,figsize=(20,15))\n\nsns.boxplot(train_info.sex, train_info.age_approx, ax=ax[0,0], palette=\"Reds_r\");\nax[0,0].set_title(\"Age per gender in train\");\n\nsns.boxplot(test_info.sex, test_info.age_approx, ax=ax[0,1], palette=\"Blues_r\");\nax[0,1].set_title(\"Age per gender in test\");\n\nsns.countplot(train_info.age_approx, hue=train_info.sex, ax=ax[1,0], palette=\"Reds_r\");\nsns.countplot(test_info.age_approx, hue=test_info.sex, ax=ax[1,1], palette=\"Blues_r\");","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Gender, Age Vs. Cancer","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"sex_and_cancer_map = train_info.groupby(\n    [\"benign_malignant\", \"sex\"]\n).size().unstack(level=0) / train_info.groupby(\"benign_malignant\").size() * 100\n\ncancer_sex_map = train_info.groupby(\n    [\"benign_malignant\", \"sex\"]\n).size().unstack(level=1) / train_info.groupby(\"sex\").size() * 100\n\n\nfig, ax = plt.subplots(1,3,figsize=(20,5))\n\nsns.boxplot(train_info.benign_malignant, train_info.age_approx, ax=ax[0], palette=\"Greens\");\nax[0].set_title(\"Age and cancer\");\nax[0].set_xlabel(\"\");\n\nsns.heatmap(sex_and_cancer_map, annot=True, cmap=\"Greens\", cbar=False, ax=ax[1])\nax[1].set_xlabel(\"\")\nax[1].set_ylabel(\"\");\n\nsns.heatmap(cancer_sex_map, annot=True, cmap=\"Greens\", cbar=False, ax=ax[2])\nax[2].set_xlabel(\"\")\nax[2].set_ylabel(\"\");","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(2,2,figsize=(20,15))\n\nsns.countplot(train_info[train_info.benign_malignant==\"benign\"].age_approx, hue=train_info.sex, palette=\"Purples_r\", ax=ax[0,0])\nax[0,0].set_title(\"Benign cases in train\");\n\nsns.countplot(train_info[train_info.benign_malignant==\"malignant\"].age_approx, hue=train_info.sex, palette=\"Oranges_r\", ax=ax[0,1])\nax[0,1].set_title(\"Malignant cases in train\");\n\nsns.violinplot(train_info.sex, train_info.age_approx, hue=train_info.benign_malignant, split=True, ax=ax[1,0], palette=\"Greens_r\");\nsns.violinplot(train_info.benign_malignant, train_info.age_approx, hue=train_info.sex, split=True, ax=ax[1,1], palette=\"RdPu\");","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Patient Information Management Individually","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"patient_gender_train = train_info.groupby(\"patient_id\").sex.unique().apply(lambda l: l[0])\npatient_gender_test = test_info.groupby(\"patient_id\").sex.unique().apply(lambda l: l[0])\n\ntrain_patients = pd.DataFrame(index=patient_gender_train.index.values, data=patient_gender_train.values, columns=[\"sex\"])\ntest_patients = pd.DataFrame(index=patient_gender_test.index.values, data=patient_gender_test.values, columns=[\"sex\"])\n\ntrain_patients.loc[:, \"num_images\"] = train_info.groupby(\"patient_id\").size()\ntest_patients.loc[:, \"num_images\"] = test_info.groupby(\"patient_id\").size()\n\ntrain_patients.loc[:, \"min_age\"] = train_info.groupby(\"patient_id\").age_approx.min()\ntrain_patients.loc[:, \"max_age\"] = train_info.groupby(\"patient_id\").age_approx.max()\ntest_patients.loc[:, \"min_age\"] = test_info.groupby(\"patient_id\").age_approx.min()\ntest_patients.loc[:, \"max_age\"] = test_info.groupby(\"patient_id\").age_approx.max()\n\ntrain_patients.loc[:, \"age_span\"] = train_patients[\"max_age\"] - train_patients[\"min_age\"]\ntest_patients.loc[:, \"age_span\"] = test_patients[\"max_age\"] - test_patients[\"min_age\"]\n\ntrain_patients.loc[:, \"benign_cases\"] = train_info.groupby([\"patient_id\", \"benign_malignant\"]).size().loc[:, \"benign\"]\ntrain_patients.loc[:, \"malignant_cases\"] = train_info.groupby([\"patient_id\", \"benign_malignant\"]).size().loc[:, \"malignant\"]\ntrain_patients[\"min_age_malignant\"] = train_info.groupby([\"patient_id\", \"benign_malignant\"]).age_approx.min().loc[:, \"malignant\"]\ntrain_patients[\"max_age_malignant\"] = train_info.groupby([\"patient_id\", \"benign_malignant\"]).age_approx.max().loc[:, \"malignant\"]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_patients.sort_values(by=\"malignant_cases\", ascending=False).head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(2,2,figsize=(20,12))\nsns.countplot(train_patients.sex, ax=ax[0,0], palette=\"Reds\")\nax[0,0].set_title(\"Gender counts with unique patient ids in train\")\nsns.countplot(test_patients.sex, ax=ax[0,1], palette=\"Blues\");\nax[0,1].set_title(\"Gender counts with unique patient ids in test\");\n\ntrain_age_span_perc = train_patients.age_span.value_counts() / train_patients.shape[0] * 100\ntest_age_span_perc = test_patients.age_span.value_counts() / test_patients.shape[0] * 100\n\nsns.barplot(train_age_span_perc.index, train_age_span_perc.values, ax=ax[1,0], color=\"Orangered\");\nsns.barplot(test_age_span_perc.index, test_age_span_perc.values, ax=ax[1,1], color=\"Lightseagreen\");\nax[1,0].set_title(\"Patients age span in train\")\nax[1,1].set_title(\"Patients age span in test\")\nfor n in range(2):\n    ax[1,n].set_ylabel(\"% in data\")\n    ax[1,n].set_xlabel(\"age span\");","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Different between Train and Test set Image","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"example_files = os.listdir(basepath + \"train/\")[0:2]\nexample_files","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_info.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_info[\"dcm_path\"] = basepath + \"train/\" + train_info.image_name + \".dcm\"\ntest_info[\"dcm_path\"] = basepath + \"test/\" + test_info.image_name + \".dcm\"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(train_info.dcm_path[0])\nprint(test_info.dcm_path[0])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"example_dcm = pydicom.dcmread(train_info.dcm_path[2])\nexample_dcm","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"image = example_dcm.pixel_array\nprint(image.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_info[\"image_path\"] = basepath + \"jpeg/train/\" + train_info.image_name + \".jpg\"\ntest_info[\"image_path\"] = basepath + \"jpeg/test/\" + test_info.image_name + \".jpg\"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Exploratory Data Analysis","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"os.listdir(imagestatspath)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_image_stats = pd.read_csv(imagestatspath +  \"test_image_stats.csv\")\ntest_image_stats.head(1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_image_stats_1 = pd.read_csv(imagestatspath + \"train_image_stats_10000.csv\")\ntrain_image_stats_2 = pd.read_csv(imagestatspath + \"train_image_stats_20000.csv\")\ntrain_image_stats_3 = pd.read_csv(imagestatspath + \"train_image_stats_toend.csv\")\ntrain_image_stats_4 = train_image_stats_1.append(train_image_stats_2)\ntrain_image_stats = train_image_stats_4.append(train_image_stats_3)\ntrain_image_stats.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Statistical Analysis","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_test = True","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if plot_test:\n    N = test_image_stats.shape[0]\n    selected_data = test_image_stats\n    my_title = \"Test image statistics\"\nelse:\n    N = train_image_stats.shape[0]\n    selected_data = train_image_stats\n    my_title = \"Train image statistics\"\n\ntrace1 = go.Scatter3d(\n    x=selected_data.img_mean.values[0:N], \n    y=selected_data.img_std.values[0:N],\n    z=selected_data.img_skew.values[0:N],\n    mode='markers',\n    text=selected_data[\"rows\"].values[0:N],\n    marker=dict(\n        color=selected_data[\"columns\"].values[0:N],\n        colorscale = \"Jet\",\n        colorbar=dict(thickness=10, title=\"image columns\", len=0.8),\n        opacity=0.4,\n        size=2\n    )\n)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"figure_data = [trace1]\nlayout = go.Layout(\n    title = my_title,\n    scene = dict(\n        xaxis = dict(title=\"Image mean\"),\n        yaxis = dict(title=\"Image standard deviation\"),\n        zaxis = dict(title=\"Image skewness\"),\n    ),\n    margin=dict(\n        l=0,\n        r=0,\n        b=0,\n        t=0\n    ),\n    showlegend=True\n)\n\nfig = go.Figure(data=figure_data, layout=layout)\npy.iplot(fig, filename='simple-3d-scatter')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_image_stats.groupby([\"rows\", \"columns\"]).size().sort_values(ascending=False).iloc[0:10] / test_image_stats.shape[0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_image_stats.groupby([\"rows\", \"columns\"]).size().sort_values(ascending=False).iloc[0:10] / train_image_stats.shape[0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"examples1 = {\"rows\": 1080, \"columns\": 1920}\nexamples2 = {\"rows\": 4000, \"columns\": 6000}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"selection1 = np.random.choice(test_image_stats[\n    (test_image_stats[\"rows\"]==examples1[\"rows\"]) & (test_image_stats[\"columns\"]==examples1[\"columns\"])\n].path.values, size=8, replace=False)\n\nfig, ax = plt.subplots(2,4,figsize=(20,8))\n\nfor n in range(2):\n    for m in range(4):\n        path = selection1[m + n*4]\n        dcm_file = pydicom.dcmread(path)\n        image = dcm_file.pixel_array\n        ax[n,m].imshow(image)\n        ax[n,m].grid(False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"selection2 = np.random.choice(test_image_stats[\n    (test_image_stats[\"rows\"]==examples2[\"rows\"]) & (test_image_stats[\"columns\"]==examples2[\"columns\"])\n].path.values, size=8, replace=False)\n\nfig, ax = plt.subplots(2,4,figsize=(20,6))\n\nfor n in range(2):\n    for m in range(4):\n        path = selection2[m + n*4]\n        dcm_file = pydicom.dcmread(path)\n        image = dcm_file.pixel_array\n        ax[n,m].imshow(image)\n        ax[n,m].grid(False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Triple Stratified KFold CV with TFRecords\nThis is a simple starter notebook for Kaggle's Melanoma Comp showing triple stratifed KFold with TFRecords. Triple stratified KFold is explained [here][2]. There are many configuration variables below to allow you to experiment. Use either GPU or TPU. You can control which size images are loaded, which efficientNets are used, and whether external data is used. You can experiment with different data augmentation, model architecture, loss, optimizers, and learning schedules. The TFRecords contain meta data, so you can input that into your CNN too. \n\n**NOTE:** this notebook lets you run a different experiment in each fold if you want to run lots of experiments. (Then it is like running multiple holdout validaiton experiments but in that case note that the overall CV score is meaningless because LB will be much higher when the multiple experiments are ensembled to predict test). **If you want a proper CV with a reliable overall CV score you need to choose the same configuration for each fold.**\n\nThis notebook follows the 5 step process presented in my \"How to compete with GPUs Workshop\" [here][1]. Some code sections have been reused from AgentAuers' great notebook [here][3]\n\n[1]: https://www.kaggle.com/cdeotte/how-to-compete-with-gpus-workshop\n[2]: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/165526\n[3]: https://www.kaggle.com/agentauers/incredible-tpus-finetune-effnetb0-b6-at-once","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Kaggle's SIIM-ISIC Melanoma Classification\nIn this competition, we need to identify melanoma in images of skin lesions. Full description [here][1]. This is a very challenging image classification task as seen by looking at the sample images below. Can you recognize the differences between images? Below are example of skin images with and without melanoma.\n\n[1]: https://www.kaggle.com/c/siim-isic-melanoma-classification","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import cv2, pandas as pd, matplotlib.pyplot as plt\ntrain = pd.read_csv('../input/siim-isic-melanoma-classification/train.csv')\nprint('Examples WITH Melanoma')\nimgs = train.loc[train.target==1].sample(10).image_name.values\nplt.figure(figsize=(20,8))\nfor i,k in enumerate(imgs):\n    img = cv2.imread('../input/jpeg-melanoma-128x128/train/%s.jpg'%k)\n    img = cv2.cvtColor(img, cv2.COLOR_RGB2BGR)\n    plt.subplot(2,5,i+1); plt.axis('off')\n    plt.imshow(img)\nplt.show()\nprint('Examples WITHOUT Melanoma')\nimgs = train.loc[train.target==0].sample(10).image_name.values\nplt.figure(figsize=(20,8))\nfor i,k in enumerate(imgs):\n    img = cv2.imread('../input/jpeg-melanoma-128x128/train/%s.jpg'%k)\n    img = cv2.cvtColor(img, cv2.COLOR_RGB2BGR)\n    plt.subplot(2,5,i+1); plt.axis('off')\n    plt.imshow(img)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd, numpy as np\nfrom kaggle_datasets import KaggleDatasets\nimport tensorflow as tf, re, math\nimport tensorflow.keras.backend as K\nimport tensorflow.keras.applications.inception_resnet_v2 as irn\nfrom sklearn.model_selection import KFold\nfrom sklearn.metrics import roc_auc_score\nimport matplotlib.pyplot as plt","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Configuration\nIn order to be a proper cross validation with a meaningful overall CV score (aligned with LB score), **you need to choose the same** `IMG_SIZES`, `INC2019`, `INC2018`, and `EFF_NETS` **for each fold**. If your goal is to just run lots of experiments, then you can choose to have a different experiment in each fold. Then each fold is like a holdout validation experiment. When you find a configuration you like, you can use that configuration for all folds. \n* DEVICE - is GPU or TPU\n* SEED - a different seed produces a different triple stratified kfold split.\n* FOLDS - number of folds. Best set to 3, 5, or 15 but can be any number between 2 and 15\n* IMG_SIZES - is a Python list of length FOLDS. These are the image sizes to use each fold\n* INC2019 - This includes the new half of the 2019 competition data. The second half of the 2019 data is the comp data from 2018 plus 2017\n* INC2018 - This includes the second half of the 2019 competition data which is the comp data from 2018 plus 2017\n* BATCH_SIZES - is a list of length FOLDS. These are batch sizes for each fold. For maximum speed, it is best to use the largest batch size your GPU or TPU allows.\n* EPOCHS - is a list of length FOLDS. These are maximum epochs. Note that each fold, the best epoch model is saved and used. So if epochs is too large, it won't matter.\n* EFF_NETS - is a list of length FOLDS. These are the EfficientNets to use each fold. The number refers to the B. So a number of `0` refers to EfficientNetB0, and `1` refers to EfficientNetB1, etc.\n* WGTS - this should be `1/FOLDS` for each fold. This is the weight when ensembling the folds to predict the test set. If you want a weird ensemble, you can use different weights.\n* TTA - test time augmentation. Each test image is randomly augmented and predicted TTA times and the average prediction is used. TTA is also applied to OOF during validation.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"DEVICE = \"TPU\" #or \"GPU\"\n\n# USE DIFFERENT SEED FOR DIFFERENT STRATIFIED KFOLD\nSEED = 42\n\n# NUMBER OF FOLDS. USE 3, 5, OR 15 \nFOLDS = 5\n\n# WHICH IMAGE SIZES TO LOAD EACH FOLD\n# CHOOSE 128, 192, 256, 384, 512, 768 \nIMG_SIZES = [256, 256, 256, 256, 256]\n\n# INCLUDE OLD COMP DATA? YES=1 NO=0\nINC2019 = [0,0,1,1,1]\nINC2018 = [1,1,0,0,0]\n\n# BATCH SIZE AND EPOCHS\nBATCH_SIZES = [32]*FOLDS\nEPOCHS = [12]*FOLDS\n\n# WHICH EFFICIENTNET B? TO USE\nEFF_NETS = [0,0,0,0,0]\n\n# WEIGHTS FOR FOLD MODELS WHEN PREDICTING TEST\nWGTS = [1/FOLDS]*FOLDS\n\n# TEST TIME AUGMENTATION STEPS\nTTA = 11","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if DEVICE == \"TPU\":\n    print(\"connecting to TPU...\")\n    try:\n        tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n        print('Running on TPU ', tpu.master())\n    except ValueError:\n        print(\"Could not connect to TPU\")\n        tpu = None\n\n    if tpu:\n        try:\n            print(\"initializing  TPU ...\")\n            tf.config.experimental_connect_to_cluster(tpu)\n            tf.tpu.experimental.initialize_tpu_system(tpu)\n            strategy = tf.distribute.experimental.TPUStrategy(tpu)\n            print(\"TPU initialized\")\n        except _:\n            print(\"failed to initialize TPU\")\n    else:\n        DEVICE = \"GPU\"\n\nif DEVICE != \"TPU\":\n    print(\"Using default strategy for CPU and single GPU\")\n    strategy = tf.distribute.get_strategy()\n\nif DEVICE == \"GPU\":\n    print(\"Num GPUs Available: \", len(tf.config.experimental.list_physical_devices('GPU')))\n    \n\nAUTO     = tf.data.experimental.AUTOTUNE\nREPLICAS = strategy.num_replicas_in_sync\nprint(f'REPLICAS: {REPLICAS}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Step 1: Preprocess\nPreprocess has already been done and saved to TFRecords. Here we choose which size to load. We can use either 128x128, 192x192, 256x256, 384x384, 512x512, 768x768 by changing the `IMG_SIZES` variable in the preceeding code section. These TFRecords are discussed [here][1]. The advantage of using different input sizes is discussed [here][2]\n\n[1]: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/155579\n[2]: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/160147","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"GCS_PATH = [None]*FOLDS; GCS_PATH2 = [None]*FOLDS\nfor i,k in enumerate(IMG_SIZES):\n    GCS_PATH[i] = KaggleDatasets().get_gcs_path('melanoma-%ix%i'%(k,k))\n    GCS_PATH2[i] = KaggleDatasets().get_gcs_path('isic2019-%ix%i'%(k,k))\nfiles_train = np.sort(np.array(tf.io.gfile.glob(GCS_PATH[0] + '/train*.tfrec')))\nfiles_test  = np.sort(np.array(tf.io.gfile.glob(GCS_PATH[0] + '/test*.tfrec')))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Step 2: Data Augmentation\nThis notebook uses rotation, sheer, zoom, shift augmentation first shown in this notebook [here][1] and successfully used in Melanoma comp by AgentAuers [here][2]. This notebook also uses horizontal flip, hue, saturation, contrast, brightness augmentation similar to last years winner and also similar to AgentAuers' notebook.\n\nAdditionally we can decide to use external data by changing the variables `INC2019` and `INC2018` in the preceeding code section. These variables respectively indicate whether to load last year 2019 data and/or year 2018 + 2017 data. These datasets are discussed [here][3]\n\nConsider experimenting with different augmenation and/or external data. The code to load TFRecords is taken from AgentAuers' notebook [here][2]. Thank you AgentAuers, this is great work.\n\n[1]: https://www.kaggle.com/cdeotte/rotation-augmentation-gpu-tpu-0-96\n[2]: https://www.kaggle.com/agentauers/incredible-tpus-finetune-effnetb0-b6-at-once\n[3]: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/164910","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"ROT_ = 180.0\nSHR_ = 2.0\nHZOOM_ = 8.0\nWZOOM_ = 8.0\nHSHIFT_ = 8.0\nWSHIFT_ = 8.0","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_mat(rotation, shear, height_zoom, width_zoom, height_shift, width_shift):\n    # returns 3x3 transformmatrix which transforms indicies\n        \n    # CONVERT DEGREES TO RADIANS\n    rotation = math.pi * rotation / 180.\n    shear    = math.pi * shear    / 180.\n\n    def get_3x3_mat(lst):\n        return tf.reshape(tf.concat([lst],axis=0), [3,3])\n    \n    # ROTATION MATRIX\n    c1   = tf.math.cos(rotation)\n    s1   = tf.math.sin(rotation)\n    one  = tf.constant([1],dtype='float32')\n    zero = tf.constant([0],dtype='float32')\n    \n    rotation_matrix = get_3x3_mat([c1,   s1,   zero, \n                                   -s1,  c1,   zero, \n                                   zero, zero, one])    \n    # SHEAR MATRIX\n    c2 = tf.math.cos(shear)\n    s2 = tf.math.sin(shear)    \n    \n    shear_matrix = get_3x3_mat([one,  s2,   zero, \n                                zero, c2,   zero, \n                                zero, zero, one])        \n    # ZOOM MATRIX\n    zoom_matrix = get_3x3_mat([one/height_zoom, zero,           zero, \n                               zero,            one/width_zoom, zero, \n                               zero,            zero,           one])    \n    # SHIFT MATRIX\n    shift_matrix = get_3x3_mat([one,  zero, height_shift, \n                                zero, one,  width_shift, \n                                zero, zero, one])\n    \n    return K.dot(K.dot(rotation_matrix, shear_matrix), \n                 K.dot(zoom_matrix,     shift_matrix))\n\n\ndef transform(image, DIM=256):    \n    # input image - is one image of size [dim,dim,3] not a batch of [b,dim,dim,3]\n    # output - image randomly rotated, sheared, zoomed, and shifted\n    XDIM = DIM%2 #fix for size 331\n    \n    rot = ROT_ * tf.random.normal([1], dtype='float32')\n    shr = SHR_ * tf.random.normal([1], dtype='float32') \n    h_zoom = 1.0 + tf.random.normal([1], dtype='float32') / HZOOM_\n    w_zoom = 1.0 + tf.random.normal([1], dtype='float32') / WZOOM_\n    h_shift = HSHIFT_ * tf.random.normal([1], dtype='float32') \n    w_shift = WSHIFT_ * tf.random.normal([1], dtype='float32') \n\n    # GET TRANSFORMATION MATRIX\n    m = get_mat(rot,shr,h_zoom,w_zoom,h_shift,w_shift) \n\n    # LIST DESTINATION PIXEL INDICES\n    x   = tf.repeat(tf.range(DIM//2, -DIM//2,-1), DIM)\n    y   = tf.tile(tf.range(-DIM//2, DIM//2), [DIM])\n    z   = tf.ones([DIM*DIM], dtype='int32')\n    idx = tf.stack( [x,y,z] )\n    \n    # ROTATE DESTINATION PIXELS ONTO ORIGIN PIXELS\n    idx2 = K.dot(m, tf.cast(idx, dtype='float32'))\n    idx2 = K.cast(idx2, dtype='int32')\n    idx2 = K.clip(idx2, -DIM//2+XDIM+1, DIM//2)\n    \n    # FIND ORIGIN PIXEL VALUES           \n    idx3 = tf.stack([DIM//2-idx2[0,], DIM//2-1+idx2[1,]])\n    d    = tf.gather_nd(image, tf.transpose(idx3))\n        \n    return tf.reshape(d,[DIM, DIM,3])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def read_labeled_tfrecord(example):\n    tfrec_format = {\n        'image'                        : tf.io.FixedLenFeature([], tf.string),\n        'image_name'                   : tf.io.FixedLenFeature([], tf.string),\n        'patient_id'                   : tf.io.FixedLenFeature([], tf.int64),\n        'sex'                          : tf.io.FixedLenFeature([], tf.int64),\n        'age_approx'                   : tf.io.FixedLenFeature([], tf.int64),\n        'anatom_site_general_challenge': tf.io.FixedLenFeature([], tf.int64),\n        'diagnosis'                    : tf.io.FixedLenFeature([], tf.int64),\n        'target'                       : tf.io.FixedLenFeature([], tf.int64)\n    }           \n    example = tf.io.parse_single_example(example, tfrec_format)\n    return example['image'], example['target']\n\n\ndef read_unlabeled_tfrecord(example, return_image_name):\n    tfrec_format = {\n        'image'                        : tf.io.FixedLenFeature([], tf.string),\n        'image_name'                   : tf.io.FixedLenFeature([], tf.string),\n    }\n    example = tf.io.parse_single_example(example, tfrec_format)\n    return example['image'], example['image_name'] if return_image_name else 0\n\n \ndef prepare_image(img, augment=True, dim=256):    \n    img = tf.image.decode_jpeg(img, channels=3)\n    img = tf.cast(img, tf.float32) / 255.0\n    \n    if augment:\n        img = transform(img,DIM=dim)\n        img = tf.image.random_flip_left_right(img)\n        #img = tf.image.random_hue(img, 0.01)\n        img = tf.image.random_saturation(img, 0.7, 1.3)\n        img = tf.image.random_contrast(img, 0.8, 1.2)\n        img = tf.image.random_brightness(img, 0.1)\n                      \n    img = tf.reshape(img, [dim,dim, 3])\n            \n    return img\n\ndef count_data_items(filenames):\n    n = [int(re.compile(r\"-([0-9]*)\\.\").search(filename).group(1)) \n         for filename in filenames]\n    return np.sum(n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_dataset(files, augment = False, shuffle = False, repeat = False, \n                labeled=True, return_image_names=True, batch_size=16, dim=256):\n    \n    ds = tf.data.TFRecordDataset(files, num_parallel_reads=AUTO)\n    ds = ds.cache()\n    \n    if repeat:\n        ds = ds.repeat()\n    \n    if shuffle: \n        ds = ds.shuffle(1024*8)\n        opt = tf.data.Options()\n        opt.experimental_deterministic = False\n        ds = ds.with_options(opt)\n        \n    if labeled: \n        ds = ds.map(read_labeled_tfrecord, num_parallel_calls=AUTO)\n    else:\n        ds = ds.map(lambda example: read_unlabeled_tfrecord(example, return_image_names), \n                    num_parallel_calls=AUTO)      \n    \n    ds = ds.map(lambda img, imgname_or_label: (prepare_image(img, augment=augment, dim=dim), \n                                               imgname_or_label), \n                num_parallel_calls=AUTO)\n    \n    ds = ds.batch(batch_size * REPLICAS)\n    ds = ds.prefetch(AUTO)\n    return ds","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Step 3: Build Model\nThis is a common model architecute. Consider experimenting with different backbones, custom heads, losses, and optimizers. Also consider inputing meta features into your CNN.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"EFNS = [irn.InceptionResNetV2]\n\ndef build_model(dim=128, ef=0):\n    inp = tf.keras.layers.Input(shape=(dim,dim,3))\n    base = EFNS[ef](input_shape=(dim,dim,3),weights='imagenet',include_top=False)\n    x = base(inp)\n    x = tf.keras.layers.GlobalMaxPooling2D()(x)\n    x = tf.keras.layers.Dropout(0.3)(x)\n    x = tf.keras.layers.Dense(1,activation='sigmoid')(x)\n    model = tf.keras.Model(inputs=inp,outputs=x)\n#     opt = tf.keras.optimizers.Adam(learning_rate=0.001)\n    opt = tf.keras.optimizers.RMSprop(learning_rate=0.1)\n    loss = tf.keras.losses.BinaryCrossentropy(label_smoothing=0.05) \n    model.compile(optimizer=opt,loss=loss,metrics=['AUC'])\n    return model\nEFNS = [irn.InceptionResNetV2] # [efn.EfficientNetB7]    \n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Step 4: Train Schedule\nThis is a common train schedule for transfer learning. The learning rate starts near zero, then increases to a maximum, then decays over time. Consider changing the schedule and/or learning rates. Note how the learning rate max is larger with larger batches sizes. This is a good practice to follow.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_lr_callback(batch_size=8):\n    lr_start   = 0.000005\n    lr_max     = 0.00000125 * REPLICAS * batch_size\n    lr_min     = 0.000001\n    lr_ramp_ep = 5\n    lr_sus_ep  = 0\n    lr_decay   = 0.8\n   \n    def lrfn(epoch):\n        if epoch < lr_ramp_ep:\n            lr = (lr_max - lr_start) / lr_ramp_ep * epoch + lr_start\n            \n        elif epoch < lr_ramp_ep + lr_sus_ep:\n            lr = lr_max\n            \n        else:\n            lr = (lr_max - lr_min) * lr_decay**(epoch - lr_ramp_ep - lr_sus_ep) + lr_min\n            \n        return lr\n\n    lr_callback = tf.keras.callbacks.LearningRateScheduler(lrfn, verbose=False)\n    return lr_callback","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Train Model\nOur model will be trained for the number of FOLDS and EPOCHS you chose in the configuration above. Each fold the model with lowest validation loss will be saved and used to predict OOF and test. Adjust the variables `VERBOSE` and `DISPLOY_PLOT` below to determine what output you want displayed. The variable `VERBOSE=1 or 2` will display the training and validation loss and auc for each epoch as text. The variable `DISPLAY_PLOT` shows this information as a plot. ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# USE VERBOSE=0 for silent, VERBOSE=1 for interactive, VERBOSE=2 for commit\nVERBOSE = 2\nDISPLAY_PLOT = False\n\nskf = KFold(n_splits=FOLDS,shuffle=True,random_state=SEED)\noof_pred = []; oof_tar = []; oof_val = []; oof_names = []; oof_folds = [] \npreds = np.zeros((count_data_items(files_test),1))\n\nfor fold,(idxT,idxV) in enumerate(skf.split(np.arange(15))):\n    \n    # DISPLAY FOLD INFO\n    if DEVICE=='TPU':\n        if tpu: tf.tpu.experimental.initialize_tpu_system(tpu)\n    print('#'*25); print('#### FOLD',fold+1)\n    print('#### Image Size %i with InceptionResNetV2 and batch_size %i'%\n          (IMG_SIZES[fold],BATCH_SIZES[fold]*REPLICAS))\n    \n    # CREATE TRAIN AND VALIDATION SUBSETS\n    files_train = tf.io.gfile.glob([GCS_PATH[fold] + '/train%.2i*.tfrec'%x for x in idxT])\n    if INC2019[fold]:\n        files_train += tf.io.gfile.glob([GCS_PATH2[fold] + '/train%.2i*.tfrec'%x for x in idxT*2+1])\n        print('#### Using 2019 external data')\n    if INC2018[fold]:\n        files_train += tf.io.gfile.glob([GCS_PATH2[fold] + '/train%.2i*.tfrec'%x for x in idxT*2])\n        print('#### Using 2018+2017 external data')\n    np.random.shuffle(files_train); print('#'*25)\n    files_valid = tf.io.gfile.glob([GCS_PATH[fold] + '/train%.2i*.tfrec'%x for x in idxV])\n    files_test = np.sort(np.array(tf.io.gfile.glob(GCS_PATH[fold] + '/test*.tfrec')))\n    \n    # BUILD MODEL\n    K.clear_session()\n    with strategy.scope():\n        model = build_model(dim=IMG_SIZES[fold],ef=EFF_NETS[fold])\n        \n    # SAVE BEST MODEL EACH FOLD\n    sv = tf.keras.callbacks.ModelCheckpoint(\n        'fold-%i.h5'%fold, monitor='val_loss', verbose=0, save_best_only=True,\n        save_weights_only=True, mode='min', save_freq='epoch')\n   \n    # TRAIN\n    print('Training...')\n    history = model.fit(\n        get_dataset(files_train, augment=True, shuffle=True, repeat=True,\n                dim=IMG_SIZES[fold], batch_size = BATCH_SIZES[fold]), \n        epochs=EPOCHS[fold], callbacks = [sv,get_lr_callback(BATCH_SIZES[fold])], \n        steps_per_epoch=count_data_items(files_train)/BATCH_SIZES[fold]//REPLICAS,\n        validation_data=get_dataset(files_valid,augment=False,shuffle=False,\n                repeat=False,dim=IMG_SIZES[fold]), #class_weight = {0:1,1:2},\n        verbose=VERBOSE\n    )\n    \n    print('Loading best model...')\n    model.load_weights('fold-%i.h5'%fold)\n    \n    # PREDICT OOF USING TTA\n    print('Predicting OOF with TTA...')\n    ds_valid = get_dataset(files_valid,labeled=False,return_image_names=False,augment=True,\n            repeat=True,shuffle=False,dim=IMG_SIZES[fold],batch_size=BATCH_SIZES[fold]*4)\n    ct_valid = count_data_items(files_valid); STEPS = TTA * ct_valid/BATCH_SIZES[fold]/4/REPLICAS\n    pred = model.predict(ds_valid,steps=STEPS,verbose=VERBOSE)[:TTA*ct_valid,] \n    oof_pred.append( np.mean(pred.reshape((ct_valid,TTA),order='F'),axis=1) )                 \n    #oof_pred.append(model.predict(get_dataset(files_valid,dim=IMG_SIZES[fold]),verbose=1))\n    \n    # GET OOF TARGETS AND NAMES\n    ds_valid = get_dataset(files_valid, augment=False, repeat=False, dim=IMG_SIZES[fold],\n            labeled=True, return_image_names=True)\n    oof_tar.append( np.array([target.numpy() for img, target in iter(ds_valid.unbatch())]) )\n    oof_folds.append( np.ones_like(oof_tar[-1],dtype='int8')*fold )\n    ds = get_dataset(files_valid, augment=False, repeat=False, dim=IMG_SIZES[fold],\n                labeled=False, return_image_names=True)\n    oof_names.append( np.array([img_name.numpy().decode(\"utf-8\") for img, img_name in iter(ds.unbatch())]))\n    \n    # PREDICT TEST USING TTA\n    print('Predicting Test with TTA...')\n    ds_test = get_dataset(files_test,labeled=False,return_image_names=False,augment=True,\n            repeat=True,shuffle=False,dim=IMG_SIZES[fold],batch_size=BATCH_SIZES[fold]*4)\n    ct_test = count_data_items(files_test); STEPS = TTA * ct_test/BATCH_SIZES[fold]/4/REPLICAS\n    pred = model.predict(ds_test,steps=STEPS,verbose=VERBOSE)[:TTA*ct_test,] \n    preds[:,0] += np.mean(pred.reshape((ct_test,TTA),order='F'),axis=1) * WGTS[fold]\n    \n    # REPORT RESULTS\n    auc = roc_auc_score(oof_tar[-1],oof_pred[-1])\n    oof_val.append(np.max( history.history['val_auc'] ))\n    print('#### FOLD %i OOF AUC without TTA = %.3f, with TTA = %.3f'%(fold+1,oof_val[-1],auc))\n    \n    # PLOT TRAINING\n    if DISPLAY_PLOT:\n        plt.figure(figsize=(15,5))\n        plt.plot(np.arange(EPOCHS[fold]),history.history['auc'],'-o',label='Train AUC',color='#ff7f0e')\n        plt.plot(np.arange(EPOCHS[fold]),history.history['val_auc'],'-o',label='Val AUC',color='#1f77b4')\n        x = np.argmax( history.history['val_auc'] ); y = np.max( history.history['val_auc'] )\n        xdist = plt.xlim()[1] - plt.xlim()[0]; ydist = plt.ylim()[1] - plt.ylim()[0]\n        plt.scatter(x,y,s=200,color='#1f77b4'); plt.text(x-0.03*xdist,y-0.13*ydist,'max auc\\n%.2f'%y,size=14)\n        plt.ylabel('AUC',size=14); plt.xlabel('Epoch',size=14)\n        plt.legend(loc=2)\n        plt2 = plt.gca().twinx()\n        plt2.plot(np.arange(EPOCHS[fold]),history.history['loss'],'-o',label='Train Loss',color='#2ca02c')\n        plt2.plot(np.arange(EPOCHS[fold]),history.history['val_loss'],'-o',label='Val Loss',color='#d62728')\n        x = np.argmin( history.history['val_loss'] ); y = np.min( history.history['val_loss'] )\n        ydist = plt.ylim()[1] - plt.ylim()[0]\n        plt.scatter(x,y,s=200,color='#d62728'); plt.text(x-0.03*xdist,y+0.05*ydist,'min loss',size=14)\n        plt.ylabel('Loss',size=14)\n        plt.title('FOLD %i - Image Size %i, InceptionResNetV2, inc2019=%i, inc2018=%i'%\n                (fold+1,IMG_SIZES[fold],INC2019[fold],INC2018[fold]),size=18)\n        plt.legend(loc=3)\n        plt.show()  ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Calculate OOF AUC\nThe OOF (out of fold) predictions are saved to disk. If you wish to ensemble multiple models, use the OOF to determine what are the best weights to blend your models with. Choose weights that maximize OOF CV score when used to blend OOF. Then use those same weights to blend your test predictions.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# COMPUTE OVERALL OOF AUC\noof = np.concatenate(oof_pred); true = np.concatenate(oof_tar);\nnames = np.concatenate(oof_names); folds = np.concatenate(oof_folds)\nauc = roc_auc_score(true,oof)\nprint('Overall OOF AUC with TTA = %.3f'%auc)\n\n# SAVE OOF TO DISK\ndf_oof = pd.DataFrame(dict(\n    image_name = names, target=true, pred = oof, fold=folds))\ndf_oof.to_csv('oof.csv',index=False)\ndf_oof.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Submit To Kaggle","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"ds = get_dataset(files_test, augment=False, repeat=False, dim=IMG_SIZES[fold],\n                 labeled=False, return_image_names=True)\n\nimage_names = np.array([img_name.numpy().decode(\"utf-8\") \n                        for img, img_name in iter(ds.unbatch())])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission = pd.DataFrame(dict(image_name=image_names, target=preds[:,0]))\nsubmission = submission.sort_values('image_name') \nsubmission.to_csv('submission.csv', index=False)\nsubmission.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.hist(submission.target,bins=100)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Thanks for [the great starter notebook.](https://www.kaggle.com/mahmudds/siim-isic-melanoma-classification)\n## -------------------- End--------------------------","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}