{"cells":[{"metadata":{},"cell_type":"markdown","source":"## cellular automata evaluation\nOriginal: https://www.kaggle.com/nroman/melanoma-pytorch-starter-efficientnet"},{"metadata":{},"cell_type":"markdown","source":"#### Versions oc:\n* v1: fork from https://www.kaggle.com/octaviomm/melanoma-pytorch-starter-efficientnet\n* v2: THISONE\n* v3: multiple changes to work only with the files from the TRAIN folder (for training and validation)\n* v4: adding WeightedRandomSampler\n\n#### Versions:\n* v9: ColorJitter transformation added **[0.896]**\n* v10: Changed the dataset to [this one](https://www.kaggle.com/shonenkov/melanoma-merged-external-data-512x512-jpeg) with external data. **[0.894]**\n* v11: Switched to [another dataset](https://www.kaggle.com/nroman/melanoma-external-malignant-256/) which I've created by myself. Also switched from StratifiedKFold to GroupKFold **[0.916]**\n* v12: Switched to efficientnet-b1 **[0.919]**\n* v13: Using meta featues: sex and age **[0.918]**\n* v14: anatom_site_general_challenge meta feature added as one-hot encoded matrix **[0.923]**\n* v16: Fixed OOF - now it contains only data from original training dataset, without extarnal data. Also switched back to StratifiedKFold. Added DrawHair augmentation. **[0.909]**\n* v18: Too many things were changed at the same time. All experiments should have only one small change each, so it would be easy to understand how changes affect the result. Said that I rolled back everything, keeping only OOF fix, to make sure it work.\n* v19: Added 'Hair' augmentation. OOF rework posponed untill the best time, since there is some bug in my code for it. **[0.925]**\n* v20: Advanced Hair Augmentation technique used. Read more about it here: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/159176 **[0.923]**\n* v21: Microscope augmentation added instead of Cutout. Read more here: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/159476 **[0.914]**\n* v22: Changed the dataset to [this one](https://www.kaggle.com/cdeotte/jpeg-melanoma-256x256) by Chris Deotte. More info [here](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/165526) **[0.900]**\n* v23: All the same as v22 but effnet-b0 instead of b1 and more epochs per fold. **[0.895]**\n* v24: effnet-b01 and more epochs. **[0.9092]**\n* v25: Fixed a mistake in a way of filling preds. See [this comment](https://www.kaggle.com/nroman/melanoma-pytorch-starter-efficientnet/comments?scriptVersionId=39125585#913846). **[0.9016]**\n* v26: Fix for another mistake. This time with a way of averaging TTA. See [this comment](https://www.kaggle.com/nroman/melanoma-pytorch-starter-efficientnet/comments#955916) **[0.915]**\n* v27: Back to [my dataset](https://www.kaggle.com/nroman/melanoma-external-malignant-256/)"},{"metadata":{},"cell_type":"markdown","source":"# * CHANGE TEST TRANSFORMS to test_transforms\n# * change roc for val loss to save model\n# * check the threshold change for imbalanced classification   "},{"metadata":{"_kg_hide-output":true,"trusted":true,"_kg_hide-input":false},"cell_type":"code","source":"!pip install -q efficientnet_pytorch torchtoolbox","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import torch\nimport torchvision\nimport torch.nn.functional as F\nimport torch.nn as nn\nimport torchtoolbox.transform as transforms\nfrom torch.utils.data import Dataset, DataLoader, Subset\nfrom torch.optim.lr_scheduler import ReduceLROnPlateau\nfrom sklearn.metrics import accuracy_score, roc_auc_score\nfrom sklearn.model_selection import StratifiedKFold, GroupKFold, KFold\nimport pandas as pd\nimport numpy as np\nimport gc\nimport os\nimport cv2\nimport time\nimport datetime\nimport warnings\nimport random\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom tqdm.notebook import tqdm\nfrom efficientnet_pytorch import EfficientNet\nfrom torch.utils.data import WeightedRandomSampler\nfrom sklearn.metrics import precision_score, recall_score\nfrom IPython.display import FileLink\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"warnings.simplefilter('ignore')\ndef seed_everything(seed):\n    random.seed(seed)\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    np.random.seed(seed)\n    torch.manual_seed(seed)\n    torch.cuda.manual_seed(seed)\n    torch.backends.cudnn.deterministic = True\n    torch.backends.cudnn.benchmark = True\n\nseed_everything(47)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def make_train_test_df_for_cea_aug2(train_df_orig, df_cea_done, test_set_size = 2000, test_set_1s = 100, add_0s_into_train_df=0, add_1s_into_train_df=0):\n    '''make a train_df that contains all the files that cea==completed & target==1\n    and a test_df that has all the files that cea!=completed & target==1\n    Optionally you can include fewer files'''\n    # make a train_df that contains all the files that cea==completed & target==1\n    image_name_cea_done = df_cea_done['image_name'].values\n    train_df = train_df_orig[train_df_orig['image_name'].isin(image_name_cea_done)]\n    # make sure test_df has all the files that cea!=completed & target==1\n    test_df_large = train_df_orig[~train_df_orig['image_name'].isin(image_name_cea_done)]\n    # divide in target==1 & target == 0\n    test_df_only1s = test_df_large[test_df_large['target']==1]\n    test_df_only0s = test_df_large[test_df_large['target']==0]\n    test_df_only0s = test_df_only0s.reset_index(drop=True)\n    test_df_only1s = test_df_only1s.reset_index(drop=True)\n    # if add_0s_into_train_df more samples are wanted in train_df (of zeros)\n    if add_0s_into_train_df >0:\n        random.seed(0)\n        rand_ints = random.sample(range(len(test_df_only0s)-1), add_0s_into_train_df)\n        extra_0s_for_train_df = test_df_only0s.iloc[rand_ints]\n        train_df = train_df.append(extra_0s_for_train_df)\n        test_df_only0s =test_df_only0s.drop(rand_ints, axis=0)\n    # if add_1s_into_train_df more samples are wanted in train_df (of ones)\n    if add_1s_into_train_df >0:\n        random.seed(0)\n        rand_ints = random.sample(range(len(test_df_only1s)-1), add_1s_into_train_df)\n        extra_1s_for_train_df = test_df_only1s.iloc[rand_ints]\n        train_df = train_df.append(extra_1s_for_train_df)\n        test_df_only1s = test_df_only1s.drop(rand_ints, axis=0)\n    # if only a subset of the target == 1 is wanted\n    if test_set_1s > 0:\n        random.seed(0)\n        rand_ints = random.sample(range(len(test_df_only1s)), test_set_1s)\n        test_df_only1s = test_df_only1s.iloc[rand_ints]\n        \n    # if add_1s_into_train_df > 0 or add_0s_into_train_df > 0\n    if add_1s_into_train_df > 0 or add_0s_into_train_df > 0:\n        test_df_large = pd.DataFrame()\n        test_df_large = test_df_large.append(test_df_only1s)\n        test_df_large = test_df_large.append(test_df_only0s)\n        random.seed(0)\n        rand_ints = random.sample(range(len(test_df_large)), len(test_df_large))\n        test_df_large.index = rand_ints\n        test_df_large = test_df_large.reindex()\n        test_df = test_df_large\n    else:\n        # get a large subset of test_df_large that contain all target 1 from test_df_large\n        test_df = test_df_large\n        if test_set_size > 0:\n            number1s_already_in_test = len(test_df_only1s)\n            random.seed(0)\n            rand_ints = random.sample(range(len(test_df_only0s)), test_set_size - number1s_already_in_test)\n            test_df_only0s_subset = test_df_only0s.iloc[rand_ints]\n            test_df = test_df_only1s.append(test_df_only0s_subset)\n    return train_df.reset_index(drop=True), test_df.reset_index(drop=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def transform_features(train_df, test_df):\n    # One-hot encoding of anatom_site_general_challenge feature\n    concat = pd.concat([train_df['anatom_site_general_challenge'], test_df['anatom_site_general_challenge']], ignore_index=True)\n    dummies = pd.get_dummies(concat, dummy_na=True, dtype=np.uint8, prefix='site')\n    train_df = pd.concat([train_df, dummies.iloc[:train_df.shape[0]]], axis=1)\n    test_df = pd.concat([test_df, dummies.iloc[train_df.shape[0]:].reset_index(drop=True)], axis=1)\n\n    # Sex features\n    train_df['sex'] = train_df['sex'].map({'male': 1, 'female': 0})\n    test_df['sex'] = test_df['sex'].map({'male': 1, 'female': 0})\n    train_df['sex'] = train_df['sex'].fillna(-1)\n    test_df['sex'] = test_df['sex'].fillna(-1)\n\n    # Age features\n    train_df['age_approx'] /= train_df['age_approx'].max()\n    test_df['age_approx'] /= test_df['age_approx'].max()\n    train_df['age_approx'] = train_df['age_approx'].fillna(0)\n    test_df['age_approx'] = test_df['age_approx'].fillna(0)\n\n    train_df['patient_id'] = train_df['patient_id'].fillna(0)\n    return train_df, test_df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_synthesize_images_for_only_images_in_fold(train_df, train_df_synthesized, train_idx):\n    #For only the images in this train fold get the synthesized ones\n    df_train_in_fold = train_df.iloc[train_idx].reset_index(drop=True)\n    names_in_train = df_train_in_fold['image_name'].values\n    names_unique_synt_all = np.unique(train_df_synthesized['image_name_root'].values)\n    names_in_train_that_have_synt = list(set(names_in_train).intersection(names_unique_synt_all))\n    # select those images from all the synthesized ones\n    train_synt_to_add_fold = train_df_synthesized[train_df_synthesized['image_name_root'].isin(names_in_train_that_have_synt)]\n    # append the synthesized images to the current train fold\n    df_train_in_fold_synt_added = df_train_in_fold.append(train_synt_to_add_fold)\n    return df_train_in_fold_synt_added","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_sampler_for_imbalanced_classification(train_df):\n    '''https://discuss.pytorch.org/t/how-to-handle-imbalanced-classes/11264/2'''\n    target = train_df['target'].values\n    # print(f'target train 0/1: {len(np.where(target == 1)[0])}/{len(np.where(target == 0)[0])}')\n    class_sample_count = np.array([len(np.where(target == t)[0]) for t in np.unique(target)])\n    weight = 1. / class_sample_count\n    samples_weight_py = np.array([weight[t] for t in target])\n    samples_weight = torch.from_numpy(samples_weight_py)\n    samples_weigth = samples_weight.double()\n    sampler = WeightedRandomSampler(samples_weight, len(samples_weight))\n    return sampler, samples_weight_py","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!ls /kaggle/input","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# OC just checking shapes and files that completed cea reconstruction\ntrain_df_orig = pd.read_csv('/kaggle/input/jpeg-melanoma-256x256/train.csv')\ndf_cea_done = pd.read_csv('/kaggle/input/files-cea-done-264/files_cea_done_264.csv')\nprint(f\"train_df_orig = {train_df_orig.shape}, train_df total 1's = {np.sum(train_df_orig['target'].values)}\")\nprint(f'df_cea_done = {df_cea_done.shape}')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# OPTION 2\ntrain_df, test_df = make_train_test_df_for_cea_aug2(train_df_orig, df_cea_done, \n                                                   test_set_size = 2000, test_set_1s = 450,\n                                                  add_0s_into_train_df = 24890-len(df_cea_done),#1500-len(df_cea_done)\n                                                  add_1s_into_train_df = 110) # 0\nnp.shape(train_df), np.shape(test_df)\nprint(f\"train_df = {train_df.shape}, train_df total 1's = {np.sum(train_df['target'].values)}, train_df total 1=0's = {np.sum(train_df['target'].values==0)}\")\nprint(f\"test_df = {test_df.shape}, test_df total 1's = {np.sum(test_df['target'].values)}, test_df total 0's = {np.sum(test_df['target'].values==0)}\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# # OPTION 3 (small, only for testing)\n# train_df, test_df = make_train_test_df_for_cea_aug2(train_df_orig, df_cea_done, \n#                                                    test_set_size = 2000, test_set_1s = 450,\n#                                                   add_0s_into_train_df = 0,#1500-len(df_cea_done)\n#                                                   add_1s_into_train_df = 0) # 0\n# np.shape(train_df), np.shape(test_df)\n# print(f\"train_df = {train_df.shape}, train_df total 1's = {np.sum(train_df['target'].values)}, train_df total 1=0's = {np.sum(train_df['target'].values==0)}\")\n# print(f\"test_df = {test_df.shape}, test_df total 1's = {np.sum(test_df['target'].values)}, test_df total 0's = {np.sum(test_df['target'].values==0)}\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Transform features (one hot encoding)\ntrain_df, test_df = transform_features(train_df, test_df)\nprint(f\"train_df = {train_df.shape}, train_df total 1's = {np.sum(train_df['target'].values)}, train_df total 1=0's = {np.sum(train_df['target'].values==0)}\")\nprint(f\"test_df = {test_df.shape}, test_df total 1's = {np.sum(test_df['target'].values)}, test_df total 0's = {np.sum(test_df['target'].values==0)}\")\nprint('============')\n# make Dataframe with the synthesized images\n# path_synthesis_selected = '/kaggle/input/cea-synthesis/cea_synthesis_selected_no_dark/'\npath_synthesis_selected = '/kaggle/input/cea-synthesis-threshold0/cea_synthesis_selected_no_dark_threshold0/'\nnames_synthesis_selected = os.listdir(path_synthesis_selected)\nnames_synt_selec = np.sort(names_synthesis_selected)\nnames_synt_selec = [i[:-4] for i in names_synt_selec]\nnames_synt_selec_root = [i[:-4] for i in names_synt_selec]\ndf_names_synt_selec = pd.DataFrame((names_synt_selec,names_synt_selec_root)).T\ndf_names_synt_selec.columns = ['synt_name', 'image_name']\n# Add the data of each lesion to the DF of the synthesized images\ntrain_df_synthesized = df_names_synt_selec.merge(train_df, on='image_name')\n# change image_name to load the correct name\ntrain_df_synthesized = train_df_synthesized.rename(columns={'image_name':'image_name_root'})\ntrain_df_synthesized = train_df_synthesized.rename(columns={'synt_name':'image_name'})\n# add a column to each dataframe to indicate the path where the images are located\ntrain_df['imfolder'] = '/kaggle/input/jpeg-melanoma-256x256/train/'\ntest_df['imfolder'] = '/kaggle/input/jpeg-melanoma-256x256/train/'\n# train_df_synthesized['imfolder'] = '/kaggle/input/cea-synthesis/cea_synthesis_selected_no_dark/'\ntrain_df_synthesized['imfolder'] = '/kaggle/input/cea-synthesis-threshold0/cea_synthesis_selected_no_dark_threshold0/'\nprint(f\"train_df = {train_df.shape}, train_df total 1's = {np.sum(train_df['target'].values)}, train_df total 1=0's = {np.sum(train_df['target'].values==0)}\")\nprint(f\"test_df = {test_df.shape}, test_df total 1's = {np.sum(test_df['target'].values)}, test_df total 0's = {np.sum(test_df['target'].values==0)}\")\nprint(f\"train_df_synthesized = {train_df_synthesized.shape}, test_df total 1's = {np.sum(train_df_synthesized['target'].values)}, test_df total 0's = {np.sum(train_df_synthesized['target'].values==0)}\")\n# By default we start training from 0\nRESUME_TRAINING = False","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# ## Test if you can Get the correct samples in train and val per fold\n\n# skf = KFold(n_splits=5, shuffle=True, random_state=47)\n# train_idx_all, val_idx_all = [], []\n# count_1s_per_train_fold, count_1s_per_val_fold = [], []\n# for fold, (train_idx, val_idx) in enumerate(skf.split(X=np.zeros(len(train_df)), y=train_df['target'], groups=train_df['patient_id'].tolist()), 1):\n#     train_idx_all.append(train_idx)\n#     val_idx_all.append(val_idx)\n#     count_1s_per_val_fold.append(np.sum(train_df.iloc[val_idx]['target'].values))\n#     count_1s_per_train_fold.append(np.sum(train_df.iloc[train_idx]['target'].values))\n#     print(f'fold={fold}, train_idx={np.shape(train_idx)}, train_idx=[{train_idx[0]}-{train_idx[-1]}], val_idx={np.shape(val_idx)}, val_idx[{val_idx[0]}-{val_idx[-1]}]')\n# #figure\n# fold_samples = np.zeros(len(train_df))\n# fold_samples[train_idx_all[2]]=1\n# fold_samples_x = np.linspace(1,len(fold_samples),len(fold_samples))\n# fig, ax = plt.subplots(2,1,figsize=(34,4))\n# ax[0].scatter(fold_samples_x, fold_samples);\n# ax[1].scatter(fold_samples_x[10000:10500], fold_samples[10000:10500]);\n# print(f'count_1s_per_val_fold = {count_1s_per_val_fold}')\n# print(f'count_1s_per_train_fold = {count_1s_per_train_fold}')\n\n\n# df_train_in_fold_synt_added = get_synthesize_images_for_only_images_in_fold(train_df, train_df_synthesized, train_idx_all[3])\n# print(f'df_train_in_fold_synt_added: {df_train_in_fold_synt_added.shape}')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\ndevice","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Dataset class adapted to get the path from the DF\nclass MelanomaDataset(Dataset):\n    def __init__(self, df: pd.DataFrame, train: bool = True, transforms = None, meta_features = None):\n        \"\"\"\n        Class initialization\n        Args:\n            df (pd.DataFrame): DataFrame with data description\n            imfolder (str): folder with images\n            train (bool): flag of whether a training dataset is being initialized or testing one\n            transforms: image transformation method to be applied\n            meta_features (list): list of features with meta information, such as sex and age\n            \n        \"\"\"\n        self.df = df\n        self.transforms = transforms\n        self.train = train\n        self.meta_features = meta_features\n        \n    def __getitem__(self, index):\n        # print(index)\n        im_path = os.path.join(self.df.iloc[index]['imfolder'], self.df.iloc[index]['image_name'] + '.jpg')\n        # print(im_path)\n        x = cv2.imread(im_path)\n        meta = np.array(self.df.iloc[index][self.meta_features].values, dtype=np.float32)\n\n        if self.transforms:\n            x = self.transforms(x)\n            \n        if self.train:\n            y = self.df.iloc[index]['target']\n            return (x, meta), y\n        else:\n            return (x, meta)\n    \n    def __len__(self):\n        return len(self.df)\n    \n    \nclass Net(nn.Module):\n    def __init__(self, arch, n_meta_features: int):\n        super(Net, self).__init__()\n        self.arch = arch\n        if 'ResNet' in str(arch.__class__):\n            self.arch.fc = nn.Linear(in_features=512, out_features=500, bias=True)\n        if 'EfficientNet' in str(arch.__class__):\n            self.arch._fc = nn.Linear(in_features=1280, out_features=500, bias=True)\n        self.meta = nn.Sequential(nn.Linear(n_meta_features, 500),\n                                  nn.BatchNorm1d(500),\n                                  nn.ReLU(),\n                                  nn.Dropout(p=0.2),\n                                  nn.Linear(500, 250),  # FC layer output will have 250 features\n                                  nn.BatchNorm1d(250),\n                                  nn.ReLU(),\n                                  nn.Dropout(p=0.2))\n        self.ouput = nn.Linear(500 + 250, 1)\n        \n    def forward(self, inputs):\n        \"\"\"\n        No sigmoid in forward because we are going to use BCEWithLogitsLoss\n        Which applies sigmoid for us when calculating a loss\n        \"\"\"\n        x, meta = inputs\n        cnn_features = self.arch(x)\n        meta_features = self.meta(meta)\n        features = torch.cat((cnn_features, meta_features), dim=1)\n        output = self.ouput(features)\n        return output","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class Microscope:\n    \"\"\"\n    Cutting out the edges around the center circle of the image\n    Imitating a picture, taken through the microscope\n\n    Args:\n        p (float): probability of applying an augmentation\n    \"\"\"\n\n    def __init__(self, p: float = 0.5):\n        self.p = p\n\n    def __call__(self, img):\n        \"\"\"\n        Args:\n            img (PIL Image): Image to apply transformation to.\n\n        Returns:\n            PIL Image: Image with transformation.\n        \"\"\"\n        if random.random() < self.p:\n            circle = cv2.circle((np.ones(img.shape) * 255).astype(np.uint8), # image placeholder\n                        (img.shape[0]//2, img.shape[1]//2), # center point of circle\n                        random.randint(img.shape[0]//2 - 3, img.shape[0]//2 + 15), # radius\n                        (0, 0, 0), # color\n                        -1)\n\n            mask = circle - 255\n            img = np.multiply(img, mask)\n        \n        return img\n\n    def __repr__(self):\n        return f'{self.__class__.__name__}(p={self.p})'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_transform = transforms.Compose([\n    #AdvancedHairAugmentation(hairs_folder='/kaggle/input/melanoma-hairs'),\n    transforms.RandomResizedCrop(size=256, scale=(0.8, 1.0)),\n    transforms.RandomHorizontalFlip(),\n    transforms.RandomVerticalFlip(),\n    Microscope(p=0.5),\n    transforms.ToTensor(),\n    transforms.Normalize(mean=[0.485, 0.456, 0.406],std=[0.229, 0.224, 0.225])\n])\ntest_transform = transforms.Compose([\n    transforms.ToTensor(),\n    transforms.Normalize(mean=[0.485, 0.456, 0.406],std=[0.229, 0.224, 0.225])\n])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"arch = EfficientNet.from_pretrained('efficientnet-b1');","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# check intersection of train and test\nbool(set(train_df['image_name'].values) & set(test_df['image_name'].values))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# # Make sure there are no train samples in test subset and vicecersa\n# for i in train_df['image_name'].values:\n#     assert(i not in test_df['image_name'].values)\n# for i in test_df['image_name'].values:\n#     assert(i not in train_df['image_name'].values)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Check target distribution \nprint(f'train_df_orig: {np.sum(train_df_orig[\"target\"].values==1)/len(train_df_orig):.4f}, {np.sum(train_df_orig[\"target\"].values==0)}, {np.sum(train_df_orig[\"target\"].values==1)}')\nprint(f'train_df: {np.sum(train_df[\"target\"].values==1)/len(train_df):.4f}, {np.sum(train_df[\"target\"].values==0)}, {np.sum(train_df[\"target\"].values==1)}')\nprint(f'test_df: {np.sum(test_df[\"target\"].values==1)/len(test_df):.4f}, {np.sum(test_df[\"target\"].values==0)}, {np.sum(test_df[\"target\"].values==1)}')\nfig, ax = plt.subplots(1,3, figsize=(9,2))\nax[0].hist(train_df_orig['target'].values);\nax[1].hist(train_df['target'].values);\nax[2].hist(test_df['target'].values);","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"meta_features = ['sex', 'age_approx'] + [col for col in train_df.columns if 'site_' in col]\nmeta_features.remove('anatom_site_general_challenge')\nnp.asarray(meta_features)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df2 = train_df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# OC check datasets and dataloaders\nrand_ints = np.random.randint(0,len(train_df2),100)\nrand_ints = np.random.randint(0,len(train_df2),100)\ndf=train_df2.iloc[rand_ints].reset_index(drop=True)\nsampler, samples_weight_py = get_sampler_for_imbalanced_classification(df)\ntrain_or_test = MelanomaDataset(df, \n#                             imfolder='/kaggle/input/melanoma-external-malignant-256/train/train/', \n#                             imfolder='/kaggle/input/jpeg-melanoma-256x256/train/', \n                            train=True, \n                            transforms=train_transform,\n                            meta_features=meta_features)\n# XX WARNING shuffle should be false \ntrain_or_test_loader = DataLoader(dataset=train_or_test, batch_size=64, shuffle=False, num_workers=2,\n                                  sampler=sampler)\ntrain_or_test_loader_item = next(iter(train_or_test_loader))\nprint(f'X and Y = {len(train_or_test_loader_item)}')\nprint(f'X = {len(train_or_test_loader_item[0])} (image) and (meta features)')\nprint(f'image ={np.shape(train_or_test_loader_item[0][0])}, meta features = { np.shape(train_or_test_loader_item[0][1])}')\nprint(f'Y len = ({len(train_or_test_loader_item[1])}) ,number 1s: {torch.sum(train_or_test_loader_item[1])} first 5: {train_or_test_loader_item[1][:5]}')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# new test dataset. we get the images from the TRAIN folder\ntest = MelanomaDataset(df=test_df.reset_index(drop=True),\n                       # imfolder='/kaggle/input/melanoma-external-malignant-256/test/test/', \n#                        imfolder='/kaggle/input/jpeg-melanoma-256x256/train/', \n                       train=False,\n                       transforms=test_transform,  # For TTA\n                       meta_features=meta_features)\ntest_loader = DataLoader(dataset=test, batch_size=16, shuffle=False, num_workers=2)\ntest_loader_item = next(iter(test_loader))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### check folds splits"},{"metadata":{"trusted":true},"cell_type":"code","source":"# skf = KFold(n_splits=5, shuffle=True, random_state=47)\n# train_idx_all = []\n# count_1s_per_val_fold = []\n# for fold, (train_idx, val_idx) in enumerate(skf.split(X=np.zeros(len(train_df2)), y=train_df2['target'], groups=train_df2['patient_id'].tolist()), 1):\n#     train_idx_all.append(train_idx)\n#     count_1s_per_val_fold.append(np.sum(train_df2.iloc[val_idx]['target'].values))\n#     print(f'fold={fold}, train_idx={np.shape(train_idx)}, train_idx=[{train_idx[0]}-{train_idx[-1]}], val_idx={np.shape(val_idx)}, val_idx[{val_idx[0]}-{val_idx[-1]}]')\n# #figure\n# fold_samples = np.zeros(len(train_df2))\n# fold_samples[train_idx_all[2]]=1\n# fold_samples_x = np.linspace(1,len(fold_samples),len(fold_samples))\n# plt.figure(figsize=(34,2))\n# plt.scatter(fold_samples_x, fold_samples);\n# print(f'count_1s_per_val_fold = {count_1s_per_val_fold}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* run for one epoch to save the LAST models, \n* then make a new main loop block that loads the previous LAST models and epochs "},{"metadata":{"trusted":true},"cell_type":"code","source":"RESUME_TRAINING = True","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true},"cell_type":"code","source":"USE_AUGMENTATIONS = True\nWEIGHTED_SAMPLER = True\nepochs = 10 # orig 12 \nBATCH_SIZE_TRAIN = 64 # orig 64\nBATCH_SIZE_VAL_TEST = 16\nes_patience = 10 # orig 3  # Early Stopping patience - for how many epochs with no improvements to wait\n# TTA = 3 # Test Time Augmentation rounds\nstart = time.time()\nskf = KFold(n_splits=5, shuffle=True, random_state=47)\n\nval_acc_all = [ [] for _ in range(skf.n_splits) ]\nval_roc_all = [ [] for _ in range(skf.n_splits) ]\nval_precision_all = [ [] for _ in range(skf.n_splits) ]\nval_recall_all = [ [] for _ in range(skf.n_splits) ]\nepoch_loss_all = [ [] for _ in range(skf.n_splits) ]\nepoch_loss_val_all = [ [] for _ in range(skf.n_splits) ]\n\noof = np.zeros((len(train_df2), 1))  # Out Of Fold predictions\noof_all = np.zeros((len(train_df2), skf.n_splits))  # Out Of Fold predictions OC\npreds = torch.zeros((len(test), 1), dtype=torch.float32, device=device)  # Predictions for test test\npreds_separate = []\n\nval_acc_all = [ [] for _ in range(skf.n_splits) ]\nval_roc_all = [ [] for _ in range(skf.n_splits) ]\nfor fold, (train_idx, val_idx) in enumerate(skf.split(X=np.zeros(len(train_df2)), y=train_df2['target'], groups=train_df2['patient_id'].tolist()), 1):\n    print('=' * 20, 'Fold', fold, '=' * 20)  \n    \n    model_path = f'model_{fold}.pth'  # Path and filename to save model to\n    best_val = 0  # Best validation score within this fold (for val_roc)\n    # best_val = 1000  # Best validation score within this fold (for val_loss)\n    patience = es_patience  # Current patience counter\n    arch = EfficientNet.from_pretrained('efficientnet-b1')\n    model = Net(arch=arch, n_meta_features=len(meta_features))  # New model for each fold\n    model = model.to(device)\n    \n    optim = torch.optim.Adam(model.parameters(), lr=0.001)\n    scheduler = ReduceLROnPlateau(optimizer=optim, mode='max', patience=1, verbose=True, factor=0.2) # for val_roc\n    # scheduler = ReduceLROnPlateau(optimizer=optim, mode='min', patience=1, verbose=True, factor=0.2) # for val_loss\n    criterion = nn.BCEWithLogitsLoss()\n    \n    if USE_AUGMENTATIONS:\n        train_df_aug = get_synthesize_images_for_only_images_in_fold(train_df, train_df_synthesized, train_idx)\n    else:\n        train_df_aug = train_df2.iloc[train_idx]\n    \n    if WEIGHTED_SAMPLER:\n        if USE_AUGMENTATIONS:\n            sampler, samples_weight_py = get_sampler_for_imbalanced_classification(train_df_aug.reset_index(drop=True))\n        else:\n            sampler, samples_weight_py = get_sampler_for_imbalanced_classification(train_df_aug.reset_index(drop=True))\n\n    train = MelanomaDataset(df=train_df_aug.reset_index(drop=True), \n                            # imfolder='/kaggle/input/melanoma-external-malignant-256/train/train/', \n                            # imfolder='/kaggle/input/jpeg-melanoma-256x256/train/', \n                            train=True, \n                            transforms=train_transform,\n                            meta_features=meta_features)\n    val = MelanomaDataset(df=train_df2.iloc[val_idx].reset_index(drop=True), \n                            # imfolder='/kaggle/input/melanoma-external-malignant-256/train/train/', \n                            # imfolder='/kaggle/input/jpeg-melanoma-256x256/train/', \n                            train=True, \n                            transforms=test_transform,\n                            meta_features=meta_features)\n    \n    if WEIGHTED_SAMPLER:\n        train_loader = DataLoader(dataset=train, batch_size=BATCH_SIZE_TRAIN, num_workers=2, sampler=sampler)\n    else:\n        train_loader = DataLoader(dataset=train, batch_size=BATCH_SIZE_TRAIN, shuffle=True, num_workers=2)\n    val_loader = DataLoader(dataset=val, batch_size=BATCH_SIZE_VAL_TEST, shuffle=False, num_workers=2)\n    test_loader = DataLoader(dataset=test, batch_size=BATCH_SIZE_VAL_TEST, shuffle=False, num_workers=2)\n    \n    for epoch in tqdm(range(epochs)):\n        start_time = time.time()\n        correct = 0\n        epoch_loss = 0\n        epoch_loss_val = 0\n        model.train()\n        \n        for x, y in tqdm(train_loader, desc='train'):\n            x[0] = torch.tensor(x[0], device=device, dtype=torch.float32)\n            x[1] = torch.tensor(x[1], device=device, dtype=torch.float32)\n            y = torch.tensor(y, device=device, dtype=torch.float32)\n            optim.zero_grad()\n            z = model(x)\n            loss = criterion(z, y.unsqueeze(1))\n            loss.backward()\n            optim.step()\n            # pred_proba_train = torch.sigmoid(z) # added by OMM\n            pred = torch.round(torch.sigmoid(z))  # round off sigmoid to obtain predictions\n            correct += (pred.cpu() == y.cpu().unsqueeze(1)).sum().item()  # tracking number of correctly predicted samples\n            epoch_loss += loss.item()\n        train_acc = correct / len(train_idx)\n        \n        model.eval()  # switch model to the evaluation mode\n        val_preds = torch.zeros((len(val_idx), 1), dtype=torch.float32, device=device)\n        with torch.no_grad():  # Do not calculate gradient since we are only predicting\n            # Predicting on validation set\n            for j, (x_val, y_val) in enumerate(val_loader):\n                x_val[0] = torch.tensor(x_val[0], device=device, dtype=torch.float32)\n                x_val[1] = torch.tensor(x_val[1], device=device, dtype=torch.float32)\n                y_val = torch.tensor(y_val, device=device, dtype=torch.float32)\n                z_val = model(x_val)\n                loss_val = criterion(z_val, y_val.unsqueeze(1))\n                val_pred = torch.sigmoid(z_val)\n                \n                val_preds[j*val_loader.batch_size:j*val_loader.batch_size + x_val[0].shape[0]] = val_pred\n                epoch_loss_val += loss_val.item()\n            val_acc = accuracy_score(train_df2.iloc[val_idx]['target'].values, torch.round(val_preds.cpu()))\n            val_roc = roc_auc_score(train_df2.iloc[val_idx]['target'].values, val_preds.cpu())\n            val_precision = precision_score(train_df2.iloc[val_idx]['target'].values, torch.round(val_preds.cpu()))\n            val_recall = recall_score(train_df2.iloc[val_idx]['target'].values, torch.round(val_preds.cpu()))\n            \n            val_acc_all[fold-1].append(val_acc)\n            val_roc_all[fold-1].append(val_roc)\n            val_precision_all[fold-1].append(val_precision)\n            val_recall_all[fold-1].append(val_recall)\n            \n            epoch_loss_all[fold-1].append(epoch_loss)\n            epoch_loss_val_all[fold-1].append(epoch_loss_val)\n            \n            print('Epoch {:03}: | Loss: {:.3f} | Train acc: {:.3f} | Val acc: {:.3f} | Val roc_auc: {:.3f} | Training time: {}'.format(\n            epoch + 1, epoch_loss, train_acc, val_acc, val_roc, str(datetime.timedelta(seconds=time.time() - start_time))[:7]))\n            \n            # scheduler.step(val_roc)\n            scheduler.step(epoch_loss_val)\n                \n            if val_roc >= best_val: #val_roc >= best_val epoch_loss_val <= best_val\n                best_val = val_roc # best_val = val_roc best_val = epoch_loss_val\n                patience = es_patience  # Resetting patience since we have new best validation accuracy\n                torch.save(model, model_path)  # Saving current best model\n            else:\n                patience -= 1\n                if patience == 0:\n                    print('Early stopping. Best Val roc_auc: {:.3f}'.format(best_val))\n                    break\n    \n#     assert(1==2) # continue in this block\n    # save model after epoch iterations\n#     model_last_epoch_path = f'model_last_epoch_{fold}.pth'\n#     torch.save(model, model_last_epoch_path)\n    \n    # TRANINIG FINISHED (FOR THIS FOLD)\n    model = torch.load(model_path)  # Loading best model of this fold\n    model.eval()  # switch model to the evaluation mode\n    val_preds = torch.zeros((len(val_idx), 1), dtype=torch.float32, device=device)\n    test_preds = torch.zeros((len(test), 1), dtype=torch.float32, device=device)\n    with torch.no_grad():\n        # Predicting on validation set once again to obtain data for OOF\n        for j, (x_val, y_val) in enumerate(val_loader):\n            x_val[0] = torch.tensor(x_val[0], device=device, dtype=torch.float32)\n            x_val[1] = torch.tensor(x_val[1], device=device, dtype=torch.float32)\n            y_val = torch.tensor(y_val, device=device, dtype=torch.float32)\n            z_val = model(x_val)\n            val_pred = torch.sigmoid(z_val)\n            val_preds[j*val_loader.batch_size:j*val_loader.batch_size + x_val[0].shape[0]] = val_pred\n        oof[val_idx] = val_preds.cpu().numpy()\n        \n        # Predicting on test set\n        \n        # Not using TTA (new block) \n        for i, x_test in tqdm(enumerate(test_loader), desc='test set', total = len(test_df)//BATCH_SIZE_VAL_TEST + 1):\n            x_test[0] = torch.tensor(x_test[0], device=device, dtype=torch.float32)\n            x_test[1] = torch.tensor(x_test[1], device=device, dtype=torch.float32)\n            z_test = model(x_test)\n            z_test = torch.sigmoid(z_test)\n            test_preds[i*test_loader.batch_size:i*test_loader.batch_size + x_test[0].shape[0]] += z_test\n        \n        # preds = test_preds # XX WARNING use this for working with one fold\n        preds += test_preds # use this for working with one 5fold\n        preds_separate.append(test_preds)\n        # TTA predictions\n        # tta_preds = torch.zeros((len(test), 1), dtype=torch.float32, device=device)\n        # for _ in range(TTA):\n            # for i, x_test in tqdm(enumerate(test_loader), desc='test'):\n                # x_test[0] = torch.tensor(x_test[0], device=device, dtype=torch.float32)\n                # x_test[1] = torch.tensor(x_test[1], device=device, dtype=torch.float32)\n                # z_test = model(x_test)\n                # z_test = torch.sigmoid(z_test)\n                # tta_preds[i*test_loader.batch_size:i*test_loader.batch_size + x_test[0].shape[0]] += z_test\n        # preds += tta_preds / TTA\n\n        \npreds /= skf.n_splits\nstop = time.time()\nprint(f'epochs {epochs}: {(stop - start)/60:.3f} mins (in {device})')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!mkdir /kaggle/working/classification_results","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!mv model_1.pth classification_results/model_1.pth\n!mv model_2.pth classification_results/model_2.pth\n!mv model_3.pth classification_results/model_3.pth\n!mv model_4.pth classification_results/model_4.pth\n!mv model_5.pth classification_results/model_5.pth","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# val_acc_all, val_roc_all, val_precision_all, val_recall_all\ndf_val_acc_all = pd.DataFrame(val_acc_all).T\ndf_val_roc_all = pd.DataFrame(val_roc_all).T\ndf_val_precision_all = pd.DataFrame(val_precision_all).T\ndf_val_recall_all = pd.DataFrame(val_recall_all).T\n# convert preds_separate from torch to DF\npreds_separate_py = [np.squeeze(i.detach().cpu().numpy()) for i in preds_separate]\ndf_preds_separate_all = pd.DataFrame(preds_separate_py).T","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# save results to folder\npath_class_results = '/kaggle/working/classification_results/'\nnp.save(f'{path_class_results}epoch_loss_all.npy',epoch_loss_all)\nnp.save(f'{path_class_results}epoch_loss_val_all.npy',epoch_loss_val_all)\nnp.save(f'{path_class_results}oof.npy',oof)\ntrain_df.to_csv(f'{path_class_results}train_df.csv')\ntest_df.to_csv(f'{path_class_results}test_df.csv')\nnp.save(f'{path_class_results}preds.npy',preds.detach().cpu().numpy())\ndf_val_acc_all.to_csv(f'{path_class_results}df_val_acc_all.csv', index=False)\ndf_val_roc_all.to_csv(f'{path_class_results}df_val_roc_all.csv', index=False)\ndf_val_precision_all.to_csv(f'{path_class_results}df_val_precision_all.csv', index=False)\ndf_val_recall_all.to_csv(f'{path_class_results}df_val_recall_all.csv', index=False)\ndf_preds_separate_all.to_csv(f'{path_class_results}df_preds_separate_all.csv', index=False)\nnp.save(f'{path_class_results}WEIGHTED_SAMPLER.npy',WEIGHTED_SAMPLER)\nnp.save(f'{path_class_results}epoch.npy',[epoch])\nnp.save(f'{path_class_results}es_patience.npy',[es_patience])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!zip -qr classification_results.zip /kaggle/working/classification_results/","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"FileLink('classification_results.zip')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# END"},{"metadata":{"trusted":true},"cell_type":"code","source":"path_source = '/kaggle/working/classification_results/'\n# read files\ndf_val_acc_all = pd.read_csv(f'{path_source}df_val_acc_all.csv')\ndf_val_precision_all = pd.read_csv(f'{path_source}df_val_precision_all.csv')\ndf_val_recall_all = pd.read_csv(f'{path_source}df_val_recall_all.csv')\ndf_val_roc_all = pd.read_csv(f'{path_source}df_val_roc_all.csv')\nepoch_loss_all = np.load(f'{path_source}epoch_loss_all.npy', allow_pickle=True)\nepoch_loss_val_all = np.load(f'{path_source}epoch_loss_val_all.npy', allow_pickle=True)\noof = np.load(f'{path_source}oof.npy')\npreds = np.load(f'{path_source}preds.npy')\nWEIGHTED_SAMPLER = np.load(f'{path_source}WEIGHTED_SAMPLER.npy')\nepoch = np.load(f'{path_source}epoch.npy', allow_pickle=True)\n# transform dataframes into list of lists\nval_acc_all = df_val_acc_all.T.values.tolist()\nval_precision_all = df_val_precision_all.T.values.tolist()\nval_recall_all = df_val_recall_all.T.values.tolist()\nval_roc_all = df_val_roc_all.T.values.tolist()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"epoch[0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"predictions = preds.detach().cpu().numpy()\npredictions /=5\nfig, ax = plt.subplots(1,2, figsize=(12,3))\nax[0].plot(oof[:,0])\nax[0].set_title(f'oof val weighted={WEIGHTED_SAMPLER}')\nax[0].plot(train_df['target'].values * np.max(oof[:,0]), c='y')\nax[1].plot(predictions)\nax[1].plot(test_df['target'].values * np.max(predictions), c='y')\nax[1].set_title(f'test set weighted={WEIGHTED_SAMPLER}')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"predictions = preds.detach().cpu().numpy()\npredictions /=5\nfig, ax = plt.subplots(1,2, figsize=(12,3))\nax[0].plot(oof[:,0])\nax[0].set_title(f'oof val weighted={WEIGHTED_SAMPLER}')\nax[0].plot(train_df['target'].values * np.max(oof[:,0]), c='y')\nax[1].plot(predictions)\nax[1].plot(test_df['target'].values * np.max(predictions), c='y')\nax[1].set_title(f'test set weighted={WEIGHTED_SAMPLER}')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"oof_val_true = train_df2['target'].values # we collect the whole train_df from the 5 folds of val\nprint('OOF_val ROC: {:.3f}'.format(roc_auc_score(oof_val_true, oof)))\nprint('OOF_val accuracy: {:.3f}'.format(accuracy_score(oof_val_true, oof.round())))\nprint('OOF_val precision: {:.5f}'.format(precision_score(oof_val_true, oof.round())))\nprint('OOF_val recall: {:.5f}'.format(recall_score(oof_val_true, oof.round())))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"oof_val_true = train_df2['target'].values # we collect the whole train_df from the 5 folds of val\nprint('OOF_val ROC: {:.3f}'.format(roc_auc_score(oof_val_true, oof)))\nprint('OOF_val accuracy: {:.3f}'.format(accuracy_score(oof_val_true, oof.round())))\nprint('OOF_val precision: {:.5f}'.format(precision_score(oof_val_true, oof.round())))\nprint('OOF_val recall: {:.5f}'.format(recall_score(oof_val_true, oof.round())))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_true = test_df['target'].values\nprint('test ROC: {:.3f}'.format(roc_auc_score(test_true, predictions)))\nprint('test accuracy: {:.3f}'.format(accuracy_score(test_true, predictions.round())))\nprint('test precision: {:.5f}'.format(precision_score(test_true, predictions.round())))\nprint('test recall: {:.5f}'.format(recall_score(test_true, predictions.round())))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_true = test_df['target'].values\nprint('test ROC: {:.3f}'.format(roc_auc_score(test_true, predictions)))\nprint('test accuracy: {:.3f}'.format(accuracy_score(test_true, predictions.round())))\nprint('test precision: {:.5f}'.format(precision_score(test_true, predictions.round())))\nprint('test recall: {:.5f}'.format(recall_score(test_true, predictions.round())))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1,4,figsize=(16,4))\nplt.style.use('seaborn-white') # seaborn-white\ntitles_metrics = ['acc', 'roc', 'precision', 'recall']\nfor idx, metrics in enumerate([val_acc_all, val_roc_all, val_precision_all, val_recall_all]):\n    for i in metrics:\n        ax[idx].plot(i)\n        ax[idx].set_title(titles_metrics[idx])\n        if idx<3:\n            ax[idx].set_ylim([.4, 1])\nplt.suptitle(f'val ROC epochs:{epochs} weighted={WEIGHTED_SAMPLER}', fontsize=18)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1,4,figsize=(16,4))\nplt.style.use('seaborn-white') # seaborn-white\ntitles_metrics = ['acc', 'roc', 'precision', 'recall']\nfor idx, metrics in enumerate([val_acc_all, val_roc_all, val_precision_all, val_recall_all]):\n    for i in metrics:\n        ax[idx].plot(i)\n        ax[idx].set_title(titles_metrics[idx])\n        if idx<3:\n            ax[idx].set_ylim([.4, 1])\nplt.suptitle(f'val ROC epochs:{epochs} weighted={WEIGHTED_SAMPLER}', fontsize=18)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1,4,figsize=(16,4))\nplt.style.use('seaborn-white') # seaborn-white\ntitles_metrics = ['acc', 'roc', 'precision', 'recall']\nfor idx, metrics in enumerate([val_acc_all, val_roc_all, val_precision_all, val_recall_all]):\n    for i in metrics:\n        ax[idx].plot(i)\n        ax[idx].set_title(titles_metrics[idx])\n        if idx<3:\n            ax[idx].set_ylim([.4, 1])\nplt.suptitle(f'val ROC epochs:{epochs} weighted={WEIGHTED_SAMPLER}', fontsize=18)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.kdeplot(pd.Series(preds.cpu().numpy().reshape(-1,)));","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.plot(tta_preds.detach().cpu().numpy())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Saving OOF predictions so stacking would be easier\npd.Series(oof.reshape(-1,)).to_csv('oof.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub = pd.read_csv('/kaggle/input/siim-isic-melanoma-classification/sample_submission.csv')\nsub['target'] = preds.cpu().numpy().reshape(-1,)\nsub.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Not used:"},{"metadata":{},"cell_type":"markdown","source":"# Extra"},{"metadata":{"trusted":true},"cell_type":"code","source":"# SEPARATE BLOCKS\n# path_synthesis_selected = '/kaggle/input/cea-synthesis/cea_synthesis_selected_no_dark/'\n# names_synthesis_selected = os.listdir(path_synthesis_selected)\n# len(names_synthesis_selected)\n#============\n# names_synt_selec = np.sort(names_synthesis_selected)\n# names_synt_selec = [i[:-4] for i in names_synt_selec]\n# names_synt_selec_root = [i[:-4] for i in names_synt_selec]\n# df_names_synt_selec = pd.DataFrame((names_synt_selec,names_synt_selec_root)).T\n# df_names_synt_selec.columns = ['synt_name', 'image_name']\n# print(df_names_synt_selec.shape)\n# df_names_synt_selec.head()\n\n# print(train_df.shape, df_names_synt_selec.shape)\n# train_df_synthesized = df_names_synt_selec.merge(train_df, on='image_name')\n# # change image_name to load the correct name\n# train_df_synthesized = train_df_synthesized.rename(columns={'image_name':'image_name_root'})\n# train_df_synthesized = train_df_synthesized.rename(columns={'synt_name':'image_name'})\n# print(train_df_synthesized.shape)\n# train_df_synthesized\n\n# train_df['imfolder'] = '/kaggle/input/jpeg-melanoma-256x256/train/'\n# test_df['imfolder'] = '/kaggle/input/jpeg-melanoma-256x256/train/'\n# train_df_synthesized['imfolder'] = '/kaggle/input/cea-synthesis/cea_synthesis_selected_no_dark/'\n\n# train_df_synt_added_BIG = train_df.append(train_df_synthesized, ignore_index=True, sort=False)\n# train_df = train_df_synt_added_BIG\n# print(train_df.shape)\n# train_df.tail()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# # One-hot encoding of anatom_site_general_challenge feature\n# concat = pd.concat([train_df['anatom_site_general_challenge'], test_df['anatom_site_general_challenge']], ignore_index=True)\n# dummies = pd.get_dummies(concat, dummy_na=True, dtype=np.uint8, prefix='site')\n# train_df = pd.concat([train_df, dummies.iloc[:train_df.shape[0]]], axis=1)\n# test_df = pd.concat([test_df, dummies.iloc[train_df.shape[0]:].reset_index(drop=True)], axis=1)\n\n# # Sex features\n# train_df['sex'] = train_df['sex'].map({'male': 1, 'female': 0})\n# test_df['sex'] = test_df['sex'].map({'male': 1, 'female': 0})\n# train_df['sex'] = train_df['sex'].fillna(-1)\n# test_df['sex'] = test_df['sex'].fillna(-1)\n\n# # Age features\n# train_df['age_approx'] /= train_df['age_approx'].max()\n# test_df['age_approx'] /= test_df['age_approx'].max()\n# train_df['age_approx'] = train_df['age_approx'].fillna(0)\n# test_df['age_approx'] = test_df['age_approx'].fillna(0)\n\n# train_df['patient_id'] = train_df['patient_id'].fillna(0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# ## get_synthesize_images_for_only_images_in_fold():\n# # These are the correct samples in the val subset\n# INDEX_FOLD = 3\n# print(f'train_idx: {train_idx_all[INDEX_FOLD][:5]}, val_idx: {val_idx_all[INDEX_FOLD][:5]}')\n# df_val_in_fold = train_df.iloc[val_idx_all[INDEX_FOLD]].reset_index(drop=True)\n# print(f'df_val_in_fold: {df_val_in_fold.shape}')\n# # Here you should add only files selected by train_idx that also have synt\n# df_train_in_fold = train_df.iloc[train_idx_all[INDEX_FOLD]].reset_index(drop=True)\n# print(f'df_train_in_fold: {df_train_in_fold.shape}')\n# print(f'train_df_synthesized: {train_df_synthesized.shape}')\n# #From the images in train this fold get the ones in synt\n# names_in_train = df_train_in_fold['image_name'].values\n# names_unique_synt_all = np.unique(train_df_synthesized['image_name_root'].values)\n# names_in_train_that_have_synt = list(set(names_in_train).intersection(names_unique_synt_all))\n# print(f'names_in_train: {len(names_in_train)}')\n# print(f'names_unique_synt_all: {len(names_unique_synt_all)}')\n# print(f'names_in_train_that_have_synt: {len(names_in_train_that_have_synt)}')\n# print('=============')\n# # select those images from all the synthesized ones\n# print(f'train_df_synthesized: {train_df_synthesized.shape}')\n# train_synt_to_add_fold = train_df_synthesized[train_df_synthesized['image_name_root'].isin(names_in_train_that_have_synt)]\n# print(f'train_synt_to_add_fold: {train_synt_to_add_fold.shape}')\n# print('=============')\n# # append the synthesized images to the current train fold\n# df_train_in_fold_synt_added = df_train_in_fold.append(train_synt_to_add_fold)\n# print(f'df_train_in_fold_synt_added: {df_train_in_fold_synt_added.shape}')\n# df_train_in_fold_synt_added.tail(2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# # OPTION 1\n# train_df, test_df = make_train_test_df_for_cea_aug2(train_df_orig, df_cea_done, \n#                                                    test_set_size = 2000, test_set_1s = 450,\n#                                                   add_0s_into_train_df = 1524-len(df_cea_done),#1500-len(df_cea_done)\n#                                                   add_1s_into_train_df = 0) # 0\n# np.shape(train_df), np.shape(test_df)\n# print(f\"train_df = {train_df.shape}, train_df total 1's = {np.sum(train_df['target'].values)}, train_df total 1=0's = {np.sum(train_df['target'].values==0)}\")\n# print(f\"test_df = {test_df.shape}, test_df total 1's = {np.sum(test_df['target'].values)}, test_df total 0's = {np.sum(test_df['target'].values==0)}\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# class MelanomaDataset(Dataset):\n#     def __init__(self, df: pd.DataFrame, imfolder: str, train: bool = True, transforms = None, meta_features = None):\n#         \"\"\"\n#         Class initialization\n#         Args:\n#             df (pd.DataFrame): DataFrame with data description\n#             imfolder (str): folder with images\n#             train (bool): flag of whether a training dataset is being initialized or testing one\n#             transforms: image transformation method to be applied\n#             meta_features (list): list of features with meta information, such as sex and age\n            \n#         \"\"\"\n#         self.df = df\n#         self.imfolder = imfolder\n#         self.transforms = transforms\n#         self.train = train\n#         self.meta_features = meta_features\n        \n#     def __getitem__(self, index):\n#         # print(index)\n#         im_path = os.path.join(self.imfolder, self.df.iloc[index]['image_name'] + '.jpg')\n#         # print(im_path)\n#         x = cv2.imread(im_path)\n#         meta = np.array(self.df.iloc[index][self.meta_features].values, dtype=np.float32)\n\n#         if self.transforms:\n#             x = self.transforms(x)\n            \n#         if self.train:\n#             y = self.df.iloc[index]['target']\n#             return (x, meta), y\n#         else:\n#             return (x, meta)\n    \n#     def __len__(self):\n#         return len(self.df)\n    \n    \n# class Net(nn.Module):\n#     def __init__(self, arch, n_meta_features: int):\n#         super(Net, self).__init__()\n#         self.arch = arch\n#         if 'ResNet' in str(arch.__class__):\n#             self.arch.fc = nn.Linear(in_features=512, out_features=500, bias=True)\n#         if 'EfficientNet' in str(arch.__class__):\n#             self.arch._fc = nn.Linear(in_features=1280, out_features=500, bias=True)\n#         self.meta = nn.Sequential(nn.Linear(n_meta_features, 500),\n#                                   nn.BatchNorm1d(500),\n#                                   nn.ReLU(),\n#                                   nn.Dropout(p=0.2),\n#                                   nn.Linear(500, 250),  # FC layer output will have 250 features\n#                                   nn.BatchNorm1d(250),\n#                                   nn.ReLU(),\n#                                   nn.Dropout(p=0.2))\n#         self.ouput = nn.Linear(500 + 250, 1)\n        \n#     def forward(self, inputs):\n#         \"\"\"\n#         No sigmoid in forward because we are going to use BCEWithLogitsLoss\n#         Which applies sigmoid for us when calculating a loss\n#         \"\"\"\n#         x, meta = inputs\n#         cnn_features = self.arch(x)\n#         meta_features = self.meta(meta)\n#         features = torch.cat((cnn_features, meta_features), dim=1)\n#         output = self.ouput(features)\n#         return output","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# # One-hot encoding of anatom_site_general_challenge feature\n# concat = pd.concat([train_df['anatom_site_general_challenge'], test_df['anatom_site_general_challenge']], ignore_index=True)\n# dummies = pd.get_dummies(concat, dummy_na=True, dtype=np.uint8, prefix='site')\n# train_df = pd.concat([train_df, dummies.iloc[:train_df.shape[0]]], axis=1)\n# test_df = pd.concat([test_df, dummies.iloc[train_df.shape[0]:].reset_index(drop=True)], axis=1)\n\n# # Sex features\n# train_df['sex'] = train_df['sex'].map({'male': 1, 'female': 0})\n# test_df['sex'] = test_df['sex'].map({'male': 1, 'female': 0})\n# train_df['sex'] = train_df['sex'].fillna(-1)\n# test_df['sex'] = test_df['sex'].fillna(-1)\n\n# # Age features\n# train_df['age_approx'] /= train_df['age_approx'].max()\n# test_df['age_approx'] /= test_df['age_approx'].max()\n# train_df['age_approx'] = train_df['age_approx'].fillna(0)\n# test_df['age_approx'] = test_df['age_approx'].fillna(0)\n\n# train_df['patient_id'] = train_df['patient_id'].fillna(0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# ORIGINAL DATAFRAME\n# for fold, (train_idx, val_idx) in enumerate(skf.split(X=np.zeros(len(train_df)), y=train_df['target'], groups=train_df['patient_id'].tolist()), 1):\n#     print(f'fold={fold}, train_idx={np.shape(train_idx)}, train_idx[0]={train_idx[0]}, val_idx={np.shape(val_idx)}, val_idx[0]={val_idx[0]}')\n# train_idx","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# # SMALLER test dataloader\n# n_ints =50\n# rand_ints = np.random.randint(0,len(test_df),n_ints)\n# test = MelanomaDataset(df=test_df.iloc[rand_ints].reset_index(drop=True),\n#                        imfolder='/kaggle/input/jpeg-melanoma-256x256/test/', \n#                        train=False,\n#                        transforms=train_transform,  # For TTA\n#                        meta_features=meta_features)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def make_train_test_df_for_cea_aug2(train_df_orig, df_cea_done, test_set_size = 2000, test_set_1s = 100, add_0s_into_train_df=0, add_1s_into_train_df=0):\n    '''make a train_df that contains all the files that cea==completed & target==1\n    and a test_df that has all the files that cea!=completed & target==1\n    Optionally you can include fewer files'''\n    # make a train_df that contains all the files that cea==completed & target==1\n    image_name_cea_done = df_cea_done['image_name'].values\n    train_df = train_df_orig[train_df_orig['image_name'].isin(image_name_cea_done)]\n    # make sure test_df has all the files that cea!=completed & target==1\n    test_df_large = train_df_orig[~train_df_orig['image_name'].isin(image_name_cea_done)]\n    # divide in target==1 & target == 0\n    test_df_only1s = test_df_large[test_df_large['target']==1]\n    test_df_only0s = test_df_large[test_df_large['target']==0]\n    test_df_only0s = test_df_only0s.reset_index(drop=True)\n    test_df_only1s = test_df_only1s.reset_index(drop=True)\n    # if add_0s_into_train_df more samples are wanted in train_df (of zeros)\n    if add_0s_into_train_df >0:\n        random.seed(0)\n        rand_ints = random.sample(range(len(test_df_only0s)-1), add_0s_into_train_df)\n        extra_0s_for_train_df = test_df_only0s.iloc[rand_ints]\n        train_df = train_df.append(extra_0s_for_train_df)\n        test_df_only0s =test_df_only0s.drop(rand_ints, axis=0)\n    # if add_1s_into_train_df more samples are wanted in train_df (of ones)\n    if add_1s_into_train_df >0:\n        random.seed(0)\n        rand_ints = random.sample(range(len(test_df_only1s)-1), add_1s_into_train_df)\n        extra_1s_for_train_df = test_df_only1s.iloc[rand_ints]\n        train_df = train_df.append(extra_1s_for_train_df)\n        test_df_only1s = test_df_only1s.drop(rand_ints, axis=0)\n    # if only a subset of the target == 1 is wanted\n    if test_set_1s > 0:\n        random.seed(0)\n        rand_ints = random.sample(range(len(test_df_only1s)), test_set_1s)\n        test_df_only1s = test_df_only1s.iloc[rand_ints]\n        \n    # if add_1s_into_train_df > 0 or add_0s_into_train_df > 0\n    if add_1s_into_train_df > 0 or add_0s_into_train_df > 0:\n        test_df_large = pd.DataFrame()\n        test_df_large = test_df_large.append(test_df_only1s)\n        test_df_large = test_df_large.append(test_df_only0s)\n        random.seed(0)\n        rand_ints = random.sample(range(len(test_df_large)), len(test_df_large))\n        test_df_large.index = rand_ints\n        test_df_large = test_df_large.reindex()\n        test_df = test_df_large\n    else:\n        # get a large subset of test_df_large that contain all target 1 from test_df_large\n        test_df = test_df_large\n        if test_set_size > 0:\n            number1s_already_in_test = len(test_df_only1s)\n            random.seed(0)\n            rand_ints = random.sample(range(len(test_df_only0s)), test_set_size - number1s_already_in_test)\n            test_df_only0s_subset = test_df_only0s.iloc[rand_ints]\n            test_df = test_df_only1s.append(test_df_only0s_subset)\n    return train_df.reset_index(drop=True), test_df.reset_index(drop=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"predictions = preds.detach().cpu().numpy()\npredictions /=5\nfig, ax = plt.subplots(1,2, figsize=(12,3))\nax[0].plot(oof[:,0])\nax[0].set_title(f'oof val weighted={WEIGHTED_SAMPLER}, ep={epochs}')\nax[1].plot(predictions)\nax[1].set_title(f'test set weighted={WEIGHTED_SAMPLER}, ep={epochs}')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"predictions = preds.detach().cpu().numpy()\npredictions /=5\nfig, ax = plt.subplots(1,2, figsize=(12,3))\nax[0].plot(oof[:,0])\nax[0].set_title(f'oof val weighted={WEIGHTED_SAMPLER}')\nax[1].plot(predictions)\nax[1].set_title(f'test set weighted={WEIGHTED_SAMPLER}')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def make_train_test_df_for_cea_aug(train_df_orig, df_cea_done, samples_subset = 2000, samples_subset_tgt1 = 100):\n    '''make a train_df that contains all the files that cea==completed & target==1\n    and a test_df that has all the files that cea!=completed & target==1\n    Optionally you can include fewer files'''\n    # make a train_df that contains all the files that cea==completed & target==1\n    image_name_cea_done = df_cea_done['image_name'].values\n    train_df = train_df_orig[train_df_orig['image_name'].isin(image_name_cea_done)]\n    # make sure test_df has all the files that cea!=completed & target==1\n    test_df_large = train_df_orig[~train_df_orig['image_name'].isin(image_name_cea_done)]\n    # divide in target==1 & target == 0\n    test_df_only1s = test_df_large[test_df_large['target']==1]\n    test_df_only0s = test_df_large[test_df_large['target']==0]\n    test_df_only0s = test_df_only0s.reset_index(drop=True)\n    test_df_only1s = test_df_only1s.reset_index(drop=True)\n    # if only a subset of the target == 1 is wanted\n    if samples_subset_tgt1 > 0:\n        random.seed(0)\n        rand_ints = random.sample(range(len(test_df_only1s)), samples_subset_tgt1)\n        test_df_only1s = test_df_only1s.iloc[rand_ints]\n    # get a large subset of test_df_large that contain all target 1 from test_df_large\n    if samples_subset > 0:\n        number1s_already_in_test = len(test_df_only1s)\n        random.seed(0)\n        rand_ints = random.sample(range(len(test_df_only0s)), samples_subset - number1s_already_in_test)\n        test_df_only0s_subset = test_df_only0s.iloc[rand_ints]\n    test_df = test_df_only1s.append(test_df_only0s_subset)\n    return train_df.reset_index(drop=True), test_df.reset_index(drop=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# # THIS IS NOT IS DONE IN make_train_test_df_for_cea_aug()\n# # make a train_df that contains all the files that cea==completed & target==1\n# image_name_cea_done = df_cea_done['image_name'].values\n# train_df = train_df_orig[train_df_orig['image_name'].isin(image_name_cea_done)]\n# # make sure test_df has all the files that cea!=completed & target==1\n# test_df_large = train_df_orig[~train_df_orig['image_name'].isin(image_name_cea_done)]\n# print(f'train_df_orig = {train_df_orig.shape}, train_df 1s = {np.sum(train_df_orig[\"target\"].values)}')\n# print(f'train_df = {train_df.shape}, train_df 1s = {np.sum(train_df[\"target\"].values)}')\n# print(f'test_df_large = {test_df_large.shape}, test_df_large 1s = {np.sum(test_df_large[\"target\"].values)}')\n# # divide in target==1 & target == 0\n# test_df_only1s = test_df_large[test_df_large['target']==1]\n# test_df_only0s = test_df_large[test_df_large['target']==0]\n# test_df_only0s = test_df_only0s.reset_index(drop=True)\n# test_df_only1s = test_df_only1s.reset_index(drop=True)\n# print(np.shape(test_df_only1s), np.shape(test_df_only0s))\n# # if only a subset of the target == 1 is wanted\n# n_ints = 100\n# if n_ints > 0:\n#     random.seed(0)\n#     rand_ints = random.sample(range(len(test_df_only1s)), n_ints)\n#     test_df_only1s = test_df_only1s.iloc[rand_ints]\n# print(np.shape(test_df_only1s))\n# # get a large subset of test_df_large that contain all target 1 from test_df_large\n# number1s_already_in_test = len(test_df_only1s)\n# n_ints = 2000\n# random.seed(0)\n# rand_ints = random.sample(range(len(test_df_only0s)), n_ints - number1s_already_in_test)\n# test_df_only0s_subset = test_df_only0s.iloc[rand_ints]\n# print(np.shape(test_df_only0s_subset))\n# df_test = test_df_only1s.append(test_df_only0s_subset)\n# print(df_test.shape)\n# df_test.tail(5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# # OC check a df with a subset\n# train_df2 = train_df[train_df['image_name'].isin(df_cea['image_name'].values)]\n# print(train_df2.shape)\n# train_df2 = train_df2[train_df2['age_approx'].notna()]\n# print(train_df2.shape)\n# train_df2.tail()\n# # OC add fake target 1s just to see if the code works\n# n_ints =20\n# rand_ints = np.random.randint(0,len(train_df2),n_ints)\n# for i in rand_ints:\n#     train_df2['target'].iloc[i] = 1\n# print(np.sum(train_df2['target'].values==1)/len(train_df2), np.sum(train_df2['target'].values==1))\n# plt.figure(figsize=(3,2))\n# plt.hist(train_df2['target'].values);","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# # OC just checking shapes\n# extra_image_path = os.path.join('/kaggle/input/melanoma-external-malignant-256/train/train', \n#                                 train_df2.iloc[10]['image_name'] + '.jpg')\n# new_image_path = os.path.join('/kaggle/input/jpeg-melanoma-256x256/train', \n#                                 train_df2.iloc[10]['image_name'] + '.jpg')\n# x1 = cv2.imread(extra_image_path)\n# x2 = cv2.imread(new_image_path)\n# fig, ax = plt.subplots(1,2)\n# ax[0].imshow(x)\n# ax[1].imshow(x)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}