{"cells":[{"metadata":{},"cell_type":"markdown","source":"# 1. OVERVIEW\n\nThe goal of this notebook is to demonstrate a technique that can improve the classification performance by learning from both training and test data. This is done by pre-training a model on the complete data set. This approach can help to reduce the impact of sampling bias by exposing the model to the test data and benefit from a larger sample size while learning. We will demonstarte how this simple technique can get an AUC improvement on CV and private LB.\n\nSo, how to make use of the test sample? The labels are only observed on the training data. Luckily, this competition also provides a bunch of meta-data per each training and test image. What we can do is the following:\n1. Pre-train a model on the complete train+test data using one of the meta-features as a surrogate label.\n2. Initialize from the pre-trained weights when training a final melanoma classification model.\n\nThe intuition behind this approach is that by learning to classify images according to one of meta variables such as `sex` or `age_approx`, the model can learn some of the visual features that might be useful for the malignant lesion classification. For instance, size of lesions and color of the skin can be helpful in determining both patient age and lesion type. Exposing the model to the test data also allows it to take a sneak peek at test images, which may help to learn patterns that are more prevalent in the test distribution.\n\nIn this notebook, we will train a model to classify `anatom_site_general_challenge` on both training and test data and store the pre-trained weights of the backbone. Next, we will build a melanoma classification model that initializes from the pre-trained weights and check its performance on CV and on the leaderboard.\n\nP.S. The notebook hevaily relies on the [great pipeline](https://www.kaggle.com/cdeotte/triple-stratified-kfold-with-tfrecords) developed by [Chris Deotte](https://www.kaggle.com/cdeotte) and reuses much of his original code. I know that many teams have been using this pipeline to train their models, so relying on it here should be familiar to you. Please allow me to thank Chris and give him credit for his hard work! Kindly refer to his notebook for general questions on the modeling pipeline where he provided extensive comments and documentation.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# 2. INITIALIZATION","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"!pip install -q efficientnet >> /dev/null","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd, numpy as np\nfrom kaggle_datasets import KaggleDatasets\nimport tensorflow as tf, re, math\nimport tensorflow.keras.backend as K\nimport efficientnet.tfkeras as efn\nfrom sklearn.model_selection import KFold\nfrom sklearn.metrics import roc_auc_score\nimport matplotlib.pyplot as plt\nfrom scipy.stats import rankdata\nimport PIL, cv2","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In addition to other training parameters, we introduce `USE_PRETRAIN_WEIGHTS` variable to reflect whether we want to train a pre-trained model on full data before training a final melanoma classification model. \n\nFor demonstartion purposes, we use EfficientNet `B0`, `128x128` image size with no TTA and no external data from previous competitions. You can easily incoroprate the external data by following the Chris' notebook and experiment with larger architectures and images sizes.","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"# DEVICE\nDEVICE = \"TPU\"\n\n# USE DIFFERENT SEED FOR DIFFERENT STRATIFIED KFOLD\nSEED = 42\n\n# NUMBER OF FOLDS. USE 3, 5, OR 15 \nFOLDS = 5\n\n# WHICH IMAGE SIZES TO LOAD EACH FOLD\nIMG_SIZES = [128]*FOLDS\n\n# BATCH SIZE AND EPOCHS\nBATCH_SIZES = [32]*FOLDS\nEPOCHS      = [10]*FOLDS\n\n# WHICH EFFICIENTNET TO USE\nEFF_NETS = [0]*FOLDS\n\n# WEIGHTS FOR FOLD MODELS WHEN PREDICTING TEST\nWGTS = [1/FOLDS]*FOLDS\n\n# PRETRAINED WEIGHTS\nUSE_PRETRAIN_WEIGHTS = True","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# CONNECT TO DEVICE\nif DEVICE == \"TPU\":\n    print(\"connecting to TPU...\")\n    try:\n        tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n        print('Running on TPU ', tpu.master())\n    except ValueError:\n        print(\"Could not connect to TPU\")\n        tpu = None\n\n    if tpu:\n        try:\n            print(\"initializing  TPU ...\")\n            tf.config.experimental_connect_to_cluster(tpu)\n            tf.tpu.experimental.initialize_tpu_system(tpu)\n            strategy = tf.distribute.experimental.TPUStrategy(tpu)\n            print(\"TPU initialized\")\n        except _:\n            print(\"failed to initialize TPU\")\n    else:\n        DEVICE = \"GPU\"\n\nif DEVICE != \"TPU\":\n    print(\"Using default strategy for CPU and single GPU\")\n    strategy = tf.distribute.get_strategy()\n\nif DEVICE == \"GPU\":\n    print(\"Num GPUs Available: \", len(tf.config.experimental.list_physical_devices('GPU')))\n\nAUTO     = tf.data.experimental.AUTOTUNE\nREPLICAS = strategy.num_replicas_in_sync\nprint(f'REPLICAS: {REPLICAS}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 3. IMAGE PROCESSING\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# IMAGE PATHS\nGCS_PATH = [None]*FOLDS\n\nfor i,k in enumerate(IMG_SIZES):\n    GCS_PATH[i]  = KaggleDatasets().get_gcs_path('melanoma-%ix%i'%(k,k))\n    \nfiles_train = np.sort(np.array(tf.io.gfile.glob(GCS_PATH[0] + '/train*.tfrec')))\nfiles_test  = np.sort(np.array(tf.io.gfile.glob(GCS_PATH[0] + '/test*.tfrec')))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The `read_labeled_tfrecord()` function is modified to provide two outputs: \n\n1. Image tensor.\n\n2. Either `anatom_site_general_challenge` or `target` as a label. The former is one-hot-encoded since it is a categorical feature with six possible values. The selection of the label is controlled by the `pretraining` argument that is read from the `get_dataset()` function provided below. Setting `pretraining = True` implies reading `anatom_site_general_challenge` as a surrogate label.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def read_labeled_tfrecord(example, pretraining = False):\n    if pretraining:\n        tfrec_format = {\n            'image'                        : tf.io.FixedLenFeature([], tf.string),\n            'image_name'                   : tf.io.FixedLenFeature([], tf.string),\n            'anatom_site_general_challenge': tf.io.FixedLenFeature([], tf.int64),\n        }      \n    else:\n        tfrec_format = {\n            'image'                        : tf.io.FixedLenFeature([], tf.string),\n            'image_name'                   : tf.io.FixedLenFeature([], tf.string),\n            'target'                       : tf.io.FixedLenFeature([], tf.int64)\n        }   \n    example = tf.io.parse_single_example(example, tfrec_format)\n    return example['image'], tf.one_hot(example['anatom_site_general_challenge'], 6) if pretraining else example['target']\n\n\ndef read_unlabeled_tfrecord(example, return_image_name=True):\n    tfrec_format = {\n        'image'                        : tf.io.FixedLenFeature([], tf.string),\n        'image_name'                   : tf.io.FixedLenFeature([], tf.string),\n    }\n    example = tf.io.parse_single_example(example, tfrec_format)\n    return example['image'], example['image_name'] if return_image_name else 0\n\n \ndef prepare_image(img, dim = 256):    \n    img = tf.image.decode_jpeg(img, channels = 3)\n    img = tf.cast(img, tf.float32) / 255.0\n    img = img * circle_mask\n    img = tf.reshape(img, [dim,dim, 3])\n            \n    return img\n\ndef count_data_items(filenames):\n    n = [int(re.compile(r\"-([0-9]*)\\.\").search(filename).group(1)) \n         for filename in filenames]\n    return np.sum(n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_dataset(files, \n                shuffle            = False, \n                repeat             = False, \n                labeled            = True, \n                pretraining        = False,\n                return_image_names = True, \n                batch_size         = 16, \n                dim                = 256):\n    \n    ds = tf.data.TFRecordDataset(files, num_parallel_reads = AUTO)\n    ds = ds.cache()\n    \n    if repeat:\n        ds = ds.repeat()\n    \n    if shuffle: \n        ds = ds.shuffle(1024*2) #if too large causes OOM in GPU CPU\n        opt = tf.data.Options()\n        opt.experimental_deterministic = False\n        ds = ds.with_options(opt)\n        \n    if labeled: \n        ds = ds = ds.map(lambda example: read_labeled_tfrecord(example, pretraining), \n                         num_parallel_calls=AUTO)\n    else:\n        ds = ds.map(lambda example: read_unlabeled_tfrecord(example, return_image_names), \n                    num_parallel_calls = AUTO)\n    \n    ds = ds.map(lambda img, imgname_or_label: (\n                prepare_image(img, dim = dim), \n                imgname_or_label), \n                num_parallel_calls = AUTO)\n    \n    ds = ds.batch(batch_size * REPLICAS)\n    ds = ds.prefetch(AUTO)\n    return ds","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We also use a circular crop (a.k.a. [microscope augmentation](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/159476)) to improve image consistency. The snippet below creates a circular mask, which is applied in the `prepare_image()` function.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# CIRCLE CROP PREPARATIONS\ncircle_img  = np.zeros((IMG_SIZES[0], IMG_SIZES[0]), np.uint8)\ncircle_img  = cv2.circle(circle_img, (int(IMG_SIZES[0]/2), int(IMG_SIZES[0]/2)), int(IMG_SIZES[0]/2), 1, thickness = -1)\ncircle_img  = np.repeat(circle_img[:, :, np.newaxis], 3, axis = 2)\ncircle_mask = tf.cast(circle_img, tf.float32)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# LOAD DATA AND APPLY AUGMENTATIONS\ndef show_dataset(thumb_size, cols, rows, ds):\n    mosaic = PIL.Image.new(mode='RGB', size=(thumb_size*cols + (cols-1), \n                                             thumb_size*rows + (rows-1)))\n    for idx, data in enumerate(iter(ds)):\n        img, target_or_imgid = data\n        ix  = idx % cols\n        iy  = idx // cols\n        img = np.clip(img.numpy() * 255, 0, 255).astype(np.uint8)\n        img = PIL.Image.fromarray(img)\n        img = img.resize((thumb_size, thumb_size), resample = PIL.Image.BILINEAR)\n        mosaic.paste(img, (ix*thumb_size + ix, \n                           iy*thumb_size + iy))\n        nn = target_or_imgid.numpy().decode(\"utf-8\")\n\n    display(mosaic)\n    return nn\n\nfiles_train = tf.io.gfile.glob(GCS_PATH[0] + '/train*.tfrec')\nds = tf.data.TFRecordDataset(files_train, num_parallel_reads = AUTO).shuffle(1024)\nds = ds.take(10).cache()\nds = ds.map(read_unlabeled_tfrecord, num_parallel_calls = AUTO)\nds = ds.map(lambda img, target: (prepare_image(img, dim = IMG_SIZES[0]),\n                                 target), num_parallel_calls = AUTO)\nds = ds.take(12*5)\nds = ds.prefetch(AUTO)\n\n# DISPLAY IMAGES\nname = show_dataset(128, 5, 2, ds)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 4. PRE-TRAINED MODEL\n\nThe `build_model()` function incorporates three important features that depend on the training regime:\n    \n1. When building a model for pre-training, we use `CategoricalCrossentropy` as a loss because `anatom_site_general_challenge` is a categorical variable. When building a model that classifies lesions as benign/malgnant, we use `BinaryCrossentropy` as a loss.\n\n2. When training a final binary classification model, we load the saved pre-trained weights by using `base.load_weights('base_weights.h5')` if `use_pretrain_weights == True`.\n\n3. We use a dense layer with six output nodes and softmax activation when doing pre-training and a dense layer with a single output node and sigmoid activation when training a final model.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"EFNS = [efn.EfficientNetB0, efn.EfficientNetB1, efn.EfficientNetB2, efn.EfficientNetB3, \n        efn.EfficientNetB4, efn.EfficientNetB5, efn.EfficientNetB6, efn.EfficientNetB7]\n\ndef build_model(dim = 256, ef = 0, pretraining = False, use_pretrain_weights = False):\n    \n    # base\n    inp  = tf.keras.layers.Input(shape = (dim,dim,3))\n    base = EFNS[ef](input_shape = (dim,dim,3), weights = 'imagenet', include_top = False)\n    \n    # base weights\n    if use_pretrain_weights:\n        base.load_weights('base_weights.h5')\n    \n    x = base(inp)\n    x = tf.keras.layers.GlobalAveragePooling2D()(x)\n    \n    if pretraining:\n        x     = tf.keras.layers.Dense(6, activation = 'softmax')(x)\n        model = tf.keras.Model(inputs = inp, outputs = x)\n        opt   = tf.keras.optimizers.Adam(learning_rate = 0.001)\n        loss  = tf.keras.losses.CategoricalCrossentropy()    \n        model.compile(optimizer = opt, loss = loss)\n    else:\n        x     = tf.keras.layers.Dense(1, activation = 'sigmoid')(x)\n        model = tf.keras.Model(inputs = inp, outputs = x)\n        opt   = tf.keras.optimizers.Adam(learning_rate = 0.001)\n        loss  = tf.keras.losses.BinaryCrossentropy(label_smoothing = 0.01)  \n        model.compile(optimizer = opt, loss = loss, metrics = ['AUC'])\n    \n    return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_lr_callback(batch_size=8):\n    \n    lr_start   = 0.000005\n    lr_max     = 0.00000125 * REPLICAS * batch_size\n    lr_min     = 0.000001\n    lr_ramp_ep = 5\n    lr_sus_ep  = 0\n    lr_decay   = 0.8\n   \n    def lrfn(epoch):\n        if epoch < lr_ramp_ep:\n            lr = (lr_max - lr_start) / lr_ramp_ep * epoch + lr_start\n            \n        elif epoch < lr_ramp_ep + lr_sus_ep:\n            lr = lr_max\n            \n        else:\n            lr = (lr_max - lr_min) * lr_decay**(epoch - lr_ramp_ep - lr_sus_ep) + lr_min\n            \n        return lr\n\n    lr_callback = tf.keras.callbacks.LearningRateScheduler(lrfn, verbose=False)\n    return lr_callback","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The pre-trained model is trained on both training and test data. Here, we use the original training data merged with the complete test set as a training sample. We fix the number of training epochs to `EPOCHS` and do not perform early stopping. You can also experiment with setting up a small validation sample from both training and test data to perform early stopping.\n\nHere, we don't produce any predictions by the pre-trained model since we will only utilize it to extract the weights. ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"### TRAIN MODEL\nif USE_PRETRAIN_WEIGHTS:\n\n    # USE VERBOSE=0 for silent, VERBOSE=1 for interactive, VERBOSE=2 for commit\n    VERBOSE = 2\n\n    # DISPLAY INFO\n    if DEVICE == 'TPU':\n        if tpu: tf.tpu.experimental.initialize_tpu_system(tpu)\n    print('#### Image Size %i with EN B%i and batch_size %i'%\n          (IMG_SIZES[0],EFF_NETS[0],BATCH_SIZES[0]*REPLICAS))\n\n    # CREATE TRAIN AND VALIDATION SUBSETS\n    files_train = tf.io.gfile.glob(GCS_PATH[0] + '/train*.tfrec')\n    print('#### Using 2020 train data')\n    files_train += tf.io.gfile.glob(GCS_PATH[0] + '/test*.tfrec')\n    print('#### Using 2020 test data')\n    np.random.shuffle(files_train)\n\n    # BUILD MODEL\n    K.clear_session()\n    tf.random.set_seed(SEED)\n    with strategy.scope():\n        model = build_model(dim         = IMG_SIZES[0],\n                            ef          = EFF_NETS[0], \n                            pretraining = True)\n\n    # SAVE BEST MODEL EACH FOLD\n    sv = tf.keras.callbacks.ModelCheckpoint(\n        'weights.h5', monitor='loss', verbose=0, save_best_only=True,\n        save_weights_only=True, mode='min', save_freq='epoch')\n\n    # TRAIN\n    print('Training...')\n    history = model.fit(\n        get_dataset(files_train, \n                    dim         = IMG_SIZES[0], \n                    batch_size  = BATCH_SIZES[0],\n                    shuffle     = True, \n                    repeat      = True, \n                    pretraining = True), \n        epochs          = EPOCHS[0], \n        callbacks       = [sv, get_lr_callback(BATCH_SIZES[0])], \n        steps_per_epoch = count_data_items(files_train)/BATCH_SIZES[0]//REPLICAS,\n        verbose = VERBOSE)\n    \nelse:\n    \n    print('#### NOT using a pre-trained model')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The pre-training is complete! Now, we need to resave weights of our pre-trained model to make it easier to load them in the future! We are not really interested in the classification head, so we only export the weights of the convolutional part of the network. We can index these layers using `model.layers[1]`.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# LOAD WEIGHTS AND CHECK MODEL\nif USE_PRETRAIN_WEIGHTS:\n    model.load_weights('weights.h5')\n    model.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# EXPORT BASE WEIGHTS\nif USE_PRETRAIN_WEIGHTS:\n    model.layers[1].save_weights('base_weights.h5')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 5. FINAL MODEL\n\nNow we can train a final classification model using a regular cross-validation framework on the training data! \n\nWe need to take care of a couple of changes:\n1. Make sure that we don't use test data in the training folds anymore\n2. Run the model on all fold combinations.\n3. Set `use_pretrain_weights = True` and `pretraining = False` in the `build_model()` function to initialize from the pre-trained weights in the beginning of each fold.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# USE VERBOSE=0 for silent, VERBOSE=1 for interactive, VERBOSE=2 for commit\nVERBOSE = 0\n\nskf = KFold(n_splits = FOLDS, shuffle = True, random_state = SEED)\noof_pred = []; oof_tar = []; oof_val = []; oof_names = []; oof_folds = []\npreds = np.zeros((count_data_items(files_test),1))\n\nfor fold,(idxT,idxV) in enumerate(skf.split(np.arange(15))):\n    \n    # DISPLAY FOLD INFO\n    if DEVICE == 'TPU':\n        if tpu: tf.tpu.experimental.initialize_tpu_system(tpu)\n    print('#'*25); print('#### FOLD',fold+1)\n    print('#### Image Size %i with EfficientNet B%i and batch_size %i'%\n          (IMG_SIZES[fold],EFF_NETS[fold],BATCH_SIZES[fold]*REPLICAS))\n    \n    # CREATE TRAIN AND VALIDATION SUBSETS\n    files_train = tf.io.gfile.glob([GCS_PATH[fold] + '/train%.2i*.tfrec'%x for x in idxT])      \n    print('#### Using 2020 train data')\n    np.random.shuffle(files_train); print('#'*25)\n    \n    files_valid = tf.io.gfile.glob([GCS_PATH[fold] + '/train%.2i*.tfrec'%x for x in idxV])\n    files_test = np.sort(np.array(tf.io.gfile.glob(GCS_PATH[fold] + '/test*.tfrec')))\n    \n    # BUILD MODEL\n    K.clear_session()\n    tf.random.set_seed(SEED)\n    with strategy.scope():\n        model = build_model(dim                  = IMG_SIZES[fold],\n                            ef                   = EFF_NETS[fold],\n                            use_pretrain_weights = USE_PRETRAIN_WEIGHTS, \n                            pretraining          = False)\n        \n    # SAVE BEST MODEL EACH FOLD\n    sv = tf.keras.callbacks.ModelCheckpoint(\n        'fold-%i.h5'%fold, monitor='val_auc', verbose=0, save_best_only=True,\n        save_weights_only=True, mode='max', save_freq='epoch')\n   \n    # TRAIN\n    print('Training...')\n    history = model.fit(\n        get_dataset(files_train, \n                    shuffle    = True, \n                    repeat     = True, \n                    dim        = IMG_SIZES[fold], \n                    batch_size = BATCH_SIZES[fold]), \n        epochs = EPOCHS[fold], \n        callbacks = [sv,get_lr_callback(BATCH_SIZES[fold])], \n        steps_per_epoch = count_data_items(files_train)/BATCH_SIZES[fold]//REPLICAS,\n        validation_data = get_dataset(files_valid,\n                                      shuffle = False,\n                                      repeat  = False, \n                                      dim     = IMG_SIZES[fold]),\n        verbose = VERBOSE\n    )\n    print('Loading best model...')\n    model.load_weights('fold-%i.h5'%fold)\n    \n    # PREDICT OOF\n    print('Predicting OOF...')\n    ds_valid = get_dataset(files_valid,labeled=False,return_image_names=False,shuffle=False,dim=IMG_SIZES[fold],batch_size=BATCH_SIZES[fold]*4)\n    ct_valid = count_data_items(files_valid); STEPS = ct_valid/BATCH_SIZES[fold]/4/REPLICAS\n    pred     = model.predict(ds_valid,steps=STEPS,verbose=VERBOSE)[:ct_valid,] \n    oof_pred.append(pred)      \n\n    # GET OOF TARGETS AND NAMES\n    ds_valid = get_dataset(files_valid,dim=IMG_SIZES[fold],labeled=True, return_image_names=True)\n    oof_tar.append(np.array([target.numpy() for img, target in iter(ds_valid.unbatch())]) )\n    oof_folds.append(np.ones_like(oof_tar[-1],dtype='int8')*fold )\n    ds = get_dataset(files_valid,dim=IMG_SIZES[fold],labeled=False,return_image_names=True)\n    oof_names.append(np.array([img_name.numpy().decode(\"utf-8\") for img, img_name in iter(ds.unbatch())]))\n    \n    # PREDICT TEST\n    print('Predicting Test...')\n    ds_test     = get_dataset(files_test,labeled=False,return_image_names=False,shuffle=False,dim=IMG_SIZES[fold],batch_size=BATCH_SIZES[fold]*4)\n    ct_test     = count_data_items(files_test); STEPS = ct_test/BATCH_SIZES[fold]/4/REPLICAS\n    pred        = model.predict(ds_test,steps=STEPS,verbose=VERBOSE)[:ct_test,]\n    preds[:,0] += (pred * WGTS[fold]).reshape(-1)\n\n    # REPORT RESULTS\n    auc = roc_auc_score(oof_tar[-1],oof_pred[-1])\n    print('#### FOLD %i OOF AUC = %.4f'%(fold+1,auc))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"How does the OOF AUC compare to a model without the pre-training stage? To check this, we can simply set `USE_PRETRAIN_WEIGHTS = False` in the begining of the notebeook. This is done [in the previous version](https://www.kaggle.com/kozodoi/pre-training-on-full-data-with-surrogate-labels?scriptVersionId=41201266) of this notebook and yields a model with a lower OOF AUC.\n\nCompared to a model initialized from the Imagenet weights, pre-training on a surrogate label brings a CV improvement, which also translates into an AUC gain on public and private LB. Great news!","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# COMPUTE OVERALL OOF AUC\noof      = np.concatenate(oof_pred);  true  = np.concatenate(oof_tar);\nnames    = np.concatenate(oof_names); folds = np.concatenate(oof_folds)\nauc      = roc_auc_score(true,oof)\nprint('Overall OOF AUC = %.4f'%auc)\n\n# SAVE OOF TO DISK\ndf_oof = pd.DataFrame(dict(image_name = names, target = true, pred = oof.reshape(-1), fold = folds))\ndf_oof.to_csv('oof.csv', index = False)\ndf_oof.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# CREATE SUBMISSION\nds = get_dataset(files_test, \n                 dim                = IMG_SIZES[fold],\n                 labeled            = False, \n                 return_image_names = True)\n\nimage_names = np.array([img_name.numpy().decode(\"utf-8\") for img, img_name in iter(ds.unbatch())])\n\nsubmission = pd.DataFrame(dict(image_name = image_names, target = preds[:,0]))\nsubmission = submission.sort_values('image_name') \nsubmission.to_csv('submission.csv', index = False)\nsubmission.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 6. CONCLUSIONS\n\nThis is the end of this notebook. We demonstrated how to use meta-data to construct a surrogate label and pre-train a CNN model on both training and test data. This technique improved the resulting performance on both CV and LB. \n\nThe pre-trained model can be further optimized to increase performance gains. Using a validation subset on the pre-training stage can help to tune the number of epochs and other learning parameters. Another idea could be to construct a surrogate label with more unique values (e.g., combination of `anatom_site_general_challenge` and `sex`) to make the pre-training task more challenging and motivate the model to learn better. On the other hand, further optimizing the final classification model may reduce the benefit of pre-training. I will leave these options to those who are interested to experiment :)\n\nPlease don't hesitate to ask questions in the comments section if something is not clear. Happy Kaggling!","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}