{"cells":[{"metadata":{"trusted":true},"cell_type":"markdown","source":"# 1. Introduction + Set Up\n\nTransfer learning in computer vision is the process of using a pre-trained model on a new dataset of images. This is extremely useful when the dataset is not that big. ML models thrive when there is a lot of data available to train on, but it many scenarios, obtaining large datasets is quite difficult. The pre-trained model is trained on a big dataset, and the weights are preserved. The trained layers of the model is used as the basis for our new model. We will be using the VGG16 model to predict whether or not a lesion is a malignant or benign tumor.\n\nAdditionally, training models take up a lot of time, and so we will utilize the TPU accelerator to reduce it time it takes to train our model.\n\nRun the following cell to import the necessary packages. Don't forget to change the accelerator to TPU v3-8 on the right.","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import re\nimport os\nimport numpy as np\nimport pandas as pd\nimport tensorflow as tf\nfrom functools import partial\nfrom kaggle_datasets import KaggleDatasets\nfrom sklearn.model_selection import train_test_split\nimport tempfile\nimport matplotlib.pyplot as plt\n\ntry:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n    print('Device:', tpu.master())\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nexcept:\n    strategy = tf.distribute.get_strategy()\nprint('Number of replicas:', strategy.num_replicas_in_sync)\n    \nprint(tf.__version__)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Running the following cell will save time in loading the data as it allows for parallel processing. TPU can only work with Google Cloud files and cannot be run with local files. We want to state a relatively bigger batch size, as our data is inbalanced. We will go over what inbalanced data is in later cells.\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"AUTOTUNE = tf.data.experimental.AUTOTUNE\nGCS_PATH = KaggleDatasets().get_gcs_path()\nBATCH_SIZE = 16 * strategy.num_replicas_in_sync\nIMAGE_SIZE = [1024, 1024]\nIMAGE_RESIZE = [256, 256]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 2. Load the data\n\nNow we will load in our data. For this notebook, we will be importing the TFRecord Files. It is good practice to divide the training set data into two. The smaller dataset will be the validation set. Having a validation set is useful to prevent overfitting as the finetuning of the model will be done by calculating metrics on the validation set and not the training set.\n\n```train_test_split``` will divide our dataset for us. For reproducible results, ```random_state``` 5 was arbitrarily chosen.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"TRAINING_FILENAMES, VALID_FILENAMES = train_test_split(\n    tf.io.gfile.glob(GCS_PATH + '/tfrecords/train*.tfrec'),\n    test_size=0.1, random_state=5\n)\nTEST_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/tfrecords/test*.tfrec')\nprint('Train TFRecord Files:', len(TRAINING_FILENAMES))\nprint('Validation TFRecord Files:', len(VALID_FILENAMES))\nprint('Test TFRecord Files:', len(TEST_FILENAMES))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 2.1 Decoding the data\n\nThe images have to be converted to tensors so that it will be a valid input in our model. As images utilize an RBG scale, we specify 3 channels.\n\nIt is also best practice to normalize data before it is is fed into the model. For our image data, we will scale it down so that the value of each pixel will range from [0, 1] instead of [0. 255].\n\nWe also reshape our data so that all of the images will be the same shape. Although the TFRecord files have already been reshaped for us, it is best practice to reshape the input so that we know exactly what's going in to our model.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def decode_image(image):\n    image = tf.image.decode_jpeg(image, channels=3)\n    image = tf.cast(image, tf.float32) / 255.0\n    image = tf.reshape(image, [*IMAGE_SIZE, 3])\n    return image","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As we load in our data, we need both our ```X``` and our ```Y```. The X is our image; the model will find features and patterns in our image dataset. We want to predict Y, the probability that the lesion in the image is malignant. When we input our training dataset into the model, it is necessary know what the labels of our images are. For our testing set, we do not need to feed in the labels because the model will predict it for us. In this particular dataset, our label is the ```target``` feature. We will to through our TFRecords and parse out the image and the target values.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def read_tfrecord(example, labeled):\n    tfrecord_format = {\n        \"image\": tf.io.FixedLenFeature([], tf.string),\n        \"target\": tf.io.FixedLenFeature([], tf.int64)\n    } if labeled else {\n        \"image\": tf.io.FixedLenFeature([], tf.string),\n        \"image_name\": tf.io.FixedLenFeature([], tf.string)\n    }\n    example = tf.io.parse_single_example(example, tfrecord_format)\n    image = decode_image(example['image'])\n    if labeled:\n        label = tf.cast(example['target'], tf.int32)\n        return image, label\n    idnum = example['image_name']\n    return image, idnum","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Our dataset is not ordered in any meaningful way, so the order can be ignored when loading our dataset. By ignoring the order and reading files as soon as they come in, it will take a shorter time to load the data.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def load_dataset(filenames, labeled=True, ordered=False):\n    ignore_order = tf.data.Options()\n    if not ordered:\n        ignore_order.experimental_deterministic = False # disable order, increase speed\n    dataset = tf.data.TFRecordDataset(filenames, num_parallel_reads=AUTOTUNE) # automatically interleaves reads from multiple files\n    dataset = dataset.with_options(ignore_order) # uses data as soon as it streams in, rather than in its original order\n    dataset = dataset.map(partial(read_tfrecord, labeled=labeled), num_parallel_calls=AUTOTUNE)\n    # returns a dataset of (image, label) pairs if labeled=True or (image, id) pairs if labeled=False\n    return dataset","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 2.2 Data augmentation\n\nImage augmentation introduces new images into our data, and generally prevents the model from overfitting and leads to higher accuracy on testing datasets. For this particular dataset, we will randomly flip the image horizontally.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def augmentation_pipeline(image, label):\n    image = tf.image.random_flip_left_right(image)\n    image = tf.image.resize(image, IMAGE_RESIZE)\n    return image, label","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 2.3 Define loading methods\n\nWe define the following three functions to get our three different datasets.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_training_dataset():\n    dataset = load_dataset(TRAINING_FILENAMES, labeled=True)\n    dataset = dataset.map(augmentation_pipeline, num_parallel_calls=AUTOTUNE)\n    dataset = dataset.repeat()\n    dataset = dataset.shuffle(2048)\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.prefetch(AUTOTUNE)\n    return dataset","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_validation_dataset(ordered=False):\n    dataset = load_dataset(VALID_FILENAMES, labeled=True, ordered=ordered)\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.cache()\n    dataset = dataset.prefetch(AUTOTUNE)\n    return dataset","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_test_dataset(ordered=False):\n    dataset = load_dataset(TEST_FILENAMES, labeled=False, ordered=ordered)\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.prefetch(AUTOTUNE)\n    return dataset","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The following cell returns the number of images we have in each dataset.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def count_data_items(filenames):\n    n = [int(re.compile(r\"-([0-9]*)\\.\").search(filename).group(1)) for filename in filenames]\n    return np.sum(n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Run the following cell to visualize how many images we have in each of our datasets. Notice that the training dataset has the most images. This is normal, as having more training points will allow the model to learn more.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"NUM_TRAINING_IMAGES = count_data_items(TRAINING_FILENAMES)\nNUM_VALIDATION_IMAGES = count_data_items(VALID_FILENAMES)\nNUM_TEST_IMAGES = count_data_items(TEST_FILENAMES)\nSTEPS_PER_EPOCH = NUM_TRAINING_IMAGES // BATCH_SIZE\nprint(\n    'Dataset: {} training images, {} validation images, {} unlabeled test images'.format(\n        NUM_TRAINING_IMAGES, NUM_VALIDATION_IMAGES, NUM_TEST_IMAGES\n    )\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 3. Building our model\n\nNow that our data has been loaded, we will now start building our model.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## 3.1 Define the learning rate\n\nThe following function allows for the model to change the learning rate as it runs each epoch. Having a learning rate that is too high will prevent our model from converging. However, having a learning rate that is too small will cause our model to run for far too long. With this function, the model will know how to change its learning rate after each epoch and update the learning rate itself to allow for increased efficiency while still allowing the model to converge.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def exponential_decay(lr0, s):\n    def exponential_decay_fn(epoch):\n        return lr0 * 0.1 **(epoch / s)\n    return exponential_decay_fn\n\nexponential_decay_fn = exponential_decay(0.01, 20)\n\nlr_scheduler = tf.keras.callbacks.LearningRateScheduler(exponential_decay_fn)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 3.2 Explore our data\n\nBefore we build our model, we want to look at the data to see what we are working with. When we look at our training and validation dataset, we see that our data is inbalanced. So what does inbalanced data mean?\n\nOur data is inbalanced because when we look at our data, we see that only 1.76% of the images are images of malignant lesions while 98.24% of the images are benign. This means that there are not enough images of malignant lesions that our model can learn from.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_csv = pd.read_csv('../input/siim-isic-melanoma-classification/train.csv')\ntest_csv = pd.read_csv('../input/siim-isic-melanoma-classification/test.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"total_img = train_csv['target'].size\n\nmalignant = np.count_nonzero(train_csv['target'])\nbenign = total_img - malignant\n\nprint('Examples:\\n    Total: {}\\n    Positive: {} ({:.2f}% of total)\\n'.format(\n    total_img, malignant, 100 * malignant / total_img))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_dataset = get_training_dataset()\nvalid_dataset = get_validation_dataset()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"image_batch, label_batch = next(iter(train_dataset))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def show_batch(image_batch, label_batch):\n    plt.figure(figsize=(10,10))\n    for n in range(25):\n        ax = plt.subplot(5,5,n+1)\n        plt.imshow(image_batch[n])\n        if label_batch[n]:\n            plt.title(\"MALIGNANT\")\n        else:\n            plt.title(\"BENIGN\")\n        plt.axis(\"off\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"show_batch(image_batch.numpy(), label_batch.numpy())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 3.3 Build our base model\n\nTransfer learning is a great way to reap the benefits of a well-trained model without having the train the model ourselves. For this notebook, we want to import the VGG16 model. As this model has already been trained, we do not want to change the weights in this model and must set its ```trainable``` attribute to ```False```. We also don't want to include the top layers because the classification that we want to accomplish in this model is not the same as the classification that the model was originally trained on. We want the a single node outputed that determines the probability thatthe lesion will be malignant. However, because the data is inbalanced, this model will not work very well.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def make_model(output_bias = None, metrics = None):    \n    if output_bias is not None:\n        output_bias = tf.keras.initializers.Constant(output_bias)\n        \n    base_model = tf.keras.applications.Xception(input_shape=(*IMAGE_RESIZE, 3),\n                                                include_top=False,\n                                                weights='imagenet')\n    \n    base_model.trainable = False\n    \n    model = tf.keras.Sequential([\n        base_model,\n        tf.keras.layers.GlobalAveragePooling2D(),\n        tf.keras.layers.Dense(8, activation='relu'),\n        tf.keras.layers.Dense(1, activation='sigmoid',\n                              bias_initializer=output_bias)\n    ])\n    \n    model.compile(optimizer='adam',\n                  loss='binary_crossentropy',\n                  metrics=metrics)\n    \n    return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"STEPS_PER_EPOCH = NUM_TRAINING_IMAGES // BATCH_SIZE\nVALID_STEPS = NUM_VALIDATION_IMAGES // BATCH_SIZE","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 3.4 Correcting for data inbalance","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### 3.4.1 Set initial bias\n\nWe want to set the correct initial bias for our model so that it will not waste time figuring out that there are not many malignant images in our dataset. We want our output layer to reflect the inbalance that we have in our data.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"initial_bias = np.log([malignant/benign])\ninitial_bias","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 3.4.2 Set class weights\n\nSince there are not enough malignant images, we want these malignant images to have more weight in our model. By increasing the weight of these malignant images, the model will pay more attention to them, and this will help balance out the difference in quantity.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"weight_for_0 = (1 / benign)*(total_img)/2.0 \nweight_for_1 = (1 / malignant)*(total_img)/2.0\n\nclass_weight = {0: weight_for_0, 1: weight_for_1}\n\nprint('Weight for class 0: {:.2f}'.format(weight_for_0))\nprint('Weight for class 1: {:.2f}'.format(weight_for_1))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 3.5 Deciding our evaluation metrics\n\nWhen we compile our model, we do not want our metric to be ```accuracy```. If we run the model, with an ```accuracy``` metric, it will give us false confidence in our model. If we look at the dataset, we see that 98% of the images are classifed as benign, ```0```. Now, if accuracy was the sole determinant of our model, a model that always outputs 0 will achieve a high accuracy although the model is not good. Additionally, for the type of prediction we are making, it is riskier to predict a malignant tumor as benign than it is to predict a benign tumor as malignant. Thus, our metric should not be just accuracy. The competition scores the model by finding the area under the ROC curve, which is why our metric will be set to ```keras.metrics.AUC```.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"with strategy.scope():\n    model = make_model(output_bias = initial_bias, metrics=tf.keras.metrics.AUC(name='auc'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can use callbacks to stop training when there are no improvements in our validation set predictions, and this stops overfitting. Additionally, we can save the best weights for our model so that it doesn't have to be retrained. At the end of the training process, the model will restore the weights of its best iteration.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"checkpoint_cb = tf.keras.callbacks.ModelCheckpoint(\"melanoma_model.h5\",\n                                                    save_best_only=True)\n\nearly_stopping_cb = tf.keras.callbacks.EarlyStopping(patience=10,\n                                                     restore_best_weights=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"history = model.fit(\n    train_dataset, epochs=100,\n    steps_per_epoch=STEPS_PER_EPOCH,\n    validation_data=valid_dataset,\n    validation_steps=VALID_STEPS,\n    callbacks=[checkpoint_cb, early_stopping_cb, lr_scheduler],\n    class_weight=class_weight\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 4. Predicting results\n\nUsing our model, let us predict the probability that a tumor is malignant based on the image of the lesion using our testing dataset.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"test_ds = get_test_dataset(ordered=True)\n\nprint('Computing predictions...')\ntest_images_ds = test_ds.map(lambda image, idnum: image)\nprobabilities = model.predict(test_images_ds)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's visualize how our predictions look like. We want our submission file to look like the sample submission file, so let's visualize how that looks.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"sub = pd.read_csv('/kaggle/input/siim-isic-melanoma-classification/sample_submission.csv')\nsub.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's create our submission file.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Generating submission.csv file...')\ntest_ids_ds = test_ds.map(lambda image, idnum: idnum).unbatch()\ntest_ids = next(iter(test_ids_ds.batch(NUM_TEST_IMAGES))).numpy().astype('U')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pred_df = pd.DataFrame({'image_name': test_ids, 'target': np.concatenate(probabilities)})\npred_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del sub['target']\nsub = sub.merge(pred_df, on='image_name')\nsub.to_csv('submission.csv', index=False)\nsub.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}