{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Introduction\n\n**References**  \nThis notebook was built using the following amazing resources created by :\n- **Martin Gorner:** [Getting Started: TPUs + Cassava Leaf Disease](https://www.kaggle.com/jessemostipak/getting-started-tpus-cassava-leaf-disease)\n\nThe Output of the notbook can be used in the inference notebook proviede earlier:\n\nhttps://www.kaggle.com/lucamtb/brain-tumor-very-basice-inference","metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","papermill":{"duration":0.039382,"end_time":"2020-11-19T21:45:23.042097","exception":false,"start_time":"2020-11-19T21:45:23.002715","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"# Set up environment","metadata":{"papermill":{"duration":0.037375,"end_time":"2020-11-19T21:45:23.192515","exception":false,"start_time":"2020-11-19T21:45:23.15514","status":"completed"},"tags":[]}},{"cell_type":"code","source":"import math, re, os, random\nimport tensorflow as tf\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nfrom kaggle_datasets import KaggleDatasets\nfrom tensorflow import keras\nfrom functools import partial\nfrom sklearn.model_selection import train_test_split\nprint(\"Tensorflow version \" + tf.__version__)","metadata":{"papermill":{"duration":6.890298,"end_time":"2020-11-19T21:45:30.119979","exception":false,"start_time":"2020-11-19T21:45:23.229681","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-08-05T13:40:34.995134Z","iopub.execute_input":"2021-08-05T13:40:34.995484Z","iopub.status.idle":"2021-08-05T13:40:35.002977Z","shell.execute_reply.started":"2021-08-05T13:40:34.995455Z","shell.execute_reply":"2021-08-05T13:40:35.001443Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"try:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n    print('Device:', tpu.master())\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nexcept:\n    strategy = tf.distribute.get_strategy()\nprint('Number of replicas:', strategy.num_replicas_in_sync)","metadata":{"papermill":{"duration":4.150374,"end_time":"2020-11-19T21:45:34.382816","exception":false,"start_time":"2020-11-19T21:45:30.232442","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-08-05T13:40:35.004676Z","iopub.execute_input":"2021-08-05T13:40:35.005219Z","iopub.status.idle":"2021-08-05T13:40:42.93265Z","shell.execute_reply.started":"2021-08-05T13:40:35.005174Z","shell.execute_reply":"2021-08-05T13:40:42.931671Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Set up variables\n","metadata":{"papermill":{"duration":0.038122,"end_time":"2020-11-19T21:45:34.458722","exception":false,"start_time":"2020-11-19T21:45:34.4206","status":"completed"},"tags":[]}},{"cell_type":"code","source":"AUTOTUNE = tf.data.experimental.AUTOTUNE\nGCS_PATH = KaggleDatasets().get_gcs_path('tfflair-records')\nBATCH_SIZE = 24 * strategy.num_replicas_in_sync\nIMAGE_SIZE = [256, 256] # was 512\nEPOCHS = 5\nfolds = 3\n\n","metadata":{"papermill":{"duration":145.219568,"end_time":"2020-11-19T21:47:59.715925","exception":false,"start_time":"2020-11-19T21:45:34.496357","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-08-05T13:40:42.934432Z","iopub.execute_input":"2021-08-05T13:40:42.934724Z","iopub.status.idle":"2021-08-05T13:40:43.27564Z","shell.execute_reply.started":"2021-08-05T13:40:42.934696Z","shell.execute_reply":"2021-08-05T13:40:43.274596Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"GCS_PATH","metadata":{"execution":{"iopub.status.busy":"2021-08-05T13:40:43.277583Z","iopub.execute_input":"2021-08-05T13:40:43.277926Z","iopub.status.idle":"2021-08-05T13:40:43.284706Z","shell.execute_reply.started":"2021-08-05T13:40:43.277894Z","shell.execute_reply":"2021-08-05T13:40:43.283588Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_gs = pd.DataFrame(tf.io.gfile.glob(GCS_PATH + '/*.tfrec'),columns = ['gs'])\n\ndf_gs","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.model_selection import GroupKFold\n\n\ngkf  = GroupKFold(n_splits = folds)\ndf_gs['fold'] = -1\nfor fold, (train_idx, val_idx) in enumerate(gkf.split(df_gs, groups = df_gs.gs.tolist())):\n\n    df_gs.loc[val_idx, 'fold'] = fold\n\ndf_gs\n\n","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Load the data\nIf you've primarily worked with notebooks in Learn, you've maybe noticed that data import and formatting is taken care of for you. But because we're working with competition data we'll have to handle this part of the pipeline ourselves.   \n\nThe data we're working with have been formatted into `TFRecords`, which are a format for storing a sequence of binary records. `TFRecords` work _really_ well with TPUs, and allow us to send a small number of large files across the TPU for processing.   \n\nIf you'd like to learn more about `TFRecords` and maybe even try creating them yourself, check out this **[TFRecords Basics notebook](https://www.kaggle.com/ryanholbrook/tfrecords-basics)** and **[corresponding video](https://youtu.be/KgjaC9VeOi8)** from Kaggle Data Scientist Ryan Holbrook.  \n\nBecause our data consists of `training` and `test` images only, we're going to split our `training` data into `training` and `validation` data using the `train_test_split()` function. ","metadata":{"papermill":{"duration":0.037843,"end_time":"2020-11-19T21:47:59.792061","exception":false,"start_time":"2020-11-19T21:47:59.754218","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"## Decode the data\nIn the code chunk below we'll set up a series of functions that allow us to convert our images into tensors so that we can utilize them in our model. We'll also normalize our data. Our images are using a \"Red, Blue, Green (RBG)\" scale that has a range of [0, 255], and by normalizing it we'll set each pixel's value to a number in the range of [0, 1]. ","metadata":{"papermill":{"duration":0.038439,"end_time":"2020-11-19T21:47:59.869037","exception":false,"start_time":"2020-11-19T21:47:59.830598","status":"completed"},"tags":[]}},{"cell_type":"code","source":"[*IMAGE_SIZE]","metadata":{"execution":{"iopub.status.busy":"2021-08-05T13:40:43.286203Z","iopub.execute_input":"2021-08-05T13:40:43.286628Z","iopub.status.idle":"2021-08-05T13:40:43.298164Z","shell.execute_reply.started":"2021-08-05T13:40:43.286587Z","shell.execute_reply":"2021-08-05T13:40:43.29722Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def decode_image(image):\n    image = tf.image.decode_png(image, channels=3)\n    image = tf.cast(image, tf.float32) / 255.0\n    image = tf.image.resize(image,[*IMAGE_SIZE])\n    #image = tf.reshape(image, [*IMAGE_SIZE])\n    return image","metadata":{"papermill":{"duration":0.04859,"end_time":"2020-11-19T21:47:59.955731","exception":false,"start_time":"2020-11-19T21:47:59.907141","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-08-05T13:40:43.300091Z","iopub.execute_input":"2021-08-05T13:40:43.300626Z","iopub.status.idle":"2021-08-05T13:40:43.310626Z","shell.execute_reply.started":"2021-08-05T13:40:43.30058Z","shell.execute_reply":"2021-08-05T13:40:43.309326Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def read_tfrecord(example, labeled):\n    tfrecord_format = {\n        \"image\": tf.io.FixedLenFeature([], tf.string),\n         \"label\": tf.io.FixedLenSequenceFeature([], tf.int64,allow_missing=True)\n    } if labeled else {\n        \"image\": tf.io.FixedLenFeature([], tf.string),\n        \"image_name\": tf.io.FixedLenFeature([], tf.string)\n    }\n    example = tf.io.parse_single_example(example, tfrecord_format)\n    image = decode_image(example['image'])\n    label = tf.cast(example['label'], tf.int32)\n    #label = tf.reshape(label,(1,num_classes))\n  \n    return image, label","metadata":{"papermill":{"duration":0.052475,"end_time":"2020-11-19T21:48:00.127039","exception":false,"start_time":"2020-11-19T21:48:00.074564","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-08-05T13:40:43.312065Z","iopub.execute_input":"2021-08-05T13:40:43.312368Z","iopub.status.idle":"2021-08-05T13:40:43.323277Z","shell.execute_reply.started":"2021-08-05T13:40:43.312341Z","shell.execute_reply":"2021-08-05T13:40:43.32227Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We'll use the following function to load our dataset. One of the advantages of a TPU is that we can run multiple files across the TPU at once, and this accounts for the speed advantages of using a TPU. To capitalize on that, we want to make sure that we're using data as soon as it streams in, rather than creating a data streaming bottleneck.","metadata":{"papermill":{"duration":0.038946,"end_time":"2020-11-19T21:48:00.205445","exception":false,"start_time":"2020-11-19T21:48:00.166499","status":"completed"},"tags":[]}},{"cell_type":"code","source":"def filter_fn(image, label):\n    \n    im_size = tf.math.reduce_mean(image)\n    \n    return tf.math.greater(im_size, 0.025)","metadata":{"execution":{"iopub.status.busy":"2021-08-05T13:40:43.326045Z","iopub.execute_input":"2021-08-05T13:40:43.326381Z","iopub.status.idle":"2021-08-05T13:40:43.341589Z","shell.execute_reply.started":"2021-08-05T13:40:43.326349Z","shell.execute_reply":"2021-08-05T13:40:43.340276Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def load_dataset(filenames, labeled=True, ordered=False):\n    ignore_order = tf.data.Options()\n    if not ordered:\n        ignore_order.experimental_deterministic = False # disable order, increase speed\n    dataset = tf.data.TFRecordDataset(filenames, num_parallel_reads=AUTOTUNE) # automatically interleaves reads from multiple files\n    dataset = dataset.with_options(ignore_order) # uses data as soon as it streams in, rather than in its original order\n    dataset = dataset.map(partial(read_tfrecord, labeled=labeled), num_parallel_calls=AUTOTUNE)\n    dataset = dataset.filter(filter_fn)\n    return dataset","metadata":{"papermill":{"duration":0.073623,"end_time":"2020-11-19T21:48:00.328703","exception":false,"start_time":"2020-11-19T21:48:00.25508","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-08-05T13:40:43.343505Z","iopub.execute_input":"2021-08-05T13:40:43.343919Z","iopub.status.idle":"2021-08-05T13:40:43.354348Z","shell.execute_reply.started":"2021-08-05T13:40:43.343887Z","shell.execute_reply":"2021-08-05T13:40:43.353403Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## A note on using train_test_split()\nWhile I used `train_test_split()` to create both a `training` and `validation` dataset, consider exploring **[cross validation instead](https://www.kaggle.com/dansbecker/cross-validation)**.","metadata":{"papermill":{"duration":0.03958,"end_time":"2020-11-19T21:48:00.416432","exception":false,"start_time":"2020-11-19T21:48:00.376852","status":"completed"},"tags":[]}},{"cell_type":"code","source":"#TRAINING_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/*.tfrec')","metadata":{"execution":{"iopub.status.busy":"2021-08-05T13:40:43.355739Z","iopub.execute_input":"2021-08-05T13:40:43.35625Z","iopub.status.idle":"2021-08-05T13:40:43.367229Z","shell.execute_reply.started":"2021-08-05T13:40:43.356218Z","shell.execute_reply":"2021-08-05T13:40:43.366409Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"TRAINING_FILENAMES, VALID_FILENAMES = train_test_split(\n    tf.io.gfile.glob(GCS_PATH + '/*.tfrec'),\n    test_size=0.2, random_state=15 ## was 0.35\n)\n\n#TEST_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/test_tfrecords/ld_test*.tfrec')","metadata":{"papermill":{"duration":0.225941,"end_time":"2020-11-19T21:48:00.687385","exception":false,"start_time":"2020-11-19T21:48:00.461444","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-08-05T13:40:43.368508Z","iopub.execute_input":"2021-08-05T13:40:43.369197Z","iopub.status.idle":"2021-08-05T13:40:43.456661Z","shell.execute_reply.started":"2021-08-05T13:40:43.369151Z","shell.execute_reply":"2021-08-05T13:40:43.45536Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dataset = load_dataset(TRAINING_FILENAMES, labeled=True)","metadata":{"execution":{"iopub.status.busy":"2021-08-05T13:40:43.458023Z","iopub.execute_input":"2021-08-05T13:40:43.458402Z","iopub.status.idle":"2021-08-05T13:40:43.494554Z","shell.execute_reply.started":"2021-08-05T13:40:43.458367Z","shell.execute_reply":"2021-08-05T13:40:43.49352Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for i in dataset.take(2):\n    \n    print(i[0].shape)","metadata":{"execution":{"iopub.status.busy":"2021-08-05T13:40:43.49605Z","iopub.execute_input":"2021-08-05T13:40:43.496501Z","iopub.status.idle":"2021-08-05T13:40:44.043922Z","shell.execute_reply.started":"2021-08-05T13:40:43.496458Z","shell.execute_reply":"2021-08-05T13:40:44.04288Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dataset = tf.data.TFRecordDataset(VALID_FILENAMES, num_parallel_reads=AUTOTUNE)","metadata":{"execution":{"iopub.status.busy":"2021-08-05T13:40:44.04546Z","iopub.execute_input":"2021-08-05T13:40:44.045764Z","iopub.status.idle":"2021-08-05T13:40:44.058858Z","shell.execute_reply.started":"2021-08-05T13:40:44.045734Z","shell.execute_reply":"2021-08-05T13:40:44.058085Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dataset = dataset.with_options(tf.data.Options())","metadata":{"execution":{"iopub.status.busy":"2021-08-05T13:40:44.060415Z","iopub.execute_input":"2021-08-05T13:40:44.060942Z","iopub.status.idle":"2021-08-05T13:40:44.069028Z","shell.execute_reply.started":"2021-08-05T13:40:44.060909Z","shell.execute_reply":"2021-08-05T13:40:44.067448Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import sys\nimport numpy\nnumpy.set_printoptions(threshold=sys.maxsize)\n\nlabeled = True\n\ntfrecord_format = {\n    \"image\": tf.io.FixedLenFeature([], tf.string),\n    \"label\": tf.io.FixedLenSequenceFeature([], tf.int64,allow_missing=True)\n} if labeled else {\n    \"image\": tf.io.FixedLenFeature([], tf.string),\n    \"image_name\": tf.io.FixedLenFeature([], tf.string)\n    }\n\nj = 0\nfor i in dataset:\n    image = tf.io.parse_single_example(i,tfrecord_format)\n    if j == 1:\n        break\n    else:\n        j = j +1\n        #print(tf.reshape(tf.reduce_max(tf.one_hot(image['image'], num_classes, dtype=tf.int32), axis=0),(1,19)))\n\n","metadata":{"execution":{"iopub.status.busy":"2021-08-05T13:40:44.070651Z","iopub.execute_input":"2021-08-05T13:40:44.071102Z","iopub.status.idle":"2021-08-05T13:40:44.157144Z","shell.execute_reply.started":"2021-08-05T13:40:44.07107Z","shell.execute_reply":"2021-08-05T13:40:44.155996Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"image = decode_image(image['image'])","metadata":{"execution":{"iopub.status.busy":"2021-08-05T13:40:44.160727Z","iopub.execute_input":"2021-08-05T13:40:44.16107Z","iopub.status.idle":"2021-08-05T13:40:44.267766Z","shell.execute_reply.started":"2021-08-05T13:40:44.161032Z","shell.execute_reply":"2021-08-05T13:40:44.266535Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tf.math.reduce_mean(image)","metadata":{"execution":{"iopub.status.busy":"2021-08-05T13:40:44.26921Z","iopub.execute_input":"2021-08-05T13:40:44.269497Z","iopub.status.idle":"2021-08-05T13:40:44.277357Z","shell.execute_reply.started":"2021-08-05T13:40:44.269469Z","shell.execute_reply":"2021-08-05T13:40:44.276007Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dataset.take(1)","metadata":{"execution":{"iopub.status.busy":"2021-08-05T13:40:44.278451Z","iopub.execute_input":"2021-08-05T13:40:44.278723Z","iopub.status.idle":"2021-08-05T13:40:44.290498Z","shell.execute_reply.started":"2021-08-05T13:40:44.278697Z","shell.execute_reply":"2021-08-05T13:40:44.289482Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"image.shape","metadata":{"execution":{"iopub.status.busy":"2021-08-05T13:40:44.291709Z","iopub.execute_input":"2021-08-05T13:40:44.291994Z","iopub.status.idle":"2021-08-05T13:40:44.305956Z","shell.execute_reply.started":"2021-08-05T13:40:44.291966Z","shell.execute_reply":"2021-08-05T13:40:44.304838Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.imshow(image)","metadata":{"execution":{"iopub.status.busy":"2021-08-05T13:40:44.30767Z","iopub.execute_input":"2021-08-05T13:40:44.308379Z","iopub.status.idle":"2021-08-05T13:40:44.668271Z","shell.execute_reply.started":"2021-08-05T13:40:44.308302Z","shell.execute_reply":"2021-08-05T13:40:44.667052Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Adding in augmentations \n","metadata":{"papermill":{"duration":0.038372,"end_time":"2020-11-19T21:48:00.765394","exception":false,"start_time":"2020-11-19T21:48:00.727022","status":"completed"},"tags":[]}},{"cell_type":"code","source":"def data_augment(image, label):\n    # Thanks to the dataset.prefetch(AUTO) statement in the following function this happens essentially for free on TPU. \n    # Data pipeline code is executed on the \"CPU\" part of the TPU while the TPU itself is computing gradients.\n    image = tf.image.random_flip_left_right(image)\n    image = tf.image.random_flip_up_down(image)\n    ##image = tf.image.random_crop(image, [int(IMAGE_SIZE[0]), int(IMAGE_SIZE[0]), 3])\n    image = tf.image.rot90(image, k= random.randint(1, 4))\n    #image = dropout(image)\n    #label = tf.one_hot(label, num_classes, dtype=tf.int32)\n    #image = tf.image.draw_bounding_boxes(image, [64, 1, 32] , colors)\n    #image = tfa.image.rotate(image, tf.constant(np.pi/8))\n    #image = tf.image.random_brightness(image, 0.2)\n    #image = tf.image.random_hue(image, 0.2)\n    #image = tf.image.random_saturation(image, 5, 10)\n    return image, label","metadata":{"papermill":{"duration":0.047715,"end_time":"2020-11-19T21:48:00.851918","exception":false,"start_time":"2020-11-19T21:48:00.804203","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-08-05T13:40:44.669965Z","iopub.execute_input":"2021-08-05T13:40:44.6704Z","iopub.status.idle":"2021-08-05T13:40:44.6769Z","shell.execute_reply.started":"2021-08-05T13:40:44.670355Z","shell.execute_reply":"2021-08-05T13:40:44.675867Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Define data loading methods\nThe following functions will be used to load our `training`, `validation`, and `test` datasets, as well as print out the number of images in each dataset.","metadata":{"papermill":{"duration":0.038742,"end_time":"2020-11-19T21:48:00.930185","exception":false,"start_time":"2020-11-19T21:48:00.891443","status":"completed"},"tags":[]}},{"cell_type":"code","source":"def get_training_dataset():\n    dataset = load_dataset(TRAINING_FILENAMES, labeled=True)  \n    #dataset = dataset.map(data_augment, num_parallel_calls=AUTOTUNE)  \n    dataset = dataset.repeat()\n    dataset = dataset.shuffle(30233)\n    #dataset = dataset.unbatch()\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.prefetch(AUTOTUNE)\n    return dataset","metadata":{"papermill":{"duration":0.052326,"end_time":"2020-11-19T21:48:01.021791","exception":false,"start_time":"2020-11-19T21:48:00.969465","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-08-05T13:40:44.680285Z","iopub.execute_input":"2021-08-05T13:40:44.68091Z","iopub.status.idle":"2021-08-05T13:40:44.69003Z","shell.execute_reply.started":"2021-08-05T13:40:44.680872Z","shell.execute_reply":"2021-08-05T13:40:44.688903Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_validation_dataset(ordered=False):\n    dataset = load_dataset(VALID_FILENAMES, labeled=True, ordered=ordered) \n    dataset = dataset.batch(BATCH_SIZE)\n    #dataset = dataset.cache()\n    dataset = dataset.prefetch(AUTOTUNE)\n    return dataset","metadata":{"papermill":{"duration":0.049787,"end_time":"2020-11-19T21:48:01.112145","exception":false,"start_time":"2020-11-19T21:48:01.062358","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-08-05T13:40:44.691717Z","iopub.execute_input":"2021-08-05T13:40:44.692281Z","iopub.status.idle":"2021-08-05T13:40:44.703473Z","shell.execute_reply.started":"2021-08-05T13:40:44.692245Z","shell.execute_reply":"2021-08-05T13:40:44.702425Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_test_dataset(ordered=False):\n    dataset = load_dataset(TEST_FILENAMES, labeled=False, ordered=ordered)\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.prefetch(AUTOTUNE)\n    return dataset","metadata":{"papermill":{"duration":0.050395,"end_time":"2020-11-19T21:48:01.207665","exception":false,"start_time":"2020-11-19T21:48:01.15727","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-08-05T13:40:44.704888Z","iopub.execute_input":"2021-08-05T13:40:44.705218Z","iopub.status.idle":"2021-08-05T13:40:44.721781Z","shell.execute_reply.started":"2021-08-05T13:40:44.705188Z","shell.execute_reply":"2021-08-05T13:40:44.720481Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def count_data_items(filenames):\n    n = [int(re.compile(r\"-([0-9]*)\\.\").search(filename).group(1)) for filename in filenames]\n    return np.sum(n)","metadata":{"papermill":{"duration":0.05422,"end_time":"2020-11-19T21:48:01.304611","exception":false,"start_time":"2020-11-19T21:48:01.250391","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-08-05T13:40:44.723625Z","iopub.execute_input":"2021-08-05T13:40:44.724062Z","iopub.status.idle":"2021-08-05T13:40:44.73447Z","shell.execute_reply.started":"2021-08-05T13:40:44.724016Z","shell.execute_reply":"2021-08-05T13:40:44.733252Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"NUM_TRAINING_IMAGES = count_data_items(TRAINING_FILENAMES)\nNUM_VALIDATION_IMAGES = count_data_items(VALID_FILENAMES)\n#NUM_TEST_IMAGES = count_data_items(TEST_FILENAMES)\n\nprint('Dataset: {} training images, {} validation images '.format(\n    NUM_TRAINING_IMAGES, NUM_VALIDATION_IMAGES))","metadata":{"papermill":{"duration":0.051209,"end_time":"2020-11-19T21:48:01.396198","exception":false,"start_time":"2020-11-19T21:48:01.344989","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-08-05T13:40:44.7361Z","iopub.execute_input":"2021-08-05T13:40:44.736735Z","iopub.status.idle":"2021-08-05T13:40:44.749081Z","shell.execute_reply.started":"2021-08-05T13:40:44.736679Z","shell.execute_reply":"2021-08-05T13:40:44.747933Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"NUM_VALIDATION_IMAGES","metadata":{"execution":{"iopub.status.busy":"2021-08-05T13:40:44.750392Z","iopub.execute_input":"2021-08-05T13:40:44.750704Z","iopub.status.idle":"2021-08-05T13:40:44.763179Z","shell.execute_reply.started":"2021-08-05T13:40:44.750672Z","shell.execute_reply":"2021-08-05T13:40:44.762156Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dataset = load_dataset(TRAINING_FILENAMES, labeled=True) \ndataset = dataset.map(data_augment, num_parallel_calls=AUTOTUNE) ","metadata":{"execution":{"iopub.status.busy":"2021-08-05T13:40:44.764619Z","iopub.execute_input":"2021-08-05T13:40:44.765033Z","iopub.status.idle":"2021-08-05T13:40:45.480192Z","shell.execute_reply.started":"2021-08-05T13:40:44.76499Z","shell.execute_reply":"2021-08-05T13:40:45.47906Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for i in dataset.take(3):\n    print((i[0].shape))","metadata":{"execution":{"iopub.status.busy":"2021-08-05T13:40:45.481666Z","iopub.execute_input":"2021-08-05T13:40:45.48198Z","iopub.status.idle":"2021-08-05T13:40:45.562408Z","shell.execute_reply.started":"2021-08-05T13:40:45.481951Z","shell.execute_reply":"2021-08-05T13:40:45.561654Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"You can also modify the above code to look at your `validation` and `test` data, like this:","metadata":{"papermill":{"duration":0.086379,"end_time":"2020-11-19T21:48:22.547481","exception":false,"start_time":"2020-11-19T21:48:22.461102","status":"completed"},"tags":[]}},{"cell_type":"code","source":"# load our validation dataset for EDA\nvalidation_dataset = get_validation_dataset()\nvalidation_dataset = validation_dataset.unbatch().batch(20)\nvalid_batch = iter(validation_dataset)","metadata":{"papermill":{"duration":0.136485,"end_time":"2020-11-19T21:48:22.769878","exception":false,"start_time":"2020-11-19T21:48:22.633393","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-08-05T13:40:45.563464Z","iopub.execute_input":"2021-08-05T13:40:45.563877Z","iopub.status.idle":"2021-08-05T13:40:45.611879Z","shell.execute_reply.started":"2021-08-05T13:40:45.563847Z","shell.execute_reply":"2021-08-05T13:40:45.610991Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Building our model\nIn order to ensure that our model is trained on the TPU, we build it using `with strategy.scope()`.    \n\nThis model was built using transfer learning, meaning that we have a _pre-trained model_ (ResNet50) as our base model and then the customizable model built using `tf.keras.Sequential`. If you're new to transfer learning I recommend setting `base_model.trainable` to **False**, but _do_ encourage you to change which base model you're using (more options are available in the **[`tf.keras.applications` Module](https://www.tensorflow.org/api_docs/python/tf/keras/applications)** documentation) as well iterate on the custom model. \n\nNote that we're using `sparse_categorical_crossentropy` as our loss function, because we did _not_ one-hot encode our labels.","metadata":{"papermill":{"duration":0.22538,"end_time":"2020-11-19T21:48:29.285377","exception":false,"start_time":"2020-11-19T21:48:29.059997","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"# Train the model\nAs our model is training you'll see a printout for each epoch, and can also monitor TPU usage by clicking on the TPU metrics in the toolbar at the top right of your notebook.","metadata":{"papermill":{"duration":0.174404,"end_time":"2020-11-19T21:48:49.513099","exception":false,"start_time":"2020-11-19T21:48:49.338695","status":"completed"},"tags":[]}},{"cell_type":"code","source":"# load data\ntrain_dataset = get_training_dataset()\nvalid_dataset = get_validation_dataset()","metadata":{"papermill":{"duration":0.249051,"end_time":"2020-11-19T21:48:49.936435","exception":false,"start_time":"2020-11-19T21:48:49.687384","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-08-05T13:40:45.612988Z","iopub.execute_input":"2021-08-05T13:40:45.613459Z","iopub.status.idle":"2021-08-05T13:40:45.684595Z","shell.execute_reply.started":"2021-08-05T13:40:45.613427Z","shell.execute_reply":"2021-08-05T13:40:45.683571Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model\n","metadata":{"papermill":{"duration":1.245239,"end_time":"2020-11-19T22:04:54.493139","exception":false,"start_time":"2020-11-19T22:04:53.2479","status":"completed"},"tags":[]}},{"cell_type":"code","source":"!pip install -U efficientnet","metadata":{"execution":{"iopub.status.busy":"2021-08-05T13:40:45.686005Z","iopub.execute_input":"2021-08-05T13:40:45.686457Z","iopub.status.idle":"2021-08-05T13:40:55.029885Z","shell.execute_reply.started":"2021-08-05T13:40:45.686415Z","shell.execute_reply":"2021-08-05T13:40:55.028693Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import efficientnet.keras as eff\nfrom keras.layers import Dropout\n","metadata":{"execution":{"iopub.status.busy":"2021-08-05T13:40:55.032104Z","iopub.execute_input":"2021-08-05T13:40:55.032458Z","iopub.status.idle":"2021-08-05T13:40:55.418374Z","shell.execute_reply.started":"2021-08-05T13:40:55.032422Z","shell.execute_reply":"2021-08-05T13:40:55.417341Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"STEPS_PER_EPOCH = NUM_TRAINING_IMAGES // BATCH_SIZE\nVALID_STEPS = NUM_VALIDATION_IMAGES // BATCH_SIZE","metadata":{"execution":{"iopub.status.busy":"2021-08-05T13:40:55.419985Z","iopub.execute_input":"2021-08-05T13:40:55.420428Z","iopub.status.idle":"2021-08-05T13:40:55.42583Z","shell.execute_reply.started":"2021-08-05T13:40:55.420384Z","shell.execute_reply":"2021-08-05T13:40:55.424285Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def unfreeze_model(model):\n    # We unfreeze the top 20 layers while leaving BatchNorm layers frozen\n    for layer in model.layers:#[-18:]:\n        if not isinstance(layer, tf.keras.layers.BatchNormalization):\n             layer.trainable = True","metadata":{"execution":{"iopub.status.busy":"2021-08-05T13:40:55.427492Z","iopub.execute_input":"2021-08-05T13:40:55.427804Z","iopub.status.idle":"2021-08-05T13:40:55.438634Z","shell.execute_reply.started":"2021-08-05T13:40:55.427774Z","shell.execute_reply":"2021-08-05T13:40:55.437379Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"num_folds = [i for i in range(folds)]\n\nlr_rates = [3e-5]\n\nepsilons = [1e-4]\n\nfor fold in num_folds:\n\n\n    for lr in lr_rates:\n\n        for e in epsilons:\n\n            my_callbacks = [\n                #tf.keras.callbacks.EarlyStopping(patience=5,verbose=1,monitor='val_binary_accuracy'),\n                tf.keras.callbacks.ModelCheckpoint(filepath=f'Brain_flair_model_effect_{lr}_{e}.h5',verbose=1,monitor='val_binary_accuracy',save_best_only=True),\n                #tf.keras.callbacks.LearningRateScheduler(lr_schedul,verbose=1)\n                tf.keras.callbacks.ReduceLROnPlateau(monitor='val_binary_accuracy',factor=0.1,patience=3,min_lr=1e-30,mode='min',verbose=1,)\n\n            ]\n\n            with strategy.scope():       \n                #img_adjust_layer = tf.keras.layers.Lambda(tf.keras.applications.efficientnet.preprocess_input, input_shape=[*IMAGE_SIZE, 3])\n                img_adjust_layer = tf.keras.layers.Lambda(tf.keras.applications.densenet.preprocess_input, input_shape=[*IMAGE_SIZE, 3])\n                #img_adjust_layer = tf.keras.layers.Lambda(tf.keras.applications.resnet.preprocess_input, input_shape=[*IMAGE_SIZE, 3])\n\n                #base_model = eff.EfficientNetB0(weights='noisy-student', include_top=False) #imagenet noisy-student\n                base_model = tf.keras.applications.DenseNet121(weights='imagenet', include_top=False)\n                #base_model = tf.keras.applications.ResNet50(weights='imagenet', include_top=False)\n                base_model.trainable = False\n\n                model = tf.keras.Sequential([\n                    tf.keras.layers.BatchNormalization(renorm=True),\n                    img_adjust_layer,\n                    base_model,\n                    tf.keras.layers.GlobalAveragePooling2D(),  ## Average\n                    tf.keras.layers.Dense(512, activation='relu', \n                    bias_regularizer=tf.keras.regularizers.L1L2(l1=0.01, l2=0.001)),\n                    Dropout(0.5),\n                    #tf.keras.layers.Lambda(lambda x: tf.math.l2_normalize(x, axis=1)), # L2 normalize embeddings\n                    #tf.keras.layers.Dense(1024, activation='relu', \n                    #bias_regularizer=tf.keras.regularizers.L1L2(l1=0.01, l2=0.001)),\n                    #Dropout(0.5),\n\n                    #tf.keras.layers.BatchNormalization(renorm=True),\n                    tf.keras.layers.Dense(1, activation='sigmoid')  \n                ])\n\n                model.compile(\n                    optimizer=tf.keras.optimizers.Adam(learning_rate=lr,epsilon=e),\n                    #loss='binary_crossentropy',#loss_func,  \n                    loss=tf.keras.losses.BinaryCrossentropy(from_logits=False),\n                    metrics=tf.keras.metrics.BinaryAccuracy()\n                )\n\n\n                unfreeze_model(model)\n\n                epochs = EPOCHS  # @param {type: \"slider\", min:8, max:50}\n\n                history = model.fit(train_dataset, \n                                    steps_per_epoch=STEPS_PER_EPOCH, \n                                    epochs=epochs,\n                                    #validation_split=0.1,\n                                    validation_data=valid_dataset,\n                                    #validation_steps=VALID_STEPS,\n                                   callbacks=my_callbacks)\n\n                history_frame = pd.DataFrame(history.history)\n                history_frame.loc[:, ['loss', 'val_loss']].plot()\n                history_frame.loc[:, ['binary_accuracy', 'val_binary_accuracy']].plot();\n\n\n\n","metadata":{"execution":{"iopub.status.busy":"2021-08-05T13:40:55.440048Z","iopub.execute_input":"2021-08-05T13:40:55.440508Z","iopub.status.idle":"2021-08-05T13:41:56.277605Z","shell.execute_reply.started":"2021-08-05T13:40:55.440378Z","shell.execute_reply":"2021-08-05T13:41:56.275289Z"},"trusted":true},"execution_count":null,"outputs":[]}]}