{"cells":[{"metadata":{},"cell_type":"markdown","source":"This notebook is oriented to show how to write a kaggle compatible tensorflow TPU training module with insights and human-readable explanations.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install -q efficientnet","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\n\nimport tensorflow as tf\nfrom tensorflow.keras import layers,models\n\nimport matplotlib.pyplot as plt\n\nfrom functools import partial\nfrom sklearn.model_selection import train_test_split\n\nfrom kaggle_datasets import KaggleDatasets\n\nimport efficientnet.tfkeras as efn","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The first step is obtaining train, validation and test filenames. As the TPU training requires oneself to load the data through GCS we have to use `KaggleDatasets().get_gcs_path()` to obtain the GCS tfrecord filenames for further parsing.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"GCS_PATH = KaggleDatasets().get_gcs_path()\nTRAINING_FILENAMES, VALIDATION_FILENAMES = train_test_split(\n    tf.io.gfile.glob(GCS_PATH + '/tfrecords/train*.tfrec'),\n    test_size=0.2, random_state=11\n)\nTEST_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/tfrecords/test*.tfrec')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Second step is actually initalizing TPUs.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"try:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n    print('Device:', tpu.master())\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nexcept:\n    strategy = tf.distribute.get_strategy()\nprint('Number of replicas:', strategy.num_replicas_in_sync)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"tensorflow AUTOTUNE allows the system to automatically tune the required parameters for better performance.\n\nmultiplying batch size with strategy.num_replicas_in_sync is in general shown to reduce the TPU waiting time and provide continious TPU pipeline. Large testing batch size is for faster proccessing thus reducing the computation time.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"AUTOTUNE=tf.data.experimental.AUTOTUNE\nBATCH_SIZE_TRAIN=16*strategy.num_replicas_in_sync\nBATCH_SIZE_TEST=1024\nEPOCHS=20","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The next step is the most overwhelming one.\n\nThe first function is view_image which simply allows me to look at the images in a single batch of train/val/test sets, its function is to make sure that everything works okay\n\nThe second function is augmentations which is used to do online augmentation and modify images on the fly. Difference between online and offline augmentation is that online augmentation does not require huge amound of space as the images are not stored, moreover, the performance of both augmentations is comparably the same.\n\nnext functions are just a standart way to parse the tensorflow tfrecords format to convert into image and pass to augmentations\n\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def view_image(batch,batch_size):\n    \n    try:\n        image,label=next(iter(batch))\n        image=image.numpy()\n        label=label.numpy()\n    except:\n        image=next(iter(batch))\n        image=image.numpy()\n    \n    fig=plt.figure(figsize=(22,22))\n    \n    for i in range(batch_size):\n            ax=fig.add_subplot(4,8,i+1)\n            ax.imshow(image[i])\n            try:\n                ax.set_title(f\"Label: {label[i]}\")\n            except:\n                pass\n\ndef augmentations(image,label,augment):\n    \n    if augment:\n        image=tf.image.random_flip_left_right(image)\n        image=tf.image.random_flip_up_down(image)\n        image=tf.image.random_hue(image,0.3)\n        image=tf.image.random_contrast(image,0.2,0.5)\n        image=tf.image.random_brightness(image,0.3)\n        image=tf.image.random_saturation(image,5,10)\n        \n    image=tf.cast(image,tf.float32)/255.0\n    image=tf.image.resize(image,[256,256])\n    return image,label\n\ndef standart_parse(image):\n    image=tf.cast(image,tf.float32)/255.0\n    image=tf.image.resize(image,[256,256])\n    return image\n\ndef decode_image(image):\n    image=tf.image.decode_jpeg(image,channels=3)\n    return image\n\ndef read_tfrecord(record, labeled):\n    # scheme for tfrecord parsing\n    \n    if labeled:\n        TFRECORD_SCHEME={\n\n            \"image\": tf.io.FixedLenFeature([],tf.string),\n            \"target\": tf.io.FixedLenFeature([],tf.int64)\n        }\n    else:\n        TFRECORD_SCHEME={\n            \"image\": tf.io.FixedLenFeature([],tf.string),\n            \"image_name\": tf.io.FixedLenFeature([],tf.string)\n        }        \n    \n    datapoint = tf.io.parse_single_example(record,TFRECORD_SCHEME)\n    image = decode_image(datapoint[\"image\"])\n    if labeled:\n        label = datapoint[\"target\"]\n        return image,label\n    else:\n        return image\n\ndef load_dataset(filenames,labeled,batch_size,augment=True):\n    \n    options=tf.data.Options()\n    options.experimental_deterministic=False\n    \n    dataset=tf.data.TFRecordDataset(filenames,num_parallel_reads=AUTOTUNE)\n    \n    dataset=dataset.with_options(options)\n    \n    dataset=dataset.map(partial(read_tfrecord,labeled=labeled),num_parallel_calls=AUTOTUNE)\n    \n    if labeled:\n        dataset=dataset.map(partial(augmentations,augment=augment),num_parallel_calls=AUTOTUNE).batch(batch_size).prefetch(AUTOTUNE)\n    else:\n        dataset=dataset.map(standart_parse,num_parallel_calls=AUTOTUNE).batch(batch_size).prefetch(AUTOTUNE)\n        \n    return dataset\n    ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next simply use our functions to create the datasets that is compatible with TPU.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"melanoma_train=load_dataset(TRAINING_FILENAMES,labeled=True,batch_size=BATCH_SIZE_TRAIN)\nmelanoma_val=load_dataset(VALIDATION_FILENAMES,labeled=True,augment=False,batch_size=BATCH_SIZE_TEST)\nmelanoma_test=load_dataset(TEST_FILENAMES,labeled=False,augment=False,batch_size=BATCH_SIZE_TEST)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next step is initlialization of our efficientnet module ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"\nwith strategy.scope():\n    B7efn=efn.EfficientNetB7(weights=\"noisy-student\",include_top=False,input_shape=[256,256,3])\n    model = models.Sequential([\n        B7efn,\n        layers.GlobalAveragePooling2D(),\n        layers.Dropout(0.1),\n        layers.Dense(128,activation=\"relu\"),\n        layers.Dropout(0.1),\n        layers.Dense(64,activation=\"relu\"),\n        layers.Dense(1,activation=\"sigmoid\")\n    ])\n    \n\n# testing=models.Sequential([\n\n\n#         layers.Conv2D(32, (3, 3), activation='relu', input_shape=[256,256,3]),\n#         layers.MaxPooling2D((2, 2)),\n#         layers.Conv2D(64, (3, 3), activation='relu'),\n#         layers.MaxPooling2D((2, 2)),\n#         layers.Conv2D(64, (3, 3), activation='relu'),\n#         layers.Flatten(),\n#         layers.Dense(64, activation='relu'),\n#         layers.Dense(1, activation='sigmoid')])\n\nmodel.compile(optimizer='adam', loss=tf.keras.losses.BinaryCrossentropy(), metrics=\"accuracy\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"And lastly we create save_the_best_model and learning rate decay modules to increase our predictive capabilities and avoid nasty zero gradient points.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"model_checkpoint=tf.keras.callbacks.ModelCheckpoint(filepath=\"best_model.h5\",save_weights_only=True,save_best_only=True,monitor=\"val_accuracy\",mode=\"max\")\nlrdecay=tf.keras.callbacks.ReduceLROnPlateau(patience=3,verbose=1)\n\nmodel.fit(melanoma_train,validation_data=melanoma_val, epochs=EPOCHS, callbacks=[model_checkpoint,lrdecay])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}