{"cells":[{"metadata":{},"cell_type":"markdown","source":"## About this kernel\n\nMost of this notebook is blatantly stolen from https://www.kaggle.com/xhlulu/alaska2-efficientnet-on-tpus\n\nI've uploaded some tfrecord datasets. All permutations of the same cover image should be separated into each tfrecord file. So two tfrecord files should never have any image that matches the cover. This should make validation better, because you don't want the model to see the cover in train and then the stega image in validation.\n\nThe main contribution here is the 6 alaska datasets that have been turned into TFRecords and a cache trick. \n\nBecause TPUs have limited ram, if you decode the JPEG and then cache it, you will run out of ram because you are caching an uncompressed JPEG, which is massive. Instead, cache the JPEG binary and decode it. TPUs have alot of CPU cores for that job and your main bottleneck is usually file access. \n\nAlso, the tfrecords stream better than 300k images as separate files. \n\nTweak it yourself and experiment!\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install -q efficientnet","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":true},"cell_type":"code","source":"import math, re, os\n\nimport numpy as np\nimport pandas as pd\nfrom matplotlib import pyplot as plt\nfrom kaggle_datasets import KaggleDatasets\nimport tensorflow as tf\nimport tensorflow.keras.layers as L\nimport efficientnet.tfkeras as efn\nfrom sklearn import metrics\nfrom sklearn.model_selection import train_test_split\nimport cv2","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## TPU Strategy and other configs ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Detect hardware, return appropriate distribution strategy\ntry:\n    # TPU detection. No parameters necessary if TPU_NAME environment variable is\n    # set: this is always the case on Kaggle.\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n    print('Running on TPU ', tpu.master())\nexcept ValueError:\n    tpu = None\n\nif tpu:\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n    # Default distribution strategy in Tensorflow. Works on CPU and single GPU.\n    strategy = tf.distribute.get_strategy()\n\nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# For tf.dataset\nAUTO = tf.data.experimental.AUTOTUNE\n\n# Data access\n\nGCS_DS_PATH = KaggleDatasets().get_gcs_path('alaska2-image-steganalysis')\n\n\n# Configuration\nEPOCHS = 30\nBATCH_SIZE = 32 * strategy.num_replicas_in_sync","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import glob\nfrom tqdm.notebook import tqdm \n\nall_records = []\nfor i in tqdm(range(7)):\n    \n    records = glob.glob('/kaggle/input/alaska0%i/*' % i )\n    TF_REC_DS_PATH = KaggleDatasets().get_gcs_path('alaska0%i' % i )\n    records = [os.path.join(TF_REC_DS_PATH, record[-14:]) for record in records]\n    all_records += records \n    \nall_records = [record for record in all_records if int(record.split('/')[-1].split('.')[0]) < 200]\n    \ntrain_records, valid_records = train_test_split(all_records, test_size = 0.1)\n\nlen(all_records)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Create Dataset objects\n\nA `tf.data.Dataset` object is needed in order to run the model smoothly on the TPUs. Here, I heavily trim down [my previous kernel](https://www.kaggle.com/xhlulu/flowers-tpu-concise-efficientnet-b7), which was inspired by [Martin's kernel](https://www.kaggle.com/mgornergoogle/getting-started-with-100-flowers-on-tpu).","execution_count":null},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"# consider random crop of the data rather than resize\n\nfeature_description = {\n    'bits': tf.io.FixedLenFeature([], tf.string, default_value=''),\n    'label' : tf.io.FixedLenFeature([], tf.int64, default_value=0),\n}\n\ndef _parse_function(example_proto):\n  # Parse the input `tf.Example` proto using the dictionary above.\n  return tf.io.parse_single_example(example_proto, feature_description)\n\ndef split(features):\n    return features['bits'], features['label']\n\n\ndef read_image(filename, label=None):\n    bits = tf.io.read_file(filename)\n    if label is None:\n        return bits\n    else:\n        return bits, label\n\nbinary = False\ndef convert_label(label):\n    if binary:\n        return tf.cast(label != 0, tf.int64)\n    else:\n        return label\n    \n    \ndef decode_image(bits, label=None, image_size=(512, 512)):\n    image = tf.image.decode_jpeg(bits, channels=3)\n    image = tf.cast(image, tf.float32) / 255.0\n    image = tf.image.resize(image, image_size)\n    \n    if label is None:\n        return image\n    else:\n        return image, convert_label(label)\n    \n\n\ndef data_augment(image, label=None):\n    image = tf.image.random_flip_left_right(image)\n    image = tf.image.random_flip_up_down(image)\n    \n    if label is None:\n        return image\n    else:\n        return image, label","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\ntrain_dataset = (\n    tf.data.TFRecordDataset(train_records, num_parallel_reads=AUTO)\n    .map(_parse_function, num_parallel_calls=AUTO)\n    .map(split, num_parallel_calls=AUTO)\n    .cache()\n    .map(decode_image, num_parallel_calls=AUTO)\n    .map(data_augment, num_parallel_calls = AUTO)\n    .repeat()\n    .shuffle(1024)\n    .batch(BATCH_SIZE)\n    .prefetch(AUTO)\n)\n\n\nvalid_dataset = (\n    tf.data.TFRecordDataset(valid_records, num_parallel_reads=AUTO)\n    .map(_parse_function, num_parallel_calls=AUTO)\n    .map(split, num_parallel_calls=AUTO)\n    .cache()\n    .map(decode_image, num_parallel_calls=AUTO)\n    .shuffle(1024)\n    .batch(BATCH_SIZE)\n    .prefetch(AUTO)\n)\n\nvalid_dataset.take(1)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub = pd.read_csv('/kaggle/input/alaska2-image-steganalysis/sample_submission.csv')\ndef append_path(pre):\n    return np.vectorize(lambda file: os.path.join(GCS_DS_PATH, pre, file))\ntest_paths = append_path('Test')(sub.Id.values)\n\ntest_dataset = (\n    tf.data.Dataset\n    .from_tensor_slices(test_paths)\n    .map(read_image, num_parallel_calls=AUTO)\n    .map(decode_image, num_parallel_calls=AUTO)\n    .batch(BATCH_SIZE)\n)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Modelling","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Helper Functions","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def build_lrfn(lr_start=0.00001, lr_max=1e-3, \n               lr_min=0.000001, lr_rampup_epochs=2, \n               lr_sustain_epochs=0, lr_exp_decay=.8):\n    lr_max = lr_max * strategy.num_replicas_in_sync\n\n    def lrfn(epoch):\n        if epoch < lr_rampup_epochs:\n            lr = (lr_max - lr_start) / lr_rampup_epochs * epoch + lr_start\n        elif epoch < lr_rampup_epochs + lr_sustain_epochs:\n            lr = lr_max\n        else:\n            lr = (lr_max - lr_min) * lr_exp_decay**(epoch - lr_rampup_epochs - lr_sustain_epochs) + lr_min\n        return lr\n    \n    return lrfn","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Load Model into TPU","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"if binary:\n    out = 1\n    loss_fn = tf.keras.losses.BinaryCrossentropy\n    metrics = ['accuracy']\nelse:\n    out = 4\n    loss_fn = tf.keras.losses.SparseCategoricalCrossentropy\n    metrics = ['sparse_categorical_accuracy']\n\nwith strategy.scope():\n    model = tf.keras.Sequential([\n        efn.EfficientNetB2(\n            input_shape=(512, 512, 3),\n            weights='imagenet',\n            include_top=False\n        ),\n        L.GlobalAveragePooling2D(),\n        L.Dense(out, activation='sigmoid')\n    ])\n\n#     optimizer = tf.keras.optimizers.Adam(learning_rate = 1e-3 * 8)\n    \n    model.compile(\n        optimizer='adam',\n        loss = loss_fn(),\n        metrics=metrics \n    )\n    model.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Start training","execution_count":null},{"metadata":{"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"lrfn = build_lrfn(lr_start = 1e-5 ,lr_max = 1e-3 * 4, lr_rampup_epochs=4, \n               lr_sustain_epochs=8, lr_exp_decay=.9)\n\nlr_schedule = tf.keras.callbacks.LearningRateScheduler(lrfn, verbose=1)\n\n# STEPS_PER_EPOCH = train_labels.shape[0] // BATCH_SIZE\n\nSTEPS_PER_EPOCH = 100\n\nhistory = model.fit(\n    train_dataset, \n    epochs=EPOCHS, \n    callbacks=[lr_schedule],\n    steps_per_epoch=STEPS_PER_EPOCH,\n    validation_data=valid_dataset,\n#     validation_steps = 120\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model.save(\"model.h5\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Evaluation","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Unhide below to see helper function `display_training_curves`:","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Submission","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"\nif not binary: \n\n    preds = model.predict(test_dataset, verbose=1)\n    y_pred = 1 - tf.nn.softmax(preds)[:, 0]\n    sub.Label = y_pred\n\n\nsub.to_csv('submission.csv', index=False)\nsub.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# np.mean(y_pred)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}