{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Some tips for using TPU\nThe competition ALASKA2 Image Steganalysis has closed. I spent a lot of time on this competition, and finally got 4th. Here is the [brief of my solution](https://www.kaggle.com/c/alaska2-image-steganalysis/discussion/168537). \n\nI publish this notebook for [Extra TPU Time](http://https://www.kaggle.com/tpu-prize) program. So this is not the whole solution, but the tips for using TPU.\n* The usage of .tfrec file\n* Converting DCT into YCbCr in training time with TF's way\n* Augmentation\n\nIf you feel it is useful, please upvote and leave your comment.","execution_count":null},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"!pip install -q git+https://github.com/qubvel/efficientnet","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\nimport json\nimport cv2\nimport glob\nimport datetime\nfrom tqdm import tqdm\nimport matplotlib.pyplot as plt\nfrom sklearn import metrics\n\nimport efficientnet.tfkeras as efn\nimport numpy as np\nimport random as rn\nimport pandas as pd\nfrom kaggle_datasets import KaggleDatasets\nfrom sklearn.model_selection import KFold, StratifiedKFold\n\nimport tensorflow as tf\nfrom tensorflow.keras import backend as K, layers, losses, optimizers, initializers, constraints, callbacks, models, utils, applications","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"try:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()  # TPU detection. No parameters necessary if TPU_NAME environment variable is set. On Kaggle this is always the case.\n    print('Running on TPU ', tpu.master())\nexcept ValueError:\n    tpu = None\n\nif tpu:\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n    strategy = tf.distribute.get_strategy() # default distribution strategy in Tensorflow. Works on CPU and single GPU.\n\nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"FOLD_ID = 0\nTRAIN_BATCH_SIZE = 128\nVALID_BATCH_SIZE = 128\nAUTO = tf.data.experimental.AUTOTUNE\noption_no_order = tf.data.Options()\noption_no_order.experimental_deterministic = False","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# The usage of .tfrec file\nThe bottleneck of using TPU is disk I/O.\n> The Tensor Processing Unit (TPU) hardware accelerators we will be using in this lab are very fast. The challenge is often to feed them data fast enough to keep them busy. Google Cloud Storage (GCS) is capable of sustaining very high throughput but as with all cloud storage systems, initiating a connection costs some network back and forth. Therefore, having our data stored as thousands of individual files is not ideal. We are going to batch them in a smaller number of files and use the power of tf.data.Dataset to read from multiple files in parallel.\n\nYou can find more detail from [here](https://codelabs.developers.google.com/codelabs/keras-flowers-data/#4).\n\nFor [some reason](https://www.kaggle.com/c/alaska2-image-steganalysis/discussion/150359#845167) I chose to use YCbCr as input. At beginning, I used [jpegio](https://www.kaggle.com/remicogranne/jpeg-explanations) to convert JPEG into DCT, The DCT data is too large(512x512x3x2 per image), so saved DCT as 16-bit PNG format(lossless) into .tfrec file. and convert DCT into YCbCr in training time. But I found it a little slow when I use tf.io.decode_png to decode 16-bit PNG. So the final solution is save the DCT data as two 8-bit PNGs. For the information about JPEG, DCT, YCbCr, and quantization, please refer to [Wikipedia](https://en.wikipedia.org/wiki/JPEG).\n\nThe whole dataset:\n* [Cover](https://www.kaggle.com/wuliaokaola/alaska2-ds-0512-cover) \n* [JMiPOD](https://www.kaggle.com/wuliaokaola/alaska2-ds-0512-jmipod)\n* [JUNIWARD](https://www.kaggle.com/wuliaokaola/alaska2-ds-0512-juniward)\n* [UERD](https://www.kaggle.com/wuliaokaola/alaska2-ds-0512-uerd)","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"GCS_DS_PATH_Cover = KaggleDatasets().get_gcs_path('alaska2-ds-0512-cover')\nGCS_DS_PATH_JMiPOD = KaggleDatasets().get_gcs_path('alaska2-ds-0512-jmipod')\nGCS_DS_PATH_JUNIWARD = KaggleDatasets().get_gcs_path('alaska2-ds-0512-juniward')\nGCS_DS_PATH_UERD = KaggleDatasets().get_gcs_path('alaska2-ds-0512-uerd')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Converting DCT into YCbCr in training time with TF's way\nHere is a notebook ([Faster YCbCr Decoding](https://www.kaggle.com/anjum48/faster-ycbcr-decoding)) do this in python's way. I rewrited it into TF's way. By using this method we can make it faster enough to do converting at training time.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# The quantization table for quality 95.\nq95 = np.array([[[ 2.,  2.,  2.],\n        [ 1.,  2.,  2.],\n        [ 1.,  2.,  2.],\n        [ 2.,  5.,  5.],\n        [ 2., 10., 10.],\n        [ 4., 10., 10.],\n        [ 5., 10., 10.],\n        [ 6., 10., 10.]],\n\n       [[ 1.,  2.,  2.],\n        [ 1.,  2.,  2.],\n        [ 1.,  3.,  3.],\n        [ 2.,  7.,  7.],\n        [ 3., 10., 10.],\n        [ 6., 10., 10.],\n        [ 6., 10., 10.],\n        [ 6., 10., 10.]],\n\n       [[ 1.,  2.,  2.],\n        [ 1.,  3.,  3.],\n        [ 2.,  6.,  6.],\n        [ 2., 10., 10.],\n        [ 4., 10., 10.],\n        [ 6., 10., 10.],\n        [ 7., 10., 10.],\n        [ 6., 10., 10.]],\n\n       [[ 1.,  5.,  5.],\n        [ 2.,  7.,  7.],\n        [ 2., 10., 10.],\n        [ 3., 10., 10.],\n        [ 5., 10., 10.],\n        [ 9., 10., 10.],\n        [ 8., 10., 10.],\n        [ 6., 10., 10.]],\n\n       [[ 2., 10., 10.],\n        [ 2., 10., 10.],\n        [ 4., 10., 10.],\n        [ 6., 10., 10.],\n        [ 7., 10., 10.],\n        [11., 10., 10.],\n        [10., 10., 10.],\n        [ 8., 10., 10.]],\n\n       [[ 2., 10., 10.],\n        [ 4., 10., 10.],\n        [ 6., 10., 10.],\n        [ 6., 10., 10.],\n        [ 8., 10., 10.],\n        [10., 10., 10.],\n        [11., 10., 10.],\n        [ 9., 10., 10.]],\n\n       [[ 5., 10., 10.],\n        [ 6., 10., 10.],\n        [ 8., 10., 10.],\n        [ 9., 10., 10.],\n        [10., 10., 10.],\n        [12., 10., 10.],\n        [12., 10., 10.],\n        [10., 10., 10.]],\n\n       [[ 7., 10., 10.],\n        [ 9., 10., 10.],\n        [10., 10., 10.],\n        [10., 10., 10.],\n        [11., 10., 10.],\n        [10., 10., 10.],\n        [10., 10., 10.],\n        [10., 10., 10.]]])\n\n# The quantization table for quality 90.\nq90 = np.array([[[ 3.,  3.,  3.],\n        [ 2.,  4.,  4.],\n        [ 2.,  5.,  5.],\n        [ 3.,  9.,  9.],\n        [ 5., 20., 20.],\n        [ 8., 20., 20.],\n        [10., 20., 20.],\n        [12., 20., 20.]],\n\n       [[ 2.,  4.,  4.],\n        [ 2.,  4.,  4.],\n        [ 3.,  5.,  5.],\n        [ 4., 13., 13.],\n        [ 5., 20., 20.],\n        [12., 20., 20.],\n        [12., 20., 20.],\n        [11., 20., 20.]],\n\n       [[ 3.,  5.,  5.],\n        [ 3.,  5.,  5.],\n        [ 3., 11., 11.],\n        [ 5., 20., 20.],\n        [ 8., 20., 20.],\n        [11., 20., 20.],\n        [14., 20., 20.],\n        [11., 20., 20.]],\n\n       [[ 3.,  9.,  9.],\n        [ 3., 13., 13.],\n        [ 4., 20., 20.],\n        [ 6., 20., 20.],\n        [10., 20., 20.],\n        [17., 20., 20.],\n        [16., 20., 20.],\n        [12., 20., 20.]],\n\n       [[ 4., 20., 20.],\n        [ 4., 20., 20.],\n        [ 7., 20., 20.],\n        [11., 20., 20.],\n        [14., 20., 20.],\n        [22., 20., 20.],\n        [21., 20., 20.],\n        [15., 20., 20.]],\n\n       [[ 5., 20., 20.],\n        [ 7., 20., 20.],\n        [11., 20., 20.],\n        [13., 20., 20.],\n        [16., 20., 20.],\n        [21., 20., 20.],\n        [23., 20., 20.],\n        [18., 20., 20.]],\n\n       [[10., 20., 20.],\n        [13., 20., 20.],\n        [16., 20., 20.],\n        [17., 20., 20.],\n        [21., 20., 20.],\n        [24., 20., 20.],\n        [24., 20., 20.],\n        [20., 20., 20.]],\n\n       [[14., 20., 20.],\n        [18., 20., 20.],\n        [19., 20., 20.],\n        [20., 20., 20.],\n        [22., 20., 20.],\n        [20., 20., 20.],\n        [21., 20., 20.],\n        [20., 20., 20.]]])\n\n# The quantization table for quality 75.\nq75 = np.array([[[ 8.,  9.,  9.],\n        [ 6.,  9.,  9.],\n        [ 5., 12., 12.],\n        [ 8., 24., 24.],\n        [12., 50., 50.],\n        [20., 50., 50.],\n        [26., 50., 50.],\n        [31., 50., 50.]],\n\n       [[ 6.,  9.,  9.],\n        [ 6., 11., 11.],\n        [ 7., 13., 13.],\n        [10., 33., 33.],\n        [13., 50., 50.],\n        [29., 50., 50.],\n        [30., 50., 50.],\n        [28., 50., 50.]],\n\n       [[ 7., 12., 12.],\n        [ 7., 13., 13.],\n        [ 8., 28., 28.],\n        [12., 50., 50.],\n        [20., 50., 50.],\n        [29., 50., 50.],\n        [35., 50., 50.],\n        [28., 50., 50.]],\n\n       [[ 7., 24., 24.],\n        [ 9., 33., 33.],\n        [11., 50., 50.],\n        [15., 50., 50.],\n        [26., 50., 50.],\n        [44., 50., 50.],\n        [40., 50., 50.],\n        [31., 50., 50.]],\n\n       [[ 9., 50., 50.],\n        [11., 50., 50.],\n        [19., 50., 50.],\n        [28., 50., 50.],\n        [34., 50., 50.],\n        [55., 50., 50.],\n        [52., 50., 50.],\n        [39., 50., 50.]],\n\n       [[12., 50., 50.],\n        [18., 50., 50.],\n        [28., 50., 50.],\n        [32., 50., 50.],\n        [41., 50., 50.],\n        [52., 50., 50.],\n        [57., 50., 50.],\n        [46., 50., 50.]],\n\n       [[25., 50., 50.],\n        [32., 50., 50.],\n        [39., 50., 50.],\n        [44., 50., 50.],\n        [52., 50., 50.],\n        [61., 50., 50.],\n        [60., 50., 50.],\n        [51., 50., 50.]],\n\n       [[36., 50., 50.],\n        [46., 50., 50.],\n        [48., 50., 50.],\n        [49., 50., 50.],\n        [56., 50., 50.],\n        [50., 50., 50.],\n        [52., 50., 50.],\n        [50., 50., 50.]]])\n\n# The image size is 512x512, so tile quantization table into 64x64\n\nq95 = tf.cast(tf.tile(q95, (64, 64, 1)), dtype=tf.float32)\nq90 = tf.cast(tf.tile(q90, (64, 64, 1)), dtype=tf.float32)\nq75 = tf.cast(tf.tile(q75, (64, 64, 1)), dtype=tf.float32)\n\n[col, row] = np.meshgrid(range(8), range(8))\nT = 0.5 * np.cos(np.pi * (2 * col + 1) * row / (2 * 8))\nT[0, :] = T[0, :] / np.sqrt(2)\nbroadcast_dims = (64, 8, 64, 8)\nt = np.broadcast_to(T.reshape(1, 8, 1, 8), broadcast_dims)\nt = tf.constant(t, dtype=tf.float32)\n\na = tf.transpose(t, (0, 2, 3, 1))\nc = tf.transpose(t, (0, 2, 1, 3))\n\n# Conver DCT into YCbCr\ndef dct_2_ycc(dct):\n    dct = tf.reshape(dct, (64, 8, 64, 8))\n    dct = tf.transpose(dct, (0, 2, 1, 3))\n    ycc = tf.matmul(tf.matmul(a, dct), c)\n    ycc = tf.transpose(ycc, (0, 2, 1, 3))\n    ycc = tf.reshape(ycc, (512, 512))\n    return ycc","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Augmentation\nIn this competition we need make model focus on wavelets of small area (24x24 / 32x32 / 40x40) but the contents of image. The method is doing random grid shuffle. Keeping the boarder(16 pixels) because UERD always change this area.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2006644%2F86c070756b810c74293e6ada0707b630%2Fshuffle.png?generation=1595300985030345&alt=media)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def grid_shuffle(image):\n    b = 2 # border size: 8x2=16\n    s = 5 # grid size: 8x5=40\n    \n    m = tf.ones([512 - b * 2 * 8, 512 - b * 2 * 8, 3], dtype=tf.float32)\n    m = tf.image.pad_to_bounding_box(m, b * 8, b * 8, 512, 512)\n    t = []\n    for i in range(b, 64-b, s):\n        for j in range(b, 64-b, s):\n            t.append([image[i*8:i*8+8*s, j*8:j*8+8*s, :]])\n    rn.shuffle(t)\n    t = tf.concat(t, axis=0)\n    t = tf.reshape(t, ((64-2*b)//s, (64-2*b)//s, 8*s, 8*s, 3))\n    t = tf.transpose(t, (4, 0, 2, 1, 3))\n    t = tf.reshape(t, (3, 512 - b * 2 * 8, 512 - b * 2 * 8))\n    \n    t = tf.transpose(t, (1, 2, 0))\n    t = tf.image.pad_to_bounding_box(t, b * 8, b * 8, 512, 512)\n    image = image * (1 - m) + t * m\n    \n    return image","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def read_tfrecord(data):\n    features = {\n        \"image\": tf.io.FixedLenFeature([], tf.string), \n        \"dct1\": tf.io.FixedLenFeature([], tf.string), \n        \"dct2\": tf.io.FixedLenFeature([], tf.string), \n        \"q\": tf.io.FixedLenFeature([], tf.int64),  \n        \"mask\": tf.io.FixedLenFeature([], tf.string), \n        \"label\": tf.io.FixedLenFeature([], tf.int64),  \n    }\n\n    # decode the TFRecord\n    tf_record = tf.io.parse_single_example(data, features)\n    \n    q = tf.cast(tf_record['q'], tf.float32)\n    label = tf.cast(tf_record['label'], tf.float32)\n    \n    dct1 = tf.image.decode_png(tf_record['dct1'])\n    dct2 = tf.image.decode_png(tf_record['dct2'])\n    dct = tf.cast(dct1, tf.int32) * 256 + tf.cast(dct2, tf.int32)\n    dct = tf.cast(dct, tf.int16)\n    \n    dct = tf.cast(dct, tf.float32)\n    dct = tf.case([(tf.math.equal(q, tf.constant(0, dtype=tf.float32)), lambda: dct * q95), \n                (tf.math.equal(q, tf.constant(1, dtype=tf.float32)), lambda: dct * q90),\n                (tf.math.equal(q, tf.constant(2, dtype=tf.float32)), lambda: dct * q75),\n                ])\n    \n    ycc = tf.stack([dct_2_ycc(dct[:,:,0]), dct_2_ycc(dct[:,:,1]), dct_2_ycc(dct[:,:,2])], axis=2) / 255.\n    \n    mask = tf.image.decode_png(tf_record['mask'], channels=1)\n    return (ycc, q), (label, mask)\n\ndef data_augment(i, o):\n    image, q = i\n    label, mask = o\n\n    c0 = tf.constant(0)\n\n    image = tf.image.random_flip_left_right(image)\n    image = tf.image.random_flip_up_down(image)\n    \n    k = tf.random.uniform([], minval=0, maxval=2, dtype=tf.dtypes.int32)\n    image = tf.cond(tf.equal(k, c0), lambda:(image), lambda:(grid_shuffle(image)))\n    \n    k = tf.random.uniform([], minval=0, maxval=2, dtype=tf.dtypes.int32)\n    image = tf.cond(tf.equal(k, c0), lambda:(image), lambda:(-image))\n    \n    image = tf.reshape(image, [512, 512, 3])\n    return (image, q), (label, mask)\n\ndef mask_2_payload(i, o):\n    image, q = i\n    label, mask = o\n    \n    mask = tf.cast(mask, tf.float32)\n    f0 = tf.constant(0, dtype=tf.float32)\n    label = tf.cond(tf.equal(K.sum(mask), f0), lambda: f0, lambda: label)\n    payload = K.mean(mask)\n    return (image, q), (label, payload)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_train_dataset(ids):\n    filenames = []\n    for i in ids:\n        filenames.append(GCS_DS_PATH_Cover + \"/Cover_{:03d}.tfrec\".format(i))\n        filenames.append(GCS_DS_PATH_JMiPOD + \"/JMiPOD_{:03d}.tfrec\".format(i))\n        filenames.append(GCS_DS_PATH_JUNIWARD + \"/JUNIWARD_{:03d}.tfrec\".format(i))\n        filenames.append(GCS_DS_PATH_UERD + \"/UERD_{:03d}.tfrec\".format(i))\n    rn.shuffle(filenames)\n    dataset = tf.data.TFRecordDataset(filenames, num_parallel_reads=AUTO)\n    dataset = dataset.with_options(option_no_order)\n    dataset = dataset.map(read_tfrecord, num_parallel_calls=AUTO)\n    dataset = dataset.map(data_augment, num_parallel_calls=AUTO)\n    dataset = dataset.map(mask_2_payload, num_parallel_calls=AUTO)\n    dataset = dataset.prefetch(AUTO)\n    dataset = dataset.shuffle(1024*1)\n    dataset = dataset.repeat()\n    dataset = dataset.batch(TRAIN_BATCH_SIZE)\n    return dataset\n\ndef get_valid_dataset(ids):\n    filenames = []\n    for i in ids:\n        filenames.append(GCS_DS_PATH_Cover + \"/Cover_{:03d}.tfrec\".format(i))\n        filenames.append(GCS_DS_PATH_JMiPOD + \"/JMiPOD_{:03d}.tfrec\".format(i))\n        filenames.append(GCS_DS_PATH_JUNIWARD + \"/JUNIWARD_{:03d}.tfrec\".format(i))\n        filenames.append(GCS_DS_PATH_UERD + \"/UERD_{:03d}.tfrec\".format(i))\n    dataset = tf.data.TFRecordDataset(filenames, num_parallel_reads=AUTO)\n    dataset = dataset.map(read_tfrecord, num_parallel_calls=AUTO)\n    dataset = dataset.map(mask_2_payload, num_parallel_calls=AUTO)\n    dataset = dataset.prefetch(AUTO)\n    dataset = dataset.batch(VALID_BATCH_SIZE)\n    return dataset","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fid = 0\nn = len(glob.glob('../input/alaska2-ds-0512-cover/*.tfrec'))\nids = np.arange(n)\nkf = KFold(n_splits=5)\nfor train_index, valid_index in kf.split(ids):\n    train_dataset = get_train_dataset(train_index)    \n    valid_dataset = get_valid_dataset(valid_index)\n    if fid == FOLD_ID:\n        break\n    fid += 1","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Some exaples (50% grid shuffled): ","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"for (images, qs), (labels, payloads) in train_dataset:\n    for i in range(10):\n        plt.figure(figsize=(10, 10))\n        plt.imshow(images[i][:, :, 0])\n        plt.show()\n    break","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def create_model():\n    q_input = layers.Input(shape=(1,))\n    x_input = layers.Input(shape=(512, 512, 3))\n    x = x_input\n    q = tf.reshape(q_input, (-1,1,1,1))\n    q = tf.tile(q, (1,512,512,1))\n    x = layers.Concatenate()([x, q])\n    \n    mt = efn.EfficientNetB0(\n                input_tensor=x,\n                weights=None,\n                include_top=False\n            )\n    x = mt.layers[-1].output\n    \n    gp = layers.GlobalAveragePooling2D()(x)\n    l = layers.Dense(128, activation='relu')(gp)\n    l = layers.Dense(4, activation='softmax', name='label')(l)\n    \n    p = layers.Dense(128, activation='relu')(gp)\n    p = layers.Dense(1, activation='relu', name='payload')(p)\n    \n    model = models.Model(inputs=[x_input, q_input], outputs=[l, p])\n    return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"with strategy.scope():\n    lr_schedule = optimizers.schedules.ExponentialDecay(\n        initial_learning_rate=0.001,\n        decay_steps=int(75000*4*0.8),\n        decay_rate=0.98)\n    optimizer = optimizers.Adamax(learning_rate=lr_schedule)\n    model = create_model()\n    model.compile(\n            optimizer=optimizer,\n            loss = {'label': 'sparse_categorical_crossentropy', 'payload':'mae'},\n            loss_weights = {'label': 1, 'payload':1},\n            metrics={'label': 'sparse_categorical_accuracy', 'payload':'accuracy'}\n        )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class ValidCB(callbacks.Callback):\n    def on_epoch_end(self, epoch, logs={}):\n        print('Calculating metric...')\n        \n\nmodel.fit(train_dataset, \n        steps_per_epoch=int(75000*4*0.8/TRAIN_BATCH_SIZE), \n        verbose=1,\n        epochs=1,\n        initial_epoch=0,\n#         validation_data=valid_dataset,\n        callbacks=[ValidCB()]\n        )","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"By using TPU v3-8, you can train one epoch (512x512, 75000*4*0.8=240000 images, EfficientNetB0) in about 15 mins including DCT-YCbCr converting and some augmentations.\n\nIf you feel this notebook is useful, please upvote and leave your comment. Thanks.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}