{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Splitting TensorFlow Dataset for Validation \n\nThis notebook shows a way to split a TensorFlow Dataset into two, for training and validation. Only training dataset is provided in this competition. So, to evaluate models, the provided dataset needs to be splitted into two.\n\nTo split dataset, [sklearn.model_selection.train_test_split()](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html) is used. This can maintain the ratio of the target value. About 98% is 0 and 1 is less than 2%. By using the result of train_test_split(), the provied TensorFlow dataset is filtered and splitted into two datasets.\n\n# Table of Contents\n\n* <a href=\"#Preparation\">Preparation</a>\n* <a href=\"#Dataset1\">Dataset 1</a>\n* <a href=\"#SplittingTensorflowDataset\">Splitting Tensorflow Dataset</a>\n* <a href=\"#Dataset2\">Dataset 2</a>\n* <a href=\"#Model\">Model</a>\n* <a href=\"#Evaluation\">Evaluation</a>\n    * <a href=\"#LossAucPrecisionRecall\">Loss, Auc, Precision, Recall</a>\n    * <a href=\"#Roc\">Roc</a>\n    * <a href=\"#ConfusionMatrix\">Confusion Matrix</a>\n* <a href=\"#Predictions\">Predictions</a>","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# References\n\n* [Getting started with 100+ flowers on TPU](https://www.kaggle.com/mgornergoogle/getting-started-with-100-flowers-on-tpu)\n* [Melanoma TPU EfficientNet B5_dense_head](https://www.kaggle.com/ajaykumar7778/melanoma-tpu-efficientnet-b5-dense-head)\n* [TFRecords 768x768, 512x512, 384x384 With Meta Data](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/155579)\n* [Rotation Augmentation GPU/TPU - [0.96+]](https://www.kaggle.com/cdeotte/rotation-augmentation-gpu-tpu-0-96)\n* [Classification on imbalanced data](https://www.tensorflow.org/tutorials/structured_data/imbalanced_data)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"<a name=\"Preparation\"></a>\n# Preparation","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install -q efficientnet","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport os\nprint(os.listdir('../input'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import tensorflow as tf\n\nprint(tf.__version__)\nprint(tf.keras.__version__)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"# Detect hardware, return appropriate distribution strategy\ntry:\n    # TPU detection. No parameters necessary if TPU_NAME environment variable is set.\n    # On Kaggle this is always the case.\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n    print('Running on TPU ', tpu.master())\nexcept ValueError:\n    tpu = None\n\nif tpu:\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n    # default distribution strategy in Tensorflow. Works on CPU and single GPU.\n    strategy = tf.distribute.get_strategy()\n\nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"For training data, Chris Deotte's 512x512 image with metadata is used.\nThanks a lot for providing this dataset!\n  - Discussion: [TFRecords 768x768, 512x512, 384x384 With Meta Data](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/155579) \n  - Dataset: [Melanoma TFRecords 512x512](https://www.kaggle.com/cdeotte/melanoma-512x512/settings)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from kaggle_datasets import KaggleDatasets\n\n# you can list the bucket with \"!gsutil ls $GCS_DS_PATH\"\nGCS_DS_PATH = KaggleDatasets().get_gcs_path('melanoma-512x512')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"EPOCHS = 25\nIMAGE_SIZE = [512, 512]\nBATCH_SIZE = 16 * strategy.num_replicas_in_sync\nGCS_PATH = GCS_DS_PATH\n\nTRAINING_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/train*.tfrec')\nTEST_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/test*.tfrec')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a name=\"Dataset1\"></a>\n# Dataset 1","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"AUTO = tf.data.experimental.AUTOTUNE\n\ndef decode_image(image_data):\n    image = tf.image.decode_jpeg(image_data, channels=3)\n    image = tf.cast(image, tf.float32) / 255.0  # convert image to floats in [0, 1] range\n    image = tf.reshape(image, [*IMAGE_SIZE, 3]) # explicit size needed for TPU\n    return image\n\ndef read_labeled_tfrecord(example):\n    LABELED_TFREC_FORMAT = {\n        \"image\": tf.io.FixedLenFeature([], tf.string), # tf.string means bytestring\n        \"image_name\": tf.io.FixedLenFeature([], tf.string),  # shape [] means single element\n        \"patient_id\": tf.io.FixedLenFeature([], tf.int64),\n        \"sex\": tf.io.FixedLenFeature([], tf.int64),\n        \"age_approx\": tf.io.FixedLenFeature([], tf.int64),\n        \"anatom_site_general_challenge\": tf.io.FixedLenFeature([], tf.int64),\n        \"diagnosis\": tf.io.FixedLenFeature([], tf.int64),\n        \"target\": tf.io.FixedLenFeature([], tf.int64),\n    }\n    example = tf.io.parse_single_example(example, LABELED_TFREC_FORMAT)\n    image = decode_image(example['image'])\n    image_name = example['image_name']\n    target = tf.cast(example['target'], tf.int32)\n    return image, target\n\ndef read_unlabeled_tfrecord(example):\n    UNLABELED_TFREC_FORMAT = {\n        \"image\": tf.io.FixedLenFeature([], tf.string), # tf.string means bytestring\n        \"image_name\": tf.io.FixedLenFeature([], tf.string),  # shape [] means single element\n        \"patient_id\": tf.io.FixedLenFeature([], tf.int64),\n        \"sex\": tf.io.FixedLenFeature([], tf.int64),\n        \"age_approx\": tf.io.FixedLenFeature([], tf.int64),\n        \"anatom_site_general_challenge\": tf.io.FixedLenFeature([], tf.int64),\n        # no 'diagnosis' and 'target' for the test data.\n    }\n    example = tf.io.parse_single_example(example, UNLABELED_TFREC_FORMAT)\n    image = decode_image(example['image'])\n    image_name = example['image_name']\n    return image, image_name # no target\n\ndef load_dataset(filenames, labeled=True, ordered=False):\n    # Read from TFRecords. For optimal performance, reading from multiple files at once and\n    # disregarding data order. Order does not matter since we will be shuffling the data anyway.\n\n    ignore_order = tf.data.Options()\n    if not ordered:\n        ignore_order.experimental_deterministic = False # disable order, increase speed\n\n    # automatically interleaves reads from multiple files\n    dataset = tf.data.TFRecordDataset(filenames, num_parallel_reads=AUTO)\n    # uses data as soon as it streams in, rather than in its original order\n    dataset = dataset.with_options(ignore_order)\n    dataset = dataset.map(\n        read_labeled_tfrecord if labeled else read_unlabeled_tfrecord, num_parallel_calls=AUTO)\n    # returns a dataset of (image, target) pairs if labeled=True or\n    # (image, image_name) pairs if labeled=False\n    return dataset","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a name=\"SplittingTensorflowDataset\"></a>\n# Splitting Tensorflow Dataset","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# At first, make a vanilla training dataset by just loading from TFRecord files.\n# The order should be maintained.\nvanilla_training_ds = load_dataset(TRAINING_FILENAMES, labeled=True, ordered=True)\n\nprint(vanilla_training_ds)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def print_target_counts(y, label):\n    _, y_counts = np.unique(y, return_counts=True)\n    y_total = len(y)\n    y_0_count = y_counts[0]\n    y_1_count = y_counts[1]\n    y_1_percent = y_1_count / y_total * 100.0\n    print(\"{0:10s}: Total={1:5d}, 0={2:5d}, 1={3:3d}, ratio of 1={4:.2f}%\".format(\n        label, y_total, y_0_count, y_1_count, y_1_percent))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Extract target values from the vanilla training dataset.\n# Indices are generated along with the target values, which are used to filter dataset.\ny_targets = np.array([ target.numpy() for _, target in iter(vanilla_training_ds) ])\nX_indices = np.arange(len(y_targets))\n\nprint_target_counts(y_targets, \"Total\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.model_selection import train_test_split\n\n# Split the generated indices and target values by train_test_split().\n# The ratio of target values should be kept in the splitted datasets.\nX_train_indices, X_val_indices, y_train_targets, y_val_targets = train_test_split(\n    X_indices, y_targets, test_size=0.1, stratify=y_targets, random_state=53)\n\nprint_target_counts(y_train_targets, \"Training\")\nprint_target_counts(y_val_targets, \"Validation\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_selected_dataset(ds, X_indices_np):\n    # Make a tensor of type tf.int64 to match the one by Dataset.enumerate(). \n    X_indices_ts = tf.constant(X_indices_np, dtype=tf.int64)\n    \n    def is_index_in(index, rest):\n        # Returns True if the specified index value is included in X_indices_ts.\n        #\n        # '==' compares the specified index value with each values in X_indices_ts.\n        # The result is a boolean tensor, looks like [ False, True, ..., False ].\n        # reduce_any() returns Ture if True is included in the specified tensor.\n        return tf.math.reduce_any(index == X_indices_ts)\n    \n    def drop_index(index, rest):\n        return rest\n\n    # Dataset.enumerate() is similter to Python's enumerate().\n    # The method adds indices to each elements. Then, the elements are filtered\n    # by using the specified indices. Finally unnecessary indices are dropped.\n    selected_ds = ds \\\n        .enumerate() \\\n        .filter(is_index_in) \\\n        .map(drop_index)\n    return selected_ds","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"splitted_train_ds = get_selected_dataset(vanilla_training_ds, X_train_indices)\nsplitted_val_ds = get_selected_dataset(vanilla_training_ds, X_val_indices)\n\nprint(splitted_train_ds)\nprint(splitted_val_ds)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a name=\"Dataset2\"></a>\n# Dataset 2","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def data_augment(image, target):\n    image = tf.image.random_flip_left_right(image)\n    image = tf.image.random_flip_up_down(image)\n    return image, target\n\ndef get_training_dataset():\n    dataset = splitted_train_ds\n    dataset = dataset.repeat() # the training dataset must repeat for several epochs\n    dataset = dataset.shuffle(2048)\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.map(data_augment, num_parallel_calls=AUTO)\n    dataset = dataset.prefetch(AUTO) # prefetch next batch while training (autotune prefetch buffer size)\n    return dataset\n\ndef get_validation_dataset():\n    dataset = splitted_val_ds\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.cache()\n    dataset = dataset.prefetch(AUTO) # prefetch next batch while training (autotune prefetch buffer size)\n    return dataset\n\ndef get_test_dataset(ordered=False):\n    dataset = load_dataset(TEST_FILENAMES, labeled=False, ordered=ordered)\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.prefetch(AUTO) # prefetch next batch while training (autotune prefetch buffer size)\n    return dataset","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"NUM_TRAINING_IMAGES = len(y_train_targets)\nNUM_VALIDATION_IMAGES = len(y_val_targets)\nSTEPS_PER_EPOCH = NUM_TRAINING_IMAGES // BATCH_SIZE\nprint('Dataset: {} training images, {} validation images'.format(\n    NUM_TRAINING_IMAGES, NUM_VALIDATION_IMAGES))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a name=\"Model\"></a>\n# Model\n\n* [AUC](https://www.tensorflow.org/api_docs/python/tf/keras/metrics/AUC), [Precision](https://www.tensorflow.org/api_docs/python/tf/keras/metrics/Precision), and [Recall](https://www.tensorflow.org/api_docs/python/tf/keras/metrics/Recall) are used as metrics.\n* Initialize the bias for the final Dense layer to reflect the imbalance of target values as shown in [Optional: Set the correct initial bias](https://www.tensorflow.org/tutorials/structured_data/imbalanced_data#optional_set_the_correct_initial_bias).","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# https://www.tensorflow.org/tutorials/structured_data/imbalanced_data#optional_set_the_correct_initial_bias\n\nimport math\nfrom tensorflow.keras.initializers import Constant\n\n_, (y_target_neg_count, y_target_pos_count) = np.unique(y_targets, return_counts=True)\ny_target_pos_ratio = y_target_pos_count / y_target_neg_count\ndence_initial_bias = math.log(y_target_pos_ratio)\ndense_bias_initializer = Constant(dence_initial_bias)\n\nprint(y_target_neg_count, y_target_pos_count)\nprint(y_target_pos_ratio)\nprint(dence_initial_bias)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import efficientnet.tfkeras as efn\nfrom tensorflow.keras.layers import Dense, Dropout\nfrom tensorflow.keras.models import Sequential\nfrom tensorflow.keras.metrics import AUC, Precision, Recall\n\nwith strategy.scope():\n    model = Sequential([\n        efn.EfficientNetB0(\n            include_top=False, weights='noisy-student',\n            input_shape=(*IMAGE_SIZE, 3), pooling='avg'),\n        Dropout(\n            0.5, name=\"dropout\"),\n        Dense(\n            1, activation='sigmoid',\n            bias_initializer=dense_bias_initializer, name='classify')\n    ])\n    metrics = [ AUC(name='auc'), Precision(name='precision'), Recall(name='recall') ]    \n\nmodel.compile(optimizer='adam', loss='binary_crossentropy', metrics=metrics)\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a name=\"Training\"></a>\n# Training\n\n* For learning rate, ramp up and exponential decay is used in [Rotation Augmentation GPU/TPU - [0.96+]](https://www.kaggle.com/cdeotte/rotation-augmentation-gpu-tpu-0-96).\n* Best result is saved by using [ModelCheckpoint](https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/ModelCheckpoint).","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# https://www.kaggle.com/cdeotte/rotation-augmentation-gpu-tpu-0-96\n\nimport matplotlib.pyplot as plt\n\nLR_START = 0.00001\nLR_MAX = 0.00005 * strategy.num_replicas_in_sync\nLR_MIN = 0.00001\nLR_RAMPUP_EPOCHS = 5\nLR_SUSTAIN_EPOCHS = 0\nLR_EXP_DECAY = .8\n\ndef lrfn(epoch):\n    if epoch < LR_RAMPUP_EPOCHS:\n        lr = (LR_MAX - LR_START) / LR_RAMPUP_EPOCHS * epoch + LR_START\n    elif epoch < LR_RAMPUP_EPOCHS + LR_SUSTAIN_EPOCHS:\n        lr = LR_MAX\n    else:\n        lr = (LR_MAX - LR_MIN) * LR_EXP_DECAY**(epoch - LR_RAMPUP_EPOCHS - LR_SUSTAIN_EPOCHS) + LR_MIN\n    return lr\n    \nlr_callback = tf.keras.callbacks.LearningRateScheduler(lrfn, verbose = True)\n\nrng = [i for i in range(25 if EPOCHS<25 else EPOCHS)]\ny = [lrfn(x) for x in rng]\nplt.figure(figsize=(10, 4))\nplt.plot(rng, y)\nprint(\"Learning rate schedule: {:.3g} to {:.3g} to {:.3g}\".format(y[0], max(y), y[-1]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from tensorflow.keras.callbacks import ModelCheckpoint\n\nbest_model_file_path = \"best_model.hdf5\"\n\ndef make_model_check_point(best_model_file_path):\n    return ModelCheckpoint(\n        best_model_file_path, monitor='val_auc', mode='max',\n        verbose=0, save_best_only=True, save_weights_only=False, period=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model_check_point = make_model_check_point(best_model_file_path)\n\nhistory = model.fit(\n    get_training_dataset(), steps_per_epoch=STEPS_PER_EPOCH,\n    epochs=EPOCHS, validation_data=get_validation_dataset(),\n    callbacks=[lr_callback, model_check_point],\n    verbose=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model.load_weights(best_model_file_path)\neval_result = model.evaluate(get_validation_dataset(), verbose=0)\nprint(eval_result)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a name=\"Evaluation\"></a>\n# Evaluation\n\n<a name=\"LossAucPrecisionRecall\"></a>\n## Loss, Auc, Precision, Recall","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def display_training_curve(training, validation, title, subplot, ylim):\n    if subplot%10==1: # set up the subplots on the first call\n        plt.subplots(figsize=(10,10), facecolor='#F0F0F0')\n        plt.tight_layout()\n    ax = plt.subplot(subplot)\n    ax.set_facecolor('#F8F8F8')\n    ax.plot(training)\n    ax.plot(validation)\n    ax.set_title('model '+ title)\n    ax.set_ylabel(title)\n    ax.set_ylim(*ylim)\n    ax.set_xlabel('epoch')\n    ax.legend(['train', 'valid.'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"display_training_curve(\n    history.history['loss'], history.history['val_loss'],\n    'loss', 221, (0.0, 0.2))\ndisplay_training_curve(\n    history.history['auc'], history.history['val_auc'],\n    'auc', 222, (0.5, 1.0))\ndisplay_training_curve(\n    history.history['precision'], history.history['val_precision'],\n    'precision', 223, (0.0, 1.0))\ndisplay_training_curve(\n    history.history['recall'], history.history['val_recall'],\n    'recall', 224, (0.0, 1.0))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a name=\"Roc\"></a>\n## Roc","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_image(image, target):\n    return image\n\nval_image_ds = get_validation_dataset().map(get_image)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y_true = np.array([\n    target.numpy() for _, target in iter(get_validation_dataset().unbatch()) ])\ny_pred = model.predict(val_image_ds).flatten()\n\nprint(y_true.shape)\nprint(y_pred.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.metrics import roc_curve\n\nfpr, tpr, thresholds = roc_curve(y_true, y_pred)\n\nplt.figure(figsize=(10, 5))\nplt.plot(fpr, tpr, label=\"ROC Curve\")\nplt.xlabel(\"FPR\")\nplt.ylabel(\"TPR (recall)\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.metrics import roc_auc_score\n\nauc = roc_auc_score(y_true, y_pred)\nprint(\"AUC: {0:.3f}\".format(auc))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a name=\"ConfusionMatrix\"></a>\n## Confusion Matrix","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.metrics import confusion_matrix\nimport seaborn as sns\n\ndef plot_cm(labels, predictions, p=0.5):\n    cm = confusion_matrix(labels, predictions > p)\n    plt.figure(figsize=(5,5))\n    sns.heatmap(cm, annot=True, fmt=\"d\")\n    plt.title('Confusion matrix @{:.2f}'.format(p))\n    plt.ylabel('Actual label')\n    plt.xlabel('Predicted label')\n\n    print('True Negatives: ', cm[0][0])\n    print('False Positives: ', cm[0][1])\n    print('False Negatives: ', cm[1][0])\n    print('True Positives: ', cm[1][1])\n    print('Total Positive Images: ', np.sum(cm[1]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_cm(y_true, y_pred, p=0.5)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a name=\"Predictions\"></a>\n# Predictions","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# since we are splitting the dataset and iterating separately on images and ids, order matters.\ntest_ds = get_test_dataset(ordered=True)\n\nprint('Computing predictions...')\ntest_images_ds = test_ds.map(lambda image, image_name: image)\npredictions = model.predict(test_images_ds).flatten()\nprint(predictions)\n\nprint('Generating submission.csv file...')\ntest_image_names_ds = test_ds.map(lambda image, image_name: image_name).unbatch()\ntest_image_names = np.array([\n    image_name.numpy() for image_name in iter(test_image_names_ds) ]).astype('U')\nnp.savetxt(\n    'submission.csv', np.rec.fromarrays([test_image_names, predictions]),\n    fmt=['%s', '%f'], delimiter=',', header='image_name,target', comments='')\n!head submission.csv","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Thanks for reading!","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}