{"cells":[{"metadata":{},"cell_type":"markdown","source":"**[Deep Learning Home Page](https://www.kaggle.com/learn/deep-learning)**\n\n---\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"In this exercise, you'll make your first submission to the [**Petals to the Metal**](https://www.kaggle.com/c/tpu-getting-started) competition.  You'll learn how to accept the competition rules, run a notebook on Kaggle that uses (free!) TPUs, and how to submit your results to the leaderboard.\n\nWe won't cover the code in detail here, but if you'd like to dive into the details, you're encouraged to check out the [tutorial notebook](https://www.kaggle.com/ryanholbrook/create-your-first-submission).\n\nBegin by running the next code cell to set up the notebook.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install -U -t /kaggle/working/ git+https://github.com/Kaggle/learntools.git\nfrom learntools.core import binder\nbinder.bind(globals())\nfrom learntools.deep_learning.ex_tpu import *\nstep_1.check()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Code #\n\nThe code reproduces the code we covered together in **[the tutorial](https://www.kaggle.com/ryanholbrook/create-your-first-submission)**.  If you commit the notebook by following the instructions above, then the code is run for you.\n\n## Load Helper Functions ##","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from petal_helper import *","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Create Distribution Strategy ##","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Detect TPU, return appropriate distribution strategy\ntry:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver() \n    print('Running on TPU ', tpu.master())\nexcept ValueError:\n    tpu = None\n\nif tpu:\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n    strategy = tf.distribute.get_strategy() \n\nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Loading the Competition Data ##","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"ds_train = get_training_dataset()\nds_valid = get_validation_dataset()\nds_test = get_test_dataset()\n\nprint(\"Training:\", ds_train)\nprint (\"Validation:\", ds_valid)\nprint(\"Test:\", ds_test)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Explore the Data ##\n\nTry using some of the helper functions described in the **Getting Started** tutorial to explore the dataset.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Number of classes: {}\".format(len(CLASSES)))\n\nprint(\"First five classes, sorted alphabetically:\")\nfor name in sorted(CLASSES)[:5]:\n    print(name)\n\nprint (\"Number of training images: {}\".format(NUM_TRAINING_IMAGES))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Examine the shape of the data.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Training data shapes:\")\nfor image, label in ds_train.take(3):\n    print(image.numpy().shape, label.numpy().shape)\nprint(\"Training data label examples:\", label.numpy())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Test data shapes:\")\nfor image, idnum in ds_test.take(3):\n    print(image.numpy().shape, idnum.numpy().shape)\nprint(\"Test data IDs:\", idnum.numpy().astype('U')) # U=unicode string","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Peek at training data.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"one_batch = next(iter(ds_train.unbatch().batch(20)))\ndisplay_batch_of_images(one_batch)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Define Model #","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"with strategy.scope():\n    pretrained_model = tf.keras.applications.VGG16(\n        include_top=False,\n        weights=\"imagenet\",\n        input_tensor=None,\n        #input_shape=None,\n        pooling=None,\n        classes=1000,\n        classifier_activation=\"softmax\",\n        input_shape=[*IMAGE_SIZE, 3]\n    )\n    pretrained_model.trainable = False\n    \n    model = tf.keras.Sequential([\n        # To a base pretrained on ImageNet to extract features from images...\n        pretrained_model,\n        #tf.keras.layers.Dense(512, activation='relu'),\n        tf.keras.layers.Conv2D(256,4, activation='relu', input_shape=IMAGE_SIZE),\n        #tf.keras.layers.Conv2D(128,2, activation='relu', input_shape=IMAGE_SIZE),\n        # ... attach a new head to act as a classifier.\n        tf.keras.layers.GlobalAveragePooling2D(),\n        tf.keras.layers.Dense(len(CLASSES), activation='softmax')\n    ])\n    model.compile(\n        optimizer='adam',\n        loss = 'sparse_categorical_crossentropy',\n        metrics=['sparse_categorical_accuracy'],\n    )\n\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Train Model ##","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Define the batch size. This will be 16 with TPU off and 128 with TPU on\nBATCH_SIZE = 8 * strategy.num_replicas_in_sync\n\n# Define training epochs for committing/submitting. (TPU on)\nEPOCHS = 12\nSTEPS_PER_EPOCH = NUM_TRAINING_IMAGES // BATCH_SIZE\n\nhistory = model.fit(\n    ds_train,\n    validation_data=ds_valid,\n    epochs=EPOCHS,\n    steps_per_epoch=STEPS_PER_EPOCH,\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Examine training curves.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"display_training_curves(\n    history.history['loss'],\n    history.history['val_loss'],\n    'loss',\n    211,\n)\ndisplay_training_curves(\n    history.history['sparse_categorical_accuracy'],\n    history.history['val_sparse_categorical_accuracy'],\n    'accuracy',\n    212,\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Validation ##\n\nCreate a confusion matrix.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"cmdataset = get_validation_dataset(ordered=True)\nimages_ds = cmdataset.map(lambda image, label: image)\nlabels_ds = cmdataset.map(lambda image, label: label).unbatch()\n\ncm_correct_labels = next(iter(labels_ds.batch(NUM_VALIDATION_IMAGES))).numpy()\ncm_probabilities = model.predict(images_ds)\ncm_predictions = np.argmax(cm_probabilities, axis=-1)\n\nlabels = range(len(CLASSES))\ncmat = confusion_matrix(\n    cm_correct_labels,\n    cm_predictions,\n    labels=labels,\n)\ncmat = (cmat.T / cmat.sum(axis=1)).T # normalize","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"score = f1_score(\n    cm_correct_labels,\n    cm_predictions,\n    labels=labels,\n    average='macro',\n)\nprecision = precision_score(\n    cm_correct_labels,\n    cm_predictions,\n    labels=labels,\n    average='macro',\n)\nrecall = recall_score(\n    cm_correct_labels,\n    cm_predictions,\n    labels=labels,\n    average='macro',\n)\ndisplay_confusion_matrix(cmat, score, precision, recall)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Look at examples from the dataset, with true and predicted classes.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"dataset = get_validation_dataset()\ndataset = dataset.unbatch().batch(20)\nbatch = iter(dataset)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"images, labels = next(batch)\nprobabilities = model.predict(images)\npredictions = np.argmax(probabilities, axis=-1)\ndisplay_batch_of_images((images, labels), predictions)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Test Predictions ##\n\nCreate predictions to submit to the competition.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"test_ds = get_test_dataset(ordered=True)\n\nprint('Computing predictions...')\ntest_images_ds = test_ds.map(lambda image, idnum: image)\nprobabilities = model.predict(test_images_ds)\npredictions = np.argmax(probabilities, axis=-1)\nprint(predictions)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Generating submission.csv file...')\n\n# Get image ids from test set and convert to integers\ntest_ids_ds = test_ds.map(lambda image, idnum: idnum).unbatch()\ntest_ids = next(iter(test_ids_ds.batch(NUM_TEST_IMAGES))).numpy().astype('U')\n\n# Write the submission file\nnp.savetxt(\n    'submission.csv',\n    np.rec.fromarrays([test_ids, predictions]),\n    fmt=['%s', '%d'],\n    delimiter=',',\n    header='id,label',\n    comments='',\n)\n\n# Look at the first few predictions\n!head submission.csv","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Going Further #\n\nNow that you've joined the **Petals to the Metal** competition, why not try your hand at improving the model and see if you can climb the ranks! If you're looking for ideas, the *original* flower competition, [Flower Classification with TPUs](https://www.kaggle.com/c/flower-classification-with-tpus), has a wealth of information in its notebooks and discussion forum. Check it out!","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"---\n**[Deep Learning Home Page](https://www.kaggle.com/learn/deep-learning)**\n\n\n\n\n\n*Have questions or comments? Visit the [Learn Discussion forum](https://www.kaggle.com/learn-forum/161321) to chat with other Learners.*","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}