{"cells":[{"metadata":{},"cell_type":"markdown","source":"**[Deep Learning Home Page](https://www.kaggle.com/learn/deep-learning)**\n\n---\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"In this exercise, you'll make your first submission to the [**Petals to the Metal**](https://www.kaggle.com/c/tpu-getting-started) competition.  You'll learn how to accept the competition rules, run a notebook on Kaggle that uses (free!) TPUs, and how to submit your results to the leaderboard.\n\nWe won't cover the code in detail here, but if you'd like to dive into the details, you're encouraged to check out the [tutorial notebook](https://www.kaggle.com/ryanholbrook/create-your-first-submission).\n\nBegin by running the next code cell to set up the notebook.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install -U -t /kaggle/working/ git+https://github.com/Kaggle/learntools.git\nfrom learntools.core import binder\nbinder.bind(globals())\nfrom learntools.deep_learning.ex_tpu import *\nstep_1.check()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"If the code cell returns `Setup complete.`, then you're ready to continue.\n\n# Join the competition #\n\nBegin by joining the competition. Open a new window with the **[competition page](https://www.kaggle.com/c/tpu-getting-started)**, and click on the **\"Rules\"** tab.\n\nThis takes you to the rules acceptance page. You must accept the competition rules in order to participate. These rules govern how many submissions you can make per day, the maximum team size, and other competition-specific details. Click on **\"I Understand and Accept\"** to indicate that you will abide by the competition rules.\n\n# Commit your Notebook #\n\n**Committing** your notebook will run a fresh copy of the notebook start to finish, saving a copy of the `submission.csv` file as output.\n\nFirst, click on the **Save Version** button in the upper right.\n\n<figure>\n<img src=\"https://i.imgur.com/ebMUMSq.png\" alt=\"The blue Save Version button.\" width=300>\n</figure>\n\nChoose **Advanced Settings**.\n\n<figure>\n<img src=\"https://i.imgur.com/sx9l1fL.png\" alt=\"Advanced Settings in the Version menu.\" width=600>\n</figure>\n\nSelect **Run with TPU for this session** from the dropdown menu and click the blue **Save** button.\n\n<figure>\n<img src=\"https://i.imgur.com/1cB5ykf.png\" alt=\"The Accelerator dropdown menu.\" width=600>\n</figure>\n\nSelect **Save & Run All (Commit)** and click the blue **Save** button.\n\n<figure>\n<img src=\"https://i.imgur.com/YeJLsNG.png\" alt=\"The Save Version menu.\" width=600>\n</figure>\n\nThe commit may take a while to finish (about 10-15 min), but there's no harm in doing something else while it's running and coming back later.\n\nThis generates a window in the bottom left corner of the notebook. After it has finished running, click on the number to the right of the **Save Version** button. This pulls up a list of versions on the right of the screen. Click on the ellipsis **(...)** to the right of the most recent version, and select **Open in Viewer**. This brings you into view mode of the same page. You will need to scroll down to get back to these instructions.\n\n# Make a Submission #\n\nNow you're ready to make a submission! Click on the **Output** heading in the menu to the right of the notebook.\n\n<figure>\n<img src=\"https://i.imgur.com/thKwt1q.png\" alt=\"The Output heading.\" width=300>\n</figure>\n\nAnd finally you'll submit the predictions! Just look for the blue **Submit** button. After clicking it, you should shortly be on the leaderboard!\n\n<figure>\n<img src=\"https://i.imgur.com/j00mDeI.png\" alt=\"The Save Version menu.\" width=600>\n</figure>\n\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Code #\n\nThe code reproduces the code we covered together in **[the tutorial](https://www.kaggle.com/ryanholbrook/create-your-first-submission)**.  If you commit the notebook by following the instructions above, then the code is run for you.\n\n## Load Helper Functions ##","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from petal_helper import *","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Create Distribution Strategy ##","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Detect TPU, return appropriate distribution strategy\ntry:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver() \n    print('Running on TPU ', tpu.master())\nexcept ValueError:\n    tpu = None\n\nif tpu:\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n    strategy = tf.distribute.get_strategy() \n\nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Loading the Competition Data ##","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"ds_train = get_training_dataset()\nds_valid = get_validation_dataset()\nds_test = get_test_dataset()\n\nprint(\"Training:\", ds_train)\nprint (\"Validation:\", ds_valid)\nprint(\"Test:\", ds_test)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Explore the Data ##\n\nTry using some of the helper functions described in the **Getting Started** tutorial to explore the dataset.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Number of classes: {}\".format(len(CLASSES)))\n\nprint(\"First five classes, sorted alphabetically:\")\nfor name in sorted(CLASSES)[:5]:\n    print(name)\n\nprint (\"Number of training images: {}\".format(NUM_TRAINING_IMAGES))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Examine the shape of the data.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Training data shapes:\")\nfor image, label in ds_train.take(3):\n    print(image.numpy().shape, label.numpy().shape)\nprint(\"Training data label examples:\", label.numpy())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Test data shapes:\")\nfor image, idnum in ds_test.take(3):\n    print(image.numpy().shape, idnum.numpy().shape)\nprint(\"Test data IDs:\", idnum.numpy().astype('U')) # U=unicode string","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Peek at training data.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"one_batch = next(iter(ds_train.unbatch().batch(20)))\ndisplay_batch_of_images(one_batch)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Define Model #","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"with strategy.scope():\n    pretrained_model = tf.keras.applications.ResNet152V2(\n        weights='imagenet',\n        include_top=False ,\n        input_shape=[*IMAGE_SIZE, 3]\n    )\n    pretrained_model.trainable = True\n    \n    model = tf.keras.Sequential([\n        # To a base pretrained on ImageNet to extract features from images...\n        pretrained_model,\n        # ... attach a new head to act as a classifier.\n        tf.keras.layers.GlobalAveragePooling2D(),\n        tf.keras.layers.Dense(len(CLASSES), activation='softmax')\n    ])\n    model.compile(\n        optimizer='adam',\n        loss = 'sparse_categorical_crossentropy',\n        metrics=['sparse_categorical_accuracy'],\n    )\n\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Train Model ##","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"\n\n\n# checkpoint = ModelCheckpoint(model_path, monitor='val_loss', mode='min', save_best_only=True)\nes = EarlyStopping(monitor='val_loss', mode='min', patience=6, \n                   restore_best_weights=True, verbose=1)\nlr = ReduceLROnPlateau(monitor='val_loss',factor=0.2,patience=4,verbose=1)\n\n\ncallbacks = [es, lr]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Define the batch size. This will be 16 with TPU off and 128 with TPU on\nBATCH_SIZE = 16 * strategy.num_replicas_in_sync\n\n# Define training epochs for committing/submitting. (TPU on)\nEPOCHS = 50\nSTEPS_PER_EPOCH = NUM_TRAINING_IMAGES // BATCH_SIZE\n\nhistory = model.fit(\n    ds_train,\n    validation_data=ds_valid,\n    epochs=EPOCHS,\n    steps_per_epoch=STEPS_PER_EPOCH,\n    callbacks=callbacks\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Examine training curves.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"display_training_curves(\n    history.history['loss'],\n    history.history['val_loss'],\n    'loss',\n    211,\n)\ndisplay_training_curves(\n    history.history['sparse_categorical_accuracy'],\n    history.history['val_sparse_categorical_accuracy'],\n    'accuracy',\n    212,\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Validation ##\n\nCreate a confusion matrix.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"cmdataset = get_validation_dataset(ordered=True)\nimages_ds = cmdataset.map(lambda image, label: image)\nlabels_ds = cmdataset.map(lambda image, label: label).unbatch()\n\ncm_correct_labels = next(iter(labels_ds.batch(NUM_VALIDATION_IMAGES))).numpy()\ncm_probabilities = model.predict(images_ds)\ncm_predictions = np.argmax(cm_probabilities, axis=-1)\n\nlabels = range(len(CLASSES))\ncmat = confusion_matrix(\n    cm_correct_labels,\n    cm_predictions,\n    labels=labels,\n)\ncmat = (cmat.T / cmat.sum(axis=1)).T # normalize","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"score = f1_score(\n    cm_correct_labels,\n    cm_predictions,\n    labels=labels,\n    average='macro',\n)\nprecision = precision_score(\n    cm_correct_labels,\n    cm_predictions,\n    labels=labels,\n    average='macro',\n)\nrecall = recall_score(\n    cm_correct_labels,\n    cm_predictions,\n    labels=labels,\n    average='macro',\n)\ndisplay_confusion_matrix(cmat, score, precision, recall)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Look at examples from the dataset, with true and predicted classes.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"dataset = get_validation_dataset()\ndataset = dataset.unbatch().batch(20)\nbatch = iter(dataset)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"images, labels = next(batch)\nprobabilities = model.predict(images)\npredictions = np.argmax(probabilities, axis=-1)\ndisplay_batch_of_images((images, labels), predictions)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Test Predictions ##\n\nCreate predictions to submit to the competition.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"test_ds = get_test_dataset(ordered=True)\n\nprint('Computing predictions...')\ntest_images_ds = test_ds.map(lambda image, idnum: image)\nprobabilities = model.predict(test_images_ds)\npredictions = np.argmax(probabilities, axis=-1)\nprint(predictions)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Generating submission.csv file...')\n\n# Get image ids from test set and convert to integers\ntest_ids_ds = test_ds.map(lambda image, idnum: idnum).unbatch()\ntest_ids = next(iter(test_ids_ds.batch(NUM_TEST_IMAGES))).numpy().astype('U')\n\n# Write the submission file\nnp.savetxt(\n    'submission.csv',\n    np.rec.fromarrays([test_ids, predictions]),\n    fmt=['%s', '%d'],\n    delimiter=',',\n    header='id,label',\n    comments='',\n)\n\n# Look at the first few predictions\n!head submission.csv","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Going Further #\n\nNow that you've joined the **Petals to the Metal** competition, why not try your hand at improving the model and see if you can climb the ranks! If you're looking for ideas, the *original* flower competition, [Flower Classification with TPUs](https://www.kaggle.com/c/flower-classification-with-tpus), has a wealth of information in its notebooks and discussion forum. Check it out!","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"---\n**[Deep Learning Home Page](https://www.kaggle.com/learn/deep-learning)**\n\n\n\n\n\n*Have questions or comments? Visit the [Learn Discussion forum](https://www.kaggle.com/learn-forum/161321) to chat with other Learners.*","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}