{"cells":[{"metadata":{},"cell_type":"markdown","source":"**[Deep Learning Home Page](https://www.kaggle.com/learn/deep-learning)**\n\n---\n"},{"metadata":{},"cell_type":"markdown","source":"\n# Introduction #\n\nWelcome to the [**Petals to the Metal**](https://www.kaggle.com/c/tpu-getting-started) competition! In this competition, you’re challenged to build a machine learning model that identifies the type of flowers in a dataset of images.\n\nIn this tutorial notebook, you'll learn how to build an image classifier in Keras and train it on a [Tensor Processing Unit (TPU)](https://www.kaggle.com/docs/tpu). Then, in the [following exercise](https://www.kaggle.com/kernels/fork/10204702), you'll create your *own* notebook and make a submission to the competition. At the end, you'll have a complete project you can build off of with ideas of your own.\n\n<blockquote style=\"margin-right:auto; margin-left:auto; background-color: #ebf9ff; padding: 1em; margin:24px;\">\n    <strong>Get Started Now!</strong><br>\nYou don't need to understand everything in this notebook before starting the exercise. If you like, go ahead and <b><a href=\"https://www.kaggle.com/kernels/fork/10204702\">open the exercise</a></b> and follow the instructions at the beginning to enter the competition. Then, return to this tutorial for a walkthrough of your starter project.\n</blockquote>\n\n# Load the Helper Functions #\n\nAttached to the notebook is a utility script called [`petal_helper`](https://www.kaggle.com/ryanholbrook/petal-helper). It contains a number of helper functions related to data loading and visualization. The following cell will import them into your notebook session. We'll also import TensorFlow, which we'll use to create our model."},{"metadata":{"trusted":true},"cell_type":"code","source":"from petal_helper import *\n\nimport tensorflow as tf","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Distribution Strategy #\n\nA TPU has eight different *cores* and each of these cores acts as its own accelerator. (A TPU is sort of like having eight GPUs in one machine.) We tell TensorFlow how to make use of all these cores at once through a **distribution strategy**. Run the following cell to create the distribution strategy that we'll later apply to our model."},{"metadata":{"lines_to_next_cell":2,"trusted":true},"cell_type":"code","source":"# Detect TPU, return appropriate distribution strategy\ntry:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver() \n    print('Running on TPU ', tpu.master())\nexcept ValueError:\n    tpu = None\n\nif tpu:\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n    strategy = tf.distribute.get_strategy() \n\nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"TensorFlow will distribute training among the eight TPU cores by creating eight different *replicas* of your model. \n\n# Loading the Competition Data #\n\nWhen used with TPUs, datasets are often serialized into [TFRecords](https://www.tensorflow.org/tutorials/load_data/tfrecord). This is a format convenient for distributing data to each of the TPUs cores.\n\nWe've included functions in the `petal_helper` script that will load the TFRecords for you and create a data pipeline you can use with your model. There is one function for each split."},{"metadata":{"trusted":true},"cell_type":"code","source":"ds_train = get_training_dataset()\nds_valid = get_validation_dataset()\nds_test = get_test_dataset()\n\nprint(\"Training:\", ds_train)\nprint (\"Validation:\", ds_valid)\nprint(\"Test:\", ds_test)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"These are `tf.data.Dataset` objects. You can think about a dataset in TensorFlow as a *stream* of data records. Check out [this guide](https://www.tensorflow.org/guide/data) for more on working with the `tf.data` API.\n\n# Define Model #\n\nNow we're ready to create a neural network for classifying images! We'll use what's known as **transfer learning**. With transfer learning, you reuse part of a pretrained model to get a head-start on a new dataset.\n\nFor this tutorial, we'll to use a model called **VGG16** pretrained on [ImageNet](http://image-net.org/)). Later, you might want to experiment with [other models](https://www.tensorflow.org/api_docs/python/tf/keras/applications) included with Keras. ([Xception](https://www.tensorflow.org/api_docs/python/tf/keras/applications/Xception) wouldn't be a bad choice.)\n\nThe distribution strategy we created earlier contains a [context manager](https://docs.python.org/3/reference/compound_stmts.html#with), `strategy.scope`. This context manager tells TensorFlow how to divide the work of training among the eight TPU cores. When using TensorFlow with a TPU, it's important to define your model in a `strategy.scope()` context."},{"metadata":{"trusted":true},"cell_type":"code","source":"with strategy.scope():\n    pretrained_model = tf.keras.applications.VGG16(\n        weights='imagenet',\n        include_top=False ,\n        input_shape=[*IMAGE_SIZE, 3]\n    )\n    pretrained_model.trainable = False\n    \n    model = tf.keras.Sequential([\n        # To a base pretrained on ImageNet to extract features from images...\n        pretrained_model,\n        # ... attach a new head to act as a classifier.\n        tf.keras.layers.GlobalAveragePooling2D(),\n        tf.keras.layers.Dense(len(CLASSES), activation='softmax')\n    ])\n    model.compile(\n        optimizer='adam',\n        loss = 'sparse_categorical_crossentropy',\n        metrics=['sparse_categorical_accuracy'],\n    )\n\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Training #\n\nAnd now we're ready to train the model! After defining a few parameters, we're good to go."},{"metadata":{"trusted":false},"cell_type":"code","source":"# Define the batch size. This will be 16 with TPU off and 128 (=16*8) with TPU on\nBATCH_SIZE = 16 * strategy.num_replicas_in_sync\n\nprint(BATCH_SIZE)\n\n# Define training epochs\nEPOCHS = 12\nSTEPS_PER_EPOCH = NUM_TRAINING_IMAGES // BATCH_SIZE\n\nhistory = model.fit(\n    ds_train,\n    validation_data=ds_valid,\n    epochs=EPOCHS,\n    steps_per_epoch=STEPS_PER_EPOCH,\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This next cell shows how the loss and metrics progressed during training. Thankfully, it converges!"},{"metadata":{"trusted":false},"cell_type":"code","source":"display_training_curves(\n    history.history['loss'],\n    history.history['val_loss'],\n    'loss',\n    211,\n)\ndisplay_training_curves(\n    history.history['sparse_categorical_accuracy'],\n    history.history['val_sparse_categorical_accuracy'],\n    'accuracy',\n    212,\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Predictions #\n\nOnce you're satisfied with everything, you're ready to make predictions on the test set."},{"metadata":{"trusted":false},"cell_type":"code","source":"test_ds = get_test_dataset(ordered=True)\n\nprint('Computing predictions...')\ntest_images_ds = test_ds.map(lambda image, idnum: image)\nprobabilities = model.predict(test_images_ds)\npredictions = np.argmax(probabilities, axis=-1)\nprint(predictions)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We'll generate a file `submission.csv`. This file is what you'll submit to get your score on the leaderboard."},{"metadata":{"trusted":false},"cell_type":"code","source":"print('Generating submission.csv file...')\n\n# Get image ids from test set and convert to unicode\ntest_ids_ds = test_ds.map(lambda image, idnum: idnum).unbatch()\ntest_ids = next(iter(test_ids_ds.batch(NUM_TEST_IMAGES))).numpy().astype('U')\n\n# Write the submission file\nnp.savetxt(\n    'submission.csv',\n    np.rec.fromarrays([test_ids, predictions]),\n    fmt=['%s', '%d'],\n    delimiter=',',\n    header='id,label',\n    comments='',\n)\n\n# Look at the first few predictions\n!head submission.csv","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Your Turn! #\n\nNow that you've seen what a typical TPU project looks like, you're ready to **[create your own](https://www.kaggle.com/kernels/fork/10204702)**! "},{"metadata":{},"cell_type":"markdown","source":"---\n**[Deep Learning Home Page](https://www.kaggle.com/learn/deep-learning)**\n\n\n\n\n\n*Have questions or comments? Visit the [Learn Discussion forum](https://www.kaggle.com/learn-forum/161321) to chat with other Learners.*"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}