{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.15","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"tpu1vmV38","dataSources":[{"sourceId":21154,"databundleVersionId":1243559,"sourceType":"competition"}],"dockerImageVersionId":30786,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current sessionme, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2024-10-18T12:52:11.806334Z","iopub.execute_input":"2024-10-18T12:52:11.806599Z","iopub.status.idle":"2024-10-18T12:52:13.693944Z","shell.execute_reply.started":"2024-10-18T12:52:11.806570Z","shell.execute_reply":"2024-10-18T12:52:13.693228Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Introduction\n\nWelcome to the Petals to the Metal competition! In this competition, you’re challenged to build a machine learning model to classify 104 types of flowers based on their images.\n\nIn this tutorial notebook, you'll learn how to build an image classifier in Keras and train it on a Tensor Processing Unit (TPU). At the end, you'll have a complete project you can build off of with ideas of your own.","metadata":{}},{"cell_type":"markdown","source":"#  Step 1: \n Imports\nWe begin by importing several Python packages.","metadata":{}},{"cell_type":"code","source":"import math, re, os\nimport numpy as np\nimport tensorflow as tf\n\nprint(\"Tensorflow version \" + tf.__version__)","metadata":{"execution":{"iopub.status.busy":"2024-10-18T12:52:21.790242Z","iopub.execute_input":"2024-10-18T12:52:21.790818Z","iopub.status.idle":"2024-10-18T12:52:35.215782Z","shell.execute_reply.started":"2024-10-18T12:52:21.790781Z","shell.execute_reply":"2024-10-18T12:52:35.214848Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Step 2: Distribution Strategy¶\nA TPU has eight different cores and each of these cores acts as its own accelerator. (A TPU is sort of like having eight GPUs in one machine.) We tell TensorFlow how to make use of all these cores at once through a distribution strategy. Run the following cell to create the distribution strategy that we'll later apply to our model.","metadata":{}},{"cell_type":"code","source":"\ntry:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()  # Detect TPU\n    print('Running on TPU:', tpu.master())\nexcept ValueError:\n    tpu = None\n\nif tpu:\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.TPUStrategy(tpu)\nelse:\n    strategy = tf.distribute.get_strategy() \n    \nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)","metadata":{"execution":{"iopub.status.busy":"2024-10-18T12:52:40.045253Z","iopub.execute_input":"2024-10-18T12:52:40.045826Z","iopub.status.idle":"2024-10-18T12:52:48.937659Z","shell.execute_reply.started":"2024-10-18T12:52:40.045794Z","shell.execute_reply":"2024-10-18T12:52:48.936909Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Step 3: Loading the Competition Data¶\nGet GCS Path\nWhen used with TPUs, datasets need to be stored in a Google Cloud Storage bucket. You can use data from any public GCS bucket by giving its path just like you would data from '/kaggle/input'. The following will retrieve the GCS path for this competition's dataset.","metadata":{}},{"cell_type":"markdown","source":"# Load Data\n\nWhen used with TPUs, datasets are often serialized into TFRecords. This is a format convenient for distributing data to each of the TPUs cores. We've hidden the cell that reads the TFRecords for our dataset since the process is a bit long. You could come back to it later for some guidance on using your own datasets with TPUs.","metadata":{}},{"cell_type":"code","source":"from kaggle_datasets import KaggleDatasets\n\nGCS_DS_PATH = KaggleDatasets().get_gcs_path('tpu-getting-started')\nprint(GCS_DS_PATH) # what do gcs paths look like?","metadata":{"execution":{"iopub.status.busy":"2024-10-18T12:53:04.467842Z","iopub.execute_input":"2024-10-18T12:53:04.468203Z","iopub.status.idle":"2024-10-18T12:53:04.477264Z","shell.execute_reply.started":"2024-10-18T12:53:04.468172Z","shell.execute_reply":"2024-10-18T12:53:04.476333Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"You can display a single batch of images from a dataset with another of our helper functions. The next cell will turn the dataset into an iterator of batches of 16 images.","metadata":{}},{"cell_type":"code","source":"IMAGE_SIZE = [192, 192] # at this size, a GPU will run out of memory. Use the TPU\nEPOCHS = 5\nBATCH_SIZE = 16 * strategy.num_replicas_in_sync\n\nNUM_TRAINING_IMAGES = 12753\nNUM_TEST_IMAGES = 7382\nSTEPS_PER_EPOCH = NUM_TRAINING_IMAGES // BATCH_SIZE","metadata":{"execution":{"iopub.status.busy":"2024-10-18T12:53:08.763113Z","iopub.execute_input":"2024-10-18T12:53:08.763462Z","iopub.status.idle":"2024-10-18T12:53:08.767701Z","shell.execute_reply.started":"2024-10-18T12:53:08.763430Z","shell.execute_reply":"2024-10-18T12:53:08.766797Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#  Create Data Pipelines¶\nIn this final step we'll use the tf.data API to define an efficient data pipeline for each of the training, validation, and test splits.\n\n","metadata":{}},{"cell_type":"markdown","source":"# Step 4: Explore Data\nLet's take a moment to look at some of the images in the dataset.","metadata":{}},{"cell_type":"code","source":"\ndef decode_image(image_data):\n    image = tf.image.decode_jpeg(image_data, channels=3)\n    image = tf.cast(image, tf.float32) / 255.0  # convert image to floats in [0, 1] range\n    image = tf.reshape(image, [*IMAGE_SIZE, 3]) # explicit size needed for TPU\n    return image\n\ndef read_labeled_tfrecord(example):\n    LABELED_TFREC_FORMAT = {\n        \"image\": tf.io.FixedLenFeature([], tf.string), # tf.string means bytestring\n        \"class\": tf.io.FixedLenFeature([], tf.int64),  # shape [] means single element\n    }\n    example = tf.io.parse_single_example(example, LABELED_TFREC_FORMAT)\n    image = decode_image(example['image'])\n    label = tf.cast(example['class'], tf.int32)\n    return image, label # returns a dataset of (image, label) pairs\n\ndef read_unlabeled_tfrecord(example):\n    UNLABELED_TFREC_FORMAT = {\n        \"image\": tf.io.FixedLenFeature([], tf.string), # tf.string means bytestring\n        \"id\": tf.io.FixedLenFeature([], tf.string),  # shape [] means single element\n        # class is missing, this competitions's challenge is to predict flower classes for the test dataset\n    }\n    example = tf.io.parse_single_example(example, UNLABELED_TFREC_FORMAT)\n    image = decode_image(example['image'])\n    idnum = example['id']\n    return image, idnum # returns a dataset of image(s)\n\ndef load_dataset(filenames, labeled=True, ordered=False):\n    ignore_order = tf.data.Options()\n    ignore_order.experimental_deterministic = False  # Allow non-deterministic order for TPU efficiency\n    \n    dataset = tf.data.TFRecordDataset(filenames)\n    dataset = dataset.with_options(ignore_order)\n    dataset = dataset.map(\n        read_labeled_tfrecord if labeled else read_unlabeled_tfrecord,\n        num_parallel_calls=tf.data.AUTOTUNE\n    )\n    return dataset\n\ndef get_training_dataset():\n    dataset = load_dataset(\n        tf.io.gfile.glob(GCS_DS_PATH + '/tfrecords-jpeg-192x192/train/*.tfrec'), \n        labeled=True\n    )\n    dataset = dataset.shuffle(2048)  # Shuffle before repeating\n    dataset = dataset.repeat()  # Ensure enough data for all epochs\n    dataset = dataset.batch(BATCH_SIZE).prefetch(tf.data.AUTOTUNE)\n    return dataset\n\ndef get_validation_dataset():\n    dataset = load_dataset(\n        tf.io.gfile.glob(GCS_DS_PATH + '/tfrecords-jpeg-192x192/val/*.tfrec'), \n        labeled=True\n    )\n    dataset = dataset.batch(BATCH_SIZE).cache().prefetch(tf.data.AUTOTUNE)\n    return dataset\n\ndef get_test_dataset(ordered=False):\n    dataset = load_dataset(tf.io.gfile.glob(GCS_DS_PATH + '/tfrecords-jpeg-192x192/test/*.tfrec'), labeled=False, ordered=ordered)\n    dataset = dataset.batch(BATCH_SIZE)\n    return dataset\n\n# Load datasets\nds_train = get_training_dataset()\nds_valid = get_validation_dataset()\n\n","metadata":{"execution":{"iopub.status.busy":"2024-10-18T12:53:12.352873Z","iopub.execute_input":"2024-10-18T12:53:12.353734Z","iopub.status.idle":"2024-10-18T12:53:12.525764Z","shell.execute_reply.started":"2024-10-18T12:53:12.353698Z","shell.execute_reply":"2024-10-18T12:53:12.524863Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This next cell will create the datasets that we'll use with Keras during training and inference. Here, we will scale the size of the batches to the number of TPU cores.","metadata":{}},{"cell_type":"markdown","source":"# Step 5: Define Model\nFor this problem, we'll use a model called DenseNet available in Keras.\n\nHere, we import the entire DenseNet except for its top and instead use our own Neural Network in its place to help with the classification.","metadata":{}},{"cell_type":"markdown","source":"# Step 6: Training\nLearning Rate Schedule\nWe'll train this network with a special learning rate schedule.","metadata":{}},{"cell_type":"code","source":"# Model creation and training within strategy scope\nwith strategy.scope():\n    pretrained_model = tf.keras.applications.VGG16(weights='imagenet', include_top=False, input_shape=[*IMAGE_SIZE, 3])\n    pretrained_model.trainable = False  # Freeze layers for transfer learning\n\n    model = tf.keras.Sequential([\n        pretrained_model,\n        tf.keras.layers.GlobalAveragePooling2D(),\n        tf.keras.layers.Dense(104, activation='softmax')\n    ])\n\n    model.compile(\n        optimizer='adam',\n        loss='sparse_categorical_crossentropy',\n        metrics=['sparse_categorical_accuracy']\n    )\n\n    model.summary()\n\n    # Train the model\n    historical = model.fit(\n        ds_train,\n        steps_per_epoch=STEPS_PER_EPOCH,\n        epochs=EPOCHS,\n        validation_data=ds_valid\n    )","metadata":{"execution":{"iopub.status.busy":"2024-10-18T12:53:18.766907Z","iopub.execute_input":"2024-10-18T12:53:18.767739Z","iopub.status.idle":"2024-10-18T12:54:07.176866Z","shell.execute_reply.started":"2024-10-18T12:53:18.767703Z","shell.execute_reply":"2024-10-18T12:54:07.175543Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The 'sparse_categorical' versions of the loss and metrics are appropriate for a classification task with more than two labels, like this one.","metadata":{}},{"cell_type":"markdown","source":"# Fit Model\n And now we're ready to train the model. After defining a few parameters, we're good to go!","metadata":{}},{"cell_type":"code","source":"test_ds = get_test_dataset(ordered=True)\n    #steps_per_test_ds = get_test_dataset(ordered=True) # since we are splitting the dataset and iterating separately on images and ids, order matters.\n\nprint('Computing predictions...')\ntest_images_ds = test_ds.map(lambda image, idnum: image)\nprobabilities = model.predict(test_images_ds)\npredictions = np.argmax(probabilities, axis=-1)\nprint(predictions)\n\nprint('Generating submission.csv file...')\ntest_ids_ds = test_ds.map(lambda image, idnum: idnum).unbatch()\ntest_ids = next(iter(test_ids_ds.batch(NUM_TEST_IMAGES))).numpy().astype('U') # all in one batch\nnp.savetxt('submission.csv', np.rec.fromarrays([test_ids, predictions]), fmt=['%s', '%d'], delimiter=',', header='id,label', comments='')\n","metadata":{"execution":{"iopub.status.busy":"2024-10-18T12:59:19.614996Z","iopub.execute_input":"2024-10-18T12:59:19.615734Z","iopub.status.idle":"2024-10-18T12:59:35.832492Z","shell.execute_reply.started":"2024-10-18T12:59:19.615694Z","shell.execute_reply":"2024-10-18T12:59:35.831265Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This next cell shows how the loss and metrics progressed during training. Thankfully, it converges!","metadata":{}},{"cell_type":"code","source":"import pandas as pd\n\nhistory_frame = pd.DataFrame( historical.history)\nhistory_frame.loc[:, ['loss', 'val_loss']].plot()\nhistory_frame.loc[:, ['sparse_categorical_accuracy', 'val_sparse_categorical_accuracy']].plot();","metadata":{"execution":{"iopub.status.busy":"2024-10-18T13:05:34.285743Z","iopub.execute_input":"2024-10-18T13:05:34.286489Z","iopub.status.idle":"2024-10-18T13:05:34.627318Z","shell.execute_reply.started":"2024-10-18T13:05:34.286451Z","shell.execute_reply":"2024-10-18T13:05:34.626486Z"},"trusted":true},"execution_count":null,"outputs":[]}]}