{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.16","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"tpu1vmV38","dataSources":[{"sourceId":21154,"databundleVersionId":1243559,"sourceType":"competition"},{"sourceId":37130068,"sourceType":"kernelVersion"}],"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"In this exercise, you'll make your first submission to the [**Petals to the Metal**](https://www.kaggle.com/c/tpu-getting-started) competition.  You'll learn how to accept the competition rules, run a notebook on Kaggle that uses (free!) TPUs, and how to submit your results to the leaderboard.\n\nWe won't cover the code in detail here, but if you'd like to dive into the details, you're encouraged to check out the [tutorial notebook](https://www.kaggle.com/ryanholbrook/create-your-first-submission).","metadata":{}},{"cell_type":"markdown","source":"# Code #\n\nThe code reproduces the code we covered together in **[the tutorial](https://www.kaggle.com/ryanholbrook/create-your-first-submission)**.  If you commit the notebook by following the instructions above, then the code is run for you.\n\n## Load Helper Functions ##","metadata":{}},{"cell_type":"code","source":"from petal_helper import *\n\nBATCH_SIZE = 12 * strategy.num_replicas_in_sync\nprint('Batchsize:',BATCH_SIZE)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-10T03:30:35.175755Z","iopub.execute_input":"2025-03-10T03:30:35.175936Z","iopub.status.idle":"2025-03-10T03:30:56.081281Z","shell.execute_reply.started":"2025-03-10T03:30:35.175914Z","shell.execute_reply":"2025-03-10T03:30:56.080312Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def data_augment(image, label):\n    # data augmentation. Thanks to the dataset.prefetch(AUTO) statement in the next function (below),\n    # this happens essentially for free on TPU. Data pipeline code is executed on the \"CPU\" part\n    # of the TPU while the TPU itself is computing gradients.\n    image = tf.image.random_flip_left_right(image)\n    image = tf.image.random_saturation(image, 0, 2)\n    return image, label   \n\ndef get_training_dataset():\n    dataset = load_dataset(TRAINING_FILENAMES, labeled=True)\n    dataset = dataset.map(data_augment, num_parallel_calls=AUTO)\n    dataset = dataset.repeat() # the training dataset must repeat for several epochs\n    dataset = dataset.shuffle(2048)\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.prefetch(AUTO) # prefetch next batch while training (autotune prefetch buffer size)\n    return dataset\n\ndef get_validation_dataset(ordered=False):\n    dataset = load_dataset(VALIDATION_FILENAMES, labeled=True, ordered=ordered)\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.cache()\n    dataset = dataset.prefetch(AUTO) # prefetch next batch while training (autotune prefetch buffer size)\n    return dataset\n\ndef get_test_dataset(ordered=False):\n    dataset = load_dataset(TEST_FILENAMES, labeled=False, ordered=ordered)\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.prefetch(AUTO) # prefetch next batch while training (autotune prefetch buffer size)\n    return dataset\n\ndef count_data_items(filenames):\n    # the number of data items is written in the name of the .tfrec files, i.e. flowers00-230.tfrec = 230 data items\n    n = [int(re.compile(r\"-([0-9]*)\\.\").search(filename).group(1)) for filename in filenames]\n    return np.sum(n)\n\n# %% [markdown]\n\n# %% [code]\nNUM_TRAINING_IMAGES = count_data_items(TRAINING_FILENAMES)\nNUM_VALIDATION_IMAGES = count_data_items(VALIDATION_FILENAMES)\nNUM_TEST_IMAGES = count_data_items(TEST_FILENAMES)\nprint('Dataset: {} training images, {} validation images, {} unlabeled test images'.format(NUM_TRAINING_IMAGES, NUM_VALIDATION_IMAGES, NUM_TEST_IMAGES))","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import tensorflow as tf\n\n# Detect and initialize TPU\ntry:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver(tpu=\"local\")  \n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.TPUStrategy(tpu)\n    print(f\"Running on TPU: {tpu.master()}\")\nexcept Exception as e:\n    print(f\"Could not initialize TPU: {e}\")\n    strategy = tf.distribute.get_strategy()  # Fallback to default CPU/GPU\n    print(\"Running on CPU or single GPU\")\n\n# Print number of available TPU cores\nprint(\"Number of devices:\", strategy.num_replicas_in_sync, \"🚀\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-10T03:30:56.082233Z","iopub.execute_input":"2025-03-10T03:30:56.082711Z","iopub.status.idle":"2025-03-10T03:31:04.613863Z","shell.execute_reply.started":"2025-03-10T03:30:56.082683Z","shell.execute_reply":"2025-03-10T03:31:04.612967Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Create Distribution Strategy ##","metadata":{}},{"cell_type":"markdown","source":"## Loading the Competition Data ##","metadata":{}},{"cell_type":"code","source":"ds_train = get_training_dataset()\nds_valid = get_validation_dataset()\nds_test = get_test_dataset()\n\nprint(\"Training:\", ds_train)\nprint (\"Validation:\", ds_valid)\nprint(\"Test:\", ds_test)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-10T03:31:04.614801Z","iopub.execute_input":"2025-03-10T03:31:04.615017Z","iopub.status.idle":"2025-03-10T03:31:04.870057Z","shell.execute_reply.started":"2025-03-10T03:31:04.614994Z","shell.execute_reply":"2025-03-10T03:31:04.869020Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## \nExplore the Data ##\n\nTry using some of the helper functions described in the **Getting Started** tutorial to explore the dataset.","metadata":{}},{"cell_type":"code","source":"print(\"Number of classes: {}\".format(len(CLASSES)))\n\nprint(\"First five classes, sorted alphabetically:\")\nfor name in sorted(CLASSES)[:5]:\n    print(name)\n\nprint (\"Number of training images: {}\".format(NUM_TRAINING_IMAGES))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-10T03:31:04.870707Z","iopub.execute_input":"2025-03-10T03:31:04.870971Z","iopub.status.idle":"2025-03-10T03:31:04.875719Z","shell.execute_reply.started":"2025-03-10T03:31:04.870943Z","shell.execute_reply":"2025-03-10T03:31:04.874646Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Examine the shape of the data.","metadata":{}},{"cell_type":"code","source":"strategy = tf.distribute.TPUStrategy(tpu)\nstrategy","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-10T03:31:04.876415Z","iopub.execute_input":"2025-03-10T03:31:04.876641Z","iopub.status.idle":"2025-03-10T03:31:04.895791Z","shell.execute_reply.started":"2025-03-10T03:31:04.876618Z","shell.execute_reply":"2025-03-10T03:31:04.895032Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"print(\"Training data shapes:\")\nfor image, label in ds_train.take(3):\n    print(image.numpy().shape, label.numpy().shape)\nprint(\"Training data label examples:\", label.numpy())","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-10T03:31:04.896506Z","iopub.execute_input":"2025-03-10T03:31:04.896711Z","iopub.status.idle":"2025-03-10T03:31:07.241583Z","shell.execute_reply.started":"2025-03-10T03:31:04.896691Z","shell.execute_reply":"2025-03-10T03:31:07.240407Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"print(\"Test data shapes:\")\nfor image, idnum in ds_test.take(3):\n    print(image.numpy().shape, idnum.numpy().shape)\nprint(\"Test data IDs:\", idnum.numpy().astype('U')) # U=unicode string","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-10T03:31:07.242762Z","iopub.execute_input":"2025-03-10T03:31:07.243154Z","iopub.status.idle":"2025-03-10T03:31:07.461744Z","shell.execute_reply.started":"2025-03-10T03:31:07.243124Z","shell.execute_reply":"2025-03-10T03:31:07.460615Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Peek at training data.","metadata":{}},{"cell_type":"code","source":"one_batch = next(iter(ds_train.unbatch().batch(16)))\ndisplay_batch_of_images(one_batch)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-10T03:31:07.462821Z","iopub.execute_input":"2025-03-10T03:31:07.463135Z","iopub.status.idle":"2025-03-10T03:31:11.191901Z","shell.execute_reply.started":"2025-03-10T03:31:07.463108Z","shell.execute_reply":"2025-03-10T03:31:11.190262Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Define Model #","metadata":{}},{"cell_type":"code","source":"def lrfn(epoch):\n    LR_START = 0.00001\n    LR_MAX = 0.00005 * strategy.num_replicas_in_sync\n    LR_MIN = 0.00001\n    LR_RAMPUP_EPOCHS = 5\n    LR_SUSTAIN_EPOCHS = 0\n    LR_EXP_DECAY = .8\n    \n    if epoch < LR_RAMPUP_EPOCHS:\n        lr = (LR_MAX - LR_START) / LR_RAMPUP_EPOCHS * epoch + LR_START\n    elif epoch < LR_RAMPUP_EPOCHS + LR_SUSTAIN_EPOCHS:\n        lr = LR_MAX\n    else:\n        lr = (LR_MAX - LR_MIN) * LR_EXP_DECAY**(epoch - LR_RAMPUP_EPOCHS - LR_SUSTAIN_EPOCHS) + LR_MIN\n    return lr\nlr_callback = tf.keras.callbacks.LearningRateScheduler(lrfn, verbose=1)\n\n","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"with strategy.scope():\n    #pretrained_model = tf.keras.applications.DenseNet201(weights='imagenet', include_top=False ,input_shape=[*IMAGE_SIZE, 3])\n    #pretrained_model = tf.keras.applications.Xception(weights='imagenet', include_top=False ,input_shape=[*IMAGE_SIZE, 3])\n    #pretrained_model = tf.keras.applications.inception_v3.InceptionV3(weights='imagenet', include_top=False ,input_shape=[*IMAGE_SIZE, 3])\n    #pretrained_model = tf.keras.applications.VGG16(weights='imagenet', include_top=False ,input_shape=[*IMAGE_SIZE, 3])\n    #pretrained_model = tf.keras.applications.ResNet50(weights='imagenet', include_top=False ,input_shape=[*IMAGE_SIZE, 3])\n    pretrained_model = tf.keras.applications.mobilenet.MobileNet(weights='imagenet', include_top=False ,input_shape=[*IMAGE_SIZE, 3])\n    pretrained_model.trainable=False\n    \n    model_201 = tf.keras.Sequential([\n        pretrained_model,\n        tf.keras.layers.GlobalAveragePooling2D(),\n        tf.keras.layers.Dense(len(CLASSES), activation='softmax')\n    ])\n        \n    model_201.compile(\n        optimizer='nadam',\n        loss = 'sparse_categorical_crossentropy',\n        metrics=['sparse_categorical_accuracy']\n    )\n    model_201.summary()","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"BATCH_SIZE = 12 * strategy.num_replicas_in_sync\nprint('Batchsize:',BATCH_SIZE)\n# Define training epochs for committing/submitting. (TPU on)\nEPOCHS = 30\nSTEPS_PER_EPOCH = NUM_TRAINING_IMAGES // BATCH_SIZE \n\n\nwith strategy.scope():\n    history_201 = model_201.fit(\n        get_training_dataset(), \n        steps_per_epoch=STEPS_PER_EPOCH,\n        epochs=30, \n        callbacks=[lr_callback],\n        validation_data=None\n    )","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# with strategy.scope():\n\n#     feature_extractor = tf.keras.applications.EfficientNetB7(\n#                     weights = 'imagenet', include_top = False, input_shape=[*IMAGE_SIZE, 3])\n#     feature_extractor.trainable = True\n#     model = tf.keras.Sequential([\n#         feature_extractor,\n#         tf.keras.layers.GlobalAveragePooling2D(),\n#         tf.keras.layers.Dense(len(CLASSES), activation='softmax', dtype='float32')\n#     ])\n    \n#     model.compile(optimizer='adam',\n#     loss = 'sparse_categorical_crossentropy',\n#     metrics=['sparse_categorical_accuracy']\n#     )\n\n# model.summary()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-10T04:00:32.438004Z","iopub.execute_input":"2025-03-10T04:00:32.438380Z","iopub.status.idle":"2025-03-10T04:00:37.575220Z","shell.execute_reply.started":"2025-03-10T04:00:32.438347Z","shell.execute_reply":"2025-03-10T04:00:37.574120Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"!pip install -U efficientnet\n","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import efficientnet.tfkeras as efn\n\n\nwith strategy.scope():\n    pretrained_model_noisy = efn.EfficientNetB7(weights='noisy-student', include_top=False ,input_shape=[*IMAGE_SIZE, 3])\n    pretrained_model_noisy.trainable = False \n    \n    model_noisy = tf.keras.Sequential([\n        pretrained_model_noisy,\n        tf.keras.layers.GlobalAveragePooling2D(),\n        tf.keras.layers.Dense(512,activation='relu'),\n        tf.keras.layers.Dropout(0.15),\n        tf.keras.layers.Dense(len(CLASSES), activation='softmax')\n    ])\n        \n    model_noisy.compile(\n    optimizer='adam',\n    loss = 'sparse_categorical_crossentropy',\n    metrics=['sparse_categorical_accuracy']\n)\n    model_noisy.summary()","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"with strategy.scope():\n    history_noisy = model_noisy.fit(\n        get_training_dataset(), \n        steps_per_epoch=STEPS_PER_EPOCH,\n        epochs=30, \n        callbacks=[lr_callback],\n        validation_data=None\n    )","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Ensemble","metadata":{}},{"cell_type":"code","source":"from sklearn.metrics import f1_score\nval_dataset = get_validation_dataset()\nimages_ds = val_dataset.map(lambda image, label: image)\nlabels_ds = val_dataset.map(lambda image, label: label).unbatch()\nval_labels = next(iter(labels_ds.batch(3712))).numpy() # get everything as one batch\nm1 = model_201.predict(images_ds)\nm2 = model_noisy.predict(images_ds)\nscores = []\nfor alpha in np.linspace(0,1,100):\n    val_probabilities = alpha*m1+(1-alpha)*m2\n    val_predictions = np.argmax(val_probabilities, axis=-1)\n    scores.append(f1_score(val_labels, val_predictions, labels=range(104), average='macro'))\n\nbest_alpha = np.argmax(scores)/100\n    \nprint('Best alpha: ' + str(best_alpha))","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Validation","metadata":{}},{"cell_type":"code","source":"from sklearn.metrics import f1_score\nval_dataset = get_validation_dataset()\nimages_ds = val_dataset.map(lambda image, label: image)\nlabels_ds = val_dataset.map(lambda image, label: label).unbatch()\nval_labels = next(iter(labels_ds.batch(3712))).numpy() # get everything as one batch\nm1 = model_201.predict(images_ds)\nm2 = model_noisy.predict(images_ds)\nscores = []\nfor alpha in np.linspace(0,1,100):\n    val_probabilities = alpha*m1+(1-alpha)*m2\n    val_predictions = np.argmax(val_probabilities, axis=-1)\n    scores.append(f1_score(val_labels, val_predictions, labels=range(104), average='macro'))\n\nbest_alpha = np.argmax(scores)/100\n    \nprint('Best alpha: ' + str(best_alpha))","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Test Predictions ##\n\nCreate predictions to submit to the competition.","metadata":{}},{"cell_type":"code","source":"test_ds = get_test_dataset(ordered=True) # since we are splitting the dataset and iterating separately on images and ids, order matters.\n\nprint('Computing predictions...')\ntest_images_ds = test_ds.map(lambda image, idnum: image)\n\nalpha = best_alpha\n\nprobabilities = (alpha*model_201.predict(test_images_ds)+(1-alpha)*model_noisy.predict(test_images_ds))\n\npredictions = np.argmax(probabilities, axis=-1)\nprint(predictions)\n\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-10T03:36:32.335401Z","iopub.execute_input":"2025-03-10T03:36:32.335725Z","iopub.status.idle":"2025-03-10T03:37:36.662329Z","shell.execute_reply.started":"2025-03-10T03:36:32.335698Z","shell.execute_reply":"2025-03-10T03:37:36.661458Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"print('Generating submission.csv file...')\n\n# Get image ids from test set and convert to integers\ntest_ids_ds = test_ds.map(lambda image, idnum: idnum).unbatch()\ntest_ids = next(iter(test_ids_ds.batch(NUM_TEST_IMAGES))).numpy().astype('U')\n\n# Write the submission file\nnp.savetxt(\n    'submission.csv',\n    np.rec.fromarrays([test_ids, predictions]),\n    fmt=['%s', '%d'],\n    delimiter=',',\n    header='id,label',\n    comments='',\n)\n\n# Look at the first few predictions\n!head submission.csv","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-10T03:38:36.504486Z","iopub.execute_input":"2025-03-10T03:38:36.504943Z","iopub.status.idle":"2025-03-10T03:38:42.788862Z","shell.execute_reply.started":"2025-03-10T03:38:36.504893Z","shell.execute_reply":"2025-03-10T03:38:42.786963Z"}},"outputs":[],"execution_count":null}]}