{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"tpu1vmV38","dataSources":[{"sourceId":21154,"databundleVersionId":1243559,"sourceType":"competition"},{"sourceId":6976363,"sourceType":"datasetVersion","datasetId":4008819}],"dockerImageVersionId":30579,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# 🤖 Computer Vision course: Practical Guide 😎\n> With Transfer Learning, custom CNN, Ensemble, TPUs, and lots of plain English comments","metadata":{}},{"cell_type":"markdown","source":"\n\n> Picture: Petals to the Metal","metadata":{}},{"cell_type":"markdown","source":"# Hello, fellow data enthusiasts 🤓!\n\nAfter completing [Computer Vision](https://www.kaggle.com/learn/computer-vision) microcourse and studying the [examplary notebook](https://www.kaggle.com/code/ryanholbrook/create-your-first-submission) I felt there might be some public interest in a work that smooths transition from the theory to [Petals to the Metal](https://www.kaggle.com/c/tpu-getting-started) competition. Personally, I needed a Notebook with more English and less Python to 'get started' with this topic. Now, you have one. \n\n_In this Notebook you will find_:\n\n- The comprehensively commented [utility code](https://www.kaggle.com/code/ryanholbrook/create-your-first-submission) to run Tensor Processing Units (TPU). The code is structured to serve as a collection of logical blocks that you can use for EDA and modeling with your own strategies.\n- An example of **transfer learning** with __Xception__\n- A custom __Convolutional Neural Network__ as adviced in the course. Models are trained and serialized for your convinience.\n- Several essential tools (__callbacks__), not mentioned in the course: _EarlyStopping, LearningRateSchedule, Checkpoints_\n- __Ensemble of models__ and a voting strategy.\n\n\n_Project Scopes_:\n- No groundbreaking decisions on achieving a high score.\n\n\n Please upvote if you find this valuable, and don't hesitate to leave comments if you have any feedback or suggestions
\n
","metadata":{}},{"cell_type":"markdown","source":"# STEP 0: Imports","metadata":{}},{"cell_type":"code","source":"import math, re, os\nimport numpy as np\nimport pandas as pd\nimport tensorflow as tf\nfrom tensorflow import keras\nfrom keras import layers as L\nfrom keras.layers.experimental import preprocessing\nfrom keras.preprocessing import image_dataset_from_directory\nfrom keras.applications import Xception\nfrom keras.callbacks import EarlyStopping, ModelCheckpoint\n\nimport joblib\n\nfrom sklearn.metrics import f1_score\n\nfrom warnings import simplefilter\nsimplefilter(\"ignore\")\n\n\nprint(f'TensorFlow version: {tf.__version__}')","metadata":{"execution":{"iopub.status.busy":"2023-11-16T07:43:14.108809Z","iopub.execute_input":"2023-11-16T07:43:14.109501Z","iopub.status.idle":"2023-11-16T07:43:14.115322Z","shell.execute_reply.started":"2023-11-16T07:43:14.109459Z","shell.execute_reply":"2023-11-16T07:43:14.114647Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# STEP 1: Connect to Tensor Processing Units (TPUs)\n\nKaggle provides a limited access to 3 types of processing units, avaliale for your models' training.\n- Central Processing Units (**CPUs**)\n- Graphics Processing Units (**GPUs**)\n- Tensor Processing Units (**TPUs**)\n\nHere is an [**article**](https://towardsdatascience.com/when-to-use-cpus-vs-gpus-vs-tpus-in-a-kaggle-competition-9af708a8c3eb) to help you figure out which is which. Long story short, \"GPUs are a great alternative to CPUs when you want to speed up a variety of data science workflows, and TPUs are best when you specifically want to train a machine learning model as fast as you possibly can\". But you will have to work on the code to make your data digestable for a TPU.\n\nA TPU has **eight cores** (it's like having eight GPUs in one machine). With **distribution strategy**, we instruct TensorFlow on how to utilize all these cores simultaneously. We will employ this object when constructing our neural network model: it will distribute the training by generating eight distinct *replicas* of the model, one for each core.","metadata":{}},{"cell_type":"code","source":"# Detect TPU, return appropriate distribution strategy\ntry:\n tpu = tf.distribute.cluster_resolver.TPUClusterResolver() # An attempt to detect an avaliable TPU (to 'resolve a TPU cluster')\n print('Running on TPU ', tpu.master()) # Tell the world that the attempt was a success\nexcept ValueError:\n tpu = None\n\nif tpu: # If TPU was detected\n tf.config.experimental_connect_to_cluster(tpu) # connect to the TPU cluster \n tf.tpu.experimental.initialize_tpu_system(tpu) # run the cluster\n strategy = tf.distribute.TPUStrategy(tpu) # create a distribution strategy for TPU training\nelse:\n strategy = tf.distribute.get_strategy() # If no TPU was found, use the default distribution strategy for CPU or GPU","metadata":{"execution":{"iopub.status.busy":"2023-11-16T07:43:41.198376Z","iopub.execute_input":"2023-11-16T07:43:41.199160Z","iopub.status.idle":"2023-11-16T07:43:55.728049Z","shell.execute_reply.started":"2023-11-16T07:43:41.199125Z","shell.execute_reply":"2023-11-16T07:43:55.727364Z"},"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Tell the world which strategy it is\nprint(\"REPLICAS: \", strategy.num_replicas_in_sync) ","metadata":{"execution":{"iopub.status.busy":"2023-11-16T07:43:55.729375Z","iopub.execute_input":"2023-11-16T07:43:55.729612Z","iopub.status.idle":"2023-11-16T07:43:55.733307Z","shell.execute_reply.started":"2023-11-16T07:43:55.729587Z","shell.execute_reply":"2023-11-16T07:43:55.732663Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Set how many files can be processed simultaniously. This will be 16 with TPU off and 128 (=16*8) with TPU on\nBATCH_SIZE = 16 * strategy.num_replicas_in_sync","metadata":{"execution":{"iopub.status.busy":"2023-11-16T07:43:55.734167Z","iopub.execute_input":"2023-11-16T07:43:55.734398Z","iopub.status.idle":"2023-11-16T07:43:57.995291Z","shell.execute_reply.started":"2023-11-16T07:43:55.734374Z","shell.execute_reply":"2023-11-16T07:43:57.994028Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Step 2: Retrieve, Load and Format Data\n\n- When used with TPUs, datasets need to be stored in a [Google Cloud Storage bucket](https://cloud.google.com/storage/) (**GCS**). You can use data from any public GCS bucket by giving its path (like `'/kaggle/input'`). \n- You can use data from any public dataset here on Kaggle in just the same way. If you'd like to use data from one of your private datasets, see [here](https://www.kaggle.com/docs/tpu#tpu3pt5).\n- When used with TPUs, datasets are serialized into [TFRecords](https://www.kaggle.com/ryanholbrook/tfrecords-basics). This is a format for distributing data to each of the TPUs cores.","metadata":{}},{"cell_type":"code","source":"# Here we create lists of paths to our training, validation and test files\nfrom kaggle_datasets import KaggleDatasets\n\nIMAGE_SIZE = [192, 192] # This is the size for GPU. For TPU use [512, 512]\nGCS_DS_PATH = KaggleDatasets().get_gcs_path('tpu-getting-started') # You can list the bucket with \"!gsutil ls $GCS_DS_PATH\"\n \nGCS_PATH_SELECT = { # Images of different sizes are strored in different directories. The dictionary connects the sizes to the paths\n 192: '/tfrecords-jpeg-192x192',\n 224: '/tfrecords-jpeg-224x224',\n 331: '/tfrecords-jpeg-331x331',\n 512: '/tfrecords-jpeg-512x512'\n}\n\nGCS_PATH_PER_SIZE = GCS_PATH_SELECT[IMAGE_SIZE[0]] # Define the path to the directory depending on the IMAGE_SIZE\nGCS_PATH_ORIGINAL = GCS_DS_PATH + GCS_PATH_PER_SIZE # This is where the original data for the competition dwells\n\n\nTRAINING_FILENAMES = tf.io.gfile.glob(GCS_PATH_ORIGINAL + '/train/*.tfrec') # Get the list of file paths for training TFRecords\nVALIDATION_FILENAMES = tf.io.gfile.glob(GCS_PATH_ORIGINAL + '/val/*.tfrec') # Get the list of file paths for validation TFRecords\nTEST_FILENAMES = tf.io.gfile.glob(GCS_PATH_ORIGINAL + '/test/*.tfrec') # Get the list of file paths for testing TFRecords","metadata":{"execution":{"iopub.status.busy":"2023-11-16T07:43:57.998389Z","iopub.execute_input":"2023-11-16T07:43:57.999077Z","iopub.status.idle":"2023-11-16T07:43:58.037573Z","shell.execute_reply.started":"2023-11-16T07:43:57.999018Z","shell.execute_reply":"2023-11-16T07:43:58.036799Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# I saved the list with flower labels in a separate .csv\nCLASSES = pd.read_csv('/kaggle/input/cv-course-practical-guide-data/flower_classes.csv', usecols = ['flowers']).squeeze().sort_values().to_list()","metadata":{"execution":{"iopub.status.busy":"2023-11-16T07:43:58.038487Z","iopub.execute_input":"2023-11-16T07:43:58.038751Z","iopub.status.idle":"2023-11-16T07:43:58.050735Z","shell.execute_reply.started":"2023-11-16T07:43:58.038723Z","shell.execute_reply":"2023-11-16T07:43:58.049652Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"'''\nThe following code allows to create sets of TFRecords. This is a special data format suitable for processing on TPU.\nIf not for TPU, we could have easily use `keras.preprocessing.image_dataset_from_directory()`\n'''\n\nAUTO = tf.data.experimental.AUTOTUNE # Configure Auto-tuning for better performance. To be applied in many functions below # 100 - 102\n\ndef decode_image(image_data): \n image = tf.image.decode_jpeg(image_data, channels=3) # Decode the JPEG image to a tensor with 3 color channels (red, green, blue)\n image = tf.cast(image, tf.float32) / 255.0 # Convert pixel values to floating-point numbers in the range [0, 1]\n image = tf.reshape(image, [*IMAGE_SIZE, 3]) # Reshape the image tensor to match the specified IMAGE_SIZE\n # This step ensures that all images have the same dimensions for consistency\n return image\n\n# This function reads a labeled TFRecord file and returns the image and its corresponding label (to be applied on training and validation sets)\ndef read_labeled_tfrecord(example): # example: A single labeled TFRecord file (labled picture to be used for training and validation)\n LABELED_TFREC_FORMAT = { # setting a dictionary that defines the format of a TFRecord (names and dtypes of its features)\n \"image\": tf.io.FixedLenFeature([], tf.string), # tf.string means bytestring\n \"class\": tf.io.FixedLenFeature([], tf.int64), # shape [] means single element\n }\n example = tf.io.parse_single_example(example, LABELED_TFREC_FORMAT) # Parse the single TFRecord example according to the specified format.\n image = decode_image(example['image']) # Decode the 'image' feature of a TFRecord file using the 'decode_image' function (previously defined) \n label = tf.cast(example['class'], tf.int32) # tf.cast converts tensors from one data type to another. Here it ensures that all elements are integers\n return image, label # returns a dataset of (image, label) pairs. In Python you get a tuple with this syntaxis automatically\n\n\n# This function reads an unlabeled TFRecord file and returns the image and its ID (to be applied on the test set)\ndef read_unlabeled_tfrecord(example): \n UNLABELED_TFREC_FORMAT = { \n \"image\": tf.io.FixedLenFeature([], tf.string), \n \"id\": tf.io.FixedLenFeature([], tf.string), # Class is missing, this competitions's challenge is to predict flower classes for the test dataset\n }\n example = tf.io.parse_single_example(example, UNLABELED_TFREC_FORMAT) # Parse the single TFRecord example according to the specified format.\n image = decode_image(example['image']) # Decode the 'image' feature of a TFRecord example using the 'decode_image' function (previously defined)\n idnum = example['id']\n return image, idnum # Returns a dataset of (image, id) pairs. In Python you get a tuple with this syntaxis automatically\n\n\n# Read from TFRecords. For optimal performance, reading from multiple files at once and disregarding data order.\ndef load_dataset(filenames, labeled=True, ordered=False): # We set values for 'labeled' and 'ordered' in the definition of the function to use them by default. However, we reserve an option to pass different values to these parameters.\n \n options = tf.data.Options() # Creating an objects here looks like a TensorFlow reference code. It is literally the same as in the documentation\n if not ordered: # If the 'ordered' parameter is 'False' (the default) and hasn't been explicitly set to 'True' when passed to the function\n options.deterministic = False # Disable order, increase speed\n\n dataset = tf.data.TFRecordDataset(filenames, num_parallel_reads=AUTO) # automatically interleaves reads from multiple files. This 'num_parallel_reads=AUTO' parameter of tf.data.TFRecordDataset() will be used many times in the below code.\n dataset = dataset.with_options(options) # uses data as soon as it streams in, rather than in its original order\n \n # returns a dataset of (image, label) pairs if labeled=True\n dataset = dataset.map(read_labeled_tfrecord if labeled \\\n # returns a dataset of (image, id) pairs if labeled=False\n else read_unlabeled_tfrecord,\n num_parallel_calls=AUTO) \n return dataset","metadata":{"execution":{"iopub.status.busy":"2023-11-16T07:43:58.052329Z","iopub.execute_input":"2023-11-16T07:43:58.052773Z","iopub.status.idle":"2023-11-16T07:43:58.071456Z","shell.execute_reply.started":"2023-11-16T07:43:58.052727Z","shell.execute_reply":"2023-11-16T07:43:58.070775Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Step 3: Create Pipelines ","metadata":{}},{"cell_type":"code","source":"def data_augment(image, label):\n seed = 42 # Setting the seed ensures reproducibility; otherwise, the learning process can produce different results each time, making it hard to control.\n image = tf.image.random_flip_left_right(image, seed=seed) # These functions are included here to make you aware of their existence, but not all of them necessarily yield optimal performance on the given dataset.\n image = tf.image.random_flip_up_down(image, seed=seed)\n# image = tf.image.random_saturation(image, 0, 2, seed=seed) # It doesn't seem a great idea to change colours of flowers. But it could work on images of a different kind\n# image = tf.image.random_brightness(image, 0.6, seed=seed)\n# image = tf.image.random_contrast(image, 0.3, 0.5, seed=seed)\n \n return image, label \n\ndef get_training_dataset():\n dataset = load_dataset(TRAINING_FILENAMES, labeled=True) # Check load_dataset function and recall that 'dataset = tf.data.TFRecordDataset()'' with its inherent parameters\n dataset = dataset.map(data_augment, num_parallel_calls=AUTO) # Apply data_augment function\n dataset = dataset.repeat() # The repeat method is called on the dataset to make it repeat indefinitely (for all the epochs)\n dataset = dataset.shuffle(2048) # Shuffling the data is important during training to prevent the model from memorizing the order \n dataset = dataset.batch(BATCH_SIZE)\n dataset = dataset.prefetch(AUTO) # Prefetch next batch while training. Thanks to this statement, data pipeline code is executed on the CPU, \n # saving the TPU capacities for computing gradients.\n return dataset\n\ndef get_validation_dataset(ordered=False):\n dataset = load_dataset(VALIDATION_FILENAMES, labeled=True, ordered=ordered) # 'ordered=ordered' passes the 'ordered' parameter's value from the overarching function 'get_validation_dataset'\n dataset = dataset.batch(BATCH_SIZE)\n dataset = dataset.cache() # Caching the dataset means that it is temporarily stored in RAM, making it faster to access during subsequent epochs \n dataset = dataset.prefetch(AUTO) \n return dataset\n\ndef get_test_dataset(ordered=False):\n dataset = load_dataset(TEST_FILENAMES, labeled=False, ordered=ordered)\n dataset = dataset.batch(BATCH_SIZE)\n dataset = dataset.prefetch(AUTO)\n return dataset\n\ndef count_data_items(filenames): # The number of data items is written in the name of the .tfrec files, i.e. flowers00-230.tfrec = 230 data items \n n = [int(re.compile(r\"-([0-9]*)\\.\") # This is a 'regular expression'. re.compile() creates here a pattern where a number appears between a hyphen and a period, \n .search(filename) # looks for the pattern in the filenames\n .group(1)) # returns what was found. re.group() regulates which part of a pattern to return: re.group(0) returns the entire matched pattern, and re.group(n) returns the respective subpattern if the pattern contains a number of them.\n for filename in filenames] \n return np.sum(n)","metadata":{"execution":{"iopub.status.busy":"2023-11-16T07:43:58.072543Z","iopub.execute_input":"2023-11-16T07:43:58.072853Z","iopub.status.idle":"2023-11-16T07:43:58.096019Z","shell.execute_reply.started":"2023-11-16T07:43:58.072825Z","shell.execute_reply":"2023-11-16T07:43:58.095316Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"NUM_TRAINING_IMAGES = count_data_items(TRAINING_FILENAMES)\nNUM_VALIDATION_IMAGES = count_data_items(VALIDATION_FILENAMES)\nNUM_TEST_IMAGES = count_data_items(TEST_FILENAMES)\n\nprint(f'Dataset: \\n'\n f'{NUM_TRAINING_IMAGES} training images \\n'\n f'{NUM_VALIDATION_IMAGES} validation images \\n'\n f'{NUM_TEST_IMAGES} unlabeled test images')","metadata":{"execution":{"iopub.status.busy":"2023-11-16T07:43:58.096934Z","iopub.execute_input":"2023-11-16T07:43:58.097199Z","iopub.status.idle":"2023-11-16T07:43:58.122249Z","shell.execute_reply.started":"2023-11-16T07:43:58.097175Z","shell.execute_reply":"2023-11-16T07:43:58.121501Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"'''\nHere we create the datasets (traning, validation and test)\n- These datasets are `tf.data.Dataset` objects. You can think about a dataset in TensorFlow as a *stream* of data records. \n- The training and validation sets are streams of `(image, label)` pairs.\n- The test set is a stream of `(image, idnum)` pairs; we'll use these `idnum` (ID numbers) later to make our submission `csv` file.\n'''\n\nds_train = get_training_dataset()\nds_valid = get_validation_dataset()\nds_test = get_test_dataset()","metadata":{"execution":{"iopub.status.busy":"2023-11-16T07:43:58.123084Z","iopub.execute_input":"2023-11-16T07:43:58.123315Z","iopub.status.idle":"2023-11-16T07:43:58.441334Z","shell.execute_reply.started":"2023-11-16T07:43:58.123292Z","shell.execute_reply":"2023-11-16T07:43:58.440348Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Let's take a look at the data shapes\nnp.set_printoptions(threshold=15, linewidth=80) # Set the print options for NumPy to control the way arrays are displayed. This is in order to display only a part rather then all the information\n\nprint(\"Training data shapes:\")\nfor image, label in ds_train.take(3): # Iterate through the first 3 elements of the training dataset\n print(image.numpy().shape, label.numpy().shape) # .numpy() converts a TensorFlow tensor to a NumPy array\nprint(\"Training data label examples:\", label.numpy())\n\nprint ('---')\n\nprint(\"Test data shapes:\")\nfor image, idnum in ds_test.take(3):\n print(image.numpy().shape, idnum.numpy().shape)\nprint(\"Test data IDs:\", idnum.numpy().astype('U')) # U=unicode string","metadata":{"execution":{"iopub.status.busy":"2023-11-16T07:43:58.444243Z","iopub.execute_input":"2023-11-16T07:43:58.444638Z","iopub.status.idle":"2023-11-16T07:43:59.752393Z","shell.execute_reply.started":"2023-11-16T07:43:58.444591Z","shell.execute_reply":"2023-11-16T07:43:59.751510Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Step 4: Explore Data #\nLet's take a moment to look at some of the images in the dataset.","metadata":{}},{"cell_type":"code","source":"from matplotlib import pyplot as plt\n\ndef batch_to_numpy_images_and_labels(data):\n images, labels = data # unpack the tuples of (image, label) and (image, idnum). See above read_labeled_tfrecord and read_unlabeled_tfrecord functions\n numpy_images = images.numpy() # .numpy() converts a TensorFlow tensor to a NumPy array\n numpy_labels = labels.numpy()\n if numpy_labels.dtype == object: # Remember,that in our case`label` is tf.int64 (numeric format) and `idnum` is tf.string (bytestring, an `object`) \n numpy_labels = [None for _ in enumerate(numpy_images)] # So, if numpy_labels ends up carring 'idnum' values (not the 'labels'), this statement sets them to None (for test data)\n return numpy_images, numpy_labels\n\n\n# A function to generate a title based on the predicted and true target values\ndef title_from_label_and_target(label, correct_label): # it takes predictions (labels) and true values (correct_label) as arguments\n if correct_label is None: # if we deal with the test set, where no correct_labels are availible \n return CLASSES[label], True # it simply returns the prediction\n \n correct = (label == correct_label) # if target value (correct_label) is availible, it compares it with the prediction and returns a boolean value (True/False)\n \n return \"{} [{}{}{}]\".format(CLASSES[label], # returns the prediction\n 'OK' if correct else 'NO', # 'OK' if it is True, 'NO' if it is False\n u\"\\u2192\" if not correct else '', # '' if it is True, '→' if it is False \n CLASSES[correct_label] if not correct else ''),correct # '' if it is True, correct_label if it is False, separate value for 'True' or 'False'\n \n \n\n \n# a function to display a single flower image with a title\ndef display_one_flower(image, title, subplot, # subplot is what you need to display several pictures at once (on one plot)\n red=False, titlesize=16):\n plt.subplot(*subplot) # '*subplot' syntax unpacks the values in the subplot tuple (rows, columns, index) that specify the subplot layout \n plt.axis('off')\n plt.imshow(image) # plt.imshow stands for 'show image'\n if len(title) > 0: # if title is avaliable\n plt.title(title, # set parameters for this title's display\n fontsize = int(titlesize) if not red else int(titlesize/1.2), # bigger fontsize for correct (black) titles, smaller fontsize for the wrong (red) titles\n color='red' if red else 'black', # depending on the argument passed to the function\n fontdict={'verticalalignment':'center'}, \n pad=int(titlesize/1.5))\n \n return (subplot[0], # the number of rows in the subplot grid\n subplot[1], # the number of columns in the subplot grid\n subplot[2]+1) # the current index (position) within the grid. +1 makes it an iterator: each time you call this funtion, it moves to the next image\n \n\n# this function makes several pictures appear on the screen at the same time\ndef display_batch_of_images(databatch, predictions=None):\n \"\"\"This functions works with following settings:\n display_batch_of_images(images)\n display_batch_of_images(images, predictions)\n display_batch_of_images((images, labels))\n display_batch_of_images((images, labels), predictions)\n \"\"\"\n images, labels = batch_to_numpy_images_and_labels(databatch) # data\n if labels is None:\n labels = [None for _ in enumerate(images)] # creates a list of None values with the same length as the images list: to ensure that there is a label for each image.\n rows = int(math.sqrt(len(images))) # auto-squaring: this will drop data from the display that does not fit into square or square-ish rectangle\n cols = len(images)//rows # calculates the number of columns based on the number of rows and the total number of images. It uses integer division (//) to ensure that the grid is as square as possible.\n \n FIGSIZE = 13.0\n SPACING = 0.1\n subplot = (rows,cols,1) # you allready know that subplot has three parameters: (rows, columns, index)\n if rows < cols: # if there are more columns then rows\n plt.figure(figsize=(FIGSIZE,FIGSIZE/cols*rows)) # set portrait (tall) orientation\n else: # if the are more rows then colums\n plt.figure(figsize=(FIGSIZE/rows*cols,FIGSIZE)) # set landscape (wide) orientation\n \n # display\n display_dict = zip(images[:rows*cols], labels[:rows*cols]) # a dictionary with a subset of images as keys and a subset of labels as values. The subsets start from the beginning and contain rows*cols elements \n for i, (image, label) in enumerate (display_dict): # an iterator\n title = '' if label is None else CLASSES[label] # determine the title for the subplot based on the label\n correct = True # set the default value for 'correct'\n if predictions is not None: # if predictions are passed to the function\n title, correct = title_from_label_and_target(predictions[i], label) # apply the above formular, passing the predictions' indexes from the iterator and the corresponding labeles \n dynamic_titlesize = FIGSIZE*SPACING/max(rows,cols)*40+3 # magic formula tested to work from 1x1 to 10x10 images\n subplot = display_one_flower(image, title, subplot, \n not correct, # this is a value for parameter 'red' of display_one_flower function. So, if the prediction is False (correct=False), were turn it around (not correct) and pass True (red=True) to the function\n titlesize=dynamic_titlesize)\n \n #layout\n plt.tight_layout() # ensure that the subplots (in this case, the displayed images and titles) fit within the figure without overlapping or being cut off.\n if label is None and predictions is None: # if there are no predictions and true labels \n plt.subplots_adjust(wspace=0, hspace=0) # no spacing between the images \n else: # otherwise\n plt.subplots_adjust(wspace=SPACING, hspace=SPACING) # make spaces\n plt.show()","metadata":{"execution":{"iopub.status.busy":"2023-11-16T07:43:59.753580Z","iopub.execute_input":"2023-11-16T07:43:59.753987Z","iopub.status.idle":"2023-11-16T07:44:00.136478Z","shell.execute_reply.started":"2023-11-16T07:43:59.753956Z","shell.execute_reply":"2023-11-16T07:44:00.135608Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"'''\n- You can display a single batch of images from a dataset with this helper functions. \n- He we turn the dataset into an iterator of batches of 20 images.\n'''\nds_iter = iter(ds_train.unbatch().batch(20))","metadata":{"execution":{"iopub.status.busy":"2023-11-16T07:44:00.137478Z","iopub.execute_input":"2023-11-16T07:44:00.137902Z","iopub.status.idle":"2023-11-16T07:44:00.171062Z","shell.execute_reply.started":"2023-11-16T07:44:00.137873Z","shell.execute_reply":"2023-11-16T07:44:00.170371Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"'''\n- `next` function allows to fetch next batch in the stream and display it with the helper function.\n- Rerun this cell to see a new batch of images.\n'''\none_batch = next(ds_iter)\ndisplay_batch_of_images(one_batch)","metadata":{"execution":{"iopub.status.busy":"2023-11-16T07:44:00.171953Z","iopub.execute_input":"2023-11-16T07:44:00.172182Z","iopub.status.idle":"2023-11-16T07:44:02.339398Z","shell.execute_reply.started":"2023-11-16T07:44:00.172159Z","shell.execute_reply":"2023-11-16T07:44:02.338545Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Step 5: Callbacks\n- A callback is a .fit() parameter, where you can pass different objects:\n- **`Learning Rate Schedule`**: adjusts the learning rate e.g. after a certain number of epochs or when the training loss plateaus or else\n- **`Early Stopping`**: stops learning if there is no improvement after several epochs\n- **`Checkpoint`**: saves weights at the end of every epoch, if it's the best seen so far during model.fit\n\n- There are [many other options](https://keras.io/api/callbacks/). But we will make these three here.\nWe will apply the same callbacks for all the models that follow.","metadata":{}},{"cell_type":"code","source":"EPOCHS = 30 # EarlyStopping should break it sooner\nSTEPS_PER_EPOCH = NUM_TRAINING_IMAGES // BATCH_SIZE # Batches per epoch","metadata":{"execution":{"iopub.status.busy":"2023-11-16T07:44:02.340381Z","iopub.execute_input":"2023-11-16T07:44:02.340672Z","iopub.status.idle":"2023-11-16T07:44:02.344213Z","shell.execute_reply.started":"2023-11-16T07:44:02.340642Z","shell.execute_reply":"2023-11-16T07:44:02.343428Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Callback 1: learning rate schedule","metadata":{}},{"cell_type":"code","source":"def exponential_lr(epoch, # The current training epoch\n start_lr = 0.00001, # The initial learning rate\n min_lr = 0.00001, # The minimum learning rate\n max_lr = 0.00005, # The maximum learning rate\n rampup_epochs = 5, # The number of epochs for a linear increase in learning rate\n sustain_epochs = 0, # The number of epochs to sustain the maximum learning rate\n exp_decay = 0.8): # The exponential decay factor for learning rate reduction\n\n # calculates the learning rate for a given epoch based on the provided parameters\n def lr(epoch, start_lr, min_lr, max_lr, rampup_epochs, sustain_epochs, exp_decay): \n \n if epoch < rampup_epochs: # For epochs less than rampup_epochs, the learning rate increases from start_lr to max_lr.\n lr = ((max_lr - start_lr) /\n rampup_epochs * epoch + start_lr)\n \n elif epoch < rampup_epochs + sustain_epochs: # From 'rampup_epochs' till 'rampup_epochs + sustain_epochs', the learning rate remains constant at max_lr\n lr = max_lr\n \n else: # exponential decay towards min_lr\n lr = ((max_lr - min_lr) *\n exp_decay**(epoch - rampup_epochs - sustain_epochs) +\n min_lr)\n return lr\n return lr(epoch, start_lr, min_lr, max_lr, rampup_epochs, sustain_epochs, exp_decay)\n\n# This is what it was all about. We pass our customary funtion to the keras LearningRateScheduler to create a callback\nlr_callback = tf.keras.callbacks.LearningRateScheduler(exponential_lr, verbose=True) \n\n# plot our customary learning rate per epoch\nrng = [i for i in range(EPOCHS)] \ny = [exponential_lr(x) for x in rng]\nplt.plot(rng, y)\nprint(f'Learning rate schedule: \\n'\n f'from {y[0]:.3g} \\n'\n f'to {max(y):.3g} \\n'\n f'and then back to {y[-1]:.3g}')","metadata":{"execution":{"iopub.status.busy":"2023-11-16T07:44:02.345173Z","iopub.execute_input":"2023-11-16T07:44:02.345460Z","iopub.status.idle":"2023-11-16T07:44:02.522574Z","shell.execute_reply.started":"2023-11-16T07:44:02.345432Z","shell.execute_reply":"2023-11-16T07:44:02.521848Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Callback 2: EarlyStopping\nIt will stop training when there is no improvement in the validation loss during the specified number of consecutive epochs. ","metadata":{}},{"cell_type":"code","source":"early_stopping = EarlyStopping(\n monitor='val_loss',\n min_delta=0.001, # minimum change to count as an improvement\n patience=5, # how many epochs to wait before stopping\n restore_best_weights=True,\n)","metadata":{"execution":{"iopub.status.busy":"2023-11-16T07:44:02.523490Z","iopub.execute_input":"2023-11-16T07:44:02.523763Z","iopub.status.idle":"2023-11-16T07:44:02.527559Z","shell.execute_reply.started":"2023-11-16T07:44:02.523735Z","shell.execute_reply":"2023-11-16T07:44:02.526813Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Callback 3. Checkpoint\n- Checkpoint allows recording the best performing configuration of a model's weights.\n- It also allows you to upload these weights to the model from a file instead of training it again.","metadata":{}},{"cell_type":"code","source":"Xception_checkpoint_filepath = 'Xception.h5'\n\nXception_checkpoint = ModelCheckpoint(\n filepath=Xception_checkpoint_filepath,\n save_weights_only=True,\n monitor='val_loss',\n mode='min',\n save_best_only=True\n)","metadata":{"execution":{"iopub.status.busy":"2023-11-16T07:44:02.528435Z","iopub.execute_input":"2023-11-16T07:44:02.528708Z","iopub.status.idle":"2023-11-16T07:44:02.538259Z","shell.execute_reply.started":"2023-11-16T07:44:02.528681Z","shell.execute_reply":"2023-11-16T07:44:02.537597Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Step 6: Model_1 - Transfer learning\n\nNow we're ready to create a neural network for classifying images \n- We'll use **transfer learning**: take a pretrained heavy model (base) and set a keras model on top of it (head)\n- The base will be **Xception**, cause it performs well on this dataset. Run the cell below to see the list of avalible bases in Keras\n- The distribution strategy we created earlier contains a [context manager](https://docs.python.org/3/reference/compound_stmts.html#with), `strategy.scope`. When using a TPU, it's important to define your model in a strategy.scope() context.","metadata":{}},{"cell_type":"code","source":"# The list of avalible pretrained models (bases)\n', '.join(tf.keras.applications.__dir__())","metadata":{"execution":{"iopub.status.busy":"2023-11-16T07:44:02.539099Z","iopub.execute_input":"2023-11-16T07:44:02.539330Z","iopub.status.idle":"2023-11-16T07:44:02.549975Z","shell.execute_reply.started":"2023-11-16T07:44:02.539306Z","shell.execute_reply":"2023-11-16T07:44:02.549330Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%capture\nwith strategy.scope():\n pretrained_model = Xception(\n weights='imagenet', \n include_top=False, # we will build our own head on top of this base, so we tell the the strategy to 'decapitate' the base\n input_shape=[*IMAGE_SIZE, 3] # '*' unpacks the IMAGE_SIZE tuple, passing it's two elements as separate values\n )\n pretrained_model.trainable = False # transfer learning\n \n Xception_ = keras.Sequential([ # Here is our eventual model:\n pretrained_model, # add the pretrained base \n L.GlobalAveragePooling2D(), # attach a new head (GlobalAveragePooling averages feature maps produced by the base down to a single value per feature. Which is just right for a classification)\n L.Dropout(0.3), # add regularization \n L.Dense( # output layer where\n len(CLASSES), # number of neurons corresponds to the number of classes\n activation='softmax') # this is the activation function you want to use for a multi-class classification task \n ])","metadata":{"execution":{"iopub.status.busy":"2023-11-16T07:44:02.550786Z","iopub.execute_input":"2023-11-16T07:44:02.551022Z","iopub.status.idle":"2023-11-16T07:44:18.539871Z","shell.execute_reply.started":"2023-11-16T07:44:02.550999Z","shell.execute_reply":"2023-11-16T07:44:18.538828Z"},"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Xception_.compile(\n optimizer= 'nadam', # Nesterov-accelerated Adaptive Moment Estimation (nadam) is an extension of Adaptive Moment Estimation (adam)\n loss = 'sparse_categorical_crossentropy', # The one you need for a multi-class classification\n metrics = ['sparse_categorical_accuracy'], # The one you need for a multi-class classification\n)\n\nXception_.summary()","metadata":{"execution":{"iopub.status.busy":"2023-11-16T07:44:18.540917Z","iopub.execute_input":"2023-11-16T07:44:18.541194Z","iopub.status.idle":"2023-11-16T07:44:18.763673Z","shell.execute_reply.started":"2023-11-16T07:44:18.541166Z","shell.execute_reply":"2023-11-16T07:44:18.762787Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"'''\nYou don't need to train the model again. It has been trained and serialized.\nThis is the difference between .pkl and .h5 in this code:\n.pkl - carries everything, including the training history (serialized object)\n.h5 - contains the weights of a model's best perfoming configuration (checkpoint)\n'''\n\n# Xception_training = Xception_.fit(\n# ds_train,\n# validation_data=ds_valid,\n# epochs=EPOCHS,\n# steps_per_epoch=STEPS_PER_EPOCH,\n# callbacks=[lr_callback, early_stopping, Xception_checkpoint] # Here is where our callbacks go\n# )\n\n# joblib.dump(Xception_training, 'Xception.pkl')\n\nXception_training = joblib.load('/kaggle/input/cv-course-practical-guide-data/Xception.pkl')","metadata":{"execution":{"iopub.status.busy":"2023-11-16T07:44:18.764655Z","iopub.execute_input":"2023-11-16T07:44:18.764907Z","iopub.status.idle":"2023-11-16T07:44:33.222045Z","shell.execute_reply.started":"2023-11-16T07:44:18.764881Z","shell.execute_reply":"2023-11-16T07:44:33.220957Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Xception_history_frame = pd.DataFrame(Xception_training.history)\nax = Xception_history_frame.loc[:, ['loss', 'val_loss']].plot()\nXception_history_frame.loc[:, ['sparse_categorical_accuracy', 'val_sparse_categorical_accuracy']].plot(ax = ax)","metadata":{"execution":{"iopub.status.busy":"2023-11-16T07:44:33.222984Z","iopub.execute_input":"2023-11-16T07:44:33.223243Z","iopub.status.idle":"2023-11-16T07:44:33.443879Z","shell.execute_reply.started":"2023-11-16T07:44:33.223218Z","shell.execute_reply":"2023-11-16T07:44:33.443067Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Step 7: Model_2 - Custom CNN","metadata":{}},{"cell_type":"code","source":"\"\"\"\nFunny thing. When I add preprocessing, TPU refuses to train the model. GPU doesn't bother.\nIf you understand this, please share your insight.\n\"\"\"\n\nwith strategy.scope():\n custom_model = keras.Sequential([\n L.InputLayer(input_shape=[*IMAGE_SIZE, 3]),\n# preprocessing.RandomFlip(mode='horizontal'), \n# preprocessing.RandomFlip(mode='vertical'), \n# preprocessing.RandomRotation(factor=0.20),\n\n # Block One\n L.BatchNormalization(renorm=True),\n L.Conv2D(filters=64, kernel_size=3, activation='relu', padding='same'),\n L.Conv2D(filters=64, kernel_size=3, activation='relu', padding='same'),\n L.MaxPool2D(),\n\n # Block Two\n L.BatchNormalization(renorm=True),\n L.Conv2D(filters=128, kernel_size=3, activation='relu', padding='same'),\n L.Conv2D(filters=64, kernel_size=3, activation='relu', padding='same'),\n L.MaxPool2D(),\n\n # Block Three\n L.BatchNormalization(renorm=True),\n L.Conv2D(filters=256, kernel_size=3, activation='relu', padding='same'),\n L.Conv2D(filters=256, kernel_size=3, activation='relu', padding='same'),\n L.MaxPool2D(),\n\n # Head\n L.GlobalAveragePooling2D(), # attach a new head (GlobalAveragePooling averages feature maps produced by the base down to a single value per feature. Which is just right for a classification)\n L.Dropout(0.3), # add regularization \n L.Dense(len(CLASSES), activation='softmax') # output layer where\n # number of neurons corresponds to the number of classes \n ])\n \n custom_model.summary()","metadata":{"execution":{"iopub.status.busy":"2023-11-16T07:44:33.444870Z","iopub.execute_input":"2023-11-16T07:44:33.445151Z","iopub.status.idle":"2023-11-16T07:44:34.804665Z","shell.execute_reply.started":"2023-11-16T07:44:33.445122Z","shell.execute_reply":"2023-11-16T07:44:34.803859Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"custom_model.compile(\n optimizer = 'nadam',\n loss='sparse_categorical_crossentropy',\n metrics=['sparse_categorical_accuracy']\n)","metadata":{"execution":{"iopub.status.busy":"2023-11-16T07:44:34.805704Z","iopub.execute_input":"2023-11-16T07:44:34.805983Z","iopub.status.idle":"2023-11-16T07:44:34.839786Z","shell.execute_reply.started":"2023-11-16T07:44:34.805955Z","shell.execute_reply":"2023-11-16T07:44:34.838989Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Checkpoint for the second model\ncustom_model_checkpoint_filepath = 'custom_model.h5'\n\ncustom_model_checkpoint = ModelCheckpoint(\n filepath=custom_model_checkpoint_filepath,\n save_weights_only=True,\n monitor='val_loss',\n mode='min',\n save_best_only=True\n)","metadata":{"execution":{"iopub.status.busy":"2023-11-16T07:44:34.840740Z","iopub.execute_input":"2023-11-16T07:44:34.841005Z","iopub.status.idle":"2023-11-16T07:44:34.844833Z","shell.execute_reply.started":"2023-11-16T07:44:34.840977Z","shell.execute_reply":"2023-11-16T07:44:34.844171Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# custom_model_training = custom_model.fit(\n# ds_train,\n# validation_data=ds_valid,\n# epochs=EPOCHS,\n# steps_per_epoch=STEPS_PER_EPOCH,\n# callbacks=[\n# lr_callback,\n# early_stopping,\n# custom_model_checkpoint\n# ]\n# )\n\n# joblib.dump (custom_model_training, 'custom_model.pkl')\ncustom_model_training = joblib.load ('/kaggle/input/cv-course-practical-guide-data/custom_model.pkl')","metadata":{"execution":{"iopub.status.busy":"2023-11-16T07:44:34.845678Z","iopub.execute_input":"2023-11-16T07:44:34.845924Z","iopub.status.idle":"2023-11-16T07:44:35.299990Z","shell.execute_reply.started":"2023-11-16T07:44:34.845900Z","shell.execute_reply":"2023-11-16T07:44:35.299024Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\"\"\"\nNaturally, the 'Custom Convolutional Neural Network' is a world apart from a pre-trained base like Xception. \nIts role here is to assert its existence — a cherished creation, born through painstaking effort and nurtured with love.\nSeriously, I belive everyone should make his own working CNN before going to 'Transfer Learning'.\nMine is working 😎\n\"\"\"\n\nhistory_frame = pd.DataFrame(custom_model_training.history)\nax = history_frame.loc[:, ['loss', 'val_loss']].plot()\nhistory_frame.loc[:, ['sparse_categorical_accuracy', 'val_sparse_categorical_accuracy']].plot(ax = ax);","metadata":{"execution":{"iopub.status.busy":"2023-11-16T07:44:35.301092Z","iopub.execute_input":"2023-11-16T07:44:35.301380Z","iopub.status.idle":"2023-11-16T07:44:35.525718Z","shell.execute_reply.started":"2023-11-16T07:44:35.301353Z","shell.execute_reply":"2023-11-16T07:44:35.524965Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Step 7: Visual Validation\n- We will apply our **display_batch_of_images()** function to see the flowers, their predicted and true classes.\n- **Visual validation** can help reveal patterns of images the model has trouble with.","metadata":{}},{"cell_type":"code","source":"# Visual validation\ndataset = get_validation_dataset()\ndataset = dataset.unbatch().batch(20) # Display 20 images at a time. Fill free to put your number\nbatch = iter(dataset)","metadata":{"execution":{"iopub.status.busy":"2023-11-16T07:44:35.528949Z","iopub.execute_input":"2023-11-16T07:44:35.529263Z","iopub.status.idle":"2023-11-16T07:44:35.590714Z","shell.execute_reply.started":"2023-11-16T07:44:35.529231Z","shell.execute_reply":"2023-11-16T07:44:35.589918Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Xception\n# Run the cell again to see another set\nimages, labels = next(batch)\nprobabilities = Xception_.predict(images)\npredictions = np.argmax(probabilities, axis=-1)\ndisplay_batch_of_images((images, labels), predictions)","metadata":{"execution":{"iopub.status.busy":"2023-11-16T07:46:02.839535Z","iopub.execute_input":"2023-11-16T07:46:02.840051Z","iopub.status.idle":"2023-11-16T07:46:05.238213Z","shell.execute_reply.started":"2023-11-16T07:46:02.839997Z","shell.execute_reply":"2023-11-16T07:46:05.237032Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# custom_model\n# Run the cell again to see another set\nimages, labels = next(batch)\nprobabilities = custom_model.predict(images)\npredictions = np.argmax(probabilities, axis=-1)\ndisplay_batch_of_images((images, labels), predictions)","metadata":{"execution":{"iopub.status.busy":"2023-11-16T07:46:35.428939Z","iopub.execute_input":"2023-11-16T07:46:35.429434Z","iopub.status.idle":"2023-11-16T07:46:37.891035Z","shell.execute_reply.started":"2023-11-16T07:46:35.429395Z","shell.execute_reply":"2023-11-16T07:46:37.889926Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Step 9: Ensemble","metadata":{}},{"cell_type":"code","source":"cmdataset = get_validation_dataset(ordered=True) # since we are splitting the dataset and iterating separately on images and labels, order matters.\nimages_ds = cmdataset.map(lambda image, label: image) # makes a data set of images only\nlabels_ds = cmdataset.map(lambda image, label: label).unbatch() # makes a data set of labels only\n\ncm_correct_labels = next(iter(labels_ds.batch(NUM_VALIDATION_IMAGES))).numpy() # gets everything as one batch in np.array format.\n\nXception_.load_weights('/kaggle/input/cv-course-practical-guide-data/Xception.h5')\ncustom_model.load_weights('/kaggle/input/cv-course-practical-guide-data/custom_model.h5')\n\npredictions_1 = Xception_.predict(images_ds)\npredictions_2 = custom_model.predict(images_ds)\n\nscores = [] # creat a list to store F1 scores for different alpha values. \nfor alpha in np.linspace(0,1,100): # alpha is one of 100 evenly spaced values in range from 0 to 1\n cm_probabilities = alpha * predictions_1 + (1-alpha) * predictions_2 # Combine predictions from two models using the alpha weight\n cm_predictions = np.argmax(cm_probabilities, axis=-1) # For each example, select the class with the highest probability as the predicted class.\n scores.append(f1_score(cm_correct_labels, # Calculate the F1 score between the correct labels \n cm_predictions, # and the combined predictions.\n labels=range(len(CLASSES)), # It computes the F1 score for each class \n average='macro')) # and returns the macro-average.\n\nprint(\"Correct labels: \", cm_correct_labels.shape, cm_correct_labels)\nprint(\"Predicted labels: \", cm_predictions.shape, cm_predictions)\nplt.plot(scores)\n\nbest_alpha = np.argmax(scores)/100 # 'scores' is a list of 100 avarage F1 values. We find the index of max F1 value. When we devide this index (e.g.35) by 100, we find our alpha, that produced that F1 index. Tricky, but elegant.\nprint (f'best_alpha: {best_alpha}')","metadata":{"execution":{"iopub.status.busy":"2023-11-16T07:46:45.990593Z","iopub.execute_input":"2023-11-16T07:46:45.991545Z","iopub.status.idle":"2023-11-16T07:46:55.876737Z","shell.execute_reply.started":"2023-11-16T07:46:45.991486Z","shell.execute_reply":"2023-11-16T07:46:55.875787Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Step 10: Make Test Predictions","metadata":{}},{"cell_type":"code","source":"test_ds = get_test_dataset(ordered=True)\n\nprint('Computing predictions...')\ntest_images_ds = test_ds.map(lambda image, idnum: image)\n\nm1 = Xception_.predict(test_images_ds)\nm2 = custom_model.predict(test_images_ds)\n\nprobabilities = best_alpha*m1+(1-best_alpha)*m2\npredictions = np.argmax(probabilities, axis=-1) # Find the class with the highest probability for each image: \n # 'predictions' is a sequence of matrixes, where each matrix (idividual prediction) consists of 2 vectors: (1) all the classes (indexes) and (2) probabilities of an image to be that class\n # .argmax() returns index of a maximum value, i.e. class related to the highest probability value\nprint(predictions)","metadata":{"execution":{"iopub.status.busy":"2023-11-16T07:46:55.878126Z","iopub.execute_input":"2023-11-16T07:46:55.878425Z","iopub.status.idle":"2023-11-16T07:47:02.221108Z","shell.execute_reply.started":"2023-11-16T07:46:55.878383Z","shell.execute_reply":"2023-11-16T07:47:02.220003Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('Generating submission.csv file...')\n\ntest_ids_ds = test_ds.map(lambda image, idnum: idnum).unbatch() # Get image ids from test set \ntest_ids = next(iter(test_ids_ds.batch(NUM_TEST_IMAGES))).numpy().astype('U') # convert them to unicode\nsubmission_df = pd.DataFrame({'id': test_ids, 'label': predictions})\nsubmission_df.to_csv('submission.csv', index=False)\n\n\n# Look at the first few predictions\n!head submission.csv","metadata":{"execution":{"iopub.status.busy":"2023-11-16T07:47:02.222931Z","iopub.execute_input":"2023-11-16T07:47:02.223457Z","iopub.status.idle":"2023-11-16T07:47:03.672975Z","shell.execute_reply.started":"2023-11-16T07:47:02.223421Z","shell.execute_reply":"2023-11-16T07:47:03.671658Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Credits**:\n- [Rayn Holbrook](https://www.kaggle.com/ryanholbrook) for the wonderful [Computer Vision course](https://www.kaggle.com/learn/computer-vision)\n- [Alexis Cook](https://www.kaggle.com/alexisbcook) [Phil Culliton](https://www.kaggle.com/philculliton) for the [examplary notebook](https://www.kaggle.com/code/ryanholbrook/create-your-first-submission) wich this work is built upon.\n- George Zoto's awesome [kernel](https://www.kaggle.com/code/georgezoto/computer-vision-petals-to-the-metal) where I borrowed many techniques to complete this Notebook.\n\n\nIf you liked this notebook, please also check the others:\n - [🗺️Geospatial Analysis Course: Practical Guide](https://www.kaggle.com/ivanlydkin/geospatial-analysis-course-practical-guide)\n - [🕒Time Series Course: A Practical Guide](https://www.kaggle.com/code/ivanlydkin/time-series-course-a-practical-guide)\n - [🛳️Original feature for Titanic](https://www.kaggle.com/code/ivanlydkin/titanic-case-with-some-original-features)\n\n*I hope it was helpful* 🤝
\n\n Please upvote if you find this valuable, and don't hesitate to leave comments if you have any feedback or suggestions
","metadata":{}}]}