{"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":4104,"databundleVersionId":46661,"sourceType":"competition"},{"sourceId":7251,"sourceType":"datasetVersion","datasetId":2798}],"dockerImageVersionId":438,"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Overview\nThe goal is to make a nice retinopathy model by using a pretrained inception v3 as a base and retraining some modified final layers with attention\n\nThis can be massively improved with \n* high-resolution images\n* better data sampling\n* ensuring there is no leaking between training and validation sets, ```sample(replace = True)``` is real dangerous\n* better target variable (age) normalization\n* pretrained models\n* attention/related techniques to focus on areas","metadata":{"_uuid":"34b6997bb115a11f47a7f54ce9d9052791b2e707","_cell_guid":"c67e806c-e0e6-415b-8cf0-ed92eba5ed37"}},{"cell_type":"code","source":"# copy the weights and configurations for the pre-trained models\n!mkdir ~/.keras\n!mkdir ~/.keras/models\n!cp ../input/keras-pretrained-models/*notop* ~/.keras/models/\n!cp ../input/keras-pretrained-models/imagenet_class_index.json ~/.keras/models/","metadata":{"_uuid":"c163e45042a69905855f7c04a65676e5aca4837b","_cell_guid":"e94de3e7-de1f-4c18-ad1f-c8b686127340","trusted":true,"execution":{"iopub.status.busy":"2024-10-22T04:14:43.950415Z","iopub.execute_input":"2024-10-22T04:14:43.950705Z","iopub.status.idle":"2024-10-22T04:14:49.102113Z","shell.execute_reply.started":"2024-10-22T04:14:43.950663Z","shell.execute_reply":"2024-10-22T04:14:49.101079Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Code Breakdown: Importing Libraries for Data Processing, File I/O, and Visualization\n\nThis code snippet imports several essential Python libraries for data processing, file handling, image manipulation, and visualization, typically used in data science or machine learning projects. Below is a breakdown of each part:\n\n### 1. **Importing Libraries for Data Processing and Visualization**\nNumPy (np): A powerful library for numerical computing in Python. It supports efficient operations on large arrays and matrices, and offers a wide variety of mathematical functions for performing linear algebra, random number generation, and more.\n\nPandas (pd): A library that provides high-level data structures and methods designed to make data analysis fast and easy. It’s especially useful for handling tabular data, such as CSV files. Commonly used functions include pd.read_csv() for loading data into DataFrames for processing and analysis.\n\nMatplotlib (plt): A plotting library that provides functionalities for creating static, interactive, and animated visualizations. It is widely used for generating figures, plots, and charts to visually represent data.\n\nskimage.io.imread: Part of the scikit-image library, this function is used to read image files (like .jpg or .png) and convert them into NumPy arrays for image processing or machine learning tasks.\n\nos: This module allows Python to interact with the operating system, enabling functions such as file and directory navigation, environment manipulation, and path handling. For example, os.listdir() lists files in a directory.\n\nglob: A function that helps in pattern matching for file paths. It’s especially useful for retrieving files that match a specific pattern (e.g., all .jpg files in a directory) by using wildcards like *.\n\n","metadata":{}},{"cell_type":"code","source":"import numpy as np  # linear algebra\nimport pandas as pd  # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt  # showing and rendering figures\nfrom skimage.io import imread  # for reading image files\nimport os  # for interacting with the operating system\nfrom glob import glob  # for file pattern matching\n%matplotlib inline","metadata":{"_uuid":"725d378daf5f836d4885d67240fc7955f113309d","_cell_guid":"c3cc4285-bfa4-4612-ac5f-13d10678c09a","trusted":true,"execution":{"iopub.status.busy":"2024-10-22T04:23:26.182919Z","iopub.execute_input":"2024-10-22T04:23:26.183217Z","iopub.status.idle":"2024-10-22T04:23:26.191735Z","shell.execute_reply.started":"2024-10-22T04:23:26.183173Z","shell.execute_reply":"2024-10-22T04:23:26.191076Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"The code provided performs data preprocessing for diabetic retinopathy detection by reading image labels, mapping images to paths, and categorizing the data. Here's a step-by-step explanation:\n\n1. **Set Image Directory**: \n   The `base_image_dir` is defined as the path to the folder containing the diabetic retinopathy images and labels, located under `'input/diabetic-retinopathy-detection'`.\n\n2. **Load Labels**: \n   The labels for the training images are loaded into a DataFrame `retina_df` from the CSV file `'trainLabels.csv'`.\n\n3. **Extract Patient ID**: \n   A new column `'PatientId'` is added, extracting the patient's ID from the image filename (before the first underscore).\n\n4. **Map Image Paths**: \n   A new column `'path'` is created, which maps each image filename to its corresponding JPEG file path using `os.path.join`.\n\n5. **Check Image Existence**: \n   The `'exists'` column checks whether each image path exists on the file system using `os.path.exists`. The number of existing images is printed out.\n\n6. **Identify Left/Right Eye**: \n   A new column `'eye'` is created, where a value of `1` indicates the image corresponds to the left eye, and `0` corresponds to the right eye, based on the filename's suffix.\n\n7. **Categorize Disease Levels**: \n   The `'level_cat'` column is created, categorizing the severity levels of diabetic retinopathy using `to_categorical` from Keras, where the severity level is one-hot encoded.\n\n8. **Clean Data**: \n   Any rows with missing values are dropped, and only rows where the images exist are retained.\n\n9. **Random Sample**: \n   A random sample of 3 rows is displayed from the processed DataFrame.\n\nThis code helps set up the data for further analysis or model training in a diabetic retinopathy detection project.\n","metadata":{}},{"cell_type":"code","source":"base_image_dir = os.path.join('..', 'input', 'diabetic-retinopathy-detection')\nretina_df = pd.read_csv(os.path.join(base_image_dir, 'trainLabels.csv'))\nretina_df['PatientId'] = retina_df['image'].map(lambda x: x.split('_')[0])\nretina_df['path'] = retina_df['image'].map(lambda x: os.path.join(base_image_dir,\n                                                         '{}.jpeg'.format(x)))\nretina_df['exists'] = retina_df['path'].map(os.path.exists)\nprint(retina_df['exists'].sum(), 'images found of', retina_df.shape[0], 'total')\nretina_df['eye'] = retina_df['image'].map(lambda x: 1 if x.split('_')[-1]=='left' else 0)\nfrom keras.utils.np_utils import to_categorical\nretina_df['level_cat'] = retina_df['level'].map(lambda x: to_categorical(x, 1+retina_df['level'].max()))\n\nretina_df.dropna(inplace = True)\nretina_df = retina_df[retina_df['exists']]\nretina_df.sample(3)","metadata":{"_uuid":"346da81db6ee7a34af8da8af245b42e681f2ba48","_cell_guid":"c4b38df6-ffa1-4847-b605-511e72b68231","trusted":true,"execution":{"iopub.status.busy":"2024-10-22T04:37:01.525664Z","iopub.execute_input":"2024-10-22T04:37:01.526023Z","iopub.status.idle":"2024-10-22T04:37:01.59935Z","shell.execute_reply.started":"2024-10-22T04:37:01.525959Z","shell.execute_reply":"2024-10-22T04:37:01.598245Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Examine the distribution of eye and severity","metadata":{"_uuid":"688e4340238e013b8459b6f6470993c7de492d83","_cell_guid":"818da6ca-bbff-4ca0-ad57-ef3a145ae863"}},{"cell_type":"markdown","source":"The code snippet `retina_df[['level', 'eye']].hist(figsize = (10, 5))` generates histograms to visualize the distribution of the `'level'` and `'eye'` columns in the `retina_df` DataFrame. Here's what it does:\n\n- **Selection of Columns**: \n   It selects two columns, `'level'` and `'eye'`, from the `retina_df` DataFrame. \n   - `'level'` represents the severity of diabetic retinopathy (with higher numbers indicating more severe conditions).\n   - `'eye'` is a binary column (1 for left eye, 0 for right eye).\n\n- **Plotting Histograms**: \n   The `.hist()` function creates histograms to show the distribution of values in both the `'level'` and `'eye'` columns. \n   \n- **Figure Size**: \n   The `figsize = (10, 5)` argument sets the size of the resulting figure to 10 inches wide and 5 inches tall, ensuring the plot is clearly visible and well-proportioned.\n\n   This visualization helps to quickly observe the frequency of different diabetic retinopathy levels and the distribution of images for left vs. right eyes.\n","metadata":{}},{"cell_type":"code","source":"retina_df[['level', 'eye']].hist(figsize = (10, 5))","metadata":{"_uuid":"60a8111c4093ca6f69d27a4499442ba7dd750839","_cell_guid":"5c8bd288-8261-4cbe-a954-e62ac795cc3e","trusted":true,"execution":{"iopub.status.busy":"2024-10-22T04:05:13.54056Z","iopub.status.idle":"2024-10-22T04:05:13.541003Z","shell.execute_reply":"2024-10-22T04:05:13.540795Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Split Data into Training and Validation","metadata":{"_uuid":"4df45776bae0b8a1bf9d3eb4eaaebce6e24d726d","_cell_guid":"0ba697ed-85bb-4e9a-9765-4c367db078d1"}},{"cell_type":"markdown","source":"The provided code snippet performs the following steps to prepare the dataset for training and validation in a machine learning context:\n\n1. **Import Function**: \n   It imports the `train_test_split` function from the `sklearn.model_selection` module, which is used for splitting datasets into training and validation sets.\n\n2. **Prepare DataFrame**: \n   A new DataFrame `rr_df` is created, which consists of unique patient IDs and their corresponding diabetic retinopathy levels. This is done by selecting the `'PatientId'` and `'level'` columns from the original `retina_df` and dropping any duplicate entries.\n\n3. **Split Data**: \n   The `train_test_split` function is called to split the patient IDs into training and validation sets:\n   - `test_size = 0.25` indicates that 25% of the data will be reserved for validation.\n   - `random_state = 2018` ensures that the split is reproducible across different runs by providing a seed for random number generation.\n   - `stratify = rr_df['level']` ensures that the distribution of diabetic retinopathy levels is preserved in both the training and validation sets, which is important for maintaining representative samples.\n\n4. **Create DataFrames**: \n   Two new DataFrames are created based on the split patient IDs:\n   - `raw_train_df` contains all rows from the original `retina_df` where the `'PatientId'` is in the training IDs.\n   - `valid_df` contains rows where the `'PatientId'` is in the validation IDs.\n\n5. **Print Summary**: \n   Finally, it prints the number of samples in the training and validation datasets using `print('train', raw_train_df.shape[0], 'validation', valid_df.shape[0])`, providing insight into how many samples are included in each set.\n\nThis code effectively organizes the dataset into training and validation subsets while ensuring that the distribution of the diabetic retinopathy levels is consistent across both sets, which is crucial for effective model training and evaluation.\n","metadata":{}},{"cell_type":"code","source":"from sklearn.model_selection import train_test_split\nrr_df = retina_df[['PatientId', 'level']].drop_duplicates()\ntrain_ids, valid_ids = train_test_split(rr_df['PatientId'], \n                                   test_size = 0.25, \n                                   random_state = 2018,\n                                   stratify = rr_df['level'])\nraw_train_df = retina_df[retina_df['PatientId'].isin(train_ids)]\nvalid_df = retina_df[retina_df['PatientId'].isin(valid_ids)]\nprint('train', raw_train_df.shape[0], 'validation', valid_df.shape[0])","metadata":{"_uuid":"a48b300ca4d37a6e8b39f82e3c172739635e4baa","_cell_guid":"1192c6b3-a940-4fa0-a498-d7e0d400a796","trusted":true,"execution":{"iopub.status.busy":"2024-10-22T04:05:13.541628Z","iopub.status.idle":"2024-10-22T04:05:13.542054Z","shell.execute_reply":"2024-10-22T04:05:13.541865Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Balance the distribution in the training set","metadata":{"_uuid":"26e566d6cec5bd41f9afe392f456ddf7ceb306ea","_cell_guid":"f8060459-da1e-4293-8f61-c7f99de1de9f"}},{"cell_type":"markdown","source":"The provided code snippet processes the training data to ensure a balanced dataset by sampling from each group defined by the diabetic retinopathy level and eye type. Here's a detailed explanation:\n\n1. **Group By and Sampling**: \n   The line `train_df = raw_train_df.groupby(['level', 'eye']).apply(lambda x: x.sample(75, replace = True))` performs the following:\n   - It groups the `raw_train_df` DataFrame by the `'level'` (severity of diabetic retinopathy) and `'eye'` (left or right eye) columns.\n   - For each group, it samples 75 rows, allowing replacement (`replace=True`). This means that if a group has fewer than 75 entries, it can sample the same entries multiple times to meet the required sample size.\n\n2. **Resetting Index**: \n   The resulting DataFrame from the sampling is reset to have a clean index with `reset_index(drop=True)`, discarding the old index.\n\n3. **Print New and Old Sizes**: \n   The code prints the new size of the `train_df` DataFrame, which reflects the number of samples after balancing, alongside the old size of the `raw_train_df`. The statement `print('New Data Size:', train_df.shape[0], 'Old Size:', raw_train_df.shape[0])` provides a comparison of the number of samples before and after sampling.\n\n4. **Histogram of New Data**: \n   Finally, the line `train_df[['level', 'eye']].hist(figsize = (10, 5))` creates histograms to visualize the distribution of the `'level'` and `'eye'` columns in the new, balanced training dataset. The `figsize` parameter specifies the dimensions of the figure.\n\nThis code ensures that the training data is balanced across different severity levels and eye types, which can help improve the performance and robustness of machine learning models trained on this data.\n","metadata":{}},{"cell_type":"code","source":"train_df = raw_train_df.groupby(['level', 'eye']).apply(lambda x: x.sample(75, replace = True)\n                                                      ).reset_index(drop = True)\nprint('New Data Size:', train_df.shape[0], 'Old Size:', raw_train_df.shape[0])\ntrain_df[['level', 'eye']].hist(figsize = (10, 5))","metadata":{"_uuid":"ba7befa238b8c11f9672e3539ac58f3da6955bd9","_cell_guid":"7a130199-fbf6-4c60-95f5-0797b2f3eaf1","trusted":true,"execution":{"iopub.status.busy":"2024-10-22T04:05:13.542684Z","iopub.status.idle":"2024-10-22T04:05:13.543045Z","shell.execute_reply":"2024-10-22T04:05:13.542897Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"The provided code defines functions for image loading and augmentation using TensorFlow and Keras. It prepares images for training a deep learning model, specifically for diabetic retinopathy detection or similar tasks. Here’s a breakdown of the key components:\n\n### Imports\n- **TensorFlow and Keras**: The code uses TensorFlow (`tf`) for image processing and Keras backend for compatibility with Keras models. \n- **NumPy**: Used for numerical operations, particularly for handling angles in radians.\n\n### Constants\n- **IMG_SIZE**: Set to `(512, 512)`, which indicates the target size for input images, slightly smaller than what VGG16 normally expects.\n\n### Image Loader Function (`tf_image_loader`)\nThis function is designed to load and preprocess images with several augmentation options:\n- **Parameters**:\n  - `out_size`: Target size of the output images.\n  - Augmentation parameters: Flags for horizontal/vertical flips, brightness, contrast, saturation, and hue adjustments.\n  - `color_mode`: Specifies if images are loaded in RGB mode.\n  - `preproc_func`: Function to preprocess images (default is `preprocess_input` from Keras).\n  - `on_batch`: A flag to indicate if the function should operate on a batch of images or individual images.\n\n- **Functionality**:\n  - Decodes images from PNG format and resizes them.\n  - Applies various augmentations based on the specified flags (e.g., random flips, brightness, saturation, hue, and contrast adjustments).\n  - Calls the preprocessing function on the final output.\n  - Returns a function tailored for either batch processing or individual image processing.\n\n### Augmentor Function (`tf_augmentor`)\nThis function creates a pipeline for data augmentation, combining the image loader with additional transformations:\n- **Parameters**:\n  - `out_size`: Final size of the output images.\n  - `intermediate_size`: Size for intermediate processing (default is `(640, 640)`).\n  - `batch_size`: The size of image batches to be processed.\n  - Similar augmentation flags and parameters as in the image loader.\n\n- **Functionality**:\n  - Defines a `load_ops` function using the `tf_image_loader` to prepare images for augmentation.\n  - In the `batch_ops` function, it performs various transformations:\n    - **Rotation**: Randomly rotates images within a specified range.\n    - **Cropping**: Randomly crops images based on specified probabilities.\n    - Applies affine transformations to images if any transformations are specified.\n    - Resizes images to the output size, either by scaling or cropping, based on the `intermediate_trans` parameter.\n\n- **Pipeline Creation**: \n  - Returns a function that creates a data pipeline for augmenting a dataset using TensorFlow's `Dataset` API, applying the defined transformations and batching the data for training.\n\n### Summary\nOverall, the code establishes a robust framework for loading, augmenting, and preprocessing images to enhance model training. By incorporating a variety of transformations, it helps in creating a more diverse training dataset, which can improve the model's performance and generalization capabilities.\n","metadata":{}},{"cell_type":"code","source":"import tensorflow as tf\nfrom keras import backend as K\nfrom keras.applications.inception_v3 import preprocess_input\nimport numpy as np\nIMG_SIZE = (512, 512) # slightly smaller than vgg16 normally expects\ndef tf_image_loader(out_size, \n                      horizontal_flip = True, \n                      vertical_flip = False, \n                     random_brightness = True,\n                     random_contrast = True,\n                    random_saturation = True,\n                    random_hue = True,\n                      color_mode = 'rgb',\n                       preproc_func = preprocess_input,\n                       on_batch = False):\n    def _func(X):\n        with tf.name_scope('image_augmentation'):\n            with tf.name_scope('input'):\n                X = tf.image.decode_png(tf.read_file(X), channels = 3 if color_mode == 'rgb' else 0)\n                X = tf.image.resize_images(X, out_size)\n            with tf.name_scope('augmentation'):\n                if horizontal_flip:\n                    X = tf.image.random_flip_left_right(X)\n                if vertical_flip:\n                    X = tf.image.random_flip_up_down(X)\n                if random_brightness:\n                    X = tf.image.random_brightness(X, max_delta = 0.1)\n                if random_saturation:\n                    X = tf.image.random_saturation(X, lower = 0.75, upper = 1.5)\n                if random_hue:\n                    X = tf.image.random_hue(X, max_delta = 0.15)\n                if random_contrast:\n                    X = tf.image.random_contrast(X, lower = 0.75, upper = 1.5)\n                return preproc_func(X)\n    if on_batch: \n        # we are meant to use it on a batch\n        def _batch_func(X, y):\n            return tf.map_fn(_func, X), y\n        return _batch_func\n    else:\n        # we apply it to everything\n        def _all_func(X, y):\n            return _func(X), y         \n        return _all_func\n    \ndef tf_augmentor(out_size,\n                intermediate_size = (640, 640),\n                 intermediate_trans = 'crop',\n                 batch_size = 16,\n                   horizontal_flip = True, \n                  vertical_flip = False, \n                 random_brightness = True,\n                 random_contrast = True,\n                 random_saturation = True,\n                    random_hue = True,\n                  color_mode = 'rgb',\n                   preproc_func = preprocess_input,\n                   min_crop_percent = 0.001,\n                   max_crop_percent = 0.005,\n                   crop_probability = 0.5,\n                   rotation_range = 10):\n    \n    load_ops = tf_image_loader(out_size = intermediate_size, \n                               horizontal_flip=horizontal_flip, \n                               vertical_flip=vertical_flip, \n                               random_brightness = random_brightness,\n                               random_contrast = random_contrast,\n                               random_saturation = random_saturation,\n                               random_hue = random_hue,\n                               color_mode = color_mode,\n                               preproc_func = preproc_func,\n                               on_batch=False)\n    def batch_ops(X, y):\n        batch_size = tf.shape(X)[0]\n        with tf.name_scope('transformation'):\n            # code borrowed from https://becominghuman.ai/data-augmentation-on-gpu-in-tensorflow-13d14ecf2b19\n            # The list of affine transformations that our image will go under.\n            # Every element is Nx8 tensor, where N is a batch size.\n            transforms = []\n            identity = tf.constant([1, 0, 0, 0, 1, 0, 0, 0], dtype=tf.float32)\n            if rotation_range > 0:\n                angle_rad = rotation_range / 180 * np.pi\n                angles = tf.random_uniform([batch_size], -angle_rad, angle_rad)\n                transforms += [tf.contrib.image.angles_to_projective_transforms(angles, intermediate_size[0], intermediate_size[1])]\n\n            if crop_probability > 0:\n                crop_pct = tf.random_uniform([batch_size], min_crop_percent, max_crop_percent)\n                left = tf.random_uniform([batch_size], 0, intermediate_size[0] * (1.0 - crop_pct))\n                top = tf.random_uniform([batch_size], 0, intermediate_size[1] * (1.0 - crop_pct))\n                crop_transform = tf.stack([\n                      crop_pct,\n                      tf.zeros([batch_size]), top,\n                      tf.zeros([batch_size]), crop_pct, left,\n                      tf.zeros([batch_size]),\n                      tf.zeros([batch_size])\n                  ], 1)\n                coin = tf.less(tf.random_uniform([batch_size], 0, 1.0), crop_probability)\n                transforms += [tf.where(coin, crop_transform, tf.tile(tf.expand_dims(identity, 0), [batch_size, 1]))]\n            if len(transforms)>0:\n                X = tf.contrib.image.transform(X,\n                      tf.contrib.image.compose_transforms(*transforms),\n                      interpolation='BILINEAR') # or 'NEAREST'\n            if intermediate_trans=='scale':\n                X = tf.image.resize_images(X, out_size)\n            elif intermediate_trans=='crop':\n                X = tf.image.resize_image_with_crop_or_pad(X, out_size[0], out_size[1])\n            else:\n                raise ValueError('Invalid Operation {}'.format(intermediate_trans))\n            return X, y\n    def _create_pipeline(in_ds):\n        batch_ds = in_ds.map(load_ops, num_parallel_calls=4).batch(batch_size)\n        return batch_ds.map(batch_ops)\n    return _create_pipeline","metadata":{"_uuid":"9529ab766763a9f122786464c24ab1ebe22c6006","_cell_guid":"9954bfda-29bd-4c4d-b526-0a972b3e43e2","trusted":true,"execution":{"iopub.status.busy":"2024-10-22T04:05:13.543834Z","iopub.status.idle":"2024-10-22T04:05:13.544189Z","shell.execute_reply":"2024-10-22T04:05:13.544054Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"The provided code defines a function `flow_from_dataframe` that generates batches of images and corresponding labels from a DataFrame for use in training a machine learning model with TensorFlow. This function leverages the capabilities of TensorFlow's `tf.data.Dataset` API and Keras' image data generators (IDG). Here's a breakdown of its components:\n\n### Parameters\n- **`idg`**: An image data generator function or instance that preprocesses images as they are loaded.\n- **`in_df`**: A DataFrame containing the image file paths and their corresponding labels.\n- **`path_col`**: The column name in the DataFrame that contains the paths to the image files.\n- **`y_col`**: The column name in the DataFrame that contains the labels or target values associated with each image.\n- **`shuffle`**: A boolean parameter indicating whether to shuffle the dataset before generating batches (default is `True`).\n- **`color_mode`**: Specifies the color mode for loading images (default is `'rgb'`).\n\n### Functionality\n1. **Create TensorFlow Dataset**:\n   - The function begins by creating a TensorFlow dataset (`files_ds`) from the specified columns of the DataFrame. It uses `tf.data.Dataset.from_tensor_slices`, which pairs image file paths (from `path_col`) with their corresponding labels (from `y_col`).\n\n2. **Determine Length**:\n   - The variable `in_len` captures the number of image entries in the DataFrame to facilitate batch processing.\n\n3. **Infinite Loop**:\n   - The function contains a `while True:` loop, allowing it to generate an infinite stream of data batches for training.\n\n4. **Shuffling**:\n   - If `shuffle` is set to `True`, the dataset is shuffled to ensure that images are presented in a random order, which can help improve model training by preventing overfitting to the order of the data.\n\n5. **Batch Generation**:\n   - `next_batch` is defined as the next set of images and labels to be yielded. The dataset is processed by the IDG function, repeating indefinitely (`repeat()`) and creating an iterator that fetches the next set of data.\n\n6. **Yield Batches**:\n   - The function yields batches of images and labels. It iterates over the dataset, with a maximum of `in_len // 32` iterations to ensure that the output is manageable. The loop is designed to be thread-safe, enabling the function to work in a multi-threaded context.\n\n7. **Session Execution**:\n   - `K.get_session().run(next_batch)` executes the batch fetching operation in a Keras session, retrieving the next batch of images and their associated labels.\n\n### Summary\nThe `flow_from_dataframe` function serves as a generator that efficiently feeds batches of preprocessed images and labels into a model during training. By integrating TensorFlow's dataset handling with Keras' image preprocessing capabilities, it provides a flexible and powerful way to manage image data pipelines. This design allows for on-the-fly data augmentation and shuffling, enhancing the training process's effectiveness.\n","metadata":{}},{"cell_type":"code","source":"def flow_from_dataframe(idg, \n                        in_df, \n                        path_col,\n                        y_col, \n                        shuffle = True, \n                        color_mode = 'rgb'):\n    files_ds = tf.data.Dataset.from_tensor_slices((in_df[path_col].values, \n                                                   np.stack(in_df[y_col].values,0)))\n    in_len = in_df[path_col].values.shape[0]\n    while True:\n        if shuffle:\n            files_ds = files_ds.shuffle(in_len) # shuffle the whole dataset\n        \n        next_batch = idg(files_ds).repeat().make_one_shot_iterator().get_next()\n        for i in range(max(in_len//32,1)):\n            # NOTE: if we loop here it is 'thread-safe-ish' if we loop on the outside it is completely unsafe\n            yield K.get_session().run(next_batch)","metadata":{"_uuid":"07851e798db3d89ba13db7d4b56ab2b759221464","_cell_guid":"b5767f42-da63-4737-8f50-749c1a25aa84","trusted":true,"execution":{"iopub.status.busy":"2024-10-22T04:05:13.544742Z","iopub.status.idle":"2024-10-22T04:05:13.54507Z","shell.execute_reply":"2024-10-22T04:05:13.544938Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"The provided code snippet sets up data generators for training and validation datasets, specifically tailored for an image classification task, such as diabetic retinopathy detection. It utilizes the previously defined augmentation and data loading functions to create pipelines for processing image data. Here’s a breakdown of the key components:\n\n### Batch Size\n- **`batch_size = 48`**: This variable sets the size of the batches for training and validation. A batch size of 48 means that the model will process 48 images at a time during training and evaluation.\n\n### Core Image Data Generator for Training (`core_idg`)\n- **`tf_augmentor`**: The function is called to create an image augmentation pipeline for the training data:\n  - **`out_size = IMG_SIZE`**: The target size for images is set to the previously defined `IMG_SIZE` of `(512, 512)`.\n  - **`color_mode = 'rgb'`**: The images will be processed in RGB color mode.\n  - **`vertical_flip = True`**: Random vertical flipping of images will be applied during training for augmentation.\n  - **`crop_probability = 0.0`**: Cropping is disabled for now, as indicated by the zero value.\n  - **`batch_size = batch_size`**: The batch size is set to 48.\n\n### Validation Image Data Generator (`valid_idg`)\n- A separate image augmentation pipeline is set up for validation data:\n  - Similar parameters are provided for `out_size` and `color_mode`.\n  - **`crop_probability = 0.0`**: Cropping remains disabled.\n  - **`horizontal_flip = False`, `vertical_flip = False`**: No random flipping is applied to validation images to ensure consistency in evaluation.\n  - **`random_brightness`, `random_contrast`, `random_saturation`, `random_hue`**: All these augmentation options are set to `False`, ensuring that the validation images are not altered.\n  - **`rotation_range = 0`**: No rotation is applied to validation images.\n  - **`batch_size = batch_size`**: Again, the batch size is set to 48.\n\n### Training Data Generator (`train_gen`)\n- **`flow_from_dataframe`**: This function generates batches of images and labels for training:\n  - **`core_idg`**: The training image data generator created earlier is passed in.\n  - **`train_df`**: The DataFrame containing training data (file paths and labels) is provided.\n  - **`path_col = 'path'`**: Specifies the column containing image file paths.\n  - **`y_col = 'level_cat'`**: Specifies the column containing the categorical labels.\n\n### Validation Data Generator (`valid_gen`)\n- Similarly, the validation data generator is created using:\n  - **`valid_idg`**: The validation image data generator.\n  - **`valid_df`**: The DataFrame containing validation data.\n  - The same `path_col` and `y_col` parameters are used to specify where to find the images and labels.\n\n### Summary\nThis code sets up the training and validation data generators using the specified augmentations and configurations. The training generator includes data augmentation to increase the diversity of the training dataset, while the validation generator maintains the original images for accurate model evaluation. By using these generators, the model can effectively learn from varied training examples while being validated on consistent, unaltered data.\n","metadata":{}},{"cell_type":"code","source":"batch_size = 48\ncore_idg = tf_augmentor(out_size = IMG_SIZE, \n                        color_mode = 'rgb', \n                 normalization       vertical_flip = True,\n                        crop_probability=0.0, # crop doesn't work yet\n                        batch_size = batch_size) \nvalid_idg = tf_augmentor(out_size = IMG_SIZE, color_mode = 'rgb', \n                         crop_probability=0.0, \n                         horizontal_flip = False, \n                         vertical_flip = False, \n                         random_brightness = False,\n                         random_contrast = False,\n                         random_saturation = False,\n                         random_hue = False,\n                         rotation_range = 0,\n                        batch_size = batch_size)\n\ntrain_gen = flow_from_dataframe(core_idg, train_df, \n                             path_col = 'path',\n                            y_col = 'level_cat')\n\nvalid_gen = flow_from_dataframe(valid_idg, valid_df, \n                             path_col = 'path',\n                            y_col = 'level_cat') # we can use much larger batches for evaluation","metadata":{"_uuid":"1848f5048a9e00668c3778a85deea97f980e4f1c","_cell_guid":"810bd229-fec9-43c4-b3bd-afd62e3e9552","trusted":true,"execution":{"iopub.status.busy":"2024-10-22T04:05:13.545561Z","iopub.status.idle":"2024-10-22T04:05:13.54594Z","shell.execute_reply":"2024-10-22T04:05:13.545791Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Validation Set\nWe do not perform augmentation at all on these images","metadata":{"_uuid":"2ad184b936d5cebae91a265a247d8e0e25920566"}},{"cell_type":"markdown","source":"The provided code snippet is responsible for visualizing a batch of images and their corresponding labels from the validation dataset. It uses Matplotlib to create a grid of subplots, displaying the processed images along with their severity labels. Here's a breakdown of the code:\n\n### Data Retrieval\n- **`t_x, t_y = next(valid_gen)`**: This line retrieves the next batch of images (`t_x`) and their associated labels (`t_y`) from the validation data generator (`valid_gen`). `t_x` will contain the image data, while `t_y` holds the categorical labels for each image.\n\n### Creating Subplots\n- **`fig, m_axs = plt.subplots(2, 4, figsize = (16, 8))`**: This line creates a figure (`fig`) with a grid of subplots arranged in 2 rows and 4 columns, resulting in a total of 8 subplots. The `figsize` parameter specifies the overall size of the figure in inches.\n\n### Visualizing Images\n- The `for` loop iterates over the images (`c_x`), their corresponding labels (`c_y`), and the axes (`c_ax`) of the subplots:\n  - **`zip(t_x, t_y, m_axs.flatten())`**: This combines the images, labels, and axes into a single iterable, allowing for simultaneous iteration over them.\n\n- **`c_ax.imshow(np.clip(c_x * 127 + 127, 0, 255).astype(np.uint8))`**: \n  - The `imshow` function is used to display the image in the current subplot (`c_ax`).\n  - The image data `c_x` is scaled from the range `[-1, 1]` (assuming the preprocessing used this scale) back to the standard RGB range `[0, 255]`. The formula `np.clip(c_x * 127 + 127, 0, 255)` adjusts the values, and `astype(np.uint8)` converts the data type to an unsigned 8-bit integer, which is required for display.\n  \n- **`c_ax.set_title('Severity {}'.format(np.argmax(c_y, -1)))`**: This line sets the title for the subplot to indicate the severity level associated with the image. The severity is determined by finding the index of the maximum value in the categorical label array `c_y` using `np.argmax`, which corresponds to the class with the highest probability.\n\n- **`c_ax.axis('off')`**: This command hides the axis ticks and labels for a cleaner visualization.\n\n### Summary\nOverall, this code snippet effectively visualizes a batch of validation images along with their corresponding severity labels in a grid layout. It provides a clear representation of the model's input data, allowing for qualitative assessment of the images and their classifications.\n","metadata":{}},{"cell_type":"code","source":"t_x, t_y = next(valid_gen)\nfig, m_axs = plt.subplots(2, 4, figsize = (16, 8))\nfor (c_x, c_y, c_ax) in zip(t_x, t_y, m_axs.flatten()):\n    c_ax.imshow(np.clip(c_x*127+127, 0, 255).astype(np.uint8))\n    c_ax.set_title('Severity {}'.format(np.argmax(c_y, -1)))\n    c_ax.axis('off')","metadata":{"trusted":true,"_uuid":"6810407e25b887dd8b352f1e46fb3faceaa58ab7","execution":{"iopub.status.busy":"2024-10-22T04:05:13.546475Z","iopub.status.idle":"2024-10-22T04:05:13.546839Z","shell.execute_reply":"2024-10-22T04:05:13.546669Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Training Set\nThese are augmented and a real mess","metadata":{"_uuid":"34ce892a19c9734511e2da1d0f2552b361dc826d"}},{"cell_type":"markdown","source":"This code snippet is designed to visualize a batch of training images along with their corresponding severity labels. Similar to the previous snippet for validation data, it uses Matplotlib to create a grid of subplots. Here’s a breakdown of each part of the code:\n\n### Data Retrieval\n- **`t_x, t_y = next(train_gen)`**: This line retrieves the next batch of images (`t_x`) and their associated labels (`t_y`) from the training data generator (`train_gen`). `t_x` contains the augmented image data, while `t_y` holds the categorical labels corresponding to each image.\n\n### Creating Subplots\n- **`fig, m_axs = plt.subplots(2, 4, figsize = (16, 8))`**: This line creates a figure (`fig`) with a grid of subplots arranged in 2 rows and 4 columns, allowing for the visualization of a total of 8 images. The `figsize` parameter specifies the dimensions of the figure in inches.\n\n### Visualizing Images\n- The `for` loop iterates over the images (`c_x`), their corresponding labels (`c_y`), and the axes (`c_ax`) of the subplots:\n  - **`zip(t_x, t_y, m_axs.flatten())`**: This combines the images, labels, and axes into a single iterable, enabling simultaneous iteration over them.\n\n- **`c_ax.imshow(np.clip(c_x * 127 + 127, 0, 255).astype(np.uint8))`**: \n  - The `imshow` function displays the current image in the subplot (`c_ax`).\n  - The image data `c_x`, which is likely scaled between `[-1, 1]` after preprocessing, is adjusted back to the standard RGB range `[0, 255]` using the formula `np.clip(c_x * 127 + 127, 0, 255)`. The `clip` function ensures that values are within this range, and `astype(np.uint8)` converts the data type to an unsigned 8-bit integer for proper display.\n\n- **`c_ax.set_title('Severity {}'.format(np.argmax(c_y, -1)))`**: This line sets the title for the subplot, indicating the severity level of the condition represented by the image. The severity is determined by finding the index of the maximum value in the categorical label array `c_y` using `np.argmax`, which corresponds to the class with the highest probability.\n\n- **`c_ax.axis('off')`**: This command removes the axis ticks and labels from the subplot for a cleaner visualization, focusing attention on the images themselves.\n\n### Summary\nIn summary, this code snippet visualizes a batch of training images and their severity labels in a structured grid layout. It provides insight into the images used for training the model, allowing for a qualitative assessment of the data and the effectiveness of the augmentation techniques applied.\n","metadata":{}},{"cell_type":"code","source":"t_x, t_y = next(train_gen)\nfig, m_axs = plt.subplots(2, 4, figsize = (16, 8))\nfor (c_x, c_y, c_ax) in zip(t_x, t_y, m_axs.flatten()):\n    c_ax.imshow(np.clip(c_x*127+127, 0, 255).astype(np.uint8))\n    c_ax.set_title('Severity {}'.format(np.argmax(c_y, -1)))\n    c_ax.axis('off')","metadata":{"_uuid":"8190b4ad60d49fa65af074dd138a19cb8787e983","scrolled":true,"_cell_guid":"2d62234f-aeb0-4eba-8a38-d713d819abf6","trusted":true,"execution":{"iopub.status.busy":"2024-10-22T04:05:13.5474Z","iopub.status.idle":"2024-10-22T04:05:13.547663Z","shell.execute_reply":"2024-10-22T04:05:13.547538Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Attention Model\nThe basic idea is that a Global Average Pooling is too simplistic since some of the regions are more relevant than others. So we build an attention mechanism to turn pixels in the GAP on an off before the pooling and then rescale (Lambda layer) the results based on the number of pixels. The model could be seen as a sort of 'global weighted average' pooling. There is probably something published about it and it is very similar to the kind of attention models used in NLP.\nIt is largely based on the insight that the winning solution annotated and trained a UNET model to segmenting the hand and transforming it. This seems very tedious if we could just learn attention.","metadata":{"_uuid":"55d665e1e8a8d83b9db005a66a965f8a90c62da1","_cell_guid":"da22790a-672c-474e-b118-9eef15b53160"}},{"cell_type":"markdown","source":"This code snippet outlines the construction of a deep learning model using a pre-trained convolutional neural network (CNN) as its backbone, specifically focusing on implementing an attention mechanism. Below is a detailed breakdown of each section of the code:\n\n### Importing Required Libraries\n- **`from keras.applications.vgg16 import VGG16 as PTModel`**: Imports the VGG16 model.\n- **`from keras.applications.inception_resnet_v2 import InceptionResNetV2 as PTModel`**: Imports the InceptionResNetV2 model.\n- **`from keras.applications.inception_v3 import InceptionV3 as PTModel`**: Imports the InceptionV3 model.\n- **`from keras.layers import ...`**: Imports necessary layers for model construction, such as `GlobalAveragePooling2D`, `Dense`, and various convolutional layers.\n- **`from keras.models import Model`**: Imports the Model class to create a Keras model.\n\n### Model Input and Pre-trained Backbone\n- **`in_lay = Input(t_x.shape[1:])`**: Defines the input layer of the model, where `t_x.shape[1:]` corresponds to the dimensions of the input images.\n- **`base_pretrained_model = PTModel(input_shape =  t_x.shape[1:], include_top = False, weights = 'imagenet')`**: Initializes the selected pre-trained model without the top layers (i.e., the classification layers) and with weights trained on the ImageNet dataset.\n- **`base_pretrained_model.trainable = False`**: Freezes the weights of the pre-trained model, preventing them from being updated during training.\n- **`pt_depth = base_pretrained_model.get_output_shape_at(0)[-1]`**: Retrieves the number of output channels from the pre-trained model.\n\n### Feature Extraction and Batch Normalization\n- **`pt_features = base_pretrained_model(in_lay)`**: Passes the input through the pre-trained model to obtain feature representations.\n- **`bn_features = BatchNormalization()(pt_features)`**: Applies batch normalization to the extracted features, improving training stability.\n\n### Attention Mechanism\n- **Attention Layer Construction**:\n  - The attention mechanism is implemented using a series of convolutional layers:\n    - **`attn_layer = Conv2D(64, kernel_size = (1,1), padding = 'same', activation = 'relu')(Dropout(0.5)(bn_features))`**: Applies a 1x1 convolution to generate attention weights after dropping out 50% of the features.\n    - **Subsequent Convolutions**: Two more convolutional layers are applied to reduce the number of channels to 16 and then 8, ending with a final convolution that outputs a single channel using the sigmoid activation function to produce attention masks.\n  \n- **Fanning Out to All Channels**:\n  - **`up_c2_w = np.ones((1, 1, 1, pt_depth))`**: Initializes weights for the up-sampling convolution layer.\n  - **`up_c2 = Conv2D(pt_depth, kernel_size = (1,1), padding = 'same', activation = 'linear', use_bias = False, weights = [up_c2_w])`**: This layer expands the attention mask back to match the depth of the original features.\n  \n- **Masking Features**:\n  - **`mask_features = multiply([attn_layer, bn_features])`**: Multiplies the attention mask with the normalized features, effectively focusing on relevant areas of the image.\n  - **`gap_features = GlobalAveragePooling2D()(mask_features)`**: Applies global average pooling to the masked features.\n  - **`gap_mask = GlobalAveragePooling2D()(attn_layer)`**: Applies global average pooling to the attention mask.\n\n### Handling Missing Values and Final Layers\n- **`gap = Lambda(lambda x: x[0]/x[1], name = 'RescaleGAP')([gap_features, gap_mask])`**: Rescales the pooled features to account for any missing values due to the attention mechanism.\n- **`gap_dr = Dropout(0.25)(gap)`**: Applies dropout to the rescaled features to reduce overfitting.\n- **`dr_steps = Dropout(0.25)(Dense(128, activation = 'relu')(gap_dr))`**: Adds a dense layer with ReLU activation followed by another dropout layer.\n- **`out_layer = Dense(t_y.shape[-1], activation = 'softmax')(dr_steps)`**: Creates the output layer with a softmax activation function for multi-class classification.\n\n### Model Compilation\n- **`retina_model = Model(inputs = [in_lay], outputs = [out_layer])`**: Defines the model with specified inputs and outputs.\n- **Custom Metric**: A custom top-2 accuracy function is defined using Keras's `top_k_categorical_accuracy`.\n- **`retina_model.compile(...)`**: Compiles the model with the Adam optimizer, categorical cross-entropy loss, and specified metrics.\n\n### Model Summary\n- **`retina_model.summary()`**: Outputs a summary of the model architecture, including layer types, output shapes, and the total number of parameters.\n\n### Summary\nOverall, this code constructs a sophisticated convolutional neural network that utilizes transfer learning with attention mechanisms for improved feature representation. The model is designed for image classification tasks, specifically in the context of diabetic retinopathy detection.\n","metadata":{}},{"cell_type":"code","source":"from keras.applications.vgg16 import VGG16 as PTModel\nfrom keras.applications.inception_resnet_v2 import InceptionResNetV2 as PTModel\nfrom keras.applications.inception_v3 import InceptionV3 as PTModel\nfrom keras.layers import GlobalAveragePooling2D, Dense, Dropout, Flatten, Input, Conv2D, multiply, LocallyConnected2D, Lambda\nfrom keras.models import Model\nin_lay = Input(t_x.shape[1:])\nbase_pretrained_model = PTModel(input_shape =  t_x.shape[1:], include_top = False, weights = 'imagenet')\nbase_pretrained_model.trainable = False\npt_depth = base_pretrained_model.get_output_shape_at(0)[-1]\npt_features = base_pretrained_model(in_lay)\nfrom keras.layers import BatchNormalization\nbn_features = BatchNormalization()(pt_features)\n\n# here we do an attention mechanism to turn pixels in the GAP on an off\n\nattn_layer = Conv2D(64, kernel_size = (1,1), padding = 'same', activation = 'relu')(Dropout(0.5)(bn_features))\nattn_layer = Conv2D(16, kernel_size = (1,1), padding = 'same', activation = 'relu')(attn_layer)\nattn_layer = Conv2D(8, kernel_size = (1,1), padding = 'same', activation = 'relu')(attn_layer)\nattn_layer = Conv2D(1, \n                    kernel_size = (1,1), \n                    padding = 'valid', \n                    activation = 'sigmoid')(attn_layer)\n# fan it out to all of the channels\nup_c2_w = np.ones((1, 1, 1, pt_depth))\nup_c2 = Conv2D(pt_depth, kernel_size = (1,1), padding = 'same', \n               activation = 'linear', use_bias = False, weights = [up_c2_w])\nup_c2.trainable = False\nattn_layer = up_c2(attn_layer)\n\nmask_features = multiply([attn_layer, bn_features])\ngap_features = GlobalAveragePooling2D()(mask_features)\ngap_mask = GlobalAveragePooling2D()(attn_layer)\n# to account for missing values from the attention model\ngap = Lambda(lambda x: x[0]/x[1], name = 'RescaleGAP')([gap_features, gap_mask])\ngap_dr = Dropout(0.25)(gap)\ndr_steps = Dropout(0.25)(Dense(128, activation = 'relu')(gap_dr))\nout_layer = Dense(t_y.shape[-1], activation = 'softmax')(dr_steps)\nretina_model = Model(inputs = [in_lay], outputs = [out_layer])\nfrom keras.metrics import top_k_categorical_accuracy\ndef top_2_accuracy(in_gt, in_pred):\n    return top_k_categorical_accuracy(in_gt, in_pred, k=2)\n\nretina_model.compile(optimizer = 'adam', loss = 'categorical_crossentropy',\n                           metrics = ['categorical_accuracy', top_2_accuracy])\nretina_model.summary()","metadata":{"_uuid":"1f0dfaccda346d7bc4758e7329d61028d254a8d6","_cell_guid":"eeb36110-0cde-4450-a43c-b8f707adb235","trusted":true,"execution":{"iopub.status.busy":"2024-10-22T04:05:13.548524Z","iopub.status.idle":"2024-10-22T04:05:13.549026Z","shell.execute_reply":"2024-10-22T04:05:13.548816Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"This code snippet sets up various callbacks for a Keras model training process. Callbacks are functions or actions that can be applied at certain stages during training, allowing for dynamic adjustments to the training process based on performance metrics. Here's a detailed breakdown of each part of the code:\n\n### Importing Callbacks\n- **`from keras.callbacks import ModelCheckpoint, LearningRateScheduler, EarlyStopping, ReduceLROnPlateau`**: This line imports specific callback classes from the Keras library that will be used during training to manage model performance and training parameters.\n\n### Defining the Weight Path\n- **`weight_path=\"{}_weights.best.hdf5\".format('retina')`**: This line specifies the file path where the best model weights will be saved. The `{}` is a placeholder that gets replaced by the string `'retina'`, resulting in the path `retina_weights.best.hdf5`.\n\n### Model Checkpoint\n- **`checkpoint = ModelCheckpoint(weight_path, monitor='val_loss', verbose=1, save_best_only=True, mode='min', save_weights_only=True)`**:\n  - **`weight_path`**: Specifies where to save the model weights.\n  - **`monitor='val_loss'`**: Indicates that the validation loss will be monitored during training.\n  - **`verbose=1`**: Enables detailed logging when a model checkpoint is created.\n  - **`save_best_only=True`**: Ensures that only the best model weights (i.e., the ones that yield the lowest validation loss) are saved.\n  - **`mode='min'`**: Configures the callback to look for a minimum validation loss.\n  - **`save_weights_only=True`**: Specifies that only the model weights should be saved, not the entire model.\n\n### Reduce Learning Rate on Plateau\n- **`reduceLROnPlat = ReduceLROnPlateau(monitor='val_loss', factor=0.8, patience=3, verbose=1, mode='auto', epsilon=0.0001, cooldown=5, min_lr=0.0001)`**:\n  - **`monitor='val_loss'`**: Monitors the validation loss.\n  - **`factor=0.8`**: Reduces the learning rate by a factor of 0.8 when the monitored quantity stops improving.\n  - **`patience=3`**: Waits for 3 epochs before reducing the learning rate.\n  - **`verbose=1`**: Logs the learning rate reduction.\n  - **`mode='auto'`**: Automatically determines whether to look for a minimum or maximum.\n  - **`epsilon=0.0001`**: The threshold for measuring the new optimum.\n  - **`cooldown=5`**: After reducing the learning rate, it will not reduce it again for 5 epochs.\n  - **`min_lr=0.0001`**: Sets a lower bound for the learning rate to prevent it from going too low.\n\n### Early Stopping\n- **`early = EarlyStopping(monitor=\"val_loss\", mode=\"min\", patience=6)`**:\n  - **`monitor=\"val_loss\"`**: Monitors the validation loss.\n  - **`mode=\"min\"`**: Configures it to stop training when the validation loss stops improving.\n  - **`patience=6`**: Stops training if there is no improvement in validation loss for 6 consecutive epochs.\n\n### Callback List\n- **`callbacks_list = [checkpoint, early, reduceLROnPlat]`**: This line creates a list containing all the defined callbacks. This list can be passed to the `fit` method of the Keras model during training.\n\n### Summary\nOverall, this snippet sets up robust mechanisms for managing the training process of a deep learning model. By using checkpoints, learning rate adjustments, and early stopping, the training process can be made more efficient and can potentially lead to better model performance while avoiding overfitting.\n","metadata":{}},{"cell_type":"code","source":"from keras.callbacks import ModelCheckpoint, LearningRateScheduler, EarlyStopping, ReduceLROnPlateau\nweight_path=\"{}_weights.best.hdf5\".format('retina')\n\ncheckpoint = ModelCheckpoint(weight_path, monitor='val_loss', verbose=1, \n                             save_best_only=True, mode='min', save_weights_only = True)\n\nreduceLROnPlat = ReduceLROnPlateau(monitor='val_loss', factor=0.8, patience=3, verbose=1, mode='auto', epsilon=0.0001, cooldown=5, min_lr=0.0001)\nearly = EarlyStopping(monitor=\"val_loss\", \n                      mode=\"min\", \n                      patience=6) # probably needs to be more patient, but kaggle time is limited\ncallbacks_list = [checkpoint, early, reduceLROnPlat]","metadata":{"_uuid":"48b9764e16fb5af52aed35c82bae6299e67d5bc7","_cell_guid":"17803ae1-bed8-41a4-9a2c-e66287a24830","trusted":true,"execution":{"iopub.status.busy":"2024-10-22T04:05:13.549907Z","iopub.status.idle":"2024-10-22T04:05:13.550306Z","shell.execute_reply":"2024-10-22T04:05:13.550113Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"!rm -rf ~/.keras # clean up before starting training","metadata":{"_uuid":"78dfa383c51777377c1f81e42017cbcca5f5736f","_cell_guid":"84f7cdec-ca00-460c-9991-55b1f7f02f20","trusted":true,"execution":{"iopub.status.busy":"2024-10-22T04:05:13.551102Z","iopub.status.idle":"2024-10-22T04:05:13.551502Z","shell.execute_reply":"2024-10-22T04:05:13.551315Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"markdown","source":"This code snippet initiates the training process of the `retina_model` using a Keras `fit_generator` method. The `fit_generator` method is used for training models on data generated batch-by-batch by a Python generator, allowing for the training of models on datasets that may not fit entirely into memory. Here's a breakdown of the parameters used in this training process:\n\n### Training the Model\n- **`retina_model.fit_generator(...)`**: This method is called to start the training process for the `retina_model`.\n\n### Parameters\n1. **`train_gen`**: \n   - This is the generator that produces training data in batches. It yields pairs of (input, target) arrays.\n\n2. **`steps_per_epoch = train_df.shape[0]//batch_size`**: \n   - This parameter defines how many batches of samples will be used in one epoch.\n   - It is calculated as the total number of training samples (from `train_df`) divided by the batch size. This helps Keras determine when to end one epoch and start the next.\n\n3. **`validation_data = valid_gen`**: \n   - This specifies the generator that will yield validation data during training. The model will evaluate its performance on this validation data at the end of each epoch.\n\n4. **`validation_steps = valid_df.shape[0]//batch_size`**: \n   - Similar to `steps_per_epoch`, this defines how many batches of validation data will be evaluated at the end of each epoch. It is calculated as the total number of validation samples (from `valid_df`) divided by the batch size.\n\n5. **`epochs = 25`**: \n   - This parameter sets the number of epochs (complete passes through the training dataset) to train the model. In this case, the model will train for 25 epochs.\n\n6. **`callbacks = callbacks_list`**: \n   - This parameter includes a list of callbacks that were defined earlier. These callbacks will be executed at specific stages during the training process, such as saving the best model weights or adjusting the learning rate.\n\n7. **`workers = 0`**: \n   - This parameter specifies the number of worker threads to use for data loading. Setting it to 0 indicates that no additional threads will be used, as TensorFlow generators are not thread-safe.\n\n8. **`use_multiprocessing=False`**: \n   - This parameter determines whether to use multiprocessing for data loading. Here, it is set to `False`, meaning that data will be loaded in the main thread.\n\n9. **`max_queue_size = 0`**: \n   - This parameter defines the maximum size of the generator queue. Setting it to 0 means that no queue will be used, which is suitable when using a single-threaded generator.\n\n### Summary\nOverall, this snippet configures and initiates the training of the `retina_model` on the training dataset while validating its performance on a separate validation dataset. The use of generators allows for efficient handling of potentially large datasets, making it suitable for deep learning tasks where memory constraints may be an issue.\n","metadata":{}},{"cell_type":"code","source":"retina_model.fit_generator(train_gen, \n                           steps_per_epoch = train_df.shape[0]//batch_size,\n                           validation_data = valid_gen, \n                           validation_steps = valid_df.shape[0]//batch_size,\n                              epochs = 25, \n                              callbacks = callbacks_list,\n                             workers = 0, # tf-generators are not thread-safe\n                             use_multiprocessing=False, \n                             max_queue_size = 0\n                            )","metadata":{"_uuid":"b2148479bfe41c5d9fd0faece4c75adea509dabe","_cell_guid":"58a75586-442b-4804-84a6-d63d5a42ea14","trusted":true,"execution":{"iopub.status.busy":"2024-10-22T04:05:13.552247Z","iopub.status.idle":"2024-10-22T04:05:13.552691Z","shell.execute_reply":"2024-10-22T04:05:13.552489Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"This code snippet loads the best weights from the training process of the `retina_model` and saves the entire model architecture along with the weights to a specified file. Here's a breakdown of each line:\n\n### Loading Weights and Saving the Model\n\n1. **`retina_model.load_weights(weight_path)`**: \n   - This line loads the weights of the `retina_model` from the file specified by `weight_path`. \n   - The `weight_path` variable contains the path to the saved weights of the model that performed the best during training, as determined by the `ModelCheckpoint` callback. This allows the model to use the parameters that resulted in the lowest validation loss.\n\n2. **`retina_model.save('full_retina_model.h5')`**: \n   - After loading the best weights, this line saves the entire model (including its architecture, weights, and training configuration) to a file named `full_retina_model.h5`.\n   - This `.h5` file can later be used to reload the model without needing to redefine the architecture or re-train it. It is particularly useful for deployment or further fine-tuning on new data.\n\n### Summary\nOverall, this snippet is crucial for preserving the trained model's state after training, enabling easy reloading and utilization in future applications or evaluations.\n","metadata":{}},{"cell_type":"code","source":"# load the best version of the model\nretina_model.load_weights(weight_path)\nretina_model.save('full_retina_model.h5')","metadata":{"_uuid":"3a90f05dd206cd76c72d8c6278ebb93da41ee45f","_cell_guid":"4d0c45b0-bb23-48d2-83eb-bc3990043e26","trusted":true,"execution":{"iopub.status.busy":"2024-10-22T04:05:13.553468Z","iopub.status.idle":"2024-10-22T04:05:13.553925Z","shell.execute_reply":"2024-10-22T04:05:13.553702Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"This code snippet is designed to create a fixed dataset for evaluating the performance of a previously trained model on a validation dataset. Here’s a detailed breakdown of the process:\n\n### Creating a Fixed Evaluation Dataset\n\n1. **`from tqdm import tqdm_notebook`**:\n   - This line imports the `tqdm_notebook` module, which provides a convenient way to display a progress bar in Jupyter notebooks. It helps to visualize the progress of loops, making it easier to track long-running processes.\n\n2. **`valid_gen = flow_from_dataframe(valid_idg, valid_df, path_col='path', y_col='level_cat')`**:\n   - A new validation data generator (`valid_gen`) is created using the `flow_from_dataframe` function. \n   - This generator yields batches of images and their corresponding labels from the `valid_df` DataFrame. The paths to the images are specified by `path_col`, and the labels (in categorical format) are given by `y_col`.\n\n3. **`vbatch_count = (valid_df.shape[0] // batch_size - 1)`**:\n   - The number of batches in the validation dataset is calculated. It divides the total number of validation samples (`valid_df.shape[0]`) by the `batch_size` and subtracts 1 to ensure that the count reflects the number of complete batches available.\n\n4. **`out_size = vbatch_count * batch_size`**:\n   - The total size of the output arrays (`out_size`) is determined by multiplying the number of batches by the batch size. This ensures that there is enough space to hold all validation samples that will be processed.\n\n5. **`test_X = np.zeros((out_size,) + t_x.shape[1:], dtype=np.float32)`**:\n   - An empty NumPy array `test_X` is created to store the images. It has a shape of `(out_size, height, width, channels)`, where `height`, `width`, and `channels` are derived from the shape of `t_x`.\n\n6. **`test_Y = np.zeros((out_size,) + t_y.shape[1:], dtype=np.float32)`**:\n   - Similarly, another empty NumPy array `test_Y` is initialized to store the corresponding labels of the images.\n\n7. **`for i, (c_x, c_y) in zip(tqdm_notebook(range(vbatch_count)), valid_gen):`**:\n   - This loop iterates over the number of batches in the validation dataset. `tqdm_notebook` is used to display a progress bar as the loop processes each batch.\n   - `c_x` contains the current batch of images, and `c_y` contains the associated labels.\n\n8. **`j = i * batch_size`**:\n   - The index `j` is calculated to determine the starting position in the `test_X` and `test_Y` arrays where the current batch of images and labels should be stored.\n\n9. **`test_X[j:(j+c_x.shape[0])] = c_x`**:\n   - The images from the current batch (`c_x`) are stored in the `test_X` array starting at index `j`.\n\n10. **`test_Y[j:(j+c_x.shape[0])] = c_y`**:\n    - Similarly, the labels from the current batch (`c_y`) are stored in the `test_Y` array starting at index `j`.\n\n### Summary\nThis snippet efficiently creates a fixed dataset of images and their labels from the validation set, preparing it for evaluation of the trained model. The use of a generator allows for dynamic loading of data in batches, minimizing memory usage while providing a straightforward method to gather all validation data for later analysis.\n","metadata":{}},{"cell_type":"code","source":"##### create one fixed dataset for evaluating\nfrom tqdm import tqdm_notebook\n# fresh valid gen\nvalid_gen = flow_from_dataframe(valid_idg, valid_df, \n                             path_col = 'path',\n                            y_col = 'level_cat') \nvbatch_count = (valid_df.shape[0]//batch_size-1)\nout_size = vbatch_count*batch_size\ntest_X = np.zeros((out_size,)+t_x.shape[1:], dtype = np.float32)\ntest_Y = np.zeros((out_size,)+t_y.shape[1:], dtype = np.float32)\nfor i, (c_x, c_y) in zip(tqdm_notebook(range(vbatch_count)), \n                         valid_gen):\n    j = i*batch_size\n    test_X[j:(j+c_x.shape[0])] = c_x\n    test_Y[j:(j+c_x.shape[0])] = c_y","metadata":{"_uuid":"2b74f4ab850c6e82549d732b6f0524724b95b53c","_cell_guid":"f37dd4d8-ecd6-487a-90d8-74fe14a9a318","trusted":true,"execution":{"iopub.status.busy":"2024-10-22T04:05:13.554705Z","iopub.status.idle":"2024-10-22T04:05:13.555172Z","shell.execute_reply":"2024-10-22T04:05:13.554969Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Show Attention\nDid our attention model learn anything useful?","metadata":{"_uuid":"cca170eb40bc591f89748ede8aa35de4308faaaf","_cell_guid":"11f33f0a-61eb-488a-b7ea-4bc9d15ba8f9"}},{"cell_type":"code","source":"# get the attention layer since it is the only one with a single output dim\nfor attn_layer in retina_model.layers:\n    c_shape = attn_layer.get_output_shape_at(0)\n    if len(c_shape)==4:\n        if c_shape[-1]==1:\n            print(attn_layer)\n            break","metadata":{"_uuid":"ad5b085d351e79b950bf0c2ddc476799d5b0692f","_cell_guid":"e41a063f-35c9-410f-be63-f66b63ff9683","trusted":true,"execution":{"iopub.status.busy":"2024-10-22T04:05:13.555877Z","iopub.status.idle":"2024-10-22T04:05:13.556352Z","shell.execute_reply":"2024-10-22T04:05:13.556147Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import keras.backend as K\nrand_idx = np.random.choice(range(len(test_X)), size = 6)\nattn_func = K.function(inputs = [retina_model.get_input_at(0), K.learning_phase()],\n           outputs = [attn_layer.get_output_at(0)]\n          )\nfig, m_axs = plt.subplots(len(rand_idx), 2, figsize = (8, 4*len(rand_idx)))\n[c_ax.axis('off') for c_ax in m_axs.flatten()]\nfor c_idx, (img_ax, attn_ax) in zip(rand_idx, m_axs):\n    cur_img = test_X[c_idx:(c_idx+1)]\n    attn_img = attn_func([cur_img, 0])[0]\n    img_ax.imshow(np.clip(cur_img[0,:,:,:]*127+127, 0, 255).astype(np.uint8))\n    attn_ax.imshow(attn_img[0, :, :, 0]/attn_img[0, :, :, 0].max(), cmap = 'viridis', \n                   vmin = 0, vmax = 1, \n                   interpolation = 'lanczos')\n    real_cat = np.argmax(test_Y[c_idx, :])\n    img_ax.set_title('Eye Image\\nCat:%2d' % (real_cat))\n    pred_cat = retina_model.predict(cur_img)\n    attn_ax.set_title('Attention Map\\nPred:%2.2f%%' % (100*pred_cat[0,real_cat]))\nfig.savefig('attention_map.png', dpi = 300)","metadata":{"_uuid":"00850972ae4298f49ed1838b3fc49c2d8fb07547","_cell_guid":"340eef36-f5b2-4b15-a59f-440061a427eb","trusted":true,"execution":{"iopub.status.busy":"2024-10-22T04:05:13.557165Z","iopub.status.idle":"2024-10-22T04:05:13.557582Z","shell.execute_reply":"2024-10-22T04:05:13.557375Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Evaluate the results\nHere we evaluate the results by loading the best version of the model and seeing how the predictions look on the results. We then visualize spec","metadata":{"_uuid":"244bac80d1ea2074e47932e367996e32cbab6a3d","_cell_guid":"24796de7-b1e9-4b3b-bcc6-d997aa3e6d16"}},{"cell_type":"code","source":"from sklearn.metrics import accuracy_score, classification_report\npred_Y = retina_model.predict(test_X, batch_size = 32, verbose = True)\npred_Y_cat = np.argmax(pred_Y, -1)\ntest_Y_cat = np.argmax(test_Y, -1)\nprint('Accuracy on Test Data: %2.2f%%' % (accuracy_score(test_Y_cat, pred_Y_cat)))\nprint(classification_report(test_Y_cat, pred_Y_cat))","metadata":{"_uuid":"b421b6183b1919a7414482f0b1ac611079e45174","_cell_guid":"d0edaf00-4b7c-4f65-af0b-e5a03b9b8428","trusted":true,"execution":{"iopub.status.busy":"2024-10-22T04:05:13.558419Z","iopub.status.idle":"2024-10-22T04:05:13.558948Z","shell.execute_reply":"2024-10-22T04:05:13.558716Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import seaborn as sns\nfrom sklearn.metrics import confusion_matrix\nsns.heatmap(confusion_matrix(test_Y_cat, pred_Y_cat), \n            annot=True, fmt=\"d\", cbar = False, cmap = plt.cm.Blues, vmax = test_X.shape[0]//16)","metadata":{"_uuid":"10162e055ca7cd52878a289bab377231787ab732","_cell_guid":"15189df2-3fed-495e-9661-97bb2b712dfd","trusted":true,"execution":{"iopub.status.busy":"2024-10-22T04:05:13.559702Z","iopub.status.idle":"2024-10-22T04:05:13.560267Z","shell.execute_reply":"2024-10-22T04:05:13.560059Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# ROC Curve for healthy vs sick\nHere we make an ROC curve for healthy (```severity == 0```) and sick (```severity>0```) to see how well the model works at just identifying the disease","metadata":{"_uuid":"12dfe39ea80194062068589699953c6645e285d6","_cell_guid":"70827da6-bf91-4b65-80e9-bf1e6b885db3"}},{"cell_type":"code","source":"\nfrom sklearn.metrics import roc_curve, roc_auc_score\nsick_vec = test_Y_cat>0\nsick_score = np.sum(pred_Y[:,1:],1)\nfpr, tpr, _ = roc_curve(sick_vec, sick_score)\nfig, ax1 = plt.subplots(1,1, figsize = (6, 6), dpi = 150)\nax1.plot(fpr, tpr, 'b.-', label = 'Model Prediction (AUC: %2.2f)' % roc_auc_score(sick_vec, sick_score))\nax1.plot(fpr, fpr, 'g-', label = 'Random Guessing')\nax1.legend()\nax1.set_xlabel('False Positive Rate')\nax1.set_ylabel('True Positive Rate');","metadata":{"_uuid":"2b2aaee6c83043f721b0c9ed5bc4229eb7165200","_cell_guid":"829475ab-7db2-4421-b9ad-1a51971fd459","trusted":true,"execution":{"iopub.status.busy":"2024-10-22T04:05:13.561001Z","iopub.status.idle":"2024-10-22T04:05:13.561497Z","shell.execute_reply":"2024-10-22T04:05:13.561288Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"fig, m_axs = plt.subplots(2, 4, figsize = (32, 20))\nfor (idx, c_ax) in enumerate(m_axs.flatten()):\n    c_ax.imshow(np.clip(test_X[idx]*127+127,0 , 255).astype(np.uint8), cmap = 'bone')\n    c_ax.set_title('Actual Severity: {}\\n{}'.format(test_Y_cat[idx], \n                                                           '\\n'.join(['Predicted %02d (%04.1f%%): %s' % (k, 100*v, '*'*int(10*v)) for k, v in sorted(enumerate(pred_Y[idx]), key = lambda x: -1*x[1])])), loc='left')\n    c_ax.axis('off')\nfig.savefig('trained_img_predictions.png', dpi = 300)","metadata":{"_uuid":"ba87d0e7c3a77181487b99ca64d13de2aa8a21ee","_cell_guid":"c34f049f-b032-45bf-9d5e-a756ecc46a82","trusted":true,"execution":{"iopub.status.busy":"2024-10-22T04:11:01.655072Z","iopub.execute_input":"2024-10-22T04:11:01.655367Z","iopub.status.idle":"2024-10-22T04:11:02.628512Z","shell.execute_reply.started":"2024-10-22T04:11:01.655324Z","shell.execute_reply":"2024-10-22T04:11:02.627298Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"","metadata":{"_uuid":"eb6752295030ba512263433f8383711e4ca1c14c","collapsed":true,"_cell_guid":"f2e189dc-f80a-4b16-bb1d-5c05a155a80b","trusted":true,"jupyter":{"outputs_hidden":true}},"outputs":[],"execution_count":null}]}