{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## Whale Recogniser - transfer learning cnn categorical approach\n\n__This exploratory notebook contains more exploration and comments, and forks will be shorter for quick training of a simple model.__\n\nThis notebook trains a model to recognise and categorise images of whale's tails. It uses a transfer learning approach - a resnet-18 model pre-trained on the ImageNet dataset is re-trained on the images of whales from the kaggle competition dataset.\nhttps://www.kaggle.com/competitions/whale-categorization-playground/data\n\n### Overview\nI'm using this competition to develop my skills in machine learning and object recognition from images.   <br> \n\nThe training data contains 9850 of images of humpback whale flukes. Individual whales have been identified by researchers and given an Id. The challenge is to predict the whale Id of images in the test set. What makes this such a challenge is that there are only a few examples for each of 3,000+ whale Ids.\n\nI have decided to use the fastai library to train a neural network, and to explore the tools provided by fastai. I believe that using image transformations at the batch level might artificially increase the amount of data per whale available for training, and I'm interested in investigating this. <br><br>\n\nSince this is a learning project for me, there are a few lines of code left in which are there to help me with future projects but which don't contribute to the training of the model - for example exporting and re-loading the model to disk<br>","metadata":{}},{"cell_type":"markdown","source":"### Imports and setup","metadata":{}},{"cell_type":"code","source":"!pip install -Uqq fastai","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-11-22T20:30:29.483165Z","iopub.execute_input":"2022-11-22T20:30:29.483612Z","iopub.status.idle":"2022-11-22T20:30:54.424133Z","shell.execute_reply.started":"2022-11-22T20:30:29.483526Z","shell.execute_reply":"2022-11-22T20:30:54.422912Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# imports\nfrom fastai.vision.all import *\nfrom fastai import *\nimport pandas as pd\nimport numpy as np\nimport os\nimport seaborn as sns\nimport albumentations\n\n# Run this code to fix intellisense and autocomplete in interactive kaggle notebok. \n%config Completer.use_jedi = False","metadata":{"execution":{"iopub.status.busy":"2022-11-22T20:30:54.428284Z","iopub.execute_input":"2022-11-22T20:30:54.428583Z","iopub.status.idle":"2022-11-22T20:30:58.596410Z","shell.execute_reply.started":"2022-11-22T20:30:54.428556Z","shell.execute_reply":"2022-11-22T20:30:58.595319Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Set seeds of random number generators","metadata":{}},{"cell_type":"code","source":"def set_seeds():\n    random.seed(42)\n    np.random.seed(42)\n    torch.manual_seed(42)\n    torch.backends.cudnn.deterministic = True\n    torch.backends.cudnn.benchmark = False\nset_seeds()","metadata":{"execution":{"iopub.status.busy":"2022-11-22T20:30:58.598194Z","iopub.execute_input":"2022-11-22T20:30:58.598767Z","iopub.status.idle":"2022-11-22T20:30:58.611032Z","shell.execute_reply.started":"2022-11-22T20:30:58.598738Z","shell.execute_reply":"2022-11-22T20:30:58.609947Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##  Set the path to the data, get a list of the files.","metadata":{}},{"cell_type":"code","source":"raw_path = Path('../input/whale-categorization-playground/')\ntrain_path = (raw_path/'train')\nimages = get_image_files(train_path)\nlen(images)","metadata":{"execution":{"iopub.status.busy":"2022-11-22T20:31:11.005758Z","iopub.execute_input":"2022-11-22T20:31:11.006172Z","iopub.status.idle":"2022-11-22T20:31:14.664884Z","shell.execute_reply.started":"2022-11-22T20:31:11.006139Z","shell.execute_reply":"2022-11-22T20:31:14.663874Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Pre - resize\nResize is done at training time, but it slows down training since large image files are being opened. \noptionally resize and scale all the images to a smaller size before passing to the model. ","metadata":{"execution":{"iopub.status.busy":"2022-11-09T23:55:35.431620Z","iopub.execute_input":"2022-11-09T23:55:35.432134Z","iopub.status.idle":"2022-11-09T23:55:35.440585Z","shell.execute_reply.started":"2022-11-09T23:55:35.432095Z","shell.execute_reply":"2022-11-09T23:55:35.439558Z"}}},{"cell_type":"code","source":"pre_resize = True\nif pre_resize:\n    resized_path = Path('./resized')\n    resize_images(train_path,max_size=400, n_channels=3, dest=resized_path)\n","metadata":{"execution":{"iopub.status.busy":"2022-11-22T20:31:28.593488Z","iopub.execute_input":"2022-11-22T20:31:28.593848Z","iopub.status.idle":"2022-11-22T20:33:50.799237Z","shell.execute_reply.started":"2022-11-22T20:31:28.593819Z","shell.execute_reply":"2022-11-22T20:33:50.798041Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Load the labels and filenames into a dataframe","metadata":{}},{"cell_type":"code","source":"df = pd.read_csv(raw_path/'train.csv')\ndf.head() ","metadata":{"execution":{"iopub.status.busy":"2022-11-22T20:33:50.801624Z","iopub.execute_input":"2022-11-22T20:33:50.802068Z","iopub.status.idle":"2022-11-22T20:33:50.844005Z","shell.execute_reply.started":"2022-11-22T20:33:50.802024Z","shell.execute_reply":"2022-11-22T20:33:50.842838Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.describe()","metadata":{"execution":{"iopub.status.busy":"2022-11-22T20:33:50.845688Z","iopub.execute_input":"2022-11-22T20:33:50.846143Z","iopub.status.idle":"2022-11-22T20:33:50.879654Z","shell.execute_reply.started":"2022-11-22T20:33:50.846106Z","shell.execute_reply":"2022-11-22T20:33:50.878484Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The dataset contains a set of 9850 images with an image Id and a whale Id. ","metadata":{}},{"cell_type":"markdown","source":"### The problem with the new_whale class\nThere are 4251 individual whale IDs, and 810 of these have the 'new_whale' label. Presumably this means that there are actually 5601 individuals in the training dataset. \n\nTo the model, the new_whale category will look as though the 861 new_whales are all the same individual, but they are not. \n\nWe might choose to discard or re-lablel these new_whale id's from the training set.\n\n__Option 1__: throw away the new_whale ids from the dataset. <br> \n__Option 2__: rename all of the new_whales with a new id so that they can be recognised as unique individuals and later input to a siamese neural network.","metadata":{}},{"cell_type":"markdown","source":"## Make some dictionaries to store the labels","metadata":{}},{"cell_type":"code","source":"filename_to_label_dict = dict(df.values)    \n\nlabel_to_filename_dict = {}\n\nfor label in df.Id.unique():\n    label_to_filename_dict[label] = df.loc[df.Id==label].Image.values\n\nnew_whale_filenames = label_to_filename_dict['new_whale']\n\nimages_minus_new_whales = images.copy()\nfor image in new_whale_filenames:\n    images_minus_new_whales.remove(Path(f'../input/whale-categorization-playground/train/{image}'))\n\n#check\nlen(images_minus_new_whales) - len(images)\n","metadata":{"execution":{"iopub.status.busy":"2022-11-22T20:33:50.882506Z","iopub.execute_input":"2022-11-22T20:33:50.883212Z","iopub.status.idle":"2022-11-22T20:33:57.399074Z","shell.execute_reply.started":"2022-11-22T20:33:50.883168Z","shell.execute_reply":"2022-11-22T20:33:57.397878Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We now have two dictionaries to map from filename to whale Id and back. \n<br>We also have an alternative training set named `images_minus_new_whales`\n","metadata":{}},{"cell_type":"code","source":"label_to_filename_dict[df.Id[2]]","metadata":{"execution":{"iopub.status.busy":"2022-11-22T20:33:57.400559Z","iopub.execute_input":"2022-11-22T20:33:57.401078Z","iopub.status.idle":"2022-11-22T20:33:57.408618Z","shell.execute_reply.started":"2022-11-22T20:33:57.401039Z","shell.execute_reply":"2022-11-22T20:33:57.407618Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Distribution of Id's ","metadata":{}},{"cell_type":"code","source":"counts = df.loc[df.Id!='new_whale'].Id.value_counts()\ncounts.describe(percentiles = np.linspace(0.05, 0.95, 19))","metadata":{"execution":{"iopub.status.busy":"2022-11-22T20:33:57.410370Z","iopub.execute_input":"2022-11-22T20:33:57.411094Z","iopub.status.idle":"2022-11-22T20:33:57.433282Z","shell.execute_reply.started":"2022-11-22T20:33:57.411050Z","shell.execute_reply":"2022-11-22T20:33:57.432046Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The largest number of images of a single whale is 34. <br>95% of the whales have 5 or less photographs per individual. ","metadata":{}},{"cell_type":"markdown","source":"## Show sample images of a whale","metadata":{}},{"cell_type":"code","source":"whale_index = 1\nimage_index = 2\n\nwhale_id = df.Id.unique()[whale_index]\nlabel_to_filename_dict[whale_id][whale_index]\nimage_ids = label_to_filename_dict[whale_id]\nImage.open(train_path/image_ids[image_index])","metadata":{"execution":{"iopub.status.busy":"2022-11-22T20:33:57.435051Z","iopub.execute_input":"2022-11-22T20:33:57.435471Z","iopub.status.idle":"2022-11-22T20:33:57.544439Z","shell.execute_reply.started":"2022-11-22T20:33:57.435434Z","shell.execute_reply":"2022-11-22T20:33:57.543526Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Make a function to get whale ID from filename\nThis function will be passed to the fastai DataBlock to get the training label from the filename.\n\nEach image has an associated whale Id which we are trying to train the model to recognise. We need a way of getting the whale Id from the filename for each image. This is contained in the file train.csv, which we earlier loaded into a dictionary named `filename_to_label_dict`. <br>\n\nEach file is given a unique file name, and is appended with its file type (.jpg) <br>","metadata":{}},{"cell_type":"code","source":"def get_label_from_filepath(filepath):\n    filename = filepath.name\n    whale_id = filename_to_label_dict[filename]\n    return whale_id\n\n# test\nget_label_from_filepath(images[3])\n","metadata":{"execution":{"iopub.status.busy":"2022-11-22T20:33:57.545406Z","iopub.execute_input":"2022-11-22T20:33:57.545799Z","iopub.status.idle":"2022-11-22T20:33:57.554330Z","shell.execute_reply.started":"2022-11-22T20:33:57.545762Z","shell.execute_reply":"2022-11-22T20:33:57.553153Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Define a fastai DataBlock named __whales__\n\nThe fastai DataBlock describes the features associated with each datapoint, such as where the data is stored, the type of input, the type of output etc. \n\nTo build a DataBlock you need to give the library four things: the types of your input/labels, and at least two functions: get_items and splitter. You may also need to include get_x and get_y or a more generic list of getters that are applied to the results of get_items.\n\nsplitter is a callable which, when called with items, returns a tuple of iterables representing the indices of the training and validation data.\n\nOnce those are provided, you automatically get a Datasets or a DataLoaders:\n\n__Recap of DataBlock functionality:__\n- `blocks` specify the type of data for input (Image), and the type of data for output (the label - in this case is a Category)\n- `get_items` load a set of input data\n- `get_y` get labels from the input data\n- `splitter` split the input data into training and validation sets\n- `item_tfms` apply transformations of the input data at item level, e.g. resize\n- `batch_tfms` apply transformations to every item in a batch all at once. \n","metadata":{}},{"cell_type":"code","source":"whales = DataBlock(\n    blocks=(ImageBlock, CategoryBlock(vocab=list(df['Id'].unique()))), \n    get_items=get_image_files, \n    splitter=RandomSplitter(valid_pct=0.2, seed=42),\n    get_y=get_label_from_filepath, \n    item_tfms=Resize(224, ResizeMethod.Squish)\n    )","metadata":{"execution":{"iopub.status.busy":"2022-11-22T20:33:57.556007Z","iopub.execute_input":"2022-11-22T20:33:57.556661Z","iopub.status.idle":"2022-11-22T20:33:57.572432Z","shell.execute_reply.started":"2022-11-22T20:33:57.556627Z","shell.execute_reply":"2022-11-22T20:33:57.571381Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dataloaders = whales.dataloaders(train_path)","metadata":{"execution":{"iopub.status.busy":"2022-11-22T20:33:57.575679Z","iopub.execute_input":"2022-11-22T20:33:57.576667Z","iopub.status.idle":"2022-11-22T20:34:02.443768Z","shell.execute_reply.started":"2022-11-22T20:33:57.576628Z","shell.execute_reply":"2022-11-22T20:34:02.442754Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Look at some photos in the training and the validation sets\nSince we used `resize method = squish`, the entire image is available for training, but it is squished along one axis to make it square, and resized. ","metadata":{}},{"cell_type":"code","source":"dataloaders.train.show_batch(max_n=5, unique=True)","metadata":{"execution":{"iopub.status.busy":"2022-11-22T20:47:04.617180Z","iopub.execute_input":"2022-11-22T20:47:04.617591Z","iopub.status.idle":"2022-11-22T20:47:05.840449Z","shell.execute_reply.started":"2022-11-22T20:47:04.617560Z","shell.execute_reply":"2022-11-22T20:47:05.839382Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The images are all being resized to 224x224px squares. \ne.g. if the original image is 448px wide and 224px tall, the resulting image will be 224x224 px and will be narrow or squished along the x axis","metadata":{}},{"cell_type":"markdown","source":"### Updating the dataloaders without re-writing datablock\nWe might prefer to crop using a different method which doesn't squish the image but throws away some of the information at the edges\n","metadata":{"execution":{"iopub.status.busy":"2022-11-22T20:56:58.237664Z","iopub.execute_input":"2022-11-22T20:56:58.238120Z","iopub.status.idle":"2022-11-22T20:56:58.243261Z","shell.execute_reply.started":"2022-11-22T20:56:58.238082Z","shell.execute_reply":"2022-11-22T20:56:58.242135Z"}}},{"cell_type":"code","source":"crop_dataloaders = whales.new(item_tfms=Resize(224, ResizeMethod.Crop)).dataloaders(train_path)\nrandom_resized_crop_dataloaders = whales.new(item_tfms=RandomResizedCrop(256, min_scale=0.3, max_scale=0.1)).dataloaders(train_path)\nrandom_resized_crop_dataloaders.show_batch(max_n=5,unique=True)","metadata":{"execution":{"iopub.status.busy":"2022-11-22T22:17:49.797528Z","iopub.execute_input":"2022-11-22T22:17:49.798015Z","iopub.status.idle":"2022-11-22T22:17:53.611626Z","shell.execute_reply.started":"2022-11-22T22:17:49.797974Z","shell.execute_reply":"2022-11-22T22:17:53.610423Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Show various images from a validation set batch","metadata":{}},{"cell_type":"code","source":"crop_dataloaders.valid.show_batch(max_n=3, unique=True)","metadata":{"execution":{"iopub.status.busy":"2022-11-22T22:18:30.007421Z","iopub.execute_input":"2022-11-22T22:18:30.007902Z","iopub.status.idle":"2022-11-22T22:18:31.986209Z","shell.execute_reply.started":"2022-11-22T22:18:30.007859Z","shell.execute_reply":"2022-11-22T22:18:31.985043Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 224x224px resize\nAll the images are resized to 224 x 224 pixels so they can all be lined up and processed by the same architecture. The resize method used is changeable, currently we're using 'squish' which takes the longest dimension, shrinks or stretches until the image is square, then rescales to 224px.","metadata":{}},{"cell_type":"markdown","source":"## Train a baseline model\n### Define and train a convolutional neural network\nWe need the following parameters: <br>\n- __data_loaders__ - this is defined above and tells PyTorch how to load new batches of data. \n- __architecture__ - This is the architecture used for the neural network. Popular choice is resnet18 since they are n [relatively quick to train](https://www.kaggle.com/code/jhoward/which-image-models-are-best/) \n- __metrics__ - This is the reported measure of performance of the model, intended to be read by a human.<br><br>\nIn addition to defining the shape, size and connectedness of the nodes in the network, the architecture can includes the parameters - either all zero, randomly initialized, or the model may be pre-trained on other image recognition tasks. In this latter case we are doing transfer learning, which is much more efficient than starting from scratch. In transfer learning we only train the head - the final layer - of the model to fit our specific task. <br> <br>\nThe cell below shows how the error rate on the validation set decreases as we train the model. So far the error rate has reduced from 95% to 80% after 36 epochs. I haven't used the competition evaluation format which allows 5 labels for the whale and a new_whale label. <br><br>","metadata":{}},{"cell_type":"markdown","source":"## Train the model \n__22 mins for 6 epochs__<br>\nWe use fine_tune to indicate that we are only updating the parameters at the head of the model.","metadata":{}},{"cell_type":"code","source":"learn = vision_learner(dataloaders, resnet18, metrics=error_rate)\n%time learn.fine_tune(6)","metadata":{"execution":{"iopub.status.busy":"2022-11-22T21:32:36.396163Z","iopub.execute_input":"2022-11-22T21:32:36.396602Z","iopub.status.idle":"2022-11-22T21:52:37.992557Z","shell.execute_reply.started":"2022-11-22T21:32:36.396564Z","shell.execute_reply":"2022-11-22T21:52:37.991021Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"22 mins for 6 epochs\n## An error rate of 82% seems pretty good as a first pass. \n","metadata":{}},{"cell_type":"markdown","source":"### How do we know the model is doing something useful?\nThe model could be accurate by just predicting the 'new_whale' label each time, or predicting the most commonly photographed whales each time. One way to test this is to feed the model a bunch of zeros instead of images of whales. The expectation is that the performance should drop right off and stay at its initial value of 94%, since there is nothing to learn from a set of zeros. \n<br><br>\nHere we define a class to take an image, convert to numpy array, multiply each pixel by zero, then convert back into an image. ","metadata":{}},{"cell_type":"code","source":"class BlackTransform(Transform):\n    def __init__(self, split_idx=0, order=2):\n        self.split_idx = split_idx\n        self.order = order\n    \n    def encodes(self, img: PILImage):\n        np_img = np.array(img)\n        black_image = np_img * 0\n        return PILImage.create(black_image)","metadata":{"execution":{"iopub.status.busy":"2022-11-22T00:21:35.737369Z","iopub.execute_input":"2022-11-22T00:21:35.738087Z","iopub.status.idle":"2022-11-22T00:21:35.749980Z","shell.execute_reply.started":"2022-11-22T00:21:35.738052Z","shell.execute_reply":"2022-11-22T00:21:35.749074Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"blacked_whales = DataBlock(\n    blocks=(ImageBlock, CategoryBlock(vocab=list(df['Id'].unique()))), \n    get_items=get_image_files, \n    splitter=RandomSplitter(valid_pct=0.2, seed=42),\n    get_y=get_label_from_filepath, \n    item_tfms=[Resize(224, ResizeMethod.Squish), BlackTransform()]\n    )","metadata":{"execution":{"iopub.status.busy":"2022-11-22T00:21:52.823872Z","iopub.execute_input":"2022-11-22T00:21:52.824238Z","iopub.status.idle":"2022-11-22T00:21:52.834774Z","shell.execute_reply.started":"2022-11-22T00:21:52.824205Z","shell.execute_reply":"2022-11-22T00:21:52.833887Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"black_dls = blacked_whales.dataloaders(train_path)","metadata":{"execution":{"iopub.status.busy":"2022-11-22T00:21:23.268898Z","iopub.execute_input":"2022-11-22T00:21:23.269256Z","iopub.status.idle":"2022-11-22T00:21:26.852071Z","shell.execute_reply.started":"2022-11-22T00:21:23.269225Z","shell.execute_reply":"2022-11-22T00:21:26.850980Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"black_dls.valid.show_batch()","metadata":{"execution":{"iopub.status.busy":"2022-11-22T00:21:26.854288Z","iopub.execute_input":"2022-11-22T00:21:26.854683Z","iopub.status.idle":"2022-11-22T00:21:28.259995Z","shell.execute_reply.started":"2022-11-22T00:21:26.854645Z","shell.execute_reply":"2022-11-22T00:21:28.259148Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"learn_baseline = vision_learner(black_dls, resnet18, metrics=error_rate)\n%time learn_baseline.fine_tune(6)","metadata":{"execution":{"iopub.status.busy":"2022-11-21T23:36:48.244161Z","iopub.execute_input":"2022-11-21T23:36:48.244874Z","iopub.status.idle":"2022-11-21T23:49:29.336128Z","shell.execute_reply.started":"2022-11-21T23:36:48.244837Z","shell.execute_reply":"2022-11-21T23:49:29.335085Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here we can see that the model is unable to learn when presented with black squares then tested on the validation set. However, the training loss scores in the black squares model look suspiciously similar to the validation losses in the previous model. \n\nThe model we just trained used black squares in the training set and images of whales for validation. \n\n### What happens if we use black squares for both sets of data?","metadata":{}},{"cell_type":"code","source":"split_index = None # both validation and training set will be black boxes\n\nblack_dls_train_val = DataBlock(\n    blocks=(ImageBlock, CategoryBlock(vocab=list(df['Id'].unique()))), \n    get_items=get_image_files, \n    splitter=RandomSplitter(valid_pct=0.2, seed=42),\n    get_y=get_label_from_filepath, \n    item_tfms=[Resize(224, ResizeMethod.Squish), BlackTransform(split_index)]\n    ).dataloaders(train_path)\nlearn_baseline = vision_learner(black_dls_train_val, resnet18, metrics=error_rate)\n%time learn_baseline.fine_tune(15)","metadata":{"execution":{"iopub.status.busy":"2022-11-22T00:23:55.076750Z","iopub.execute_input":"2022-11-22T00:23:55.077151Z","iopub.status.idle":"2022-11-22T00:36:48.858164Z","shell.execute_reply.started":"2022-11-22T00:23:55.077120Z","shell.execute_reply":"2022-11-22T00:36:48.857039Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The results are similar when replacing the entire dataset with black squares as when we replaced only the training portion with black squares and left the validation set images in place. The initial run of this showed 90% error rate and a steep improvement on the validation set on the 6th final epoch. What is the model learning? I have decided to run it for more epochs to find out whether the model converges much further. ","metadata":{}},{"cell_type":"markdown","source":"## Train on lower resolution images\nI'd like to see how much reducing the image size speeds up training. The images will still go through a re-size operation but they will start out smaller - so this will indicate whether the large image files take a while to open.","metadata":{}},{"cell_type":"markdown","source":"<br>\nI'd also like to make sure that the model is making sensible predictions, and not just predicting 'new whale' each time.","metadata":{}},{"cell_type":"code","source":"# pass the path to re-sized images instead of full-resolution images\ndataloaders = whales.dataloaders(resized_path)\ndataloaders.show_batch()","metadata":{"execution":{"iopub.status.busy":"2022-11-22T00:54:51.974823Z","iopub.execute_input":"2022-11-22T00:54:51.975184Z","iopub.status.idle":"2022-11-22T00:54:53.009180Z","shell.execute_reply.started":"2022-11-22T00:54:51.975153Z","shell.execute_reply":"2022-11-22T00:54:53.008300Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"learn = vision_learner(dataloaders, resnet18, metrics=error_rate)\n%time learn.fine_tune(6)","metadata":{"execution":{"iopub.status.busy":"2022-11-22T00:54:56.473902Z","iopub.execute_input":"2022-11-22T00:54:56.474642Z","iopub.status.idle":"2022-11-22T01:00:06.346607Z","shell.execute_reply.started":"2022-11-22T00:54:56.474597Z","shell.execute_reply":"2022-11-22T01:00:06.345390Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### The model trains twice as quickly on the resized images and doesn't suffer a huge drop in performance. ","metadata":{}},{"cell_type":"markdown","source":"Training on the smaller images and resizing to 400px instead of 224px slowed down training. It took 12 mins to train for 6 epochs for images with higher resolution. The size of the file on disk, and the size of the file fed to the model both impact the training time.","metadata":{}},{"cell_type":"markdown","source":"### augmentation transforms\nRandomly __warping, shrinking, translating__ - the original training images can make the model more resillient to overfitting on the specifics of the individual training images, since it sees the same image in multiple transformed ways, rather than seeing the same image n_epochs times. It can help the model to generalise more about the subject of the images. We can test the effect of applying these transforms by comparing our baseline model against one which had transformations applied. ","metadata":{}},{"cell_type":"code","source":"whale_dls_transforms = DataBlock(\n    blocks=(ImageBlock, CategoryBlock(vocab=list(df['Id'].unique()))), \n    get_items=get_image_files, \n    splitter=RandomSplitter(valid_pct=0.2, seed=42),\n    get_y=get_label_from_filepath, \n    item_tfms=Resize(224, ResizeMethod.Squish), \n    batch_tfms=[*aug_transforms()]\n    ).dataloaders(resized_path)\n\nwhale_dls_transforms.show_batch()","metadata":{"execution":{"iopub.status.busy":"2022-11-22T01:09:24.885074Z","iopub.execute_input":"2022-11-22T01:09:24.885516Z","iopub.status.idle":"2022-11-22T01:09:25.974035Z","shell.execute_reply.started":"2022-11-22T01:09:24.885482Z","shell.execute_reply":"2022-11-22T01:09:25.973187Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"learn = vision_learner(whale_dls_transforms, resnet18, metrics=error_rate)\n%time learn.fine_tune(6)","metadata":{"execution":{"iopub.status.busy":"2022-11-22T01:10:04.547737Z","iopub.execute_input":"2022-11-22T01:10:04.548099Z","iopub.status.idle":"2022-11-22T01:15:20.532806Z","shell.execute_reply.started":"2022-11-22T01:10:04.548068Z","shell.execute_reply":"2022-11-22T01:15:20.531563Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Train a model without `new_whale` class","metadata":{}},{"cell_type":"markdown","source":"To remove all the new_whale images, use the `unlink` method","metadata":{"execution":{"iopub.status.busy":"2022-11-22T01:46:20.289227Z","iopub.execute_input":"2022-11-22T01:46:20.291636Z","iopub.status.idle":"2022-11-22T01:46:20.296125Z","shell.execute_reply.started":"2022-11-22T01:46:20.291593Z","shell.execute_reply":"2022-11-22T01:46:20.294956Z"}}},{"cell_type":"code","source":"new_whales = label_to_filename_dict['new_whale']","metadata":{"execution":{"iopub.status.busy":"2022-11-22T02:09:21.870582Z","iopub.execute_input":"2022-11-22T02:09:21.870952Z","iopub.status.idle":"2022-11-22T02:09:21.876091Z","shell.execute_reply.started":"2022-11-22T02:09:21.870913Z","shell.execute_reply":"2022-11-22T02:09:21.874833Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"new_whales_paths = L([resized_path/f for f in new_whales])","metadata":{"execution":{"iopub.status.busy":"2022-11-22T02:13:02.520217Z","iopub.execute_input":"2022-11-22T02:13:02.522879Z","iopub.status.idle":"2022-11-22T02:13:02.532378Z","shell.execute_reply.started":"2022-11-22T02:13:02.522840Z","shell.execute_reply":"2022-11-22T02:13:02.531254Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"new_whales_paths.map(Path.unlink) ","metadata":{"execution":{"iopub.status.busy":"2022-11-22T02:14:22.872331Z","iopub.execute_input":"2022-11-22T02:14:22.872709Z","iopub.status.idle":"2022-11-22T02:14:22.898471Z","shell.execute_reply.started":"2022-11-22T02:14:22.872679Z","shell.execute_reply":"2022-11-22T02:14:22.897281Z"},"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we've removed all the new_whale classes from the resized dataset. This should stop the model from being able to artificially increase accuracy by guessing new_whale each time. ","metadata":{}},{"cell_type":"code","source":"whales = DataBlock(\n    blocks=(ImageBlock, CategoryBlock(vocab=list(df['Id'].unique()))), \n    get_items=get_image_files, \n    splitter=RandomSplitter(valid_pct=0.2, seed=42),\n    get_y=get_label_from_filepath, \n    item_tfms=Resize(224, ResizeMethod.Squish), \n    batch_tfms=[*aug_transforms()]\n    )","metadata":{"execution":{"iopub.status.busy":"2022-11-22T02:14:51.619254Z","iopub.execute_input":"2022-11-22T02:14:51.619642Z","iopub.status.idle":"2022-11-22T02:15:01.386421Z","shell.execute_reply.started":"2022-11-22T02:14:51.619611Z","shell.execute_reply":"2022-11-22T02:15:01.385273Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dls = whales.dataloaders(resized_path)","metadata":{"execution":{"iopub.status.busy":"2022-11-22T02:16:21.424787Z","iopub.execute_input":"2022-11-22T02:16:21.425146Z","iopub.status.idle":"2022-11-22T02:16:21.546247Z","shell.execute_reply.started":"2022-11-22T02:16:21.425115Z","shell.execute_reply":"2022-11-22T02:16:21.545244Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"learn = vision_learner(dls, resnet18, metrics=error_rate)\n%time learn.fine_tune(6)","metadata":{"execution":{"iopub.status.busy":"2022-11-22T02:16:45.781193Z","iopub.execute_input":"2022-11-22T02:16:45.781901Z","iopub.status.idle":"2022-11-22T02:21:51.327718Z","shell.execute_reply.started":"2022-11-22T02:16:45.781864Z","shell.execute_reply":"2022-11-22T02:21:51.326394Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Plot the loss curves","metadata":{}},{"cell_type":"code","source":"learn.recorder.plot_loss()","metadata":{"execution":{"iopub.status.busy":"2022-11-22T02:21:59.753737Z","iopub.execute_input":"2022-11-22T02:21:59.754731Z","iopub.status.idle":"2022-11-22T02:22:00.033984Z","shell.execute_reply.started":"2022-11-22T02:21:59.754684Z","shell.execute_reply":"2022-11-22T02:22:00.032569Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here we can see the loss curves changing as the model trains. We only ran for 6 and the error rate reduced from 92% to 85%. Note these figures might be different in this run. One way to increase the accuracy might be to train for longer, change the learning rate. First we should clean up the input data and remove top anomalies, and play with the crop options on small batches to find crop and transformation settings which allow the model to converge. \n<br><br>\nThe training loss curve has a stepped shape. This might indicate that the data isn't suffled, and this should be double checked. Also check what changing the batch size does to the shape of these steps. It looks as though the loss on the training set improves quickly at the beginning of each epoch and slows down towards the end of the epoch. \n","metadata":{}},{"cell_type":"code","source":"test_images = get_image_files(raw_path/'test')","metadata":{"execution":{"iopub.status.busy":"2022-11-22T02:22:01.589169Z","iopub.execute_input":"2022-11-22T02:22:01.589899Z","iopub.status.idle":"2022-11-22T02:22:13.535291Z","shell.execute_reply.started":"2022-11-22T02:22:01.589861Z","shell.execute_reply":"2022-11-22T02:22:13.534245Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_images[0]","metadata":{"execution":{"iopub.status.busy":"2022-11-22T02:22:13.537441Z","iopub.execute_input":"2022-11-22T02:22:13.537757Z","iopub.status.idle":"2022-11-22T02:22:13.543788Z","shell.execute_reply.started":"2022-11-22T02:22:13.537730Z","shell.execute_reply":"2022-11-22T02:22:13.542772Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Use the model to make a prediction on an item from the test set","metadata":{}},{"cell_type":"code","source":"prediction, prediction_idx, probs = learn.predict(test_images[0])\nprint(f\"prediction = {prediction}\")\nprint(f\"prediction index = {prediction_idx}\")\nprint(f\"first ten probs = {probs[:10]}\")","metadata":{"execution":{"iopub.status.busy":"2022-11-22T02:22:13.545263Z","iopub.execute_input":"2022-11-22T02:22:13.545911Z","iopub.status.idle":"2022-11-22T02:22:13.631693Z","shell.execute_reply.started":"2022-11-22T02:22:13.545875Z","shell.execute_reply":"2022-11-22T02:22:13.630506Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Look at the predictions for a subset of test daata\n- does the model always predict one common whale? \n- Does it always predict new whale?","metadata":{"execution":{"iopub.status.busy":"2022-11-07T21:26:39.404358Z","iopub.execute_input":"2022-11-07T21:26:39.404848Z","iopub.status.idle":"2022-11-07T21:26:39.409976Z","shell.execute_reply.started":"2022-11-07T21:26:39.404770Z","shell.execute_reply":"2022-11-07T21:26:39.408851Z"}}},{"cell_type":"code","source":"sample_preds = [learn.predict(image)[0] for image in test_images[:100]]","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-11-22T02:22:13.634694Z","iopub.execute_input":"2022-11-22T02:22:13.635102Z","iopub.status.idle":"2022-11-22T02:22:19.907105Z","shell.execute_reply.started":"2022-11-22T02:22:13.635063Z","shell.execute_reply":"2022-11-22T02:22:19.906019Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pd.DataFrame(sample_preds).value_counts()[:10]","metadata":{"execution":{"iopub.status.busy":"2022-11-22T02:22:19.908731Z","iopub.execute_input":"2022-11-22T02:22:19.909363Z","iopub.status.idle":"2022-11-22T02:22:19.921876Z","shell.execute_reply.started":"2022-11-22T02:22:19.909323Z","shell.execute_reply":"2022-11-22T02:22:19.920614Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Plot top losses:\nHere we can see the images which confused the model the most - either where it predicted correctly but had low confidence, or where it predicted the wrong prediction with a high probability. ","metadata":{}},{"cell_type":"code","source":"interp = ClassificationInterpretation.from_learner(learn)\ninterp.plot_top_losses(10)","metadata":{"execution":{"iopub.status.busy":"2022-11-22T02:22:19.923792Z","iopub.execute_input":"2022-11-22T02:22:19.924156Z","iopub.status.idle":"2022-11-22T02:22:28.315921Z","shell.execute_reply.started":"2022-11-22T02:22:19.924121Z","shell.execute_reply":"2022-11-22T02:22:28.314956Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Choosing a learning rate\nThe learning rate is increased incrememntally until the loss begins to increase, then we pick a learning rate one order of magnitude before the exponential increas in loss. ","metadata":{}},{"cell_type":"code","source":"lr = learn.lr_find()","metadata":{"execution":{"iopub.status.busy":"2022-11-22T02:22:28.317780Z","iopub.execute_input":"2022-11-22T02:22:28.318397Z","iopub.status.idle":"2022-11-22T02:22:59.765738Z","shell.execute_reply.started":"2022-11-22T02:22:28.318354Z","shell.execute_reply":"2022-11-22T02:22:59.764521Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## fine_tune() \nfine_tune() teaches the model the differences between the dataset being used for training, and the dataset used to pretrain the model","metadata":{}},{"cell_type":"code","source":"learn.fine_tune(4)","metadata":{"execution":{"iopub.status.busy":"2022-11-22T02:22:59.767577Z","iopub.execute_input":"2022-11-22T02:22:59.768492Z","iopub.status.idle":"2022-11-22T02:26:39.309544Z","shell.execute_reply.started":"2022-11-22T02:22:59.768448Z","shell.execute_reply":"2022-11-22T02:26:39.308399Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"learn.recorder.plot_loss()","metadata":{"execution":{"iopub.status.busy":"2022-11-22T02:26:39.313657Z","iopub.execute_input":"2022-11-22T02:26:39.313966Z","iopub.status.idle":"2022-11-22T02:26:39.562451Z","shell.execute_reply.started":"2022-11-22T02:26:39.313935Z","shell.execute_reply":"2022-11-22T02:26:39.561380Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Export a model\nfor use elsewhere or for loading into inference notebook","metadata":{}},{"cell_type":"code","source":"learn.export()","metadata":{"execution":{"iopub.status.busy":"2022-11-22T02:26:39.563753Z","iopub.execute_input":"2022-11-22T02:26:39.564228Z","iopub.status.idle":"2022-11-22T02:26:39.779969Z","shell.execute_reply.started":"2022-11-22T02:26:39.564188Z","shell.execute_reply":"2022-11-22T02:26:39.778948Z"},"trusted":true},"execution_count":null,"outputs":[]}]}