{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Cassava Classification - fastai Starter\n\n\nThe datasets for this notebook are from [this](https://www.kaggle.com/tanlikesmath/cassava-classification-eda-fastai-starter) notebook, while this tutorial will be a twist on what has already been done!"},{"metadata":{},"cell_type":"markdown","source":"## What will this tutorial cover?\n\nWe will be looking at how to use the high-level `DataBlock` API for this challenge, how to use some advanced training features in the library, as well as some advanced inference features."},{"metadata":{},"cell_type":"markdown","source":"## Importing the Library\n\nFirst, let's import the `fastai.vision` library for us to use and work with:"},{"metadata":{"trusted":true},"cell_type":"code","source":"from fastai.vision.all import *","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"For these results to be reproducable on your end, we will go ahead and set the `random`, `torch`, and `numpy` seeds with the `set_seed` function"},{"metadata":{"trusted":true},"cell_type":"code","source":"set_seed(16)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Setting up our data\n\nAlright, now that we have our imports let's go ahead and look at our data.\n\nFirst we'll make a `Path` object so we can see what all we have available to us:"},{"metadata":{"trusted":true},"cell_type":"code","source":"path = Path(\"../input\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can then run an `ls()` (a monkey-patched function by `fastcore`) to see all the files and directories in here:"},{"metadata":{"trusted":true},"cell_type":"code","source":"path.ls()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see that the resnet50 pretrained model is there and the competition data. We'll make a `data_path` to point to this directory and take another peek:"},{"metadata":{"trusted":true},"cell_type":"code","source":"data_path = path/'cassava-leaf-disease-classification'\ndata_path.ls()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Our images are stored away in `train_images` and `test_images`, and we have a `train.csv` for our labels. Let's load it into `pandas`:"},{"metadata":{"trusted":true},"cell_type":"code","source":"df = pd.read_csv(data_path/'train.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"And take a look:"},{"metadata":{"trusted":true},"cell_type":"code","source":"df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Adjusting the `image_id`"},{"metadata":{},"cell_type":"markdown","source":"We have an `image_id` and a `label`. We're going to modify our values in `image_id` to make our lives easier when it comes to running inference. \n\nWhy? \n\n\nIn fastai we have a `get_x` and a `get_y` and this will dictate how it will *always* look for our data, regardless of how it is stored. If we built a `get_y` based on the current `DataFrame`, it would look something like so:"},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_x(row): return data_path/'train_images'/row['image_id']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"And we can see it work below:"},{"metadata":{"trusted":true},"cell_type":"code","source":"PILImage.create(get_x(df.iloc[0]))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"That's great! ***But*** there is a very large issue here. We always have our `get_x` tied to the training directory which makes it more complicated for us to work with our `test_images` directory.\n\nWhat's the solution? \n\nAdd `train_images` into the dataframe through a `lambda` function:"},{"metadata":{"trusted":true},"cell_type":"code","source":"df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df['image_id'] = df['image_id'].apply(lambda x: f'train_images/{x}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"And now we can see our new table:"},{"metadata":{"trusted":true},"cell_type":"code","source":"df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now we won't run into an issue when we're testing. \n"},{"metadata":{},"cell_type":"markdown","source":"### Adjusting our label\n\nWhat else can we do?\n\nLet's change our lables into something more readable through a dictionary (these come from the `json` file):"},{"metadata":{"trusted":true},"cell_type":"code","source":"idx2lbl = {0:\"Cassava Bacterial Blight (CBB)\",\n          1:\"Cassava Brown Streak Disease (CBSD)\",\n          2:\"Cassava Green Mottle (CGM)\",\n          3:\"Cassava Mosaic Disease (CMD)\",\n          4:\"Healthy\"}\n\ndf['label'].replace(idx2lbl, inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"And now we're good to go! Let's build the `DataBlock`"},{"metadata":{},"cell_type":"markdown","source":"## Building the `DataBlock`\n\nLet's think about how our problem looks. `fastai` provides blocks to center around *most* situations, and this is no exception.\n\nWe know our input is an image and our output is a category, so let's use `ImageBlock` and `CategoryBlock`:"},{"metadata":{"trusted":true},"cell_type":"code","source":"blocks = (ImageBlock, CategoryBlock)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next we'll want to split our data somehow. We'll use a `RandomSplitter` and split our data 80/20"},{"metadata":{"trusted":true},"cell_type":"code","source":"splitter = RandomSplitter(valid_pct=0.2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Our `DataBlock` is also going to want to know how to get our data. Since our data all stems from a `csv`, we will make a `get_x` and `get_y` function:\n\n> We already made our `get_x` earlier, so I have brought it down here"},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_x(row): return data_path/row['image_id']\n\ndef get_y(row): return row['label']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see that when we write custom `get_` functions, it will accept one *row* of our `DataFrame` to look at, and so we can filter as a result.\n\nNext we'll come up with some basic data augmentations. \n\nOur `item_tfms` should ensure everything is ready to go into a batch, so we will use `Resize`.\n\nOur `batch_tfms` should apply any extra augmentations we may want. We'll use `RandomResizedCropGPU`, `aug_transforms`, and apply our `Normalize`:\n> We will normalize our data based on ImageNet, since that is what our pretrained model was trained with"},{"metadata":{"trusted":true},"cell_type":"code","source":"item_tfms = [Resize(448)]\nbatch_tfms = [RandomResizedCropGPU(224), *aug_transforms(), Normalize.from_stats(*imagenet_stats)]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Finally, let's build the `DataBlock`!"},{"metadata":{"trusted":true},"cell_type":"code","source":"block = DataBlock(blocks = blocks,\n                 get_x = get_x,\n                 get_y = get_y,\n                 splitter = splitter,\n                 item_tfms = item_tfms,\n                 batch_tfms = batch_tfms)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"And now we can turn this into some `DataLoaders`. We're going to pass in some items (which in our case is our `DataFrame`) and a batch size to use. We will use 64:"},{"metadata":{"trusted":true},"cell_type":"code","source":"dls = block.dataloaders(df, bs=64)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's look at a batch of data to make sure everything looks alright:"},{"metadata":{"trusted":true},"cell_type":"code","source":"dls.show_batch(figsize=(12,12))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Looks great! Let's move onto training our model"},{"metadata":{},"cell_type":"markdown","source":"## The Model\n\nThe code here is from tanlikesmath's notebook linked at the start. This will move our pretrained weights to where fastai will expect it:"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Making pretrained weights work without needing to find the default filename\nif not os.path.exists('/root/.cache/torch/hub/checkpoints/'):\n        os.makedirs('/root/.cache/torch/hub/checkpoints/')\n!cp '../input/resnet50/resnet50.pth' '/root/.cache/torch/hub/checkpoints/resnet50-19c8e357.pth'","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now that our weights are setup, let's look at how to use `cnn_learner`. We're going to use a few tricks during our training that fastai can help us out with. \n\nSpecifically we will be using the `ranger` optimizer function and `LabelSmoothingCrossEntropy` as our loss function.\n\nAlong with these we'll be using the `accuracy` metric as this is how this competition will grade our results with:"},{"metadata":{"trusted":true},"cell_type":"code","source":"learn = cnn_learner(dls, resnet50, opt_func=ranger, loss_func=LabelSmoothingCrossEntropy(), metrics=accuracy)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"`fastai` has a `fit_flat_cos` function designed to best utilize the `ranger` optimizer function. Jeremy and Sylvain also came up with a `fine_tune` function best utilized for transfer learning which uses the `fit_one_cycle`, or One-Cycle Policy. We're going to create our own hybrid `fine_tune` method that will do a similar paradigm.\n\nWe can also tie it to our `Learner` objects through the `@patch` functionality. First we'll look at what `fine_tune`'s source code looks like, and rewrite it for `fit_flat_cos`:"},{"metadata":{"trusted":true},"cell_type":"code","source":"def fine_tune(self:Learner, epochs, base_lr=2e-3, freeze_epochs=1, lr_mult=100,\n              pct_start=0.3, div=5.0, **kwargs):\n    \"Fine tune with `freeze` for `freeze_epochs` then with `unfreeze` from `epochs` using discriminative LR\"\n    self.freeze()\n    self.fit_one_cycle(freeze_epochs, slice(base_lr), pct_start=0.99, **kwargs)\n    base_lr /= 2\n    self.unfreeze()\n    self.fit_one_cycle(epochs, slice(base_lr/lr_mult, base_lr), pct_start=pct_start, div=div, **kwargs)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here's what it rewritten as looks like:"},{"metadata":{},"cell_type":"markdown","source":"> I have also added in the potential for callbacks, we will see more on why later"},{"metadata":{"trusted":true},"cell_type":"code","source":"@patch\ndef fine_tune_flat(self:Learner, epochs, base_lr=4e-3, freeze_epochs=1, lr_mult=100, pct_start=0.75, \n                   first_callbacks = [], second_callbacks = [], **kwargs):\n    \"Fine-tune applied to `fit_flat_cos`\"\n    self.freeze()\n    self.fit_flat_cos(freeze_epochs, slice(base_lr), pct_start=0.99, cbs=first_callbacks, **kwargs)\n    base_lr /= 2\n    self.unfreeze()\n    self.fit_flat_cos(epochs, slice(base_lr/lr_mult, base_lr), pct_start=pct_start, cbs=second_callbacks)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now we're good to train! Let's find a learning rate first:"},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.lr_find()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We'll choose a learning rate of roughly 4e-3 to start. \n\nFinally, remember how we had those extra callback parameters? We're going to utilize the `MixUp` training methodology with a decreasing `MixUp` percentage:"},{"metadata":{"trusted":true},"cell_type":"code","source":"cbs1 = [MixUp(alpha = 0.7)]\ncbs2 = [MixUp(alpha = 0.3)]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's train for 1 epoch frozen and 10 unfrozen and a `start_pct` of 0.72:"},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.fine_tune_flat(5, base_lr=1e-3, start_pct=0.72, first_callbacks=cbs1, second_callbacks=cbs2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next up we'll move to submissions"},{"metadata":{},"cell_type":"markdown","source":"## Submitting some results"},{"metadata":{},"cell_type":"markdown","source":"Let's look at the sample submission dataframe first:"},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_df = pd.read_csv(data_path/'sample_submission.csv')\nsample_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We'll want this to be similar to our training data so let's prepend that `test`:"},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_copy = sample_df.copy()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_copy['image_id'] = sample_copy['image_id'].apply(lambda x: f'test_images/{x}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next we'll make an inference dataloader through the `test_dl` method:"},{"metadata":{"trusted":true},"cell_type":"code","source":"test_dl = learn.dls.test_dl(sample_copy)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We'll look at a batch of data to make sure it all looks okay:"},{"metadata":{"trusted":true},"cell_type":"code","source":"test_dl.show_batch()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Great! Next we'll grab some predictions. We will use the `.tta` method to run test-time-augmentation which can help boost our accuracy some:"},{"metadata":{"trusted":true},"cell_type":"code","source":"preds, _ = learn.tta(dl=test_dl)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"And now we can submit them:"},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_df['label'] = preds.argmax(dim=-1).numpy()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_df.to_csv('submission.csv',index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"And that's it! We looked at a few of the neat tricks fastai can offer while also taking a look at how the `DataBlock` API can be used for such a problem.\n\nIf you enjoyed this notebook or it helped you get started please leave an upvote and if there are any questions please leave a comment! Thanks!"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}