{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## What is this ?\n* A starter notebook for folks new to machine learning.\n* A vanilla CNN created for whale species recognition using fastai.\n* Aim is to observe the behavior of a simple CNN trained to recognize the whale species.\n* Here I am trying to show that you can use a simple CNN to know what areas of an image does the CNN \"sees\". You can use this to analyze what needs to be improved or engineered in the data to create a better model.","metadata":{}},{"cell_type":"code","source":"# fastai in kaggle has issue if this is not used-https://www.kaggle.com/product-feedback/279990\n!pip install --user torch==1.9.0 torchvision==0.10.0 torchaudio==0.9.0 torchtext==0.10.0","metadata":{"execution":{"iopub.status.busy":"2022-02-16T17:19:09.743663Z","iopub.execute_input":"2022-02-16T17:19:09.744249Z","iopub.status.idle":"2022-02-16T17:20:08.871591Z","shell.execute_reply.started":"2022-02-16T17:19:09.744206Z","shell.execute_reply":"2022-02-16T17:20:08.870737Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Import required things.","metadata":{}},{"cell_type":"code","source":"from fastai.vision.all import *","metadata":{"execution":{"iopub.status.busy":"2022-02-16T17:20:43.380876Z","iopub.execute_input":"2022-02-16T17:20:43.381168Z","iopub.status.idle":"2022-02-16T17:20:45.224966Z","shell.execute_reply.started":"2022-02-16T17:20:43.381136Z","shell.execute_reply":"2022-02-16T17:20:45.224106Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Required paths","metadata":{}},{"cell_type":"code","source":"root_path = Path(\"../input\")\ntrain_csv_path = root_path/'happy-whale-and-dolphin/train.csv'\ntrain_img_path = root_path/\"jpeg-happywhale-128x128/train_images-128-128/train_images-128-128\"","metadata":{"execution":{"iopub.status.busy":"2022-02-16T17:20:46.080667Z","iopub.execute_input":"2022-02-16T17:20:46.080994Z","iopub.status.idle":"2022-02-16T17:20:46.086617Z","shell.execute_reply.started":"2022-02-16T17:20:46.080951Z","shell.execute_reply":"2022-02-16T17:20:46.085185Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Get a glimpse of the files\n`get_image_files` is a convenience function that goes through different folders and subfolders and gathers all the image file path.","metadata":{}},{"cell_type":"code","source":"train_imgs = get_image_files(train_img_path)\ntrain_imgs[:10]","metadata":{"execution":{"iopub.status.busy":"2022-02-16T17:20:46.639825Z","iopub.execute_input":"2022-02-16T17:20:46.640387Z","iopub.status.idle":"2022-02-16T17:21:29.885595Z","shell.execute_reply.started":"2022-02-16T17:20:46.640338Z","shell.execute_reply":"2022-02-16T17:21:29.884916Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Image.open(train_imgs[1])","metadata":{"execution":{"iopub.status.busy":"2022-02-16T17:21:29.887247Z","iopub.execute_input":"2022-02-16T17:21:29.887509Z","iopub.status.idle":"2022-02-16T17:21:29.911921Z","shell.execute_reply.started":"2022-02-16T17:21:29.887475Z","shell.execute_reply":"2022-02-16T17:21:29.911322Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"let's see what is there in the csv file.","metadata":{}},{"cell_type":"code","source":"train_path_df = pd.read_csv(train_csv_path)","metadata":{"execution":{"iopub.status.busy":"2022-02-16T17:21:29.912973Z","iopub.execute_input":"2022-02-16T17:21:29.913798Z","iopub.status.idle":"2022-02-16T17:21:29.998272Z","shell.execute_reply.started":"2022-02-16T17:21:29.913758Z","shell.execute_reply":"2022-02-16T17:21:29.997527Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_path_df.head(5)","metadata":{"execution":{"iopub.status.busy":"2022-02-16T17:21:32.020217Z","iopub.execute_input":"2022-02-16T17:21:32.020485Z","iopub.status.idle":"2022-02-16T17:21:32.036875Z","shell.execute_reply.started":"2022-02-16T17:21:32.020457Z","shell.execute_reply":"2022-02-16T17:21:32.036226Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"If I can catch hold of the image filename from the dataframe and attach this to the `train_img_path` then I can get the full image path.","metadata":{}},{"cell_type":"markdown","source":"Replacing filnames with the filepath","metadata":{}},{"cell_type":"code","source":"train_path_df['image'] = train_path_df['image'].apply(lambda x:train_img_path/x)","metadata":{"execution":{"iopub.status.busy":"2022-02-16T17:21:34.608845Z","iopub.execute_input":"2022-02-16T17:21:34.609394Z","iopub.status.idle":"2022-02-16T17:21:35.253515Z","shell.execute_reply.started":"2022-02-16T17:21:34.609355Z","shell.execute_reply":"2022-02-16T17:21:35.252724Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_path_df.head(2)","metadata":{"execution":{"iopub.status.busy":"2022-02-16T17:21:35.436034Z","iopub.execute_input":"2022-02-16T17:21:35.436732Z","iopub.status.idle":"2022-02-16T17:21:35.445561Z","shell.execute_reply.started":"2022-02-16T17:21:35.436696Z","shell.execute_reply":"2022-02-16T17:21:35.444879Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Image.open(train_path_df.image[0])","metadata":{"execution":{"iopub.status.busy":"2022-02-16T17:21:36.152722Z","iopub.execute_input":"2022-02-16T17:21:36.153239Z","iopub.status.idle":"2022-02-16T17:21:36.173563Z","shell.execute_reply.started":"2022-02-16T17:21:36.153197Z","shell.execute_reply":"2022-02-16T17:21:36.172631Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Datablock\nA `DataBlock` is like a blueprint that tells fastai-->\n* What kind of data are we dealing with. Is it image, text etc.\n* What kind of labels we have. For example categorical labels, continuous labels etc.\n* From where to get the inputs.\n* From where to get the targets.\n* How to split the data into train and test.\n* The transforms that you want to do.","metadata":{}},{"cell_type":"code","source":"data = DataBlock(blocks=(ImageBlock, CategoryBlock),\n                splitter=TrainTestSplitter(),\n                 get_x = ColReader(0),\n                 get_y = ColReader(1),\n                 item_tfms=Resize(224),\n                 batch_tfms=aug_transforms(size=128))","metadata":{"execution":{"iopub.status.busy":"2022-02-16T17:21:38.208257Z","iopub.execute_input":"2022-02-16T17:21:38.208892Z","iopub.status.idle":"2022-02-16T17:21:38.215798Z","shell.execute_reply.started":"2022-02-16T17:21:38.208852Z","shell.execute_reply":"2022-02-16T17:21:38.215117Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In the above code--> \n* The `ImageBlock`  tells fastai that we are dealing with image data.\n* `CategoryBlock` means our targets are categorical in nature.\n* The  `get_x` and `get_y` tells fastai to get the x i.e. the image path from column 0 of dataframe and the target from column 1 of the dataframe.\n* `splitter` is the way in which train and test data are split. check [here](https://docs.fast.ai/data.transforms.html#TrainTestSplitter) for more details.\n* The next two lines are not that important to understand now but they are needed when you are working on image augmentation. Refer to the [fastbook](https://github.com/fastai/fastbook) to understand more about this.","metadata":{}},{"cell_type":"code","source":"# this is not necessary. I was just testing if my datablocks are correct or not\n#data.summary(train_path_df)","metadata":{"execution":{"iopub.status.busy":"2022-02-16T17:22:22.952509Z","iopub.execute_input":"2022-02-16T17:22:22.952764Z","iopub.status.idle":"2022-02-16T17:22:22.955858Z","shell.execute_reply.started":"2022-02-16T17:22:22.952736Z","shell.execute_reply":"2022-02-16T17:22:22.955196Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Dataloader\nThink of a dataloader as a mechanism that picks up your images, does all the things that you had described in the datablock and then loads your data to the device(cpu or gpu).\n\nAt the minimum, you need to provide the source of data, the `train_path_df` in our case. Here we give the `bs` i.e. the batch size. For example if we say that the batch size is 8 then it means that the dataloader will load 8 images at a time onto the device.","metadata":{}},{"cell_type":"code","source":"dls = data.dataloaders(train_path_df, bs=128)","metadata":{"execution":{"iopub.status.busy":"2022-02-16T17:22:31.242311Z","iopub.execute_input":"2022-02-16T17:22:31.242613Z","iopub.status.idle":"2022-02-16T17:22:37.18423Z","shell.execute_reply.started":"2022-02-16T17:22:31.24258Z","shell.execute_reply":"2022-02-16T17:22:37.183353Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dls.show_batch()","metadata":{"execution":{"iopub.status.busy":"2022-02-16T17:23:17.287773Z","iopub.execute_input":"2022-02-16T17:23:17.288039Z","iopub.status.idle":"2022-02-16T17:23:19.146949Z","shell.execute_reply.started":"2022-02-16T17:23:17.288009Z","shell.execute_reply":"2022-02-16T17:23:19.146141Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## The model\n\nSimple CNN to classify whale/dolphin species. The intuition is the following-->\n* a model which knows to recognize whale/dolphin species should also be able to learn the features to distinguish the different species.\n* such a model can be queried to learn what all portions of an image is taken into account. For example, if somehow the background is tricking the model such that the model considers the background as an area of interest then such data needs to be further engineered.\n\n* such model can then be used as a pre-trained model to recognize individuals .","metadata":{}},{"cell_type":"markdown","source":"## Creating a simple CNN learner\n`cnn_learner` creates a convolutional neural network for you. At the least you provide it the dataloader, a model architecture and a loss function.\n\n`models.resnet18` is a \"pre-trained\" neural network. It's a network that was already trained by some good folks (folks with huge computational hardware) on a very large  number of images. This particular network knows how to recognize different images. So, we take this network and then ask fastai to train this network on our image data (whale, dolphins). \n\nThe intuition is that since the pre-trained network already knows how to distinguish between different images, we have to spend less time and computation to make it understand how to recognize different whales and dolphins. This is **Transfer learning**","metadata":{}},{"cell_type":"code","source":"learn = cnn_learner(dls, models.resnet18, loss_func=CrossEntropyLossFlat(), ps=0.25)","metadata":{"execution":{"iopub.status.busy":"2022-02-16T17:23:34.215818Z","iopub.execute_input":"2022-02-16T17:23:34.216127Z","iopub.status.idle":"2022-02-16T17:23:36.313741Z","shell.execute_reply.started":"2022-02-16T17:23:34.216093Z","shell.execute_reply":"2022-02-16T17:23:36.313045Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"For now understand that `fine_tune` means \"take my network and make it look at each image a specified number of times. This \"number of times\" is the number that is provided as the first argument.\n\nLet's not dive into the second argument at this moment in time.\n\n> For the curious mind, the second argument is known as the learning rate. It's a hyperparameter of a neural network. More information can be found [here](https://github.com/fastai/fastbook/blob/master/04_mnist_basics.ipynb)","metadata":{}},{"cell_type":"code","source":"learn.fine_tune(2, 3e-3)","metadata":{"execution":{"iopub.status.busy":"2022-02-16T17:23:37.478657Z","iopub.execute_input":"2022-02-16T17:23:37.479467Z","iopub.status.idle":"2022-02-16T17:32:38.720327Z","shell.execute_reply.started":"2022-02-16T17:23:37.479409Z","shell.execute_reply":"2022-02-16T17:32:38.716233Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see what the model predicted vs the actual target.","metadata":{}},{"cell_type":"code","source":"learn.show_results()","metadata":{"execution":{"iopub.status.busy":"2022-02-16T17:32:46.541083Z","iopub.execute_input":"2022-02-16T17:32:46.541616Z","iopub.status.idle":"2022-02-16T17:32:47.765872Z","shell.execute_reply.started":"2022-02-16T17:32:46.541576Z","shell.execute_reply":"2022-02-16T17:32:47.76495Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"interp = Interpretation.from_learner(learn)\ninterp.plot_top_losses(3, figsize=(15,10))","metadata":{"execution":{"iopub.status.busy":"2022-02-16T17:33:39.02569Z","iopub.execute_input":"2022-02-16T17:33:39.025956Z","iopub.status.idle":"2022-02-16T17:34:12.044695Z","shell.execute_reply.started":"2022-02-16T17:33:39.025923Z","shell.execute_reply":"2022-02-16T17:34:12.043901Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Why the neural network makes the decision?\nWe can use class activation maps (CAM) to see which regions of images are of interest to the neural network. Class activation map (CAM) was first introduced by Bolei Zhou et al. in [\"Learning Deep Features for Discriminative Localization\"](https://arxiv.org/abs/1512.04150). Cam uses the output of the last convolutional layer with the predictions to give a heatmap of the regions of interest of the network in an image.\n\nThe intuition behind CAM is that if we do the dot product of the activations of the final layer with the final weights, for each location on our feature map then we can get the score of the feature that was used to make a decision. \n\nPytorch provides hooks to hook onto the any layer of a network. We can attach a hook to any layer of a neural network and it will be executed during the forward pass i.e. the moment when the output is computed or during the backward pass i.e. the moment when the weights are being re-adjusted.\n\nI would highly recommend to read more about CAM in this chapter of [fastbook](https://github.com/fastai/fastbook/blob/master/18_CAM.ipynb) before proceeding on with the code.","metadata":{}},{"cell_type":"code","source":"Image.open(train_path_df.image[6])","metadata":{"execution":{"iopub.status.busy":"2022-02-16T17:35:19.246451Z","iopub.execute_input":"2022-02-16T17:35:19.246714Z","iopub.status.idle":"2022-02-16T17:35:19.265745Z","shell.execute_reply.started":"2022-02-16T17:35:19.246685Z","shell.execute_reply":"2022-02-16T17:35:19.264966Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's grab a batch.","metadata":{}},{"cell_type":"code","source":"img = PILImage.create(train_path_df.image[6])\n# grab a batch\nx, = first(dls.test_dl([img]))","metadata":{"execution":{"iopub.status.busy":"2022-02-16T17:35:24.873964Z","iopub.execute_input":"2022-02-16T17:35:24.874716Z","iopub.status.idle":"2022-02-16T17:35:25.141434Z","shell.execute_reply.started":"2022-02-16T17:35:24.874665Z","shell.execute_reply":"2022-02-16T17:35:25.140499Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We hook into the network . A forward hook takes into three things --> the model , its input and it's output. The fourth line in the below code is the hook function.\n","metadata":{}},{"cell_type":"code","source":"class Hook():\n    def __init__(self, m):\n        self.hook = m.register_forward_hook(self.hook_func)   \n    def hook_func(self, m, i, o): self.stored = o.detach().clone()\n    def __enter__(self, *args): return self\n    def __exit__(self, *args): self.hook.remove()","metadata":{"execution":{"iopub.status.busy":"2022-02-16T17:35:26.16549Z","iopub.execute_input":"2022-02-16T17:35:26.166037Z","iopub.status.idle":"2022-02-16T17:35:26.171601Z","shell.execute_reply.started":"2022-02-16T17:35:26.165998Z","shell.execute_reply":"2022-02-16T17:35:26.170906Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we do the dot product of our weight matrix  with the activations.\n\nThe code below takes the model, hooks into the last layer of our model using our hook function and stores the activations in the `act` variable. Then we do the dot product of the stored activations with the weights using `torch.einsum`.","metadata":{}},{"cell_type":"code","source":"with Hook(learn.model[0]) as hook:\n    with torch.no_grad(): \n        output = learn.model.eval()(x.cuda())\n        act = hook.stored[0]\n    cam_map = torch.einsum('ck,kij->cij', learn.model[1][-1].weight, act)\n    cam_map.shape","metadata":{"execution":{"iopub.status.busy":"2022-02-16T17:35:34.630728Z","iopub.execute_input":"2022-02-16T17:35:34.630985Z","iopub.status.idle":"2022-02-16T17:35:34.64734Z","shell.execute_reply.started":"2022-02-16T17:35:34.630955Z","shell.execute_reply":"2022-02-16T17:35:34.646631Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"For each image in our batch, and for each class, we get a 7×7 feature map that tells us where the activations were higher and where they were lower. This will let us see which areas of the pictures influenced the model's decision.","metadata":{}},{"cell_type":"code","source":"x_dec = TensorImage(dls.train.decode((x,))[0][0])\n_,ax = plt.subplots()\nx_dec.show(ctx=ax)\nax.imshow(cam_map[1].detach().cpu(), alpha=0.6, extent=(0,128,128,0),\n              interpolation='bilinear', cmap='magma');","metadata":{"execution":{"iopub.status.busy":"2022-02-16T17:35:35.969504Z","iopub.execute_input":"2022-02-16T17:35:35.969776Z","iopub.status.idle":"2022-02-16T17:35:36.152729Z","shell.execute_reply.started":"2022-02-16T17:35:35.969745Z","shell.execute_reply":"2022-02-16T17:35:36.151967Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The bright areas in the activations are somehow spilling over to the ocean. In my opinion this should not be the case. The activations should be more around the of the dorsal fins and the surface of the fin. In layman terms the network should focus more on the object i.e. the fin features and not on the surrounding water.\n\nAn important takeaway from my perspective is that if we can restrict the region of interest to the whale fin and not the surrounding water then the network would be able to learn better.\n\n## Conclusion\nBesides tasks like classifying images or detecting objects, a neural networks can be used for analyzing the data to device strategies for pre-processing, data engineering and data cleaning. \n\nHere I have bounced off one idea about how a simple CNN can be used to understand the regions in an image that the neural network focuses to make it's decisions and then analyzing these things we can observer if the network is focusing on areas of images that it needs to focus and if not, then we can think of the steps that we can take to help the network to focus to generalize better.\n\nI have just scratched the surface of what's possible here and there can be many more ways in which deep learning can be used to take better decisions on data. So, I would emphasize you to expand on this idea and think of other ways in which you can use simple networks to analyze the data before building an architecture for the actual task at hand.","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}