{"cells":[{"metadata":{},"cell_type":"markdown","source":"# iMet 2020 — FGVC7 dataset\n\n## A small test using fastai\n\nHello World! This is my second submission for Kaggle's [iMet 2020 — FGVC7 competition](https://www.kaggle.com/c/imet-2020-fgvc7/overview). Here we are checking some approaches taught on fastai's classes. There are a lot of inspiration from other places to this; you can find the references at the bottom, ok?\n\n* __Notes:__ in this third iteration, we increased the batch size a little bit more, and also number of epochs on the first and second training rounds.\n\nWe start importing everything from `fastai.vision`. Hey, not very PEP8-ish, but stay with me.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from fastai.vision import *","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's check if we have a GPU available.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now we use the Jupyter magic `%matplotlib inline`, so all figures we generate will appear inside this notebook.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's choose a batch size here. In my tests, I had better results with a large number — larger than 32, although the literature says otherwise here. I am not sure why, but I will check it.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"batch_size = 400","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Setting up our input and output paths, train and test folders.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"path_input = Path('/kaggle/input/imet-2020-fgvc7')\nfolder_train = 'train'\nfolder_test = 'test'\n\npath_output = Path('/kaggle/working')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's take a look on the file `train.csv`. Check that a filename (`id`) can have several labels (`attribute_ids`); these labels are separated by spaces from each other.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_csv = pd.read_csv(path_input/'train.csv')\ntrain_csv.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now we define our source image list, using `ImageList`.\n\n* First, we set a seed — then you will generate the same images.\n* We use `from_csv()` to read data on `path`, gather labels from `train.csv`, and label images on `folder_train` using these labels.\n* We will also use 20% of the images on that folder as the validation dataset (`split_by_rand_pct(0.2)`).\n* `label_from_df(label_delim=' ')` ensures that labels will be split whenever we have a space between them.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"np.random.seed(42)  # the \"Answer\" is 42! \nsource = (ImageList.from_csv(path=path_input,\n                             csv_name='train.csv',\n                             folder=folder_train,\n                             suffix='.png')\n                   .split_by_rand_pct(0.2)\n                   .label_from_df(label_delim=' ')\n         )","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now we define some transforms for data augmentation. We will allow some variation on lighting and zoom, and disable warping.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"aug_transforms = get_transforms(max_lighting=0.1,\n                                max_zoom=1.05,\n                                max_warp=0.)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"OK, with source and transforms ready, it's time to create our dataset.\n* First, we will use `transform()` to apply our augmentation transforms and set an image size of `(128, 128)`.\n* Then, we use `databunch()` to create objects with 32 images each (`bs=batch_size`).\n* We use also `normalize(imagenet_stats)` to use normalization parameters from ImageNet. These are `mean = [0.485, 0.456, 0.406]` and `std = [0.229, 0.224, 0.225]`, as seen in [`fastai.vision.data`](https://github.com/fastai/fastai/blob/aeae6a0ae0cb975d916d43b3e1d71afd42894977/fastai/vision/data.py#L78). Each coordinate is applied to a dimension of the RGB input images.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"data = (source.transform(aug_transforms, size=128)\n              .databunch(bs=batch_size)\n              .normalize(imagenet_stats)\n       )","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Great! Our data is ready. Let's see some examples using `data.show_batch()`.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"data.show_batch(rows=3, figsize=(10, 10))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now we create folders to receive the models, since we cannot use the internet.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# copying the ResNet-50 structure...\n!mkdir -p /root/.cache/torch/checkpoints\n!cp /kaggle/input/resnet/resnet50-19c8e357.pth /root/.cache/torch/checkpoints/resnet50-19c8e357.pth\n\n# ... and the previously trained models.\n!cp /kaggle/input/pretrained-models/imet-stage1_size128.pth /kaggle/working\n!cp /kaggle/input/pretrained-models/imet-stage2_size128.pth /kaggle/working","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now we define the architecture we will use to train the dataset. We will use the ResNet with 50 layers:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"architecture = models.resnet50","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's define some metrics to evaluate our model.\n* Here we use accuracy with a threshold of 0.2 (`accuracy_thresh`), and the F-2 score (`fbeta`), the function used to evaluate inputs in this competition. `beta`'s default in `fbeta` is already 2.\n* `partial()` helps us calculating these measures for each epoch.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"acc_02 = partial(accuracy_thresh, thresh=0.2) \nf_score = partial(fbeta, thresh=0.2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Time to create our model! We use `cnn_learner` for that. The inputs are our data, the architecture, and the metrics.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"learn = cnn_learner(data,\n                    architecture,\n                    metrics=[acc_02, f_score],\n                    model_dir=path_output)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can use `lr_find()` to check a suitable learning rate. Using `suggestion=True`, fastai returns a suggestion for that value — the min numerical gradient — as well.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# learn.lr_find()\n# learn.recorder.plot(suggestion=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learning_rate = 3.31E-02  # as suggested by fastai's lr_find()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's start fitting! First we fit during 100 epochs. We use the 1cycle policy to fit.\n\nAfter fitting, we save the partial results using `learn.save()`.\n\nTo fit and save, please uncomment (remove the `#`) on the lines below.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# learn.fit_one_cycle(100, slice(learning_rate))\n# learn.save('imet-stage1_size128')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"My results during this training stage were:\n\n```\nepoch     train_loss  valid_loss  accuracy_thresh  fbeta     time\n(...)\n97        0.002964    0.003352    0.998843         0.587971  09:15\n98        0.002957    0.003352    0.998839         0.588603  09:16\n99        0.002953    0.003348    0.998832         0.589520  09:15\n```\n\nI am using a nVidia Tesla K80, 12 GB of RAM, on an Ubuntu Linux 20.04 machine. My driver version is 440.64, my CUDA version is 10.2.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"We already fitted this network, so we use the command `learn.load()` to load it.  After loading, we unfreeze (allowing access to the layers of) the     network, to continue fitting a little bit more. A more precise definition of `unfreeze()`, from the [fastai documentation](https://docs.fast.ai/basic_train.html):\n\n> * when we say `unfreeze`, we mean that in the specified layer groups the `requires_grad` of all layers with weights (except BatchNorm layers) are set `True`, so the layer weights will be updated during training.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.load('imet-stage1_size128')\nlearn.save('imet-stage1_size128')\nlearn.unfreeze()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now we use `lr_find()` to return a new learning rate suggestion.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# learn.lr_find()\n# learn.recorder.plot(suggestion=True)\n\nlearning_rate = 1.74E-05  # fastai's lr_find() suggested 8.32E-06 this time. I was sleeping, so I'll try this on a next iteration :)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"With the new learning rate, we come back to fitting. Now we fit for 100 more epochs.\n\nTo fit and save, please uncomment (remove the `#`) on the lines below.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# learn.fit_one_cycle(100, slice(learning_rate, learning_rate/10), wd=0.1)\n# learn.save('imet-stage2_size128')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"My results during this training stage were:\n\n```\nepoch     train_loss  valid_loss  accuracy_thresh  fbeta     time\n(...)\n97        0.002891    0.003315    0.998829         0.595562  11:57\n98        0.002901    0.003321    0.998842         0.593342  11:56\n99        0.002903    0.003327    0.998858         0.590404  11:57\n```","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"We already fitted this network, so we use the command `learn.load()` one more time.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.load('imet-stage2_size128')\nlearn.save('imet-stage2_size128')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We finish our training session freezing the network and exporting our coefficients.  A definition of `freeze()`, from the [fastai documentation](https://docs.fast.ai/basic_train.html):\n\n> When we say `freeze`, we mean that in the specified layer groups the `requires_grad` of all layers with weights (except BatchNorm layers) are set `False`, so the layer weights won't be updated during training. ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.freeze()\nlearn.export(path_output/'export.pkl')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Generating `submission.csv`\n\nFinally, we use the training coefficients to gather the results on the test data and generate the submission file.\nTo predict on the test images, we:\n* Generate a list with all test images, using `ItemList.from_folder()`.\n* Load our learner and point it to the test images, using `load_learner()`.\n* Predict (infer) on the test images, using `learn.get_preds()`.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"itemlist_test = ItemList.from_folder(path_input/folder_test)\nlearn = load_learner(path=path_output, test=itemlist_test)\npredictions, _ = learn.get_preds(ds_type=DatasetType.Test)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The next steps are:\n* Set a threshold to consider valid results.\n* Gather predictions higher than the threshold we set, and group them according to the input file.\n* Gather the filenames for processed test images.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"threshold = 0.1\nlabelled_preds = [' '.join([learn.data.classes[idx] for idx, pred in enumerate(prediction) if pred > threshold]) for prediction in predictions]\nfilenames = [f.name[:-4] for f in learn.data.test_ds.items]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Finally, we create a pandas `DataFrame` to receive the variables, and transform it to a `.csv` — the submission file.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"data_frame = pd.DataFrame({'id':filenames, 'attribute_ids':labelled_preds},\n                           columns=['id', 'attribute_ids'])\ndata_frame.to_csv(path_output/'submission.csv',\n                  index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## References\n\nSome references I used throughout the process. Thank you y'all!\n\n* The basics, and all the structure of the notebook, came from Jeremy Howard's awesome course on `fastai`. Some notes — more succinct, from @PoonamV, and more detailed, from @hiromi — follow as well:\n\n[1] https://course.fast.ai/videos/?lesson=3\n\n[2a] https://forums.fast.ai/t/deep-learning-lesson-3-notes/29829\n\n[2b] https://github.com/hiromis/notes/blob/master/Lesson3.md\n\n* Where I learned how to copy the offline neural network, and the submission code as well:\n\n[3] https://www.kaggle.com/aminyakubu/aptos-2019-blindness-detection-fast-ai\n\n* Why the batch size should not matter. Masters & Luschi (2018) show that a small batch size (up to 32) could even have _better performance_:\n\n[4] https://blog.janestreet.com/does-batch-size-matter/\n\n[5] https://arxiv.org/abs/1804.07612\n\n* The great ResNet paper:\n\n[6] https://arxiv.org/abs/1512.03385\n\n* The 1cycle learning rate policy, from Smith (2018):\n\n[7] https://arxiv.org/abs/1803.09820\n\n* Where I learned how to use trained-elsewhere nets on Kaggle:\n\n[8] https://www.kaggle.com/finlaymacrae/fastai-resnet34-transfer-learning\n\n* The Hitchhiker's Guide th the Galaxy and 42 as the \"Answer to the Ultimate Question of Life, the Universe, and Everything\":\n\n[42] https://en.wikipedia.org/wiki/42_(number)#The_Hitchhiker's_Guide_to_the_Galaxy","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"_This notebook is licensed under the BSD 3-Clause License._\n\nRedistribution and use in source and binary forms, with or without\nmodification, are permitted provided that the following conditions are met:\n\n  * Redistributions of source code must retain the above copyright\n    notice, this list of conditions and the following disclaimer.\n  * Redistributions in binary form must reproduce the above copyright\n    notice, this list of conditions and the following disclaimer in the\n    documentation and/or other materials provided with the distribution.\n  * Neither the name of the <organization> nor the\n    names of its contributors may be used to endorse or promote products\n    derived from this software without specific prior written permission.\n\nTHIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\nAND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\nIMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\nARE DISCLAIMED. IN NO EVENT SHALL <COPYRIGHT HOLDER> BE LIABLE FOR ANY\nDIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES\n(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;\nLOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND\nON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\n(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF\nTHIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}