{"cells":[{"metadata":{"id":"SRcGmc76fjLJ"},"cell_type":"markdown","source":"# fastai Efficientnet B3 with Ranger\n\nThis kernel will train a model that can achieve a score of ~83.5%"},{"metadata":{"id":"WPrIFbzffXXs","trusted":false},"cell_type":"code","source":"from fastai.vision.all import *\nset_seed(999)","execution_count":null,"outputs":[]},{"metadata":{"id":"9DecJQM8gZsg"},"cell_type":"markdown","source":"We're going to use the `wwf` library with `timm` integrated\n\n> Note: this is a training *only* kernel"},{"metadata":{"id":"Halq6HLEggxB","trusted":false},"cell_type":"code","source":"!pip install wwf timm -qqq","execution_count":null,"outputs":[]},{"metadata":{"id":"VOvpIjgrfXXt","trusted":false},"cell_type":"code","source":"from wwf.vision.timm import *","execution_count":null,"outputs":[]},{"metadata":{"id":"vacOc5M_gjz3"},"cell_type":"markdown","source":"We'll read in our data similar to the original kernel I made [here](https://www.kaggle.com/muellerzr/cassava-fastai-starter)"},{"metadata":{"id":"oIJ5buSwfXXt","trusted":false},"cell_type":"code","source":"df = pd.read_csv('../input/cassava-leaf-disease-classification/train.csv')","execution_count":null,"outputs":[]},{"metadata":{"id":"Fvez0XeYfXXt","trusted":false},"cell_type":"code","source":"idx2lbl = {0:\"Cassava Bacterial Blight (CBB)\",\n          1:\"Cassava Brown Streak Disease (CBSD)\",\n          2:\"Cassava Green Mottle (CGM)\",\n          3:\"Cassava Mosaic Disease (CMD)\",\n          4:\"Healthy\"}\n\ndf['label'].replace(idx2lbl, inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"id":"OhZwnXlMfXXt","trusted":false},"cell_type":"code","source":"data_path = Path('../input/cassava-leaf-disease-classification/')","execution_count":null,"outputs":[]},{"metadata":{"id":"5dM84jvCfXXt","trusted":false},"cell_type":"code","source":"blocks = (ImageBlock, CategoryBlock)\nsplitter = RandomSplitter(0.2, seed=999)\ndef get_x(row): return data_path/'train_images'/row['image_id']\ndef get_y(row): return row['label']\nitem_tfms = RandomResizedCrop(460, min_scale=0.75, ratio=(1.,1.))\nbatch_tfms = [*aug_transforms(size=224, max_warp=0), Normalize.from_stats(*imagenet_stats)]","execution_count":null,"outputs":[]},{"metadata":{"id":"VXqHz3xafXXt","trusted":false},"cell_type":"code","source":"block = DataBlock(blocks = blocks,\n                 get_x = get_x,\n                 get_y = get_y,\n                 splitter = splitter,\n                 item_tfms = item_tfms,\n                 batch_tfms = batch_tfms)","execution_count":null,"outputs":[]},{"metadata":{"id":"uaXSaAPvfXXt","trusted":false},"cell_type":"code","source":"dls = block.dataloaders(df, bs=32)","execution_count":null,"outputs":[]},{"metadata":{"id":"kz0XsSqogtaN"},"cell_type":"markdown","source":"Next we'll train our model:"},{"metadata":{"id":"xeGcPKRdfXXu","outputId":"dad45229-2119-4fee-be81-63a50052211c","trusted":false},"cell_type":"code","source":"learn = timm_learner(dls, 'efficientnet_b3', loss_func=LabelSmoothingCrossEntropy(), metrics=[accuracy], opt_func=ranger)\nlearn.to_native_fp16()","execution_count":null,"outputs":[]},{"metadata":{"id":"Ffk2O3UQfXXu","outputId":"5b78c73a-3585-42e1-88f7-f8b77144b989","trusted":false},"cell_type":"code","source":"learn.lr_find()","execution_count":null,"outputs":[]},{"metadata":{"id":"LfTzw0p1gvvn"},"cell_type":"markdown","source":"1e-2 works fairly well here, in the future I want to play with weight-decay some to see if it can improve the performance. We'll train with Gradient Accumulation, MixUp, and ReduceLROnPlateu"},{"metadata":{"id":"g6I9pmxCfXXv","outputId":"3a089b9b-b9ae-4a24-d2c6-adee2b3212b3","trusted":false},"cell_type":"code","source":"learn.fit_flat_cos(1,1e-2, cbs=[GradientAccumulation(), MixUp(), ReduceLROnPlateau()])","execution_count":null,"outputs":[]},{"metadata":{"id":"vNHjhI4yfXXv","outputId":"61c59540-5620-40ec-9d2e-8a3b53a7f975","trusted":false},"cell_type":"code","source":"learn.recorder.plot_loss()","execution_count":null,"outputs":[]},{"metadata":{"id":"x6yDCumzg38X"},"cell_type":"markdown","source":"We'll unfreeze and see how the lr plot looks"},{"metadata":{"id":"wC2DMrDefXXv","outputId":"9a18a0a2-7283-4a72-cc54-241483e35e79","trusted":false},"cell_type":"code","source":"learn.unfreeze()\nlearn.lr_find()","execution_count":null,"outputs":[]},{"metadata":{"id":"pcFdLDbyg59h"},"cell_type":"markdown","source":"Based on emprical work I've done, 1e-3 will be stable enough for what we want. While the metric does get a bit funky here notice the train and valid loss mostly just keep going down together"},{"metadata":{"id":"qX9e91pnfXXw","outputId":"f5d3ce40-cf18-4dc8-d757-ef80571f4096","trusted":false},"cell_type":"code","source":"learn.fit_flat_cos(10,1e-3, cbs=[GradientAccumulation(), MixUp(), ReduceLROnPlateau()])","execution_count":null,"outputs":[]},{"metadata":{"id":"hZnST076fXXw","outputId":"223056fb-50d6-408b-b584-c8d238af4167","trusted":false},"cell_type":"code","source":"learn.to_native_fp32()","execution_count":null,"outputs":[]},{"metadata":{"id":"JEB4YUINhCbb"},"cell_type":"markdown","source":"Finally you can save and export your model"},{"metadata":{"id":"HogKm3zGfXXw","outputId":"0067f6a8-935c-4602-a647-ffa4db735aa2","trusted":false},"cell_type":"code","source":"learn.save('b3')","execution_count":null,"outputs":[]},{"metadata":{"id":"f7U8K6dvhJJ3"},"cell_type":"markdown","source":"## Inference\n\nThis should be adapted in a seperate kernel, but here is a general guideline for submitting your code I have been doing:\n\n\n1. Create a dataset with your saved model\n2. Follow [this](https://www.kaggle.com/muellerzr/submission-notebook) notebook for generating your submission."}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}