{"cells":[{"metadata":{},"cell_type":"markdown","source":"Versions:\n\n* v3 - First running version of the code. However, runtime is about 2 hours which will be too long for submission.\n* v4 - Reduced number of epochs to train\n* v5 - turn off internet connection for submission."},{"metadata":{},"cell_type":"markdown","source":"# Pretraining For APTOS Blindness Detection\n\nIn this kernel, I use my dataset of cropped and resized images from the previous [retinopathy detection competition](https://www.kaggle.com/c/diabetic-retinopathy-detection) to train a ResNet50 model. I then take that model and train on this competition dataset. I am not sure if this will work (as the image distributions are different) but we will see!"},{"metadata":{"trusted":true,"_uuid":"875c2659dde79c26db51f02c0fe336a89d2aca86","_kg_hide-input":false,"_kg_hide-output":false},"cell_type":"code","source":"import os\nfiles = os.listdir('../input/diabetic-retinopathy-resized/resized_train_cropped/resized_train_cropped')\nprint('Number of files: ',len(files)) ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Making pretrained weights work without needing to find the default filename\nif not os.path.exists('/tmp/.cache/torch/checkpoints/'):\n        os.makedirs('/tmp/.cache/torch/checkpoints/')\n!cp '../input/resnet50/resnet50.pth' '/tmp/.cache/torch/checkpoints/resnet50-19c8e357.pth'","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"%reload_ext autoreload\n%autoreload 2\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"from fastai import *\nfrom fastai.vision import *\nimport pandas as pd\nimport matplotlib.pyplot as plt","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def seed_everything(seed):\n    random.seed(seed)\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    np.random.seed(seed)\n    torch.manual_seed(seed)\n    torch.cuda.manual_seed(seed)\n    torch.backends.cudnn.deterministic = True\n\nSEED = 999\nseed_everything(SEED)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"84a40b83ea5c81e7b9787f90a57dcc5795542485"},"cell_type":"code","source":"print('Make sure cuda is installed:', torch.cuda.is_available())\nprint('Make sure cudnn is enabled:', torch.backends.cudnn.enabled)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9e88f9a81e9613964544b8cbc219dc3c503a3948"},"cell_type":"markdown","source":"# Reading data\nHere I am going to open the dataset with pandas and check distribution of labels."},{"metadata":{"trusted":true,"_uuid":"9881dcf38a5f61a9c4fc1de79a42b1e956a262c6"},"cell_type":"code","source":"base_image_dir = os.path.join('..', 'input/diabetic-retinopathy-resized')\ndf = pd.read_csv(os.path.join(base_image_dir, 'trainLabels_cropped.csv'))\ndf['path'] = df['image'].map(lambda x: os.path.join(base_image_dir,'resized_train_cropped/resized_train_cropped','{}.jpeg'.format(x)))\ndf = df.drop(columns=['image'])\ndf = df.sample(frac=1).reset_index(drop=True) #shuffle dataframe\ndf.head(10)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cbb0d08b783f23bcd527f4b0891ccdfab9bc84d3"},"cell_type":"markdown","source":"The dataset is highly imbalanced, with many samples with no disease:"},{"metadata":{"trusted":true,"_uuid":"9ad4b31e5e9ba6a730558bb2234cb633660cb534"},"cell_type":"code","source":"df['level'].hist(figsize = (10, 5))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"cf4e8e212ed4bd9f37c9a7e9bc6e3f02ad3fa777"},"cell_type":"code","source":"bs =16 #smaller batch size is better for training, but may take longer\nsz=512","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"196aaa8dec9e793975e1cdc9ea560b3779dc7772"},"cell_type":"markdown","source":"Here, I load the dataset into the `ImageItemList` class provided by `fastai`. The fastai library also implements various transforms for data augmentation to improve training. While there are some defaults that I leave intact, I add vertical flipping (`do_flip=True`) as this has been commonly used for this particular problem."},{"metadata":{"trusted":true,"_uuid":"5ffa729a70f57ffea3542ec4ab80c262bba090de"},"cell_type":"code","source":"tfms = get_transforms(do_flip=True,flip_vert=True,max_rotate=360,max_warp=0,max_zoom=1.1,max_lighting=0.1,p_lighting=0.5)\nsrc = (ImageList.from_df(df=df,path='./',cols='path') #get dataset from dataset\n        .split_by_rand_pct(0.2) #Splitting the dataset\n        .label_from_df(cols='level') #obtain labels from the level column\n      )\ndata= (src.transform(tfms,size=sz,resize_method=ResizeMethod.SQUISH,padding_mode='zeros') #Data augmentation\n        .databunch(bs=bs,num_workers=4) #DataBunch\n        .normalize(imagenet_stats) #Normalize     \n       )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"fa6ee04a0efbe631dea1d55d370c017b1a2aea7e"},"cell_type":"code","source":"data.show_batch(rows=3, figsize=(7,6))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9675456a91e8bef188ba3329864057c8e04ed935"},"cell_type":"code","source":"print(data.classes)\nlen(data.classes),data.c","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1dbea921d4ad7da5e234f55da26b052a68f2b689"},"cell_type":"markdown","source":"# Training on previous dataset"},{"metadata":{"_uuid":"c251dde59220e5e6c4b6713bbacc9b57fd8d6348"},"cell_type":"markdown","source":"**Training:**\n\nWe use transfer learning, where we retrain the last layers of a pretrained neural network. I use the ResNet50 architecture trained on the ImageNet dataset, which has been commonly used for pre-training applications in computer vision. Fastai makes it quite simple to create a model and train:"},{"metadata":{"trusted":true,"_uuid":"3d0f3de01697fa7cce617ce6c8a272dd15182e6b"},"cell_type":"code","source":"import torchvision\nfrom fastai.metrics import *\nfrom fastai.callbacks import *\nlearn = cnn_learner(data, models.resnet50, wd = 1e-5, metrics = [accuracy,KappaScore(weights='quadratic')],callback_fns=[partial(CSVLogger,append=True)])\nlearn.unfreeze()\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"24f2e105338b12c64794f60282245caaccd0b5e7"},"cell_type":"code","source":"learn.lr_find()\nlearn.recorder.plot(suggestion=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3d2f180fc85239106097d06bc909db5f8e8ebfbf"},"cell_type":"code","source":"learn.fit_one_cycle(2, max_lr=slice(1e-6,4e-2))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"efbf51af0dda825cf335541b0d26a150a24902df"},"cell_type":"code","source":"learn.recorder.plot_losses()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9078f2d2090d9b385f1ec63c7672df2fd06e3ffd"},"cell_type":"code","source":"learn.recorder.plot_metrics()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"49b0c6d09518e687547346a8c8248c29bd8be866"},"cell_type":"code","source":"learn.save('prev-dataset')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"444040e745f7685c3dd6d3a4121984acef62c75d"},"cell_type":"markdown","source":"# Checking results on previous dataset"},{"metadata":{"_uuid":"cf99796e2197d141a0a7d640552b7e2cdcdc5261"},"cell_type":"markdown","source":"We look at our predictions and make a confusion matrix."},{"metadata":{"trusted":true,"_uuid":"cb2ead6c938d4c4f197574296fc86f808622a4a0"},"cell_type":"code","source":"interp = ClassificationInterpretation.from_learner(learn)\n\nlosses,idxs = interp.top_losses()\n\nlen(data.valid_ds)==len(losses)==len(idxs)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"interp.plot_confusion_matrix(figsize=(12,12), dpi=60)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Loading APTOS data\n\nNow let's move on to the APTOS competition dataset. Here I am going to open the dataset with pandas, check distribution of labels.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"base_image_dir = os.path.join('..', 'input/aptos2019-blindness-detection/')\ntrain_dir = os.path.join(base_image_dir,'train_images/')\ndf = pd.read_csv(os.path.join(base_image_dir, 'train.csv'))\ndf['path'] = df['id_code'].map(lambda x: os.path.join(train_dir,'{}.png'.format(x)))\ndf = df.drop(columns=['id_code'])\ndf = df.sample(frac=1).reset_index(drop=True) #shuffle dataframe\ndf.head(10)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Note that the distribution of images are slightly different:"},{"metadata":{"trusted":true},"cell_type":"code","source":"df['diagnosis'].hist(figsize = (10, 5))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"bs = 64 #smaller batch size is better for training, but may take longer\nsz=224","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tfms = get_transforms(do_flip=True,flip_vert=True,max_rotate=360,max_warp=0,max_zoom=1.1,max_lighting=0.1,p_lighting=0.5)\nsrc = (ImageList.from_df(df=df,path='./',cols='path') #get dataset from dataset\n        .split_by_rand_pct(0.2) #Splitting the dataset\n        .label_from_df(cols='diagnosis') #obtain labels from the level column\n      )\ndata= (src.transform(tfms,size=sz,resize_method=ResizeMethod.SQUISH,padding_mode='zeros') #Data augmentation\n        .databunch(bs=bs,num_workers=4) #DataBunch\n        .normalize(imagenet_stats) #Normalize     \n       )","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Training\n\nTime to train on our competition dataset:"},{"metadata":{"trusted":true},"cell_type":"code","source":"learn = cnn_learner(data, base_arch=models.resnet50, metrics = [KappaScore(weights='quadratic')],callback_fns=[partial(CSVLogger,append=True)])\nlearn.load('prev-dataset')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.lr_find()\nlearn.recorder.plot(suggestion=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.fit_one_cycle(1,max_lr = 1e-2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.recorder.plot_losses()\nlearn.recorder.plot_metrics()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.unfreeze()\nlearn.lr_find()\nlearn.recorder.plot(suggestion=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.fit_one_cycle(3,max_lr = slice(1e-6,1e-3))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.recorder.plot_losses()\nlearn.recorder.plot_metrics()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"For both datasets, we should train longer, but because of time constraints for submitting, I have limited training time. I will create a two-part kernel with longer training later."},{"metadata":{},"cell_type":"markdown","source":"# Create Submission\n\nNow that we have our model working on the APTOS dataset, we can create a submission and see how this model fares."},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_df = pd.read_csv('../input/aptos2019-blindness-detection/sample_submission.csv')\nsample_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.data.add_test(ImageList.from_df(sample_df,'../input/aptos2019-blindness-detection',folder='test_images',suffix='.png'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"preds,y = learn.get_preds(ds_type=DatasetType.Test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_df.diagnosis = preds.argmax(dim=-1).numpy().astype(int)\nsample_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_df.to_csv('submission.csv',index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}