{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Recursion Cellular Image Classification\n### CellSignal: Disentangling biological signal from experimental noise in cellular images\n\n> **Work done by**: Nwachukwu Anthony  \n> **Email**: nwachukwuanthony2015@gmail.com  \n> **Inspired by**: *Fastai online courses on Deep Learning*  \n> **Data from kaggle** competition, link below\n\nThe cost of some drugs and medical treatments has risen so high in recent years that many patients are having to go without. You can help with a classification project that could make researchers more efficient.\n\nOne of the more surprising reasons behind the cost is how long it takes to bring new treatments to market. Despite improvements in technology and science, research and development continues to lag. In fact, finding new treatments takes, on average, more than 10 years and costs hundreds of millions of dollars.\n\nRecursion Pharmaceuticals, creators of the industry’s largest dataset of biological images, generated entirely in-house, believes AI has the potential to dramatically improve and expedite the drug discovery process. More specifically, your efforts could help them understand how drugs interact with human cells.\n\nThis will have you disentangling experimental noise from real biological signals. Your entry will classify images of cells under one of 1,108 different genetic perturbations. You can help eliminate the noise introduced by technical execution and environmental variation between experiments.\n\nIf successful, you could dramatically improve the industry’s ability to model cellular images according to their relevant biology. In turn, applying AI could greatly decrease the cost of treatments, and ensure these treatments get to patients faster.\n\n\nYou will find the dataset on this website: https://www.kaggle.com/c/recursion-cellular-image-classification/data"},{"metadata":{},"cell_type":"markdown","source":"### Import Libraries"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"%reload_ext autoreload\n%autoreload 2\n%matplotlib inline\nimport numpy as np \nimport pandas as pd\nfrom fastai.metrics import accuracy\nfrom fastai.vision import *\nimport os\nprint(os.listdir(\"../input/recursion-cellular-image-classification\"))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Set the paths and Prepare the data"},{"metadata":{"trusted":true},"cell_type":"code","source":"path = '../input/recursion-cellular-image-classification'\ndftrain = pd.read_csv(path+'/train.csv')\ndftrain = dftrain[['id_code','sirna']]\n\ni = 0\ndic = {}\ndictest = {}\nfor fold1 in os.listdir(path+'/train'):\n    for fold2 in os.listdir(path+'/train/'+fold1):\n        for image in os.listdir(path+'/train/'+fold1+'/'+fold2):\n            dic[str(fold1)+'_'+fold2[5:]+'_'+image[0:3]] = str(fold1)+'/'+fold2+'/'+image\ndf = pd.DataFrame(list(dic.items()), columns=['id_code','Item'])\ndftraindf = pd.merge(df, dftrain)\ntrainData = dftraindf[['Item','sirna']]\n#df = df.astype({\"a\": int, \"b\": complex})\n#trainData = trainData.astype({'sirna': str})\ntrainData.to_csv(r'../working/trainData.csv', index = None, header=True);\n#print(pd. read_csv ('../working/trainData.csv'))\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"path = '../input/recursion-cellular-image-classification'\ndftest = pd.read_csv(path+'/test.csv')\ndftest = dftest['id_code']\ni = 0\ndictest = {}\nfor fold1 in os.listdir(path+'/test'):\n    for fold2 in os.listdir(path+'/test/'+fold1):\n        for image in os.listdir(path+'/test/'+fold1+'/'+fold2):\n            dictest[str(fold1)+'_'+fold2[5:]+'_'+image[0:3]] = str(fold1)+'/'+fold2+'/'+image\ndf = pd.DataFrame(list(dictest.items()), columns=['id_code','foldPath'])\n\"\"\"dftraindf = pd.merge(df, dftest)\ntestData = dftraindf[['Item']]\"\"\"\ndf.to_csv(r'../working/testData.csv', index = None, header=True);\n#print(pd. read_csv ('../working/testData.csv'))\n#print(df.head())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tfms = get_transforms()\n#df = pd.read_csv(path/'labels.csv', header='infer')\n#path = Path(path)\n# Set the parameters and create the data for the model\nnp.random.seed(42) #makes sure you get same results each time you run the code\nsrc = (ImageList.from_csv('../', 'working/trainData.csv', folder='input/recursion-cellular-image-classification/train')\n       .split_by_rand_pct(0.2)\n       .label_from_df(label_delim=' '))\ntfms = get_transforms()\ndata = (src.transform(tfms, size=128)\n        .databunch().normalize(imagenet_stats))\n\n\n\n#data = ImageDataBunch.from_df('../','working/trainData.csv',folder=path+'/train', ds_tfms=tfms, size=128)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Since kaggle doesn't allow write on the iput directory, we create a new directory outside it where\n# we can freely work and make it the path\n#path = Path(\"../working\")\n#path","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"### Visualize the Data"},{"metadata":{"trusted":true},"cell_type":"code","source":"#img = open_image(path+'/train/HEPG2-04/Plate1/O23_s2_w4.png')\n#img","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#trainData\n#trainData.loc[trainData['sirna'] == 810]\n#len(data.classes)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#print(data.classes)\nprint((len(data.train_ds),len(data.valid_ds)))\ndata.show_batch(rows=3, figsize=(7,8)) #View portion of dataset","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Tain"},{"metadata":{"trusted":true},"cell_type":"code","source":"def accuracy(input:Tensor, targs:Tensor)->Rank0Tensor:\n    \"Computes accuracy with `targs` when `input` is bs * n_classes.\"\n    n = targs.shape[0]\n    input = input.argmax(dim=-1).view(n,-1)\n    targs = targs.view(n,-1)\n    return (input==targs.long()).float().mean()\n\n\"\"\"From https://www.kaggle.com/leighplt/densenet121-pytorch\"\"\"\ndef accuracy1(output, target, topk=(1,)):\n    \"\"\"Computes the accuracy over the k top predictions for the specified values of k\"\"\"\n    with torch.no_grad():\n        maxk = max(topk)\n        batch_size = target.size(0)\n\n        _, pred = output.topk(maxk, 1, True, True)\n        pred = pred.t()\n        correct = pred.eq(target.view(1, -1).expand_as(pred))\n\n        res = []\n        for k in topk:\n            correct_k = correct[:k].view(-1).float().sum(0, keepdim=True)\n            res.append(correct_k.mul_(100.0 / batch_size).item())\n        return torch.Tensor(np.array(res))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Set the metrics. Use F-score\n#acc_02 = partial(accuracy_thresh, thresh=0.2)\n#f_score = partial(fbeta, thresh=0.2)\n#Use CNN (Convolutional Neural Network) and pretrained model (resnet50)  to train\n#learn = cnn_learner(data, models.resnet50, metrics=[acc_02,f_score])\nlearn = cnn_learner(data, models.resnet50, metrics=[accuracy])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Find and plot learning rate\nlearn.lr_find()\nlearn.recorder.plot()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#set learning rate\nlr = 0.05","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Fit the model\nlearn.fit_one_cycle(5,slice(lr))#5","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Save it\nlearn.save('stage-1-rn50')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"####learn.load('stage-1-rn50');","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### More training"},{"metadata":{"trusted":true},"cell_type":"code","source":"\n# Unfreeze the model, that is, traing afresh without the pretrained model\nlearn.unfreeze()\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Find and plot the learning rate\nlearn.lr_find()\nlearn.recorder.plot()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Fit the model\nlearn.fit_one_cycle(5, slice(1e-5, 1e-4))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Save this latest trained model\nlearn.save('stage-2-rn50')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Create a new dataset with batch size = 256\ndata = (src.transform(tfms, size=256)\n        .databunch().normalize(imagenet_stats))\n# Set the learners data as data\nlearn.data = data\ndata.train_ds[0][0].shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Freeze and find learning rate\nlearn.freeze()\nlearn.lr_find()\nlearn.recorder.plot()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Fit and save the model\nlr=1e-2/2\nlearn.fit_one_cycle(5, slice(lr))\nlearn.save('stage-1-256-rn50')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Freeze and find learning rate\nlearn.unfreeze()\nlearn.lr_find()\nlearn.recorder.plot()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Fit and save the model\nlearn.fit_one_cycle(10, slice(2e-5, 4e-5))\nlearn.recorder.plot_losses()\nlearn.save('stage-2-256-rn50')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.unfreeze()\nlearn.fit_one_cycle(10, slice(1e-5, lr/5))\nlearn.save('stage-3-256-rn50')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Export the Model"},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.export()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"### Test the Model"},{"metadata":{"trusted":true},"cell_type":"code","source":"test = ImageList.from_csv('../', 'working/testData.csv', cols='foldPath', folder='input/recursion-cellular-image-classification/test')\nlearn = load_learner('../', test=test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Find the prediction\npreds,_ = learn.get_preds(ds_type=DatasetType.Test)\nlabelled_preds = [learn.data.classes[(pred).tolist().index(max((pred).tolist()))] for pred in preds]\n#Althernatively, you can replace line two with these two lines of code below\n#labels = np.argmax(preds, 1)\n#labelled_preds = [data.classes[int(x)] for x in labels]\n#print(labelled_preds)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"lsttest = []\nfor item in learn.data.test_ds.items:\n    lst = item.split('/')[-3:]\n    lsttest.append(str(lst[0])+'_'+lst[1][5:]+'_'+lst[-1].split('_')[0])\ndf = pd.DataFrame(lsttest, columns=['id_code'])\n#print(df.head())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"path = '../input/recursion-cellular-image-classification'\ndftestcsv = pd.read_csv(path+'/test.csv')\n\ntes = OrderedDict([('id_code',lsttest), ('sirna', labelled_preds)] )\ndf = pd.DataFrame.from_dict(tes)\n\ndftestcsv = pd.DataFrame(list(dftestcsv['id_code']), columns=['id_code'])\ndftestdfcsv = pd.merge(dftestcsv, df)\ndftestdfcsv.to_csv('../working/submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dftestdfcsv.tail()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"Thank you"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}