{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Overview\nThis notebook uses [Darek Kłeczek](https://www.kaggle.com/thedrcat) prototyping [solution](https://www.kaggle.com/c/hpa-single-cell-image-classification/discussion/221550) with [Fastai](https://docs.fast.ai/). I leveraged this strategy along with Darek's [datasets](https://www.kaggle.com/thedrcat/datasets) and created a training loop over multiple folds.  I also provide a small framework to train your models using Kaggle, Colab or local setup; in my case Windows on a 6GB GPU NVIDIA Dell laptop.  \n\nAlso modified originator's logic to enable different folds to be evaluated.  \n\nPlease provide any suggestions or enhancements.  I hope you find this beneficial."},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nfrom fastai.vision.all import *\nimport pickle\nimport os","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from datetime import datetime\nfrom pytz import timezone\ntz = timezone(\"US/Eastern\")\nprint(datetime.now(tz).strftime('%y%m%d-%H:%M:%S:'))\n\npd.set_option('display.max_columns', None)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Versions:  \n> 210402-11:03:12: initial shared non-run version"},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\nfrom pathlib import Path\n\nif 'COLAB_GPU' in os.environ:\n    ENV = 'COLAB'\n    from google.colab import drive\n    drive.mount('/content/drive')\nelif 'KAGGLE_KERNEL_INTEGRATIONS' in os.environ:\n    ENV = 'KAGGLE'\nelif 'GNOME_SHELL_SESSION_MODE' in os.environ:\n    ENV = 'LINUX'\nelif os.environ['OS'] == 'Windows_NT':\n    ENV = 'WIN'\nelse:\n    ENV = 'UNDEFINED'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if ENV == 'KAGGLE': \n    !pip install -q /kaggle/input/iterative-stratification/iterative-stratification-master/\n    # Making pretrained weights work without needing to find the default filename\n    if not os.path.exists('/root/.cache/torch/hub/checkpoints/'):\n            os.makedirs('/root/.cache/torch/hub/checkpoints/')\n    !cp '../input/resnet50/resnet50.pth' '/root/.cache/torch/hub/checkpoints/resnet50-19c8e357.pth'\n    # !cp '../input/resnet34/resnet34.pth' '/root/.cache/torch/hub/checkpoints/resnet34-333f7ec4.pth'\n    \n    train_path = Path('../input/hpa-cell-tiles-sample-balanced-dataset')\n    test_path = Path('../input/hpa-cell-tiles-test-with-enc-dataset')\n    sub_path = Path('../input/hpa-single-cell-image-classification')\n    gen_path = Path('/kaggle/working')\n    model_path = Path('/kaggle/working')\n    \n    bs=256\n    TRAIN_FRAC = .1 #Change frac=1 to run on whole training sample\n    \nelif ENV == 'COLAB':\n    if (torch.__version__ < '1.7.1'):\n        !pip uninstall torch torchvision torchaudio torchtext -y\n        !pip install torch torchvision #torchaudio\n        #!pip install torch==1.8.1+cu111 torchvision==0.9.1+cu111 torchaudio==0.8.1 -f https://download.pytorch.org/whl/torch_stable.html\n\n    if (fastai.__version__ < '2.3.0'):\n        !pip uninstall fastai -y\n        !pip install fastai -q\n\n    if not (os.path.isdir(\"/content/bal_ds\")):\n        print(\"extracting balanced-dataset files\")\n        !unzip /content/drive/MyDrive/data/hpa-cell-tiles-sample-balanced-dataset.zip -d /content/bal_ds\n    \n    if not (os.path.isdir(\"/content/enc_ds\")):\n        print(\"extracting enc-dataset files\")\n        !unzip /content/drive/MyDrive/data/hpa-cell-tiles-test-with-enc-dataset.zip -d /content/enc_ds\n\n    train_path = Path('/content/bal_ds')\n    test_path = Path('/content/enc_ds')\n    sub_path = Path('/content/drive/MyDrive/data')\n    gen_path = Path('/content/drive/MyDrive/data')\n    model_path = Path('/content/drive/MyDrive/data')\n    \n    bs=256 #14min/epoch on 100% train\n    TRAIN_FRAC = .1\n\nelif ENV == 'WIN':\n    train_path = Path('D:/data/hpa-2021/hpa-cell-tiles-sample-balanced-dataset')\n    test_path = Path('D:/data/hpa-2021/hpa-cell-tiles-test-with-enc-dataset')\n    sub_path = Path('D:/data/hpa-2021')\n    gen_path = Path('D:/data/hpa-2021/generated')\n    model_path = Path('D:/data/hpa-2021/generated')\n    \n    bs=256 #40min/epoch at 100% train\n    TRAIN_FRAC = .05 \n    \nelse: ENV = 'UNDEFINED'\n\nfrom fastai.vision.all import *\nimport fastai\nprint(f\"fast.ai verion = {fastai.__version__}\")\nprint(f'Environment is {ENV}')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"path = train_path\ndf = pd.read_csv(path/'cell_df.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"labels = [str(i) for i in range(19)]\nfor x in labels: df[x] = df['image_labels'].apply(lambda r: int(x in r.split('|')))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dfs = df.sample(frac=TRAIN_FRAC, random_state=42)\ndfs = dfs.reset_index(drop=True)\nlen(dfs)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"| ID | Name                      | ID | Name                   | ID | Name                                     |\n|----|:--------------------------|----|:-----------------------|----|:-----------------------------------------|\n| 0  | Nucleoplasm               | 6  | Endoplasmic reticulum  | 12 | Centrosome                               |\n| 1  | Nuclear membrane          | 7  | Golgi apparatus        | 13 | Plasma membrane                          |\n| 2  | Nucleoli                  | 8  | Intermediate filaments | 14 | Mitochondria                             |\n| 3  | Nucleoli fibrillar center | 9  | Actin filaments        | 15 | Aggresome                                |\n| 4  | Nuclear speckles          | 10 | Microtubules           | 16 | Cytosol                                  |\n| 5  | Nuclear bodies            | 11 | Mitotic spindle        | 17 | Vesicles and punctate cytosolic patterns |\n|    |                           |    |                        | 18 | Negative                                 |"},{"metadata":{"trusted":true},"cell_type":"code","source":"unique_counts = {}\nfor lbl in labels:\n    unique_counts[lbl] = len(dfs[dfs.image_labels == lbl])\n\nfull_counts = {}\nfor lbl in labels:\n    count = 0\n    for row_label in dfs['image_labels']:\n        if lbl in row_label.split('|'): count += 1\n    full_counts[lbl] = count\n    \ncounts = list(zip(full_counts.keys(), full_counts.values(), unique_counts.values()))\ncounts = np.array(sorted(counts, key=lambda x:-x[1]))\ncounts = pd.DataFrame(counts, columns=['label', 'full_count', 'unique_count'])\ncounts.set_index('label').T","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"len(dfs)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"nfold = 5\nseed = 42\n\ny = dfs[labels].values\nX = dfs[['image_id', 'cell_id']].values\n\ndfs['fold'] = np.nan\n\n#from iterstrat.ml_stratifiers import MultilabelStratifiedKFold\ntry:\n    from iterstrat.ml_stratifiers import MultilabelStratifiedKFold\nexcept:\n    !pip install git+https://github.com/trent-b/iterative-stratification.git\n    from iterstrat.ml_stratifiers import MultilabelStratifiedKFold\n\n#GETTING ERROR when not setting Shuffle=True below\nmskf = MultilabelStratifiedKFold(n_splits=nfold, random_state=seed, shuffle=True)\n#mskf = MultilabelStratifiedKFold(n_splits=nfold, random_state=seed)\nfor i, (_, test_index) in enumerate(mskf.split(X, y)):\n    dfs.iloc[test_index, -1] = i\n    \ndfs['fold'] = dfs['fold'].astype('int')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_x(r): return path/'cells'/(r['image_id']+'_'+str(r['cell_id'])+'.jpg')\ndef get_y(r): return r['image_labels'].split('|')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_stats = ([0.07237246, 0.04476176, 0.07661699], [0.17179589, 0.10284516, 0.14199627])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"item_tfms = RandomResizedCrop(224, min_scale=0.75, ratio=(1.,1.))\nbatch_tfms = [*aug_transforms(flip_vert=True, size=128, max_warp=0), Normalize.from_stats(*sample_stats)]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import warnings\nwarnings.filterwarnings(\"ignore\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def run_training(df=dfs, fold=0, net=resnet50, lr=3e-2, epochs=2):\n    cbs = [EarlyStoppingCallback(patience=3), SaveModelCallback()]\n    df['is_valid'] = False\n    df['is_valid'][df['fold'] == fold] = True\n    \n    dblock = DataBlock(blocks=(ImageBlock, MultiCategoryBlock(vocab=labels)),\n                splitter=ColSplitter(col='is_valid'),\n                get_x=get_x,\n                get_y=get_y,\n                item_tfms=item_tfms,\n                batch_tfms=batch_tfms\n                )\n    dls = dblock.dataloaders(df, bs=bs)\n    learn = cnn_learner(dls, net, metrics=[accuracy_multi, PrecisionMulti()]).to_fp16()\n    learn.model_dir=model_path\n    _lr = learn.lr_find(show_plot=False, suggestions=True).lr_steep if lr == None else lr\n    print(f'\\nStage1 learning rate used {_lr}')\n    print(f'Training fold: {fold}')\n    learn.fine_tune(epochs,base_lr=_lr,cbs=cbs)\n    learn.save(f'{model_path}/fold_{fold}_{net.__name__}')\n    learn.recorder.plot_loss()\n    return learn, dls, df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for fold in range(5):\n    learn, dls, dfs = run_training(fold=fold, epochs=2)\n    torch.save(dls,f'{gen_path}/dls_{fold}.pkl')\n    dfs.to_pickle(f'{gen_path}/dfs_{fold}')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"path = test_path\ncell_df = pd.read_csv(path/'cell_df.csv')\ntest_dl = learn.dls.test_dl(cell_df)\ntest_dl.show_batch()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"weights = ['fold_0_resnet50','fold_1_resnet50','fold_2_resnet50','fold_3_resnet50','fold_4_resnet50']\nfinal_preds = []\nfor w in weights:\n    learn.load(model_path/w)\n    _preds, _ = learn.tta(dl=test_dl, n=4) #n=4, beta=0.25 (defaults)\n    #print(f'Pred {w}: {_preds}')\n    final_preds.append(_preds)\npreds = torch.mean(torch.stack(final_preds), dim=0)\n#print(f'Mean Preds: {preds}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Will score zero on private test set.\nSee [Darek's solution discussion:](https://www.kaggle.com/c/hpa-single-cell-image-classification/discussion/221550)    \nboth cell tiles and encoding strings for the public test data are created in a separate notebook: https://www.kaggle.com/thedrcat/hpa-cell-tiles-test-with-enc\nJust a reminder this approach will score zero on private, the final solution based on this approach needs to segment the cell tiles and create encodings on private test set."},{"metadata":{},"cell_type":"markdown","source":"# Inference\nSubmission logic from...  \n[Darek Kłeczek's](https://www.kaggle.com/thedrcat) [Notebook](https://www.kaggle.com/thedrcat/fastai-quick-submission-template/notebook)"},{"metadata":{"trusted":true},"cell_type":"code","source":"cell_df['cls'] = ''\nthreshold = 0.0\n\nfor i in range(preds.shape[0]): \n    p = torch.nonzero(preds[i] > threshold).squeeze().numpy().tolist()\n    if type(p) != list: p = [p]\n    if len(p) == 0: cls = [(preds[i].argmax().item(), preds[i].max().item())]\n    else: cls = [(x, preds[i][x].item()) for x in p]\n    cell_df['cls'].loc[i] = cls","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def combine(r):\n    cls = r[0]\n    enc = r[1]\n    classes = [str(c[0]) + ' ' + str(c[1]) + ' ' + enc for c in cls]\n    return ' '.join(classes)\n\ncombine(cell_df[['cls', 'enc']].loc[24]);","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"cell_df['pred'] = cell_df[['cls', 'enc']].apply(combine, axis=1)\ncell_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"subm = cell_df.groupby(['image_id'])['pred'].apply(lambda x: ' '.join(x)).reset_index()\n# subm = subm.loc[3:]\nsubm.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_submission = pd.read_csv(sub_path/'sample_submission.csv')\nsample_submission.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub = pd.merge(\n    sample_submission,\n    subm,\n    how=\"left\",\n    left_on='ID',\n    right_on='image_id',\n)\nsub.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def isNaN(num):\n    return num != num\n\nfor i, row in sub.iterrows():\n    if isNaN(row['pred']): continue\n    sub.PredictionString.loc[i] = row['pred']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub = sub[sample_submission.columns]\nsub.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub.to_csv(gen_path/'submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Where are the mistakes?\nSet ```fold_num``` to evaluate performance over each fold.  "},{"metadata":{"trusted":true},"cell_type":"code","source":"fold_num = 4\nlearn.load(f'{gen_path}/fold_{fold_num}_resnet50')\ndfs = pd.read_pickle(f'{gen_path}/dfs_{fold_num}')\ndls = torch.load(f'{gen_path}/dls_{fold_num}.pkl')\n\npath = train_path","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"val_targ = torch.stack([x[1] for x in learn.dls.valid_ds], dim=0).numpy()\nval_targ.shape\n\n#val_targ = dfs[labels][dfs.is_valid == True].values","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"val_targ.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"val_preds_all = learn.get_preds(dl=learn.dls.valid)\nval_preds = val_preds_all[0].numpy()\nval_preds = val_preds > 0.5\nfull_preds = val_preds_all[0].numpy()\nvis_arr = cm(val_targ, val_preds)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\n\ndef print_confusion_matrix(confusion_matrix, axes, class_label, class_names, fontsize=14):\n\n    df_cm = pd.DataFrame(\n        confusion_matrix, index=class_names, columns=class_names,\n    )\n\n    try:\n        heatmap = sns.heatmap(df_cm, annot=True, fmt=\"d\", cbar=False, ax=axes)\n    except ValueError:\n        raise ValueError(\"Confusion matrix values must be integers.\")\n    heatmap.yaxis.set_ticklabels(heatmap.yaxis.get_ticklabels(), rotation=0, ha='right', fontsize=fontsize)\n    heatmap.xaxis.set_ticklabels(heatmap.xaxis.get_ticklabels(), rotation=45, ha='right', fontsize=fontsize)\n    axes.set_ylabel('True label')\n    axes.set_xlabel('Predicted label')\n    axes.set_title(\"Confusion Matrix for the class - \" + class_label)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(5, 4, figsize=(12, 16))\n    \nfor axes, cfs_matrix, label in zip(ax.flatten(), vis_arr, labels):\n    print_confusion_matrix(cfs_matrix, axes, label, [\"0\", \"1\"])\n\nfig.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"val = dfs[dfs.is_valid==True]\nlen(val[val['16'] == 1])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.metrics import average_precision_score\naverage_precision = average_precision_score(val_targ, val_preds)\naverage_precision","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.metrics import precision_recall_curve\n\nprecision = dict()\nrecall = dict()\naverage_precision = dict()\nfor i in range(19):\n    precision[i], recall[i], _ = precision_recall_curve(val_targ[:, i], val_preds[:, i])\n    average_precision[i] = average_precision_score(val_targ[:, i], val_preds[:, i])\n\n# A \"micro-average\": quantifying score on all classes jointly\nprecision[\"micro\"], recall[\"micro\"], _ = precision_recall_curve(val_targ.ravel(), val_preds.ravel())\naverage_precision[\"micro\"] = average_precision_score(val_targ, val_preds, average=\"micro\")\nprint('Average precision score, micro-averaged over all classes: {0:0.2f}'.format(average_precision[\"micro\"]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"average_precision","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Example outputs..."},{"metadata":{},"cell_type":"markdown","source":"Fold 0:  \n```{0: 0.2167366997440679,\n 1: 0.2812805713716858,\n 10: 0.4868564932757389,\n 11: 0.011422044545973729,\n 12: 0.0713877784123358,\n 13: 0.08623643632210165,\n 14: 0.07024557395773844,\n 15: 0.03712164477441462,\n 16: 0.09251856082238721,\n 17: 0.05653912050256996,\n 18: 0.006282124500285551,\n 2: 0.1880749913690423,\n 3: 0.1505034583412653,\n 4: 0.17727102441633766,\n 5: 0.0759565962307253,\n 6: 0.06420992515555021,\n 7: 0.07310108509423187,\n 8: 0.14442435010681726,\n 9: 0.051970302684180465,\n 'micro': 0.12614287998476054}```"},{"metadata":{},"cell_type":"markdown","source":"Fold 2:  \n```{0: 0.3043858958182949,\n 1: 0.2794941275861746,\n 10: 0.5751811762284811,\n 11: 0.011422044545973729,\n 12: 0.0713877784123358,\n 13: 0.11427077693468515,\n 14: 0.14484892101919306,\n 15: 0.03712164477441462,\n 16: 0.16464123336386804,\n 17: 0.05653912050256996,\n 18: 0.006282124500285551,\n 2: 0.28014791174704406,\n 3: 0.17177427864349623,\n 4: 0.30107426298272905,\n 5: 0.0759565962307253,\n 6: 0.08618572976279001,\n 7: 0.07310108509423187,\n 8: 0.1585756992476962,\n 9: 0.051970302684180465,\n 'micro': 0.15912726712147027}```"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}