{"cells":[{"metadata":{},"cell_type":"markdown","source":"**Training code has been put on [github](https://github.com/mikelkl/APTOS2019)**\n\n**For detailed summary, please refer to [this blog](https://zhuanlan.zhihu.com/p/81695773)**\n\n# General\nThis is a not bad solution to get top2% without TTA or coefficient optimization.\n\n# Our Solution\n## Data Augumentation\n-  Introduce [2015 Diabetic Retinopathy competition data](https://www.kaggle.com/tanlikesmath/diabetic-retinopathy-resized)\n- Conduct regular transformations that create less black padding\n  - do_flip\n  - flip_vert\n  - max_zoom\n  \n## Preprocessing\n- Thanks to the [@Neuron Engineer](https://www.kaggle.com/ratthachat), we refer to his [APTOS [UpdatedV14] Preprocessing- Ben's & Cropping](https://www.kaggle.com/ratthachat/aptos-updatedv14-preprocessing-ben-s-cropping), and set `sigmaX=10`\n\n## Pretrained Model\n- We choose [EfficientNet-PyTorch](https://github.com/lukemelas/EfficientNet-PyTorch) as our base model, this series model are quite accurate and fast to train.\n\n## Training\n- Because this is a ordinal classification task, we train it as regression problem.\n- We first pretrain model on 2015 data, then finetune on 2019 data\n\n## Ensemble\n### Stage 1\n- Train `efficientnet-b3, efficientnet-b4, efficientnet-b5` models on splitted 5-fold data resulting in 15 base models.\n\n### Stage 2\n- Train [xgboost](https://github.com/dmlc/xgboost), [svr](https://scikit-learn.org/stable/modules/generated/sklearn.svm.SVR.html), [catboost](https://github.com/catboost/catboost) models on logits output of stage 1 base model.\n\n### Stage 3\n- Bagging from stage 2 models"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"%reload_ext autoreload\n%autoreload 2\n%matplotlib inline\nimport pandas as pd\n\nimport os\nimport cv2\nimport torch\nimport numpy as np\n\nimport sys\npackage_dir = '../input/efficientnet/efficientnet-pytorch/EfficientNet-PyTorch/'\nsys.path.insert(0, package_dir)\n\nimport efficientnet_pytorch \n\nfrom tqdm import tqdm_notebook\nfrom sklearn.metrics import cohen_kappa_score\nfrom fastai.vision import *\nfrom torch.nn import functional as F\nimport time\nimport pickle as pk\n\n#time clock counting\nstart = time.time()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import random\n\ndef seed_everything(seed):\n    random.seed(seed)\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    np.random.seed(seed)\n    torch.manual_seed(seed)\n    torch.cuda.manual_seed(seed)\n    torch.backends.cudnn.deterministic = True\nSEED = 999\nseed_everything(SEED)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"deployment_dir = \"../input/efficientnetstacking\"\n\ndef qk(y_pred, y):\n    k = torch.tensor(cohen_kappa_score(torch.round(y_pred), y, weights='quadratic'), device='cuda:0')\n    k[k != k] = 0\n    k[torch.isinf(k)] = 0\n    \n    return k\n\ntest_df = pd.read_csv('../input/aptos2019-blindness-detection/sample_submission.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Preprocess Image and save on disk"},{"metadata":{"trusted":true},"cell_type":"code","source":"def crop_image_from_gray(img,tol=7):\n    if img.ndim ==2:\n        mask = img>tol\n        return img[np.ix_(mask.any(1),mask.any(0))]\n    elif img.ndim==3:\n        gray_img = cv2.cvtColor(img, cv2.COLOR_RGB2GRAY)\n        mask = gray_img>tol\n        \n        check_shape = img[:,:,0][np.ix_(mask.any(1),mask.any(0))].shape[0]\n        if (check_shape == 0): # image is too dark so that we crop out everything,\n            return img # return original image\n        else:\n            img1=img[:,:,0][np.ix_(mask.any(1),mask.any(0))]\n            img2=img[:,:,1][np.ix_(mask.any(1),mask.any(0))]\n            img3=img[:,:,2][np.ix_(mask.any(1),mask.any(0))]\n    #         print(img1.shape,img2.shape,img3.shape)\n            img = np.stack([img1,img2,img3],axis=-1)\n    #         print(img.shape)\n        return img\n    \nIMG_SIZE = 512\n\ndef load_ben_color(path, sigmaX=10):\n    image = cv2.imread(path)\n    image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n    image = crop_image_from_gray(image)\n    image = cv2.resize(image, (IMG_SIZE, IMG_SIZE))\n    image=cv2.addWeighted ( image,4, cv2.GaussianBlur( image , (0,0) , sigmaX) ,-4 ,128)\n        \n    return image","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from tqdm import tqdm_notebook\nfrom PIL import Image\nimport multiprocessing as mp\nsave_dir = \"../input/test_images_ben_preprocessing_sigmaX10\"\nif not os.path.exists(save_dir):\n    os.mkdir(save_dir)\nNCORE = 20","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sv_t1 = time.time()\ndef process(q, iolock):\n    while True:\n        stuff = q.get()\n        if stuff is None:\n            break\n        idx, row = stuff\n        path=f\"../input/aptos2019-blindness-detection/test_images/{row['id_code']}.png\"\n        image = load_ben_color(path,sigmaX=10)\n        Image.fromarray(image).save(os.path.join(save_dir, \"{}.png\".format(row['id_code'])))\n\nq = mp.Queue(maxsize=NCORE)\niolock = mp.Lock()\npool = mp.Pool(NCORE, initializer=process, initargs=(q, iolock))\nfor idx, row in tqdm_notebook(test_df.iterrows()):\n    stuff = (idx, row)\n    q.put(stuff)  # blocks until q below its max size\nfor _ in range(NCORE):  # tell workers we're done\n    q.put(None)\npool.close()\npool.join()  \nsv_t2 = time.time()\nsv_dur = sv_t2 - sv_t1\nprint('Preprocessing and save takes time ... {} seconds , {} hours'.format(sv_dur,sv_dur/3600))\n##Preprocessing and save takes time ... 349.69464921951294 seconds , 0.09713740256097582 hours","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!du -sh ../input/test_images_ben_preprocessing_sigmaX10","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Stage1&2 Inference"},{"metadata":{"trusted":true},"cell_type":"code","source":"inf1_t1 = time.time()\n\nb3_models = [\"efficientnet-b3_0901_16-45-51_stage2_f1\", \"efficientnet-b3_0901_16-45-51_stage2_f2\",\n                  \"efficientnet-b3_0901_16-45-51_stage2_f3\", \"efficientnet-b3_0901_16-45-51_stage2_f4\",\n                  \"efficientnet-b3_0901_16-45-51_stage2_f5\"]\n\nb3_test_logits_list = []\nfor m in b3_models:\n    learn = load_learner(deployment_dir, \"{}.pkl\".format(m))\n\n    learn.data.add_test(ImageList.from_df(test_df,\n                                      '../input',\n                                      folder='test_images_ben_preprocessing_sigmaX10',\n                                      suffix='.png'))\n\n    logits,_ = learn.get_preds(DatasetType.Test)\n    logits = logits.numpy()\n    b3_test_logits_list.append(logits)\n    \n    print(logits.shape)\n    \nb4_models = [\"efficientnet-b4_0820_01-09-57_stage2_f1\", \"efficientnet-b4_0820_01-09-57_stage2_f2\",\n                  \"efficientnet-b4_0820_01-09-57_stage2_f3\", \"efficientnet-b4_0820_01-09-57_stage2_f4\",\n                  \"efficientnet-b4_0821_00-02-25_stage2_f5\"]\n\nb4_test_logits_list = []\nfor m in b4_models:\n    learn = load_learner(deployment_dir, \"{}.pkl\".format(m))\n\n    learn.data.add_test(ImageList.from_df(test_df,\n                                      '../input',\n                                      folder='test_images_ben_preprocessing_sigmaX10',\n                                      suffix='.png'))\n\n    logits,_ = learn.get_preds(DatasetType.Test)\n    logits = logits.numpy()\n    b4_test_logits_list.append(logits)\n    \n    print(logits.shape)\n    \nb5_models = [\"efficientnet-b5_0820_01-32-30_stage2_f1\", \"efficientnet-b5_0903_01-03-41_stage2_f2\",\n                  \"efficientnet-b5_0820_22-13-07_stage2_f3\", \"efficientnet-b5_0821_01-30-37_stage2_f4\",\n                  \"efficientnet-b5_0821_00-26-51_stage2_f5\"]\n\nb5_test_logits_list = []\nfor m in b5_models:\n    learn = load_learner(deployment_dir, \"{}.pkl\".format(m))\n\n    learn.data.add_test(ImageList.from_df(test_df,\n                                      '../input',\n                                      folder='test_images_ben_preprocessing_sigmaX10',\n                                      suffix='.png'))\n    logits,_ = learn.get_preds(DatasetType.Test)\n    logits = logits.numpy()\n    b5_test_logits_list.append(logits)\n    \n    print(logits.shape)\n    \nwith open(os.path.join(deployment_dir, \"xgboost-0903_05-26-03.pkl\"), \"rb\") as f:\n    xlf = pk.load(f)\n    \nwith open(os.path.join(deployment_dir, \"svr-0903_05-26-03.pkl\"), \"rb\") as f:\n    svr = pk.load(f)\n\nwith open(os.path.join(deployment_dir, \"cb-0906_06-09-42.pkl\"), \"rb\") as f:\n    cb = pk.load(f)\n    \n# 5 test feature then avg\nresults_xlf = []\nresults_svr = []\nresults_cb = []\nfor b3, b4, b5 in zip(b3_test_logits_list, b4_test_logits_list, b5_test_logits_list):\n    X_test = np.concatenate([b3, b4, b5], axis=1)\n    res_xlf = xlf.predict(X_test)\n    results_xlf.append(res_xlf)\n    \n    res_svr = svr.predict(X_test)\n    results_svr.append(res_svr)\n    \n    res_cb = cb.predict(X_test)\n    results_cb.append(res_cb)\n\navg_res_xlf = np.average(results_xlf, axis=0)\navg_res_svr = np.average(results_svr, axis=0)\navg_res_cb = np.average(results_cb, axis=0)\n\ny_pred_xlf = np.round(avg_res_xlf).astype(int)\ny_pred_svr = np.round(avg_res_svr).astype(int)\ny_pred_cb = np.round(avg_res_cb).astype(int)\n\ntest_df.diagnosis = y_pred_xlf\ntest_df.hist()\nplt.show()\n\ntest_df.diagnosis = y_pred_svr\ntest_df.hist()\nplt.show()\n\ntest_df.diagnosis = y_pred_cb\ntest_df.hist()\nplt.show()\n\ninf1_t2 = time.time()\ninf1_dur = inf1_t2 - inf1_t1\nprint('Inference duration ',inf1_dur, inf1_dur/3600)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Stage3 Bagging"},{"metadata":{},"cell_type":"markdown","source":"## Correlation Analysis"},{"metadata":{"trusted":true},"cell_type":"code","source":"np.corrcoef([avg_res_xlf, avg_res_svr, avg_res_cb])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"np.corrcoef([y_pred_xlf, y_pred_svr, y_pred_cb])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def voting(labels, Tweights = None):\n    if isinstance(labels, list):\n        X = np.array(labels, dtype=np.int64)   \n        maj = np.argmax(np.bincount(X, Tweights))\n    return maj\n\ndef average(logits, Tweights = None):\n    avg = np.average(logits, axis=0, weights = Tweights)\n    return avg","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"vote_list = [voting([int(avg_res_xlf[idx] + 0.5), \n                    int(avg_res_svr[idx] + 0.5), \n                    int(avg_res_cb[idx] + 0.5)]) \n            for idx in range(len(avg_res_xlf))]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Submission"},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df.diagnosis = vote_list\ntest_df.hist()\nplt.show()\n\ntest_df.to_csv('submission.csv',index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df['diagnosis'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dur = time.time() -start\nprint('Whole procedure takes {} seconds long ...'.format(dur))\nprint(dur/3600)\nprint ('done')","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}