{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Summary\n\n* Using only image size and pixel counts, local validation kappa can easily reach 0.70+\n* This does NOT generalise to the test data\n* Simple re-sizing is not enough to combat the issue, and CNN can easily overfit\n* Use pre-processing wisely to overcome this"},{"metadata":{},"cell_type":"markdown","source":" # Libraries"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"# libraries\n\nimport cv2\nimport shap\nimport numpy as np\nimport pandas as pd\nimport xgboost as xgb\nimport seaborn as sns\nfrom matplotlib import pyplot as plt\nfrom tqdm import tqdm_notebook as tqdm\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import cohen_kappa_score, confusion_matrix\n\nsns.set(rc={'figure.figsize':(11.7,8.27)})","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Image Properties\n\nLet's loop through the training and test data and grab some summary statistics about the images:\n\n- Height\n- Width\n- Height to Width Ratio\n- Total Number of Pixels\n- Total Number of Black Pixels\n- Total Proportion of Black Pixels\n- Channel Means"},{"metadata":{"trusted":true},"cell_type":"code","source":"# training features\n\ntrain = pd.read_csv('../input/train.csv')\ntrain_results = []\n\nfor index, row in tqdm(train.iterrows(), total=train.shape[0]):\n    img = cv2.imread('../input//train_images/{}.png'.format(row['id_code']))\n\n    height, width, channels = img.shape\n    ratio = width/height\n    pixel_count = width*height\n    gray_scaled = cv2.cvtColor(img, cv2.COLOR_RGB2GRAY)\n    black_cnt = pixel_count-cv2.countNonZero(gray_scaled)\n    black_pct = black_cnt/pixel_count\n    mean0, mean1, mean2, _ = cv2.mean(img)\n\n    observation = np.array(\n        (row['diagnosis'], height, width, ratio, pixel_count, black_cnt, black_pct, mean0, mean1, mean2))\n\n    train_results.append(observation)\n\ntrain_results_df = pd.DataFrame(train_results)\ntrain_results_df.columns = [\n    'diagnosis','height','width','ratio','pixel_count','black_cnt','black_pct','mean_c0','mean_c1','mean_c2']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# test features\n\ntest = pd.read_csv('../input/test.csv')\ntest_results = []\n\nfor index, row in tqdm(test.iterrows(), total=test.shape[0]):\n    img = cv2.imread('../input//test_images/{}.png'.format(row['id_code']))\n\n    height, width, channels = img.shape\n    ratio = width/height\n    pixel_count = width*height\n    gray_scaled = cv2.cvtColor(img, cv2.COLOR_RGB2GRAY)\n    black_cnt = pixel_count-cv2.countNonZero(gray_scaled)\n    black_pct = black_cnt/pixel_count\n    mean0, mean1, mean2, _ = cv2.mean(img)\n\n    observation = np.array(\n        (np.nan, height, width, ratio, pixel_count, black_cnt, black_pct, mean0, mean1, mean2))\n\n    test_results.append(observation)\n\ntest_results_df = pd.DataFrame(test_results)\ntest_results_df.columns = [\n    'diagnosis','height','width','ratio','pixel_count','black_cnt','black_pct','mean_c0','mean_c1','mean_c2']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"# Model\n\nLet's see what kind of score we get using just meta features of the images. The higher the score, the more worried we need to be about our models picking up information irrelevant for the purpose of diagnosis."},{"metadata":{"trusted":true},"cell_type":"code","source":"params = {\n    'booster': 'gbtree',\n    'objective': 'multi:softprob',\n    'eval_metric': 'mlogloss',\n    'eta': 0.005,\n    'max_depth': 10,\n    'subsample': 1.0,\n    'colsample_bytree': 1.0,\n    'tree_method': 'gpu_hist',\n    'num_class': 5}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X = train_results_df.drop(columns=['diagnosis'])\ny = train_results_df['diagnosis']\n\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=1337)\n\ndtrain = xgb.DMatrix(X_train, label=y_train)\ndvalid = xgb.DMatrix(X_test, label=y_test)\n\nwatchlist = [(dtrain, 'train'), (dvalid, 'valid')]\n\nbst = xgb.train(\n    params=params,\n    dtrain=dtrain,\n    num_boost_round=100000,\n    evals=watchlist,\n    early_stopping_rounds=250,\n    verbose_eval=100\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pred = pd.DataFrame(np.argmax(bst.predict(dvalid), axis=1))\nresults = pd.concat([y_test.reset_index(drop=True), pred], axis=1)\nscore = cohen_kappa_score(results.iloc[:,0], results.iloc[:,1], weights=\"quadratic\")\n\nprint('Validation Kappa:', score)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"cm = confusion_matrix(y_true=results.iloc[:,0], y_pred=results.iloc[:,1])\ncm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n\nfig, ax = plt.subplots()\nsns.heatmap(cm, annot=True)\nax.set(ylabel='True label', xlabel='Predicted label')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Classes 0-2 can be predicted extremely well using just the image meta data. What proportion of the data do they make up?\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"train['diagnosis'].value_counts()/train.shape[0]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So approximately 90% of the data. Uh-oh. This is not looking good. But what features is it exactly? Let's look at the feature importance."},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(12,18))\nxgb.plot_importance(bst, importance_type='gain', height=0.8, ax=ax)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It seems the most important feature is the image size, specifically the ratio of height to width. Now, when running our models we often resize to e.g. 224x224, so is this a problem? Let's start by looking at the ratios present in the training data."},{"metadata":{"trusted":true},"cell_type":"code","source":"train_results_df.groupby(['ratio', 'diagnosis'])['pixel_count'].count()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"You can clearly see the effect here. Square images are nearly always class zero, while some of the rectangular images have far higher numbers of higher classes (look at 1.505618!). \n\nBut does this matter if we resize? Yes, it will. To labour the point, let's take an image from r={1.000000, 1.333333, 1.505618} and re-size."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"\nfig, ax = plt.subplots(nrows=3, ncols=3, figsize=(20,10))\n\nimg0a = cv2.imread(\n    '../input/train_images/{}.png'.format(train.iloc[train_results_df[np.round(train_results_df['ratio'], 6)==1.000000].index[0], 0]))\n\nax[0,0].imshow(cv2.cvtColor(img0a, cv2.COLOR_BGR2RGB))\nax[0,0].axis('off')\n\nimg0b = cv2.imread(\n    '../input/train_images/{}.png'.format(train.iloc[train_results_df[np.round(train_results_df['ratio'], 6)==1.000000].index[1], 0]))\n\nax[1,0].imshow(cv2.cvtColor(img0b, cv2.COLOR_BGR2RGB))\nax[1,0].axis('off')\n\nimg0c = cv2.imread(\n    '../input/train_images/{}.png'.format(train.iloc[train_results_df[np.round(train_results_df['ratio'], 6)==1.000000].index[2], 0]))\n\nax[2,0].imshow(cv2.cvtColor(img0c, cv2.COLOR_BGR2RGB))\nax[2,0].axis('off')\n\nimg1a = cv2.imread(\n    '../input/train_images/{}.png'.format(train.iloc[train_results_df[np.round(train_results_df['ratio'], 6)==1.333333].index[0], 0]))\n\nax[0,1].imshow(cv2.cvtColor(img1a, cv2.COLOR_BGR2RGB))\nax[0,1].axis('off')\n\nimg1b = cv2.imread(\n    '../input/train_images/{}.png'.format(train.iloc[train_results_df[np.round(train_results_df['ratio'], 6)==1.333333].index[1], 0]))\n\nax[1,1].imshow(cv2.cvtColor(img1b, cv2.COLOR_BGR2RGB))\nax[1,1].axis('off')\n\nimg1c = cv2.imread(\n    '../input/train_images/{}.png'.format(train.iloc[train_results_df[np.round(train_results_df['ratio'], 6)==1.333333].index[2], 0]))\n\nax[2,1].imshow(cv2.cvtColor(img1c, cv2.COLOR_BGR2RGB))\nax[2,1].axis('off')\n\nimg2a = cv2.imread(\n    '../input/train_images/{}.png'.format(train.iloc[train_results_df[np.round(train_results_df['ratio'], 6)==1.505618].index[0], 0]))\n\nax[0,2].imshow(cv2.cvtColor(img2a, cv2.COLOR_BGR2RGB))\nax[0,2].axis('off')\n\nimg2b = cv2.imread(\n    '../input/train_images/{}.png'.format(train.iloc[train_results_df[np.round(train_results_df['ratio'], 6)==1.505618].index[1], 0]))\n\nax[1,2].imshow(cv2.cvtColor(img2b, cv2.COLOR_BGR2RGB))\nax[1,2].axis('off')\n\nimg2c = cv2.imread(\n    '../input/train_images/{}.png'.format(train.iloc[train_results_df[np.round(train_results_df['ratio'], 6)==1.505618].index[2], 0]))\n\nax[2,2].imshow(cv2.cvtColor(img2c, cv2.COLOR_BGR2RGB))\nax[2,2].axis('off')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now let's resize to 224x224"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"\nfig, ax = plt.subplots(nrows=3, ncols=3, figsize=(20,10))\n\nimg0a = cv2.imread(\n    '../input/train_images/{}.png'.format(train.iloc[train_results_df[np.round(train_results_df['ratio'], 6)==1.000000].index[0], 0]))\nimg0a = cv2.resize(img0a, (224,224))\nax[0,0].imshow(cv2.cvtColor(img0a, cv2.COLOR_BGR2RGB))\nax[0,0].axis('off')\n\nimg0b = cv2.imread(\n    '../input/train_images/{}.png'.format(train.iloc[train_results_df[np.round(train_results_df['ratio'], 6)==1.000000].index[1], 0]))\nimg0b = cv2.resize(img0b, (224,224))\nax[1,0].imshow(cv2.cvtColor(img0b, cv2.COLOR_BGR2RGB))\nax[1,0].axis('off')\n\nimg0c = cv2.imread(\n    '../input/train_images/{}.png'.format(train.iloc[train_results_df[np.round(train_results_df['ratio'], 6)==1.000000].index[2], 0]))\nimg0c = cv2.resize(img0c, (224,224))\nax[2,0].imshow(cv2.cvtColor(img0c, cv2.COLOR_BGR2RGB))\nax[2,0].axis('off')\n\nimg1a = cv2.imread(\n    '../input/train_images/{}.png'.format(train.iloc[train_results_df[np.round(train_results_df['ratio'], 6)==1.333333].index[0], 0]))\nimg1a = cv2.resize(img1a, (224,224))\nax[0,1].imshow(cv2.cvtColor(img1a, cv2.COLOR_BGR2RGB))\nax[0,1].axis('off')\n\nimg1b = cv2.imread(\n    '../input/train_images/{}.png'.format(train.iloc[train_results_df[np.round(train_results_df['ratio'], 6)==1.333333].index[1], 0]))\nimg1b = cv2.resize(img1b, (224,224))\nax[1,1].imshow(cv2.cvtColor(img1b, cv2.COLOR_BGR2RGB))\nax[1,1].axis('off')\n\nimg1c = cv2.imread(\n    '../input/train_images/{}.png'.format(train.iloc[train_results_df[np.round(train_results_df['ratio'], 6)==1.333333].index[2], 0]))\nimg1c = cv2.resize(img1c, (224,224))\nax[2,1].imshow(cv2.cvtColor(img1c, cv2.COLOR_BGR2RGB))\nax[2,1].axis('off')\n\nimg2a = cv2.imread(\n    '../input/train_images/{}.png'.format(train.iloc[train_results_df[np.round(train_results_df['ratio'], 6)==1.505618].index[0], 0]))\nimg2a = cv2.resize(img2a, (224,224))\nax[0,2].imshow(cv2.cvtColor(img2a, cv2.COLOR_BGR2RGB))\nax[0,2].axis('off')\n\nimg2b = cv2.imread(\n    '../input/train_images/{}.png'.format(train.iloc[train_results_df[np.round(train_results_df['ratio'], 6)==1.505618].index[1], 0]))\nimg2b = cv2.resize(img2b, (224,224))\nax[1,2].imshow(cv2.cvtColor(img2b, cv2.COLOR_BGR2RGB))\nax[1,2].axis('off')\n\nimg2c = cv2.imread(\n    '../input/train_images/{}.png'.format(train.iloc[train_results_df[np.round(train_results_df['ratio'], 6)==1.505618].index[2], 0]))\nimg2c = cv2.resize(img2c, (224,224))\nax[2,2].imshow(cv2.cvtColor(img2c, cv2.COLOR_BGR2RGB))\nax[2,2].axis('off')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"If this was the only pre-processing we did, it would not be hard for a deep CNN to learn to differentiate classes based on the eyeball edges and the space surrounding them.\n\nFor an example of where the model can go wrong in this way have a look at the excellent kernel by [dimitreoliveira](https://www.kaggle.com/dimitreoliveira) here:\n\nhttps://www.kaggle.com/dimitreoliveira/diabetic-retinopathy-shap-model-explainability\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1182060%2F4cc45db867d9ec1b983c353c7f23b633%2Fshap.PNG?generation=1563462787163185&alt=media)"},{"metadata":{},"cell_type":"markdown","source":"We clearly see here that the model tends in part to focus on what is going on around the edge of the eyeball, and who can blame it? It's a great way to make predictions in the training data.\nAn important thing to note here is that this weird relationship between meta-features and target does **NOT** extend to the test data. Let's generate predictions to demonstrate this:"},{"metadata":{"trusted":true},"cell_type":"code","source":"dtest = xgb.DMatrix(test_results_df.drop(columns=['diagnosis']))\ntest_pred = pd.DataFrame(np.argmax(bst.predict(dtest), axis=1))\nsubmission = pd.concat([test['id_code'], test_pred], axis=1)\nsubmission.columns = ['id_code','diagnosis']\nsubmission.to_csv('submission.csv', index=False)\nsubmission.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This will almost certainly get a negative score, despite the local kappa estimates.  I think is why we've seen quite a few posts where people have local cv of 0.80+ but sumission gives terrible results.\n\nWhile public test set is different, we don't know whether or not it extends to the hidden test dataset. I really hope not since obviously the aim is to help with diagnosis, and it would be a shame if the winning model ends up being something entirely overfit to the meta-data of the images."}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}