{"cells":[{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","collapsed":true,"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":false},"cell_type":"markdown","source":"## Pseudo Labeling by Predicted ROC Cutoff\n\nPseudo labeling in this post is basically the same as suggested below reference links,\n\n1. Predict target in test with built model and training data.\n2. Add confident predicted test observations to training data. \n3. Predict target with new built model and the training data added confident predicted test observations.\n\nbut different method is used to determine confident predicted test observation. \n\nIn this [post](https://www.kaggle.com/cdeotte/pseudo-labeling-qda-0-969), pseudo labeled test observations are determined by predicted probability, $Pr(y=0|x) > 0.99$ and $Pr(y=1|x) > 0.99$. And, in this [post](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/168383), values are labeled by intersection from ensemble of predicted values with top and bottom N, since the metric for this competition is AUC score, which is based on rank of the target. \n\n\n\nThis kernel will demonstrate pseudo labeling by predicted optimal ROC thresholds.\nThis is from an idea that the optimal ROC cutoff must be somewhere around the average of predicted target value, then the optimal ROC cutoff for test can be predicted with the optimal ROC cutoff for train and the fitted values for train and test. \nThe process is as follows,\n\n1. Predict target in train and test with built model and training data.  \n2. Find the optimal ROC cutoff using predicted target in train and real target values for each models. \n3. Predict ROC cutoff for the predicted test target with optimal ROC cutoff and predicted target in train for each models. \n4. Determine confident predicted test observations. \n\n\nIn this post, the predicted target values are from total 14 models, which are EfficientNets, and the models and data are borrowed from this [post by Chris](https://www.kaggle.com/cdeotte/triple-stratified-kfold-with-tfrecords), but different augmentations and image size. Their CV and Public LB scores are vary around 0.90~0.92\n\nIt was just experimental attempt, nevertheless I share this because I found some pattern in doing this and eventually hope this is helpful.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Reference\n\nAUC score Reference link:\n[Mann–Whitney U test](https://en.wikipedia.org/wiki/Mann%E2%80%93Whitney_U_test)\n\nModels + others(Triple Stratified Folds, etc) Reference links:\n\n[Triple Stratified KFold with TFRecords by Chris Deotte](https://www.kaggle.com/cdeotte/triple-stratified-kfold-with-tfrecords)\n\nPseudo Labeling Reference links: \n\n[Pseudo Labeling by Chris Deotte](https://www.kaggle.com/cdeotte/pseudo-labeling-qda-0-969)\n\n[Pseudo Labeling by Saksham Gupta](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/168383)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport os\nimport time\nfrom tqdm import tqdm\nfrom sklearn.metrics import roc_auc_score, roc_curve\nfrom scipy import stats\n\nimport matplotlib.pyplot as plt\nimport seaborn as sns","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"These are predicted by efficientnets with different settings, such as crops and image size, and the models are borrowed from this [post](https://www.kaggle.com/cdeotte/triple-stratified-kfold-with-tfrecords)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train = pd.read_csv('../input/siim-isic-melanoma-classification/train.csv')\ntest = pd.read_csv('../input/siim-isic-melanoma-classification/test.csv')\ndpl = pd.read_csv('../input/siim-duplicates/siim_dpl2020.csv') \n#duplicate IDs including 9 more images as Chris mentioned here, https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/161943\n\ntrain['is_train'] = 0\ntest['is_train'] = 1\n\n#removing duplicates in train\ntrain = train[~train['image_name'].isin(dpl['ISIC_id_paired'])]\n\ntrain = train[['image_name', 'is_train', 'target']]\ntest = test[['image_name', 'is_train']]\n\npath = '../input/models'\n\n#predicted values for train and test, total 14 models\nfiles_tr = [x for x in os.listdir(path) if x.endswith('_tr.csv')]\nfiles_te = [x for x in os.listdir(path) if x.endswith('_te.csv')]\n\ntr_models = {}\nte_models = {}\n\nfor i in files_tr:\n    tr_models[i] = pd.read_csv(path + '/' + i)\n    tr_models[i]['predicted_' + i.replace('_tr.csv', '')] = tr_models[i]['pred']\n    tr_models[i] = tr_models[i][['image_name', 'predicted_' + i.replace('_tr.csv', '')]]\n    train = train.merge(tr_models[i], on = 'image_name', how = 'left')\n\nfor i in files_te:\n    te_models[i] = pd.read_csv(path + '/' + i)\n    te_models[i]['predicted_' + i.replace('_te.csv', '')] = te_models[i]['target']\n    te_models[i] = te_models[i][['image_name','predicted_' + i.replace('_te.csv', '')]]\n    \n    test = test.merge(te_models[i], on = 'image_name', how = 'left')\n    \ntest['target'] = 0\n\ncombine = pd.concat([train,test])\n\nprint(train.shape, test.shape)\n\ntrain.columns","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.iloc[:,:5].head(5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test.iloc[:, :5].head(5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"predicted_val = ['predicted_model' + str(x) for x in range(1,15)]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Different distribution, Train vs Test\n\nPredicted target values in train and test have different distributions, and some of you might also notice hard overfitting in models with meta data. Kolmogorov-Smirnov test and T test are used.\n\n##### Since their distributions are different, the ROC cutoff for train and predicted ROC cutoff for test would have similar gap such the differences of their distributions.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def ttest(col, *, data=combine, group=\"site\", group_val=[\"1\", \"2\"]):\n    res = stats.ttest_ind(\n        combine[combine[group] == group_val[0]][col],\n        combine[combine[group] == group_val[1]][col],\n        equal_var=False,\n    )\n\n    return np.round(res[1], 5)\n\ndef k2(col, *, data=combine, group=\"site\", group_val=[\"1\", \"2\"]):\n    res = stats.ks_2samp(\n        data[data[group] == group_val[0]][col], \n        data[data[group] == group_val[1]][col]\n    )\n\n    return np.round(res[1], 5)\n\n\n#predicted target values in train and test have different distributions\n#Kolmogorov-Smirnov test and T test are used\n\nfor i in predicted_val:\n    print(f'model, {i}')\n    print('--ks test p value --')\n    print(k2(i, data = combine, group = 'is_train', group_val = [0, 1]))\n    print('--t test p-value --')\n    print(ttest(i, data = combine, group = 'is_train', group_val = [0, 1]))\n    print()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Optimal ROC cutoff\n\n\nFind optimal ROC cutoff for all models, total 14.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def classify(y_true, y_pred, threshold):\n    \n    new_pred = y_pred >= threshold\n    \n    pos_pred = new_pred \n    neg_pred = 1-new_pred\n    \n    tp = np.sum(y_true * pos_pred)     #True Positive\n    fp = np.sum((1-y_true) * pos_pred) #False Positive\n    tn = np.sum((1-y_true) * neg_pred) #True Negative\n    fn = np.sum(y_true * neg_pred)     #False Negative\n    \n    sensitivity = tp / (tp + fn)       #TP Ratio\n    specificity = tn / (tn + fp)       #TN Ratio\n    \n    tpr = sensitivity\n    fpr = fp/(fp + tn) #1 - specificity\n    tnr = specificity\n    \n    return tpr, fpr, tnr, tp, tn\n\n#simple ex\ny = np.array([0, 0, 1, 1])\nscores = np.array([0.1, 0.4, 0.35, 0.8])\n\ntpr, fpr, tnr, tp, tn = classify(y, scores, 0.35)\n\ntpr, fpr, tnr, tp, tn\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def opt_cutoff(train, feat, n_thresholds, do_print = True):\n    #searching cutoff from min to max of predicted target values\n    cutoff = np.linspace(train[feat].min(),\n                         train[feat].max(),\n                         n_thresholds)\n    \n    \n    tpr = np.zeros(n_thresholds)\n    fpr = np.zeros(n_thresholds)\n    tnr = np.zeros(n_thresholds)\n    \n    for c, i in enumerate(cutoff):\n        tpr1, fpr1, tnr1, tp, tn = classify(train['target'].values, train[feat].values, i)\n        \n        tpr[c] = tpr1\n        fpr[c] = fpr1\n        tnr[c] = tnr1\n    \n    optimal_idx = np.argmax(tpr + (-fpr))\n    optimal_cutoff = cutoff[optimal_idx]\n    \n    \n    new_tpr, new_fpr, new_tnr, tp, tn = classify(train['target'].values, train[feat].values, optimal_cutoff)\n    \n    if do_print:\n        print(f'optimal cutoff, {optimal_cutoff}')\n    \n        plt.plot(fpr, tpr)\n        plt.scatter(new_fpr, new_tpr)\n        plt.title(f'{feat}, roc curve')\n    \n        print(f'tpr with cutoff {np.round(new_tpr, 5)}')\n        print(f'fpr with cutoff {np.round(new_fpr, 5)}')\n        print(f'tnr with cutoff {np.round(new_tnr, 5)}')\n    \n    feat_mean = train[feat].mean()\n    feat_std = train[feat].std()\n    \n    return optimal_cutoff, feat_mean, feat_std","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"optimal_cutoff, col_avg, col_std = opt_cutoff(train, predicted_val[1], 5000) ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\noptimal_cutoff = np.zeros(len(predicted_val))\ncol_avg = np.zeros(len(predicted_val))\ncol_std = np.zeros(len(predicted_val))\n\nfor c, i in enumerate(tqdm(predicted_val)):\n    oc, ca, cs = opt_cutoff(train, i, 10000, do_print = False)\n    optimal_cutoff[c] = oc\n    col_avg[c] = ca\n    col_std[c] = cs","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('========================')\nprint('Real target value counts')\nprint(train['target'].value_counts())\nprint('========================')\n\n\ntpr_s = []\nfpr_s = []\ntnr_s = []\nsc = []\n\nfor c, i in enumerate(predicted_val):\n    \n    tpr, fpr, tnr, tp, tn = classify(train['target'].values, train[i].values, optimal_cutoff[c])\n    score = np.round(roc_auc_score(train['target'].values, train[i].values),4)\n    \n    tpr_s.append(tpr)\n    fpr_s.append(fpr)\n    tnr_s.append(tnr)\n    sc.append(score)\n    \n    print()\n    print('---------------------')\n    print(f'Model :: {i}')\n    print(f'True Negative       : {tn}')\n    print(f'True Negative Ratio : {np.int(np.round(tnr,2) * 100)}%')\n    print()\n    print(f'True Positive       : {tp}')\n    print(f'True Positive Ratio : {np.int(np.round(tpr,2) * 100)}%')\n    print(f'False Positive Ratio: {np.int(np.round(fpr,2) * 100)}%')\n    print()\n    print(f'OOF AUC score           : {score}')\n    print('---------------------')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(num=None, figsize=(10, 8), dpi=80)\nplt.scatter(optimal_cutoff, col_avg)\nfor p, o, c in zip(predicted_val, optimal_cutoff, col_avg):\n    plt.annotate(p, (o, c))\nplt.title('Optimal Cutoff vs Predicted Value Average')\n\ncor = np.round(np.corrcoef(optimal_cutoff, col_avg)[1,0], 7)\n\n\nprint(f'correlation between cutoff and prediction average, {cor}')\nprint('Linearity between those two seems acceptable, but not sure for now')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(num=None, figsize=(10, 8), dpi=80)\nplt.scatter(optimal_cutoff, col_avg + col_std)\nfor p, o, ca, cs in zip(predicted_val, optimal_cutoff, col_avg, col_std):\n    plt.annotate(p, (o, ca + cs))\nplt.title('Optimal Cutoff vs (Predicted Value Average + Standard Deviation)')\n\ncor = np.round(np.corrcoef(optimal_cutoff, col_avg + col_std)[1,0], 7)\nprint(f'correlation between cutoff and prediction average + prediction std, {cor}')\nprint('No patterns')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(num=None, figsize=(10, 8), dpi=80)\nplt.scatter(optimal_cutoff-col_std, col_avg - col_std)\nfor p, o, ca, cs in zip(predicted_val, optimal_cutoff, col_avg, col_std):\n    plt.annotate(p, (o - cs, ca - cs))\n\nplt.title('(Optimal Cutoff - Std) vs (Predicted Value Average - Std)')\n\ncor = np.round((np.corrcoef(optimal_cutoff-col_std, col_avg - col_std)[1,0]), 7)\n\nprint(f'correlation between cutoff - prediction std and prediction average - prediction std, {cor}')\nprint('Interesting pattern found here')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"te_col_avg = np.zeros(len(predicted_val))\nte_col_std = np.zeros(len(predicted_val))\n\nfor c, i in enumerate(tqdm(predicted_val)):\n    te_col_avg[c] = test[i].mean()\n    te_col_std[c] = test[i].std()\n\ntr = pd.DataFrame({'cutoff' : optimal_cutoff, 'avg': col_avg, 'std' : col_std, 'cutoff-avg' : optimal_cutoff - col_avg, \n                   'cutoff-std': optimal_cutoff - col_std, 'avg+std':col_avg + col_std, 'avg-std':col_avg - col_std})\nte = pd.DataFrame({'avg': te_col_avg, 'std' : te_col_std, 'avg-std' : te_col_avg - te_col_std})","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(num=None, figsize=(10, 8), dpi=80)\nmask = np.zeros_like(tr.corr())\nmask[np.triu_indices_from(mask)] = True\nsns.heatmap(tr.corr(),mask = mask, annot = True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def mae(x,y):\n    return np.sum(np.abs(x - y))/len(x)\n\nprint(mae(tr['cutoff-std'], tr['avg-std']))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Predict ROC cutoff\n\nSince there is high correlation between the ROC cutoff subtracted by standard deviation of predicted values and the average of predicted values subtracted by standard deviation of predicted values, we predict ROC cutoff subtracted by standard deviation of predicted values from the average of predicted values subtracted by standard deviation of predicted values.\n\nIn summary, \n\nPredict ROC cutoff - Predicted Std \n\nfrom Predicted Avg - Predicted Std.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from xgboost import XGBRegressor\n\ntree = XGBRegressor().fit(tr[['avg-std']], tr[['cutoff-std']])\n\nscore = np.int(np.round((tree.score(tr[['avg-std']], tr[['cutoff-std']]))*100, 0))\n\nprint(f'R2 score, {score}% variation explained')\n\ntr_cutoff_std = tree.predict(tr[['avg-std']]).flatten()\nte_cutoff_std = tree.predict(te[['avg-std']]).flatten()\n\n#Since we predict the value of (ROC cutoff - target std) value,\n#we add (target std) to the predicted (ROC cutoff - target std)\ntr_cutoff = tr_cutoff_std + tr['std']\nte_cutoff = te_cutoff_std + te['std']\n\ntr_error = np.round(mae(tr_cutoff, optimal_cutoff), 8)\ntrte_error = np.round(mae(tr_cutoff, te_cutoff), 8)\ntrte_avg = np.round(mae(tr['avg'], te['avg']),8)\n\nprint(f'MAE,  predicted train optimal ROC cutoff vs true train optimal ROC cutoff     {tr_error}')\nprint(f'MAE,  predicted train optimal ROC cutoff vs predicted test optimal ROC cutoff {trte_error}')\nprint(f'MAE,  predicted target average for train vs predicted target average for test {trte_avg}')\nprint()\nprint('Gap of predicted ROC cutoff between train and test is simliar with the gap of predicted target average between train and test')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, (ax1, ax2) = plt.subplots(ncols=2, figsize=(8,4), dpi=100)\n\nsns.distplot(tr['avg'], hist = False, ax = ax1, label = 'train')\nsns.distplot(te['avg'], hist = False, ax = ax1, label = 'test')\nax1.title.set_text('Average of Predicted target')\n\nsns.distplot(tr_cutoff, hist = False, ax = ax2, label = 'train')\nsns.distplot(te_cutoff, hist = False, ax = ax2, label = 'test')\nax2.title.set_text('ROC Cutoff of Predicted target')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Determine confident predicted test observation by predicted ROC cutoff for all models","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"pseudo_test = pd.DataFrame([])\npseudo_test['image_name'] = test['image_name']\n\nfor c, i in enumerate(predicted_val):\n    pseudo_test[i + '_target'] = (test[i] >= te_cutoff[c]) * 1\n    \npseudo_test['sum_by_cutoff'] = pseudo_test.drop(['image_name'], axis = 1).apply(lambda x: x.sum(), axis = 1)\npseudo_test[['image_name', 'sum_by_cutoff']].sort_values(['sum_by_cutoff'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"n_pos_pred = len(pseudo_test.loc[pseudo_test['sum_by_cutoff'] == 14, 'image_name'])\n\nprint(f'{n_pos_pred} confident predicted melanoma observations in test by all 14 models with predicted ROC cutoff')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"n_neg_pred = len(pseudo_test.loc[pseudo_test['sum_by_cutoff'] == 0, 'image_name'])\n\nprint(f'{n_neg_pred} confident predicted benign observations in test by all 14 models with predicted ROC cutoff')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Conclusion\n\n\nThis found can be just for the predicted values from my models, or number of samples can be too small to ensure whether this works or not. So you should check with your predicted values if you want to give it a try this. ","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}