{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"After some discussions in this [thread](https://www.kaggle.com/c/rsna-miccai-brain-tumor-radiogenomic-classification/discussion/275233) whether OOF or CV is better, I want to clarify some points by looking at the results empirically:\n\n- CV AUC gives a better bias estimate than OOF AUC (in particular for high K)\n- the high std can be explained by the bias-variance tradeoff\n\n","metadata":{}},{"cell_type":"markdown","source":"# Regular 5 and 200-fold CV","metadata":{}},{"cell_type":"code","source":"from sklearn.linear_model import LogisticRegression\nfrom sklearn.model_selection import StratifiedKFold\nfrom sklearn.metrics import roc_auc_score\nfrom sklearn.metrics import log_loss\nimport numpy as np\nimport pandas as pd\n\nX = pd.read_csv(\"../input/leak-in-metadata/X_train.csv\", usecols=['T2w_Percent Phase Field of View',\n                                        'FLAIR_Echo Train Length',\n                                        'T2w_shape',\n                                        'target'])\nX.fillna(0, inplace=True)\ny = X[\"target\"].values\nX.drop(\"target\", axis=1, inplace=True)\nX = X.values\n\no = []\no2 = []\nfor fold, (train_index, val_index) in enumerate(StratifiedKFold(n_splits=200).split(X, y)):\n    X_train, X_val = X[train_index], X[val_index]\n    y_train, y_val = y[train_index], y[val_index]\n\n    regr = LogisticRegression()\n    regr.fit(X_train, y_train)\n    \n    y_pred = regr.predict_proba(X_val)[...,1]\n    \n    auc = roc_auc_score(y_val, y_pred)\n    val_loss = log_loss(y_val, y_pred)\n    \n    o.append(auc)\n    o2.append(val_loss)\n\nprint(\"Loss\", np.mean(o2), np.std(o2))\nprint(\"AUC\", np.mean(o), np.std(o))","metadata":{"execution":{"iopub.status.busy":"2021-09-30T18:25:56.491751Z","iopub.execute_input":"2021-09-30T18:25:56.492314Z","iopub.status.idle":"2021-09-30T18:25:59.453987Z","shell.execute_reply.started":"2021-09-30T18:25:56.492225Z","shell.execute_reply":"2021-09-30T18:25:59.453306Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see that the standard deviation is extremely high. However, this is not surprising by the [bias-variance-tradeoff](https://stats.stackexchange.com/questions/61783/bias-and-variance-in-leave-one-out-vs-k-fold-cross-validation). Even the proper scoring rule log loss is affected by the high number of folds. Let us reduce the folds to 5.","metadata":{}},{"cell_type":"code","source":"o = []\no2 = []\nfor fold, (train_index, val_index) in enumerate(StratifiedKFold(n_splits=5).split(X, y)):\n    X_train, X_val = X[train_index], X[val_index]\n    y_train, y_val = y[train_index], y[val_index]\n\n    regr = LogisticRegression()\n    regr.fit(X_train, y_train)\n    \n    y_pred = regr.predict_proba(X_val)[...,1]\n    \n    auc = roc_auc_score(y_val, y_pred)\n    val_loss = log_loss(y_val, y_pred)\n    \n    o.append(auc)\n    o2.append(val_loss)\n\nprint(\"Loss\", np.mean(o2), np.std(o2))\nprint(\"AUC\", np.mean(o), np.std(o))","metadata":{"execution":{"iopub.status.busy":"2021-09-30T18:25:59.455623Z","iopub.execute_input":"2021-09-30T18:25:59.456440Z","iopub.status.idle":"2021-09-30T18:25:59.523093Z","shell.execute_reply.started":"2021-09-30T18:25:59.456383Z","shell.execute_reply":"2021-09-30T18:25:59.521968Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The standard deviation is closer to the true standard deviation. However, we increased the bias. The AUC has decreased to 0.57.","metadata":{}},{"cell_type":"markdown","source":"# OOF 5 and 200-fold CV","metadata":{}},{"cell_type":"code","source":"oof_score = np.zeros((X.shape[0],))\nfor fold, (train_index, val_index) in enumerate(StratifiedKFold(n_splits=200).split(X, y)):\n    X_train, X_val = X[train_index], X[val_index]\n    y_train, y_val = y[train_index], y[val_index]\n\n    regr = LogisticRegression()\n    regr.fit(X_train, y_train)\n    \n    y_pred = regr.predict_proba(X_val)[...,1]\n    \n    oof_score[val_index] = y_pred\n\nprint(\"OOF Loss\", roc_auc_score(y, oof_score))\nprint(\"OOF AUC\", log_loss(y, oof_score))","metadata":{"execution":{"iopub.status.busy":"2021-09-30T18:25:59.524706Z","iopub.execute_input":"2021-09-30T18:25:59.524910Z","iopub.status.idle":"2021-09-30T18:26:01.308349Z","shell.execute_reply.started":"2021-09-30T18:25:59.524886Z","shell.execute_reply":"2021-09-30T18:26:01.307509Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"By using OOF, we actually get a higher AUC! OOF AUC is **0.6806**, while regular CV AUC **0.6575**. Let us look at 5-fold CV again.","metadata":{}},{"cell_type":"code","source":"oof_score = np.zeros((X.shape[0],))\nfor fold, (train_index, val_index) in enumerate(StratifiedKFold(n_splits=5).split(X, y)):\n    X_train, X_val = X[train_index], X[val_index]\n    y_train, y_val = y[train_index], y[val_index]\n\n    regr = LogisticRegression()\n    regr.fit(X_train, y_train)\n    \n    y_pred = regr.predict_proba(X_val)[...,1]\n    \n    oof_score[val_index] = y_pred\n\nprint(\"OOF Loss\", roc_auc_score(y, oof_score))\nprint(\"OOF AUC\", log_loss(y, oof_score))","metadata":{"execution":{"iopub.status.busy":"2021-09-30T18:26:01.310528Z","iopub.execute_input":"2021-09-30T18:26:01.311103Z","iopub.status.idle":"2021-09-30T18:26:01.367701Z","shell.execute_reply.started":"2021-09-30T18:26:01.311061Z","shell.execute_reply":"2021-09-30T18:26:01.367163Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The AUC has increased from 0.68 to 0.7. In contrast, the regular 5-fold CV AUC is 0.5768147135783306.","metadata":{}},{"cell_type":"markdown","source":"# Train-test-split and CV","metadata":{}},{"cell_type":"markdown","source":"From the last section, we found out that:\n- CV AUC < OOF AUC\n- By increasing the number of folds, we increase the variance\n\nNow, I want to look at the number of folds in relation to the bias of the estimator.","metadata":{}},{"cell_type":"markdown","source":"The train dataset is 78% of the total data and the test dataset is 22% of the total data. I am using 60 folds because the dataset is too small.","metadata":{}},{"cell_type":"code","source":"from sklearn.model_selection import train_test_split\nfrom tqdm import tqdm\n\nFOLDS = 60\n\nX = pd.read_csv(\"../input/leak-in-metadata/X_train.csv\", usecols=['T2w_Percent Phase Field of View',\n                                        'FLAIR_Echo Train Length',\n                                        'T2w_shape',\n                                        'target'])\nX.fillna(0, inplace=True)\ny = X[\"target\"].values\nX.drop(\"target\", axis=1, inplace=True)\nX = X.values\n\nX, X_test, y, y_test = train_test_split(X, y, test_size=0.78,\n                                        train_size=0.22, random_state=3,\n                                        shuffle=True, stratify=y)\nprint(\"train\", X.shape, \"test\", X_test.shape)\n\navg_auc_shape = []\nroc = []\nloss = []\nskf = StratifiedKFold(n_splits=FOLDS, random_state=None, shuffle=False)\noof = np.zeros((X.shape[0],), dtype=np.float64)\ny_pred_test = np.zeros((y_test.shape[0],), dtype=np.float64)\nfor train_index, val_index in tqdm(skf.split(X, y), total=FOLDS):\n    X_train, X_val = X[train_index], X[val_index]\n    y_train, y_val = y[train_index], y[val_index]\n\n    model = LogisticRegression()\n    model.fit(X_train, y_train)\n\n    y_pred = model.predict_proba(X_val)[...,1]\n\n    y_pred_test += model.predict_proba(X_test)[...,1] / FOLDS\n\n    oof[val_index] = y_pred\n\n    roc.append(roc_auc_score(y_val, y_pred))\n    loss.append(log_loss(y_val, y_pred))\n    avg_auc_shape.append(y_val.shape[0])\n\nprint(\"Each AUC was computed on\", np.mean(avg_auc_shape), \"samples\")\nprint(\"CV AUC\", np.mean(roc), np.std(roc))\nprint(\"CV loss\", np.mean(loss), np.std(loss))\nprint()\nprint(\"OOF AUC\", roc_auc_score(y, oof))\nprint(\"OOF loss\", log_loss(y, oof))\nprint()\nprint(\"AUC test\", roc_auc_score(y_test, y_pred_test))\nprint(\"Loss test\", log_loss(y_test, y_pred_test))","metadata":{"execution":{"iopub.status.busy":"2021-09-30T18:36:44.147070Z","iopub.execute_input":"2021-09-30T18:36:44.147367Z","iopub.status.idle":"2021-09-30T18:36:44.805228Z","shell.execute_reply.started":"2021-09-30T18:36:44.147335Z","shell.execute_reply":"2021-09-30T18:36:44.804486Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This time we see that OOF AUC < AUC but the regular CV AUC is much closer to the test dataset. Again the estimation of the variance is wrong.\n\nLet us reduce the number of folds to 5.","metadata":{}},{"cell_type":"code","source":"FOLDS = 5\n\navg_auc_shape = []\nroc = []\nloss = []\nskf = StratifiedKFold(n_splits=FOLDS, random_state=None, shuffle=False)\noof = np.zeros((X.shape[0],), dtype=np.float64)\ny_pred_test = np.zeros((y_test.shape[0],), dtype=np.float64)\nfor train_index, val_index in tqdm(skf.split(X, y), total=FOLDS):\n    X_train, X_val = X[train_index], X[val_index]\n    y_train, y_val = y[train_index], y[val_index]\n\n    model = LogisticRegression()\n    model.fit(X_train, y_train)\n\n    y_pred = model.predict_proba(X_val)[...,1]\n\n    y_pred_test += model.predict_proba(X_test)[...,1] / FOLDS\n\n    oof[val_index] = y_pred\n\n    roc.append(roc_auc_score(y_val, y_pred))\n    loss.append(log_loss(y_val, y_pred))\n    avg_auc_shape.append(y_val.shape[0])\n\nprint(\"Each AUC was computed on\", np.mean(avg_auc_shape), \"samples\")\nprint(\"CV AUC\", np.mean(roc), np.std(roc))\nprint(\"CV loss\", np.mean(loss), np.std(loss))\nprint()\nprint(\"OOF AUC\", roc_auc_score(y, oof))\nprint(\"OOF loss\", log_loss(y, oof))\nprint()\nprint(\"AUC test\", roc_auc_score(y_test, y_pred_test))\nprint(\"Loss test\", log_loss(y_test, y_pred_test))","metadata":{"execution":{"iopub.status.busy":"2021-09-30T18:36:46.874450Z","iopub.execute_input":"2021-09-30T18:36:46.874730Z","iopub.status.idle":"2021-09-30T18:36:46.955705Z","shell.execute_reply.started":"2021-09-30T18:36:46.874693Z","shell.execute_reply":"2021-09-30T18:36:46.955158Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We have a better estimation of the standard deviation by reducing the number of folds (bias-variance tradeoff). 0.522 + 0.0622 = 0.5842 which is quite close to 0.5756.\n\nHowever, the bias is again too high. Note that OOF AUC > CV AUC.","metadata":{}},{"cell_type":"markdown","source":"# Distribution","metadata":{}},{"cell_type":"markdown","source":"In the last section, we only considered one dataset by using train_test_split with 1 seed. However, this is only one particular instance of the dataset. In the next experiment, we sample 300 datasets from the whole dataset distribution to get a better estimate of the CV.\n\nWe use 60 folds again.","metadata":{}},{"cell_type":"code","source":"FOLDS = 60\n\navg_auc_shape = []\ncv_auc = []\ncv_loss = []\n\noof_auc = []\noof_loss = []\n\ntest_auc = []\ntest_loss = []\nfor i in tqdm(range(300)):\n    X = pd.read_csv(\"../input/leak-in-metadata/X_train.csv\", usecols=['T2w_Percent Phase Field of View',\n                                            'FLAIR_Echo Train Length',\n                                            'T2w_shape',\n                                            'target'])\n    X.fillna(0, inplace=True)\n    y = X[\"target\"].values\n    X.drop(\"target\", axis=1, inplace=True)\n    X = X.values\n\n    X, X_test, y, y_test = train_test_split(X, y, test_size=0.78,\n                                            train_size=0.22, random_state=i,\n                                            shuffle=True, stratify=y)\n    #print(\"train\", X.shape, \"test\", X_test.shape)\n\n    skf = StratifiedKFold(n_splits=FOLDS, random_state=None, shuffle=False)\n    oof = np.zeros((X.shape[0],), dtype=np.float64)\n    y_pred_test = np.zeros((y_test.shape[0],), dtype=np.float64)\n    for train_index, val_index in skf.split(X, y):\n        X_train, X_val = X[train_index], X[val_index]\n        y_train, y_val = y[train_index], y[val_index]\n\n        model = LogisticRegression()\n        model.fit(X_train, y_train)\n\n        y_pred = model.predict_proba(X_val)[...,1]\n\n        y_pred_test += model.predict_proba(X_test)[...,1] / FOLDS\n\n        oof[val_index] = y_pred\n\n        cv_auc.append(roc_auc_score(y_val, y_pred))\n        cv_loss.append(log_loss(y_val, y_pred))\n        avg_auc_shape.append(y_val.shape[0])\n    \n    oof_auc.append(roc_auc_score(y, oof))\n    oof_loss.append(log_loss(y, oof))\n    \n    test_auc.append(roc_auc_score(y_test, y_pred_test))\n    test_loss.append(log_loss(y_test, y_pred_test))\n\nprint(\"Each AUC was computed on\", np.mean(avg_auc_shape), \"samples\")\nprint(\"CV AUC\", np.mean(cv_auc), np.std(cv_auc))\nprint(\"CV loss\", np.mean(cv_loss), np.std(cv_loss))\nprint()\nprint(\"OOF AUC\", np.mean(oof_auc))\nprint(\"OOF loss\", np.mean(oof_loss))\nprint()\nprint(\"AUC test\", np.mean(test_auc))\nprint(\"Loss test\", np.mean(test_loss))","metadata":{"execution":{"iopub.status.busy":"2021-09-30T18:43:58.813779Z","iopub.execute_input":"2021-09-30T18:43:58.814379Z","iopub.status.idle":"2021-09-30T18:46:34.384140Z","shell.execute_reply.started":"2021-09-30T18:43:58.814339Z","shell.execute_reply":"2021-09-30T18:46:34.380753Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Again we see that CV AUC is closer to AUC test than OOF AUC. Next, we test the results with 5 folds.","metadata":{}},{"cell_type":"code","source":"FOLDS = 5\n\navg_auc_shape = []\ncv_auc = []\ncv_loss = []\n\noof_auc = []\noof_loss = []\n\ntest_auc = []\ntest_loss = []\nfor i in tqdm(range(300)):\n    X = pd.read_csv(\"../input/leak-in-metadata/X_train.csv\", usecols=['T2w_Percent Phase Field of View',\n                                            'FLAIR_Echo Train Length',\n                                            'T2w_shape',\n                                            'target'])\n    X.fillna(0, inplace=True)\n    y = X[\"target\"].values\n    X.drop(\"target\", axis=1, inplace=True)\n    X = X.values\n\n    X, X_test, y, y_test = train_test_split(X, y, test_size=0.78,\n                                            train_size=0.22, random_state=i,\n                                            shuffle=True, stratify=y)\n    #print(\"train\", X.shape, \"test\", X_test.shape)\n\n    skf = StratifiedKFold(n_splits=FOLDS, random_state=None, shuffle=False)\n    oof = np.zeros((X.shape[0],), dtype=np.float64)\n    y_pred_test = np.zeros((y_test.shape[0],), dtype=np.float64)\n    for train_index, val_index in skf.split(X, y):\n        X_train, X_val = X[train_index], X[val_index]\n        y_train, y_val = y[train_index], y[val_index]\n\n        model = LogisticRegression()\n        model.fit(X_train, y_train)\n\n        y_pred = model.predict_proba(X_val)[...,1]\n\n        y_pred_test += model.predict_proba(X_test)[...,1] / FOLDS\n\n        oof[val_index] = y_pred\n\n        cv_auc.append(roc_auc_score(y_val, y_pred))\n        cv_loss.append(log_loss(y_val, y_pred))\n        avg_auc_shape.append(y_val.shape[0])\n    \n    oof_auc.append(roc_auc_score(y, oof))\n    oof_loss.append(log_loss(y, oof))\n    \n    test_auc.append(roc_auc_score(y_test, y_pred_test))\n    test_loss.append(log_loss(y_test, y_pred_test))\n\nprint(\"Each AUC was computed on\", np.mean(avg_auc_shape), \"samples\")\nprint(\"CV AUC\", np.mean(cv_auc), np.std(cv_auc))\nprint(\"CV loss\", np.mean(cv_loss), np.std(cv_loss))\nprint()\nprint(\"OOF AUC\", np.mean(oof_auc))\nprint(\"OOF loss\", np.mean(oof_loss))\nprint()\nprint(\"AUC test\", np.mean(test_auc))\nprint(\"Loss test\", np.mean(test_loss))","metadata":{"execution":{"iopub.status.busy":"2021-09-30T18:47:27.050638Z","iopub.execute_input":"2021-09-30T18:47:27.050942Z","iopub.status.idle":"2021-09-30T18:47:42.581453Z","shell.execute_reply.started":"2021-09-30T18:47:27.050891Z","shell.execute_reply":"2021-09-30T18:47:42.580059Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Before we saw a strong effect on the bias. This time the effect is not as strong. For 60-fold CV we had 0.575583, here we have 0.57384. AUC test is 0.57694. Then |0.575583 - 0.57694| = 0.001357 (60-fold) and |0.57384 - 0.57694| = 0.0031 (5-fold CV). Hence, 60-fold CV is closer to AUC test.","metadata":{}}]}