{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.14","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":81933,"databundleVersionId":9643020,"sourceType":"competition"}],"dockerImageVersionId":30786,"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Grab a cuppa and let's understand kappa \n![](https://thelondoncuppa.com/cdn/shop/files/tea_cup_white940.jpg?v=1652855496)","metadata":{}},{"cell_type":"markdown","source":"# **From the Competition Organisers**\n\n**Evaluation**\n\nSubmissions are scored based on the quadratic weighted kappa, which measures the agreement between two outcomes. This metric typically varies from 0 (random agreement) to 1 (complete agreement). In the event that there is less agreement than expected by chance, the metric may go below 0.\n","metadata":{}},{"cell_type":"markdown","source":"# History of Cohen's Kappa\n\nThe seminal paper that introduced **Cohen's Kappa** as a new technique was published by **Jacob Cohen** in **1960** in the journal *Educational and Psychological Measurement*. Cohen developed this statistic to address a significant limitation in the existing methods of measuring agreement between raters. You can find the paper here: [Cohen's 1960 paper](https://www.tesble.com/10.1177/001316446002000104).\n\n## Motivation for Development\n\nPrior to Cohen's work, the primary method for assessing **inter-rater reliability** was **percent agreement**. However, this method didn't account for agreement that could occur by **chance**.\n\nCohen recognized this issue and sought to create a more robust measure. He hypothesized that when raters were uncertain, they might resort to **guessing**, leading to some level of **random agreement**. Cohen developed the **kappa statistic** specifically to control for this random agreement factor, making it a more accurate representation of **true inter-rater reliability**.\n\n## How it applies to CMI - Problematic use\nSince problematic internet use can lead to a variety of negative outcomes like poor academic performance, social isolation, or mental health issues, early detection through reliable models can lead to timely interventions. \nA high Kappa value ensures that our predictive models aren't making arbitrary predictions (high penalisation for predicting values further away as we will see below)","metadata":{}},{"cell_type":"markdown","source":"### Step 1: Calculating, Visualizing, and Interpreting the Confusion Matrix\n\nIn this step, we generate a **confusion matrix** that shows the **count** of how **often** the **predicted labels match the true labels** for each class. \n\nWe have SII values as 0, 1, 2 or 3 i.e **4 classes**, let's assume y_true (actual values) and y_pred(predicted values)","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nfrom sklearn.metrics import cohen_kappa_score\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\ny_true = np.array([0, 1, 2, 2, 1, 0, 1, 2, 0, 3])\ny_pred = np.array([0, 1, 2, 1, 1, 1, 2, 2, 0, 1])\n\n\nN = 4\n\n\ndef confusion_matrix_O(y_true, y_pred, N):\n    O = np.zeros((N, N), dtype=int)\n    for i in range(len(y_true)):\n        O[int(y_true[i]), int(y_pred[i])] += 1\n    return O\nO_matrix = confusion_matrix_O(y_true, y_pred, N)\n\n\nplt.figure(figsize=(6, 4))\nsns.heatmap(O_matrix, annot=True, fmt=\"d\", cmap=\"coolwarm\", cbar=True)\nplt.title(\"Confusion Matrix (O matrix)\")\nplt.xlabel(\"Predicted\")\nplt.ylabel(\"Actual\")\nplt.show()\n\n","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2024-10-13T18:08:26.033953Z","iopub.execute_input":"2024-10-13T18:08:26.034427Z","iopub.status.idle":"2024-10-13T18:08:26.350833Z","shell.execute_reply.started":"2024-10-13T18:08:26.034366Z","shell.execute_reply":"2024-10-13T18:08:26.349493Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Step 2: Building the Weight Matrix (W Matrix)\n\nIn this step, we create a **weight matrix** (W matrix), which is used to assign different penalties to misclassifications based on how far off the predictions are from the actual values.\n- The matrix is built using a double loop over the number of classes (`N`), where each element `W[i, j]` represents the penalty for predicting class `j` when the actual class is `i`.\n- The penalty is calculated as the **squared difference** between the indices of the actual class (`i`) and the predicted class (`j`), normalized by the square of `(N - 1)`. \n\nThe penalty for misclassification increases quadratically as the difference between the true class and predicted class grows. For instance, predicting class 0 instead of 3 (a large difference) is penalized more heavily than predicting class 0 instead of 1 (a small difference). ","metadata":{}},{"cell_type":"code","source":"def weight_matrix_W(N):\n    W = np.zeros((N, N))\n    for i in range(N):\n        for j in range(N):\n            W[i, j] = ((i - j) ** 2) / (N - 1) ** 2\n    return W\n\n\nW_matrix = weight_matrix_W(N)\n\nplt.figure(figsize=(6, 4))\nsns.heatmap(W_matrix, annot=True, cmap=\"coolwarm\", cbar=True)\nplt.title(\"Weight Matrix (W matrix)\")\nplt.xlabel(\"Predicted\")\nplt.ylabel(\"Actual\")\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2024-10-13T17:59:20.119035Z","iopub.execute_input":"2024-10-13T17:59:20.119982Z","iopub.status.idle":"2024-10-13T17:59:20.431928Z","shell.execute_reply.started":"2024-10-13T17:59:20.119933Z","shell.execute_reply":"2024-10-13T17:59:20.430676Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Step 3: Building the Expected Matrix (E Matrix)\n\nIn this step, we calculate the **expected matrix** (E matrix), which represents the number of agreements we would expect purely by chance, based on the distribution of true and predicted labels.\n\n\n- The expected matrix is calculated using the **marginals** of the confusion matrix (O matrix), which are the row sums (true class distribution) and column sums (predicted class distribution).\n- For each element `E[i, j]` in the matrix, the value is computed as:\n- This formula estimates how often class `i` would be predicted as class `j` by chance, given the distribution of true and predicted classes.\n\n","metadata":{}},{"cell_type":"code","source":"def expected_matrix_E(y_true, y_pred, N):\n    true_hist = np.bincount(y_true, minlength=N)\n    pred_hist = np.bincount(y_pred, minlength=N)\n    E = np.outer(true_hist, pred_hist)\n    E = E / E.sum() * np.sum(confusion_matrix_O(y_true, y_pred, N)) \n    return E\n\n\nE_matrix = expected_matrix_E(y_true, y_pred, N)\n\nplt.figure(figsize=(6, 4))\nsns.heatmap(E_matrix, annot=True, cmap=\"coolwarm\", cbar=True)\nplt.title(\"Expected Matrix (E matrix)\")\nplt.xlabel(\"Predicted\")\nplt.ylabel(\"Actual\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-10-13T18:06:47.395918Z","iopub.execute_input":"2024-10-13T18:06:47.396973Z","iopub.status.idle":"2024-10-13T18:06:47.704610Z","shell.execute_reply.started":"2024-10-13T18:06:47.396923Z","shell.execute_reply":"2024-10-13T18:06:47.703455Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Step 4: Calculating the Quadratic Weighted Kappa (QWK)\n\nIn this step, we bring everything together \n\n1. **Confusion Matrix (O)**: We have the confusion matrix using `confusion_matrix_O`, which counts how often each class is predicted correctly or incorrectly.\n2. **Weight Matrix (W)**: The `weight_matrix_W` assigns penalties based on how far the predicted class is from the true class, with larger penalties for more distant misclassifications.\n3. **Expected Matrix (E)**: The `expected_matrix_E` calculates the agreements that would happen by chance based on the distribution of true and predicted labels.\n4. **Numerator and Denominator**:\n   - `num = np.sum(W * O)`: The weighted sum of the observed confusion matrix (O), penalizing misclassifications based on their distance.\n   - `den = np.sum(W * E)`: The weighted sum of the expected matrix (E), representing chance agreement.\n","metadata":{}},{"cell_type":"code","source":"def quadratic_weighted_kappa(y_true, y_pred, N):\n    O = confusion_matrix_O(y_true, y_pred, N)\n    W = weight_matrix_W(N)\n    E = expected_matrix_E(y_true, y_pred, N)\n    \n    num = np.sum(W * O)\n    den = np.sum(W * E)\n    \n    return 1 - (num / den)\n\ncustom_qwk = quadratic_weighted_kappa(y_true, y_pred, N)","metadata":{"execution":{"iopub.status.busy":"2024-10-13T18:23:19.946378Z","iopub.execute_input":"2024-10-13T18:23:19.947235Z","iopub.status.idle":"2024-10-13T18:23:19.954462Z","shell.execute_reply.started":"2024-10-13T18:23:19.947188Z","shell.execute_reply":"2024-10-13T18:23:19.953163Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(f'Our custom implementation QWK: {custom_qwk:.4f}')","metadata":{"execution":{"iopub.status.busy":"2024-10-13T18:25:06.500426Z","iopub.execute_input":"2024-10-13T18:25:06.501426Z","iopub.status.idle":"2024-10-13T18:25:06.507124Z","shell.execute_reply.started":"2024-10-13T18:25:06.501343Z","shell.execute_reply":"2024-10-13T18:25:06.505851Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sklearn_qwk = cohen_kappa_score(y_true, y_pred, weights='quadratic')\nprint(f'Sklearn implementation QWK: {sklearn_qwk:.4f}')","metadata":{"execution":{"iopub.status.busy":"2024-10-13T18:25:09.684266Z","iopub.execute_input":"2024-10-13T18:25:09.684749Z","iopub.status.idle":"2024-10-13T18:25:09.692971Z","shell.execute_reply.started":"2024-10-13T18:25:09.684705Z","shell.execute_reply":"2024-10-13T18:25:09.691871Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nfrom sklearn.impute import SimpleImputer\nfrom sklearn.ensemble import RandomForestClassifier\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import cohen_kappa_score\nfrom scipy.optimize import minimize\n\n\ntrain = pd.read_csv('/kaggle/input/child-mind-institute-problematic-internet-use/train.csv')\ntest = pd.read_csv('/kaggle/input/child-mind-institute-problematic-internet-use/test.csv')\ntrain = train.dropna(subset='sii')\n\n\nfeatures = ['Basic_Demos-Age', 'Basic_Demos-Sex', 'Physical-BMI', 'CGAS-CGAS_Score']\ntarget = 'sii'\n\n\nX_train, X_val, y_train, y_val = train_test_split(train[features], train[target], test_size=0.2, random_state=42)\n\n\nimputer = SimpleImputer(strategy='median')\n\n\nX_train = imputer.fit_transform(X_train)\nX_val = imputer.transform(X_val)\n\n\nrf = RandomForestClassifier(random_state=42)\nrf.fit(X_train, y_train)\n\n\ny_val_pred = rf.predict(X_val)\n\n\nX_test = imputer.transform(test[features]) \ny_test_pred = rf.predict(X_test)\n\n\nqwk_sklearn_val = cohen_kappa_score(y_val, y_val_pred, weights='quadratic')\nprint(f'QWK on Validation Data: {qwk_sklearn_val:.4f}')\n","metadata":{"execution":{"iopub.status.busy":"2024-10-13T18:34:28.828173Z","iopub.execute_input":"2024-10-13T18:34:28.828650Z","iopub.status.idle":"2024-10-13T18:34:29.479230Z","shell.execute_reply.started":"2024-10-13T18:34:28.828607Z","shell.execute_reply":"2024-10-13T18:34:29.478147Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_test_pred\n","metadata":{"execution":{"iopub.status.busy":"2024-10-13T16:53:56.877901Z","iopub.execute_input":"2024-10-13T16:53:56.878337Z","iopub.status.idle":"2024-10-13T16:53:56.885707Z","shell.execute_reply.started":"2024-10-13T16:53:56.878294Z","shell.execute_reply":"2024-10-13T16:53:56.884482Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample = pd.read_csv('/kaggle/input/child-mind-institute-problematic-internet-use/sample_submission.csv')\nsubmission = pd.DataFrame({\n        'id': sample['id'],\n        'sii': y_test_pred\n    })\n\n","metadata":{"execution":{"iopub.status.busy":"2024-10-13T18:34:34.242611Z","iopub.execute_input":"2024-10-13T18:34:34.243071Z","iopub.status.idle":"2024-10-13T18:34:34.252713Z","shell.execute_reply.started":"2024-10-13T18:34:34.243025Z","shell.execute_reply":"2024-10-13T18:34:34.251540Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission.to_csv('submission.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2024-10-13T16:55:48.795715Z","iopub.execute_input":"2024-10-13T16:55:48.796606Z","iopub.status.idle":"2024-10-13T16:55:48.804802Z","shell.execute_reply.started":"2024-10-13T16:55:48.796557Z","shell.execute_reply":"2024-10-13T16:55:48.803646Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Happy Kaggling! \n\nIf you found my notebook useful, please upvote :) thanks!","metadata":{}}]}