{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.14","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":81933,"databundleVersionId":9643020,"sourceType":"competition"}],"dockerImageVersionId":30775,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Understanding Quadratic Weighted Kappa: A Playaround\n\nHello everyone!\n\nAmong the challenges in this competition, I wanted to fully grasp the evaluation metric. From the provided formula and variable descriptions, it’s evident that the result depends on the difference between class indices (since they indicate a certain degree) and how closely the predictions match the expected class distribution. However, the combination of several matrices introduces some slight uncertainty. Therefore, I decided to use toy examples to test my understanding of this metric.\n\nI thought this code might be helpful to others as well, so I suggest visualizing edge cases to see how they influence the final result.\n\n**Note:** This is not a math guide but rather a look at how data changes affect this metric.\n\n## General Formula of the Metric\n\nBased on the formula, we need to preserve the leading 1, which means ensuring the smallest possible value in the fractional expression. Logically, this can be achieved either by reducing the numerator or increasing the denominator. If you’re seeing this formula for the first time, some interdependencies might cause slight confusion. I suggest we examine each matrix separately. Once we confidently understand each component, we’ll combine them into the final calculation.\n\n$$\n\\kappa = 1 - \\frac{\\sum_{i,j} W_{i,j} O_{i,j}}{\\sum_{i,j} W_{i,j} E_{i,j}}\n$$\n\n## O-Matrix: Prediction Results\n\nThe first matrix, \"O\", is the easiest to understand, as it resembles a confusion matrix, but for multiclass classification.\n\nErrors are located outside the diagonal. From the cells where the errors are displayed, we can determine which class was supposed to be predicted and which class the model mistakenly identified.","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nfrom matplotlib.patches import Rectangle\n\nconfuse_mtx = np.array([[39, 0, 0, 1],\n               [0, 30, 0, 0],\n               [0, 5, 15, 0],\n               [0, 0, 0, 10]])\n\n\n\ndef plot_o_matrix(confuse_matrix, ax):\n    N = confuse_matrix.shape[0]\n    labels = [f'Class{i+1}' for i in range(N)]\n    df = pd.DataFrame(confuse_matrix, index=labels, columns=labels)\n    try:\n        sns.heatmap(df, ax=ax, annot=True, fmt=\"d\", cmap=\"Greens\", cbar=False, linewidths=1.5)\n    except Exception:\n        sns.heatmap(df, ax=ax, annot=True, fmt=\".2f\", cmap=\"Greens\", cbar=False, linewidths=1.5)\n    for i in range(N):\n        for j in range(N):\n            if i != j and confuse_matrix[i, j] > 0:\n                rect = Rectangle((j, i), 1, 1, fill=False, edgecolor='red', lw=2)\n                ax.add_patch(rect)\n\nfig, ax = plt.subplots(1, 1, figsize=(4, 4))\nax.set_title('O-matrix')\nplot_o_matrix(confuse_matrix=confuse_mtx, ax=ax)","metadata":{"execution":{"iopub.status.busy":"2024-09-25T20:05:46.871046Z","iopub.execute_input":"2024-09-25T20:05:46.871439Z","iopub.status.idle":"2024-09-25T20:05:50.917481Z","shell.execute_reply.started":"2024-09-25T20:05:46.871400Z","shell.execute_reply":"2024-09-25T20:05:50.915943Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This matrix shows that all predictions were made correctly except for 5 instances. They were expected to be classified as Class 3 but were instead classified as Class 2. This part should be clear to everyone, as it’s quite straightforward. I'm showing it just for illustration.\n\n## W-matrix: Errors' weights\n\nThe second matrix is more interesting in terms of its meaning. The formula for its calculation is provided in the competition description. Here, I’ll just mention that the further we are from the correct prediction (the larger the difference in class indices), the greater the weight of such an error. This is an excellent way to evaluate errors when the predicted classes indicate some kind of degree or answer the question, 'How strong is our target'?","metadata":{}},{"cell_type":"code","source":"from matplotlib.colors import LinearSegmentedColormap\nfrom matplotlib.collections import LineCollection\n\ndef compute_weight_matrix(N):\n    W = np.zeros((N, N))\n    for i in range(N):\n        for j in range(N):\n            W[i, j] = ((i - j) ** 2) / ((N - 1) ** 2)\n    return W\n\n\ndef plot_weight_matrix(matrix, ax, punish=True, cbar=True):\n    N = matrix.shape[0]\n    labels = [f'Class{i+1}' for i in range(N)]\n    df = pd.DataFrame(matrix, index=labels, columns=labels)\n    cmap = LinearSegmentedColormap.from_list(\"custom\",\n                                             [(0, \"green\" if punish else 'red'),\n                                              (0.0001, \"yellow\"),\n                                              (1, \"red\" if punish else 'green')])\n    sns.heatmap(df, ax=ax, annot=True, fmt=\".3f\", cmap=cmap, cbar=cbar, linewidths=1.5)\n\n\ndef calc_numerator(o_matrix, do_normalize=True):\n    w_matrix = compute_weight_matrix(o_matrix.shape[0])\n    use_o_matrix = o_matrix/np.sum(o_matrix) if do_normalize else o_matrix\n    return np.sum(w_matrix * use_o_matrix)\n\n\ndef calc_denominator(o_matrix, do_normalize=True):\n    w_matrix = compute_weight_matrix(o_matrix.shape[0])\n    e_matrix = compute_expected_matrix(o_matrix)\n    use_e_matrix = e_matrix/np.sum(o_matrix) if do_normalize else e_matrix\n    return np.sum(w_matrix * use_e_matrix)\n\n\ndef _plot_results(val1, val2, desc1, desc2):\n    data = [[desc1, f'{val1:.3f}'],\n            [desc2, f'{val2:.3f}']]\n    fig, ax = plt.subplots(figsize=(5, 1))\n    ax.axis('tight')\n    ax.axis('off')\n    table = ax.table(cellText=data, colLabels=['Description', 'Value'], loc='center')\n    table.auto_set_font_size(False)\n    table.set_fontsize(12)\n    table.scale(1, 1.5)\n    plt.show()\n    \n    \ndef plot_results_numerator(o_matrix):\n    numerator = calc_numerator(o_matrix, do_normalize=False)\n    numerator_norm = calc_numerator(o_matrix, do_normalize=True)\n    _plot_results(numerator, numerator_norm, 'Numerator (abs)', 'Numerator (norm)')\n\n\ndef plot_results_denominator(o_matrix):\n    denominator = calc_denominator(o_matrix, do_normalize=False)\n    denominator_norm = calc_denominator(o_matrix, do_normalize=True)\n    _plot_results(denominator, denominator_norm, 'Denominator (abs)', 'Denominator (norm)')\n    \n\ndef plot_weight_depend(matrix, count_from_class=0, title='Distance cost'):\n    row_values = matrix[count_from_class]\n    n_class = len(row_values)\n    x = np.arange(len(row_values))\n    points = np.array([x, row_values]).T.reshape(-1, 1, 2)\n    segments = np.concatenate([points[:-1], points[1:]], axis=1)\n    cmap = plt.get_cmap('RdYlGn_r')\n    norm = plt.Normalize(0, 1)\n    lc = LineCollection(segments, cmap=cmap, norm=norm)\n    lc.set_array(row_values[:-1])\n    lc.set_linewidth(3)\n    plt.figure(figsize=(8, 6))\n    plt.gca().add_collection(lc)\n    plt.xlim(x.min(), x.max())\n    plt.ylim(0, 1)\n    plt.axvline(x=count_from_class, color='purple', linestyle='--', linewidth=3)\n    plt.xlabel(\"Class position\")\n    plt.ylabel(\"Error weight\")\n    plt.title(f\"{title} (for class {count_from_class+1} of {n_class})\")\n    plt.colorbar(lc)\n    plt.show()\n\n\nplt.close(fig)\nfig, ax = plt.subplots(1, 1, figsize=(5, 4))\nax.set_title('W-matrix')\nplot_weight_matrix(compute_weight_matrix(N=4), ax=ax, cbar=True)","metadata":{"execution":{"iopub.status.busy":"2024-09-25T20:07:06.337599Z","iopub.execute_input":"2024-09-25T20:07:06.338289Z","iopub.status.idle":"2024-09-25T20:07:06.767942Z","shell.execute_reply.started":"2024-09-25T20:07:06.338243Z","shell.execute_reply":"2024-09-25T20:07:06.766788Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This matrix does not depend on the dataset size or the number of predictions. It is determined by the number of classes, which defines its dimensions.\n\n### Playaround: another W-matrix size","metadata":{}},{"cell_type":"code","source":"MATRIX_SIZE = 12\n\nplt.close(fig)\nfig, ax = plt.subplots(1, 1, figsize=(8, 7))\nax.set_title('W-matrix')\nplot_weight_matrix(compute_weight_matrix(N=MATRIX_SIZE), ax=ax, cbar=True)","metadata":{"execution":{"iopub.status.busy":"2024-09-25T20:07:46.284210Z","iopub.execute_input":"2024-09-25T20:07:46.284637Z","iopub.status.idle":"2024-09-25T20:07:47.281291Z","shell.execute_reply.started":"2024-09-25T20:07:46.284598Z","shell.execute_reply":"2024-09-25T20:07:47.280076Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Playaround: class distance - error weight","metadata":{}},{"cell_type":"code","source":"MATRIX_SIZE = 31\n\nw_matrix = compute_weight_matrix(N=MATRIX_SIZE)\nplot_weight_depend(w_matrix, count_from_class=0)","metadata":{"execution":{"iopub.status.busy":"2024-09-25T20:08:31.602647Z","iopub.execute_input":"2024-09-25T20:08:31.603137Z","iopub.status.idle":"2024-09-25T20:08:32.030326Z","shell.execute_reply.started":"2024-09-25T20:08:31.603091Z","shell.execute_reply":"2024-09-25T20:08:32.029071Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"MATRIX_SIZE = 31\nFOR_CLASS = 15\n\nw_matrix = compute_weight_matrix(N=MATRIX_SIZE)\nplot_weight_depend(w_matrix, count_from_class=FOR_CLASS)","metadata":{"execution":{"iopub.status.busy":"2024-09-25T20:08:47.913434Z","iopub.execute_input":"2024-09-25T20:08:47.913870Z","iopub.status.idle":"2024-09-25T20:08:48.340492Z","shell.execute_reply.started":"2024-09-25T20:08:47.913827Z","shell.execute_reply":"2024-09-25T20:08:48.339165Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Numerator\n\nNow that we've seen how the O-matrix and W-matrix are formed, we can proceed to calculate the numerator of the fraction (which we aim to minimize). Let’s look at the initial example:\n\n$$\nnumerator = {\\sum_{i,j} W_{i,j} O_{i,j}}\n$$","metadata":{}},{"cell_type":"code","source":"o_matrix = np.array([[39, 0, 0, 1],\n                        [0, 30, 0, 0],\n                        [0, 5, 15, 0],\n                        [0, 0, 0, 10]])\n\nplt.close(fig)\nfig, ax = plt.subplots(1, 1, figsize=(4, 4))\nax.set_title('O-matrix')\nplot_o_matrix(confuse_matrix=o_matrix, ax=ax)","metadata":{"execution":{"iopub.status.busy":"2024-09-25T20:09:24.133166Z","iopub.execute_input":"2024-09-25T20:09:24.134107Z","iopub.status.idle":"2024-09-25T20:09:24.416356Z","shell.execute_reply.started":"2024-09-25T20:09:24.134055Z","shell.execute_reply":"2024-09-25T20:09:24.415062Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We multiply by the weight matrix:","metadata":{}},{"cell_type":"code","source":"w_matrix = compute_weight_matrix(o_matrix.shape[0])\nplt.close(fig)\nfig, ax = plt.subplots(1, 1, figsize=(5, 4))\nax.set_title('W-matrix')\nplot_weight_matrix(w_matrix, ax=ax, cbar=True)","metadata":{"execution":{"iopub.status.busy":"2024-09-25T20:09:44.892986Z","iopub.execute_input":"2024-09-25T20:09:44.893559Z","iopub.status.idle":"2024-09-25T20:09:45.315400Z","shell.execute_reply.started":"2024-09-25T20:09:44.893501Z","shell.execute_reply":"2024-09-25T20:09:45.313914Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We obtain the product of the matrices. The sum of these elements forms the numerator.","metadata":{}},{"cell_type":"code","source":"plt.close(fig)\nfig, ax = plt.subplots(1, 1, figsize=(5, 4))\nax.set_title('Numerator matrix prod')\nplot_weight_matrix(w_matrix*o_matrix, ax=ax, cbar=True)","metadata":{"execution":{"iopub.status.busy":"2024-09-25T20:10:05.824523Z","iopub.execute_input":"2024-09-25T20:10:05.824950Z","iopub.status.idle":"2024-09-25T20:10:06.248025Z","shell.execute_reply.started":"2024-09-25T20:10:05.824908Z","shell.execute_reply":"2024-09-25T20:10:06.246467Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_results_numerator(o_matrix)","metadata":{"execution":{"iopub.status.busy":"2024-09-25T20:10:19.190588Z","iopub.execute_input":"2024-09-25T20:10:19.191040Z","iopub.status.idle":"2024-09-25T20:10:19.510529Z","shell.execute_reply.started":"2024-09-25T20:10:19.190980Z","shell.execute_reply":"2024-09-25T20:10:19.509061Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Thus, 5 errors in classifying similar classes have less significance than one gross error (note: for calculating the numerator of the expression).","metadata":{}},{"cell_type":"code","source":"def describe_numerator(o_matrix, do_normalize=True):\n    w_matrix = compute_weight_matrix(N=o_matrix.shape[0])\n    fig, axs = plt.subplots(1, 4, figsize=(14, 3.5))\n\n    ax1 = axs[0]\n    ax1.set_title('O-matrix')\n    ax2 = axs[1]\n    ax2.set_title('W-matrix')\n    ax3 = axs[2]\n    ax3.set_title('Prod')\n    ax4 = axs[3]\n    ax4.set_title('Prod(normalized)')\n\n    plot_o_matrix(o_matrix, ax1)\n    plot_weight_matrix(w_matrix, ax2, punish=True, cbar=False)\n    plot_weight_matrix(w_matrix*o_matrix, ax3, punish=True, cbar=False)\n    plot_weight_matrix(w_matrix*o_matrix/np.sum(o_matrix), ax4, punish=True, cbar=False)\n    plt.show()\n\n    plot_results_numerator(o_matrix)","metadata":{"execution":{"iopub.status.busy":"2024-09-25T20:10:48.976409Z","iopub.execute_input":"2024-09-25T20:10:48.976873Z","iopub.status.idle":"2024-09-25T20:10:48.987080Z","shell.execute_reply.started":"2024-09-25T20:10:48.976829Z","shell.execute_reply":"2024-09-25T20:10:48.985586Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Playaround: numerator values","metadata":{}},{"cell_type":"code","source":"o_mtx = np.array([[390, 0, 0, 10],\n                   [0, 300, 0, 0],\n                   [0, 50, 150, 0],\n                   [0, 0, 0, 100]])\ndescribe_numerator(o_mtx)","metadata":{"execution":{"iopub.status.busy":"2024-09-25T20:11:06.021058Z","iopub.execute_input":"2024-09-25T20:11:06.021511Z","iopub.status.idle":"2024-09-25T20:11:07.240141Z","shell.execute_reply.started":"2024-09-25T20:11:06.021461Z","shell.execute_reply":"2024-09-25T20:11:07.238436Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## E-Matrix and the Denominator\n\nIf the numerator generally represents the cost of error for each individual instance among our predictions, then the E-matrix (which is the main factor influencing the denominator) adjusts the result based on expected distributions.\nIn fact, the E-Matrix is formed by the outer product of the actual class distribution and the predicted distribution. The resulting matrix is then divided by the total number of elements in the sample. There are a few ways to normalize, and I propose using normalization at this step.\n\n$$\nE_{i,j} = \\frac{n_{i} m_{j}}{N}\n$$\n\n### Uniform Distribution: Perfect Balance, Perfect Prediction\n\nLet’s examine a few cases for the matrix. First, the matrix for an ideal scenario with a fully balanced dataset, where all predictions are correct:","metadata":{}},{"cell_type":"code","source":"o_matrix = np.array([[100, 0, 0, 0],\n                  [0, 100, 0, 0],\n                  [0, 0, 100, 0],\n                  [0, 0, 0, 100]])\n\n\ndef compute_expected_matrix(O):\n    N = O.shape[0]\n    actual_histogram = O.sum(axis=1)\n    predicted_histogram = O.sum(axis=0)\n    total_samples = O.sum()\n    E = np.outer(actual_histogram, predicted_histogram) / total_samples\n    return E\n\n\ndef visualize_histograms(O, ax):\n    actual_histogram = O.sum(axis=1)\n    predicted_histogram = O.sum(axis=0)\n    labels = [f'Class{i+1}' for i in range(O.shape[0])]\n    x = np.arange(len(labels))\n    width = 0.35\n    ax.bar(x - width/2, actual_histogram, width, \n           label='True', color='blue', alpha=0.7)\n    ax.bar(x + width/2, predicted_histogram, width, \n           label='Predict', color='green', alpha=0.7)\n    ax.set_xticks(x, labels)\n    ax.legend(loc='upper right')\n\n\ndef plot_matrix(matrix, ax, positive_good=True, cbar=True, pivot_zero=False):\n    N = matrix.shape[0]\n    labels = [f'Class{i+1}' for i in range(N)]\n    df = pd.DataFrame(matrix, index=labels, columns=labels)\n    if not positive_good:\n        cmap = LinearSegmentedColormap.from_list(\"inverted\", [\"green\", \"yellow\", \"red\"])\n    else:\n        cmap = LinearSegmentedColormap.from_list(\"normal\", [\"red\", \"yellow\", \"green\"])\n    if pivot_zero:\n        sns.heatmap(df, ax=ax, annot=True, fmt=\".3f\", cmap=cmap, cbar=cbar, linewidths=1.5, center=0)\n    else:\n        sns.heatmap(df, ax=ax, annot=True, fmt=\".3f\", cmap=cmap, cbar=cbar, linewidths=1.5)\n    \n    \ndef describe_e_matrix(o_matrix):\n    e_matrix = compute_expected_matrix(o_matrix)\n    e_matrix_norm = compute_expected_matrix(o_matrix)/np.sum(o_matrix)\n    fig, axs = plt.subplots(1, 4, figsize=(14, 3.5))\n    axs[0].set_title('O-matrix')\n    axs[1].set_title('Class balance')\n    axs[2].set_title('E-matrix')\n    axs[3].set_title('E-matrix (normalized)')\n    plot_o_matrix(o_matrix, axs[0])\n    visualize_histograms(o_matrix, axs[1])\n    plot_matrix(e_matrix, axs[2], positive_good=True, cbar=False)\n    plot_matrix(e_matrix_norm, axs[3], positive_good=True, cbar=False)\n    plt.show()\n\nplt.close(fig)\ndescribe_e_matrix(o_matrix)\ne_matrix = compute_expected_matrix(o_matrix)\nw_matrix = compute_weight_matrix(o_matrix.shape[0])\n\nplot_results_denominator(o_matrix)","metadata":{"execution":{"iopub.status.busy":"2024-09-25T20:12:30.800659Z","iopub.execute_input":"2024-09-25T20:12:30.801888Z","iopub.status.idle":"2024-09-25T20:12:32.155170Z","shell.execute_reply.started":"2024-09-25T20:12:30.801835Z","shell.execute_reply":"2024-09-25T20:12:32.153058Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In reality, it doesn't matter since all our values are located on the diagonal, so they will be multiplied by 0 (the weight matrix) in the numerator. However, this gives an idea of the simplest distribution of the E-matrix values.\n\n### Distribution for Non-balanced Dataset\n\nAnother case is for an unbalanced sample, but still with all correct predictions.","metadata":{}},{"cell_type":"code","source":"o_matrix = np.array([[100, 0, 0, 0],\n                     [0, 75, 0, 0],\n                     [0, 0, 50, 0],\n                     [0, 0, 0, 25]])\nplt.close(fig)\ndescribe_e_matrix(o_matrix)\nplot_results_denominator(o_matrix)","metadata":{"execution":{"iopub.status.busy":"2024-09-25T20:13:12.520211Z","iopub.execute_input":"2024-09-25T20:13:12.520819Z","iopub.status.idle":"2024-09-25T20:13:13.815438Z","shell.execute_reply.started":"2024-09-25T20:13:12.520768Z","shell.execute_reply":"2024-09-25T20:13:13.813415Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Again, this doesn't make much sense since all values will be multiplied by zero in the weight matrix in the numerator.\n\n### Case: Problem with the Most Common Class","metadata":{}},{"cell_type":"code","source":"o_matrix = np.array([[65, 3, 13, 13],\n                     [0, 75, 0, 0],\n                     [0, 0, 50, 0],\n                     [0, 0, 0, 25]])\nplt.close(fig)\ndescribe_e_matrix(o_matrix)\nplot_results_denominator(o_matrix)","metadata":{"execution":{"iopub.status.busy":"2024-09-25T20:13:45.720706Z","iopub.execute_input":"2024-09-25T20:13:45.721180Z","iopub.status.idle":"2024-09-25T20:13:47.035729Z","shell.execute_reply.started":"2024-09-25T20:13:45.721139Z","shell.execute_reply":"2024-09-25T20:13:47.032935Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Observe Changes: Our E-matrix vs. Perfect Predict E-matrix\n\nLet’s look at how the E-matrix changes, considering the difference between perfect prediction and actual results.","metadata":{}},{"cell_type":"code","source":"def create_ideal_matrix(o_matrix):\n    ideal_matrix = np.zeros_like(o_matrix)\n    np.fill_diagonal(ideal_matrix, o_matrix.sum(axis=1))\n    return ideal_matrix\n\n\ndef describe_matrix_diff(mtx_f, mtx_s, \n                         title_f, title_s, title_diff, common_title='',\n                         positive_good=True):\n    fig, axs = plt.subplots(1, 3, figsize=(13, 4))\n    ax1 = axs[0]\n    ax1.set_title(title_f)\n    ax2 = axs[1]\n    ax2.set_title(title_s)\n    ax3 = axs[2]\n    ax3.set_title(title_diff)\n    plot_matrix(mtx_f, ax1, positive_good=positive_good, cbar=False)\n    plot_matrix(mtx_s, ax2, positive_good=positive_good, cbar=False)\n    plot_matrix(mtx_f - mtx_s, ax3, positive_good=positive_good, cbar=False, pivot_zero=True)\n    if common_title:\n        plt.suptitle(common_title, fontsize=16)\n    plt.show()\n\n\ndef describe_e_diff(o_matrix):\n    e_matrix = compute_expected_matrix(o_matrix)/np.sum(o_matrix)\n    o_ideal = create_ideal_matrix(o_matrix)\n    e_ideal = compute_expected_matrix(o_ideal)/np.sum(o_matrix)\n    fig, ax = plt.subplots(1, 3, figsize=(13, 4))\n\n    plot_o_matrix(o_matrix, ax[0])\n    ax[0].set_title('O-matrix')\n    plot_o_matrix(o_matrix/np.sum(o_matrix), ax[1])\n    ax[1].set_title('O-matrix (norm)')\n    visualize_histograms(o_matrix, ax[2])\n    ax[2].set_title('Class balance')\n    plt.suptitle('Input data view (class balance)', fontsize=16)\n    describe_matrix_diff(e_matrix, e_ideal, 'E-matrix', 'E-ideal', 'Diff (E_matrix - E_ideal)', \"Diff for our and ideal cases (both norm)\")\n\ndef describe_ew_prod(o_matrix):\n    e_matrix = compute_expected_matrix(o_matrix)\n    e_matrix_norm = e_matrix/np.sum(o_matrix)\n    w_matrix = compute_weight_matrix(o_matrix.shape[0])\n    prod_matrix = e_matrix_norm * w_matrix\n    fig, ax = plt.subplots(1, 3, figsize=(13, 4))\n    plot_weight_matrix(w_matrix, ax[0], cbar=False, punish=False)\n    ax[0].set_title('W-matrix')\n    plot_matrix(e_matrix_norm, ax[1], cbar=False)\n    ax[1].set_title('E-matrix (norm)')\n    plot_matrix(prod_matrix, ax[2], cbar=False)\n    ax[2].set_title('Prod result')\n    plt.suptitle('Determinator prod matrix', fontsize=16)\n    \n    \n\no_matrix = np.array([[65, 3, 13, 13],\n                     [0, 75, 0, 0],\n                     [0, 0, 50, 0],\n                     [0, 0, 0, 25]])\ndescribe_e_diff(o_matrix)","metadata":{"execution":{"iopub.status.busy":"2024-09-25T20:14:26.091225Z","iopub.execute_input":"2024-09-25T20:14:26.091650Z","iopub.status.idle":"2024-09-25T20:14:27.969835Z","shell.execute_reply.started":"2024-09-25T20:14:26.091611Z","shell.execute_reply":"2024-09-25T20:14:27.967959Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Observe W-matrix Effects","metadata":{}},{"cell_type":"code","source":"describe_ew_prod(o_matrix)\nplot_results_denominator(o_matrix)","metadata":{"execution":{"iopub.status.busy":"2024-09-25T20:14:44.503124Z","iopub.execute_input":"2024-09-25T20:14:44.503562Z","iopub.status.idle":"2024-09-25T20:14:45.377706Z","shell.execute_reply.started":"2024-09-25T20:14:44.503510Z","shell.execute_reply":"2024-09-25T20:14:45.376025Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Case: Problem with a Rare Class","metadata":{}},{"cell_type":"code","source":"o_matrix = np.array([[100, 0, 0, 0],\n                     [0, 75, 0, 0],\n                     [0, 0, 50, 0],\n                     [5, 5, 5, 10]])\ndescribe_e_diff(o_matrix)\ndescribe_ew_prod(o_matrix)\nplot_results_denominator(o_matrix)","metadata":{"execution":{"iopub.status.busy":"2024-09-25T20:15:05.354159Z","iopub.execute_input":"2024-09-25T20:15:05.354656Z","iopub.status.idle":"2024-09-25T20:15:07.737404Z","shell.execute_reply.started":"2024-09-25T20:15:05.354603Z","shell.execute_reply":"2024-09-25T20:15:07.735420Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Playaround: Denominator Matrices","metadata":{}},{"cell_type":"code","source":"# o_matrix = np.array([[100, 0, 0, 0],\n#                      [0, 75, 0, 0],\n#                      [0, 0, 50, 0],\n#                      [0, 0, 0, 5]])\n# describe_e_diff(o_matrix)\n# describe_ew_prod(o_matrix)\n# plot_results_denominator(o_matrix)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Putting It All Together\n\nHere, we present various prediction outcomes and a detailed calculation of the formula's components.\n\n### Demo datasets presets","metadata":{}},{"cell_type":"code","source":"examples = [\n    {\n    'title': 'Uniform, Balanced, Perfect predict',\n    'o_matrix': np.array([[100, 0, 0, 0],\n                          [0, 100, 0, 0],\n                          [0, 0, 100, 0],\n                          [0, 0, 0, 100]])\n    },\n    {\n    'title': 'Balanced, Random Prediction',\n    'o_matrix': np.array([[25, 25, 25, 25],\n                          [25, 25, 25, 25],\n                          [25, 25, 25, 25],\n                          [25, 25, 25, 25]])        \n    },\n    {\n    'title': 'Non-Balanced, Perfect Prediction',\n    'o_matrix': np.array([[400, 0, 0, 0],\n                          [0, 300, 0, 0],\n                          [0, 0, 200, 0],\n                          [0, 0, 0, 100]])\n    },\n    {\n    'title': 'Non-Balanced, Preserved Class Distribution, Random Prediction',\n    'o_matrix': np.array([[100, 100, 100, 100],\n                          [75, 75, 75, 75],\n                          [50, 50, 50, 50],\n                          [25, 25, 25, 25]])  \n    },\n    {\n        'title': 'Preference for Middle Classes',\n        'o_matrix': np.array([[200, 200, 0, 0],\n                              [0, 200, 100, 0],\n                              [0, 50, 150, 0],\n                              [0, 0, 50, 50]])\n    },\n    {\n        'title': 'Predicting Near Borders',\n        'o_matrix': np.array([[400, 0, 0, 0],\n                              [200, 100, 0, 0],\n                              [0, 0, 65, 135],\n                              [0, 0, 0, 50]])\n    },\n    {\n        'title': 'Rare Class Bad Prediction',\n        'o_matrix': np.array([[400, 0, 0, 0],\n                              [0, 300, 0, 0],\n                              [0, 0, 200, 0],\n                              [10, 15, 25, 50]])\n    },\n    {\n        'title': 'Common Class Bad Prediction',\n        'o_matrix': np.array([[200, 100, 60, 40],\n                              [0, 300, 0, 0],\n                              [0, 0, 200, 0],\n                              [0, 0, 0, 100]])\n    },\n    {\n        'title': 'Common Class Large Index Error',\n        'o_matrix': np.array([[200, 25, 25, 150],\n                              [0, 300, 0, 0],\n                              [0, 0, 200, 0],\n                              [0, 0, 0, 100]])\n    },\n    {\n        'title': 'Common Class Small Index Error',\n        'o_matrix': np.array([[200, 150, 25, 25],\n                              [0, 300, 0, 0],\n                              [0, 0, 200, 0],\n                              [0, 0, 0, 100]])\n    },\n    {\n        'title': 'Rare Class Large Index Error',\n        'o_matrix': np.array([[400, 0, 0, 0],\n                              [0, 300, 0, 0],\n                              [0, 0, 200, 0],\n                              [40, 5, 5, 50]])\n    },\n    {\n        'title': 'Rare Class Small Index Error',\n        'o_matrix': np.array([[400, 0, 0, 0],\n                              [0, 300, 0, 0],\n                              [0, 0, 200, 0],\n                              [5, 5, 40, 50]])\n    },\n    {\n        'title': 'Some Errors in All Classes',\n        'o_matrix': np.array([[300, 50, 30, 20],\n                              [30, 225, 30, 15],\n                              [10, 20, 150, 20],\n                              [2, 5, 5, 38]])\n    }\n]","metadata":{"execution":{"iopub.status.busy":"2024-09-25T20:17:04.486637Z","iopub.execute_input":"2024-09-25T20:17:04.487075Z","iopub.status.idle":"2024-09-25T20:17:04.509074Z","shell.execute_reply.started":"2024-09-25T20:17:04.487035Z","shell.execute_reply":"2024-09-25T20:17:04.507751Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def _report_overview(o_matrix):\n    plt.close()\n    fig, ax = plt.subplots(1, 3, figsize=(10.5, 4), gridspec_kw={'width_ratios': [1, 2, 2]})\n    ax[0].axis('off')\n    plot_o_matrix(o_matrix, ax[1])\n    ax[1].set_title('O-matrix')\n    visualize_histograms(o_matrix, ax[2])\n    ax[2].set_title('Class balance')\n    plt.suptitle('Input data view (class balance)', fontsize=16)\n\n\ndef _report_numerator(o_matrix):\n    w_matrix = compute_weight_matrix(o_matrix.shape[0])\n    o_matrix_norm = o_matrix/np.sum(o_matrix)\n    fig, ax = plt.subplots(1, 3, figsize=(13, 4))\n    plot_o_matrix(o_matrix_norm, ax[0])\n    ax[0].set_title('O-matrix (norm)')\n    plot_weight_matrix(w_matrix, ax[1], punish=True, cbar=False)\n    ax[1].set_title('W-matrix')\n    plot_weight_matrix(w_matrix * o_matrix_norm, ax[2], punish=True, cbar=False)\n    ax[2].set_title('Numerator matrix')\n    plt.suptitle('Numerator', fontsize=16)\n\n\ndef _report_denominator(o_matrix):\n    w_matrix = compute_weight_matrix(o_matrix.shape[0])\n    e_matrix = compute_expected_matrix(o_matrix)/np.sum(o_matrix)\n    o_ideal = create_ideal_matrix(o_matrix)\n    e_ideal = compute_expected_matrix(o_ideal)/np.sum(o_matrix)\n    fig, ax = plt.subplots(1, 3, figsize=(13, 4))\n    plot_matrix(e_matrix, ax[0], cbar=False, pivot_zero=False, positive_good=True)\n    ax[0].set_title('E-matrix (norm)')\n    plot_weight_matrix(w_matrix, ax[1], punish=False, cbar=False)\n    ax[1].set_title('W-matrix')\n    plot_matrix(w_matrix * e_matrix, ax[2], cbar=False, pivot_zero=True, positive_good=True)\n    ax[2].set_title('Denominator matrix')\n    plt.suptitle('Denominator', fontsize=16)\n\n\ndef _report_calculation(o_matrix):\n    plot_results_numerator(o_matrix)\n    plot_results_denominator(o_matrix)\n    \n    d = calc_denominator(o_matrix, do_normalize=False)\n    d_norm = calc_denominator(o_matrix, do_normalize=True)\n    n = calc_numerator(o_matrix, do_normalize=False)\n    n_norm = calc_numerator(o_matrix, do_normalize=True)\n \n    _plot_results(1-(n/d), 1-(n_norm/d_norm), 'QWK (abs way)', 'QWK (norm way)')\n    \n\n\ndef report(title, o_matrix):\n    plt.figure(figsize=(6, 2))\n    plt.axis('off')  \n    plt.text(0.5, 0.5, title, fontsize=24, color='purple', ha='center', va='center') \n    plt.show()\n\n    _report_overview(o_matrix)\n    _report_numerator(o_matrix)\n    _report_denominator(o_matrix)\n    _report_calculation(o_matrix)\n    \n\nfor item in examples:\n    o_matrix = item['o_matrix']\n    title = item['title']\n    report(title, o_matrix)","metadata":{"execution":{"iopub.status.busy":"2024-09-25T20:17:36.519714Z","iopub.execute_input":"2024-09-25T20:17:36.520181Z","iopub.status.idle":"2024-09-25T20:18:10.313868Z","shell.execute_reply.started":"2024-09-25T20:17:36.520135Z","shell.execute_reply":"2024-09-25T20:18:10.312012Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Playaround: your matrix demo","metadata":{}},{"cell_type":"code","source":"# o_matrix = np.array([])\n# report('Demo', o_matrix)","metadata":{"execution":{"iopub.status.busy":"2024-09-25T20:18:42.129150Z","iopub.execute_input":"2024-09-25T20:18:42.129585Z","iopub.status.idle":"2024-09-25T20:18:42.134646Z","shell.execute_reply.started":"2024-09-25T20:18:42.129543Z","shell.execute_reply":"2024-09-25T20:18:42.133556Z"},"trusted":true},"execution_count":null,"outputs":[]}]}