\n
\n
🍂Ensembling & Correlation
\n
\n By: Somayyeh Gholami & Mehran Kazeminia\n
\n
\n \n##
Ensembling with Correlation Guidance - ECG
","metadata":{}},{"cell_type":"code","source":"from IPython.display import HTML\nimport time\n\nhandle = display(HTML(\"\"\"
\"\"\"), display_id='html_marquee1')\ntime.sleep(2)\nhandle = display(HTML(\"\"\"
\"\"\"), display_id='html_marquee1', update=True)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-12-28T23:02:41.374654Z","iopub.execute_input":"2023-12-28T23:02:41.375079Z","iopub.status.idle":"2023-12-28T23:02:43.3875Z","shell.execute_reply.started":"2023-12-28T23:02:41.375046Z","shell.execute_reply":"2023-12-28T23:02:43.386275Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n#
\n\n- Validation data is used for tuning hyperparameters, determining coefficients for Ensembling, etc. But in many cases (such as Kaggle challenges, etc.) only the results and their public scores are known, and the determination of Ensembling coefficients is practically **done in the dark**.\n\n- The participants of the challenges, in order to get the best score, Ensembling the results of their notebooks or others and usually use the trial and error method to determine the best coefficients.\n\n- But Ensembling the best scores is not always successful. That is, in most cases, only by Ensembling some results, the score will improve. Also, the only option for most participants to choose the right results for Ensembling is the trial and error method.\n\n- When the number of answer columns is more than one, the darkness increases. Because it is not known that after finding the right coefficient for Ensembling the first columns, the same coefficient is optimal for Ensembling the next columns.\n\n- Recently, the [Open Problems - Single-Cell Perturbations challenge](https://www.kaggle.com/competitions/open-problems-single-cell-perturbations/overview) was held in Kaggle, which includes the prediction of more than eighteen thousand columns. That is, the issue of Ensembling in these cases becomes very complicated, because if a coefficient is chosen for Ensembling the first column, it cannot be sure that the same coefficient is optimal for more than eighteen thousand other Ensembling.\n\n- In this notebook, we want to share our experience of **using Correlation** with you. Correlation of projected columns, when you want to Ensembling two columns together, like a candle in the dark can help you reach your destination.\n\n- For example, when the correlation between column A and column B is negative, Ensembling should not be performed on these two columns. That is, if the public score for result A is better than the public score for result B, the first choice is result A alone and the second choice is result B alone, and no linear combination between these two columns can be good.\n\n- It is obvious that when the correlation of column A and column B is a positive number, it is still necessary to see how far the correlation value is from zero and how close it is to one. In addition, it should be seen how much the public score of result A is better than the public score for result B. For example, if the correlation is very close to one and the public score of result A is much better, result A alone is a good option and Ensembling cannot help in improving the score.\n\n- In this notebook, we try to clarify the issue with some examples. You will see in the examples that even within a fixed notebook netbook, some prediction columns can be very good and some can be really bad, but the public score we see is the score of all the good and bad columns. Of course, finding these good and bad columns is done by calculating the correlation of similar columns in two separate notebooks.\n\n- It should be noted that Ensembling should not increase the difference between public score and private score. That is, so to speak, the model should not become unstable and overfitting should occur. For example, using the results of notebooks that are Ensembling products, usually increase this risk. We will also explore and explain this issue in the notebook.\n\n- Please note that in many cases Ensembling is complex and there is no unique answer to all challenges. Of course, when there are thousands of prediction columns, using correlation seems necessary, but we have also used other methods in different challenges, which have been effective, and we will mention them below:\n\n- We previously used Comparative Method and Snap to Grid in the Indoor Location & Navigation challenge:\n\n> https://www.kaggle.com/code/mehrankazeminia/1-3-indoor-navigation-cost-minimization-floor/notebook\n\n> https://www.kaggle.com/code/mehrankazeminia/2-3-indoor-navigation-comparative-method\n\n> https://www.kaggle.com/code/mehrankazeminia/3-3-g6-snap-to-grid-fix-the-timestamps\n\n- In the Tabular Playground Series challenge - Jul 2021, we used the Smart Ensembling method:\n\n> https://www.kaggle.com/code/mehrankazeminia/2-tps-jul-21-smart-ensembling\n\n- In the Tabular Playground Series - Jul 2022 challenge, we used the Clustering-Ensembling method:\n\n> https://www.kaggle.com/code/mehrankazeminia/3-3-tps22jul-clustering-ensembling\n\n- If these notebooks appeal to you, please don't forget to **upvote** them.\n","metadata":{}},{"cell_type":"markdown","source":"
","metadata":{}},{"cell_type":"code","source":"#:::::::::::::::::::::::::::::::::::\nimport warnings # suppress warnings\nwarnings.filterwarnings('ignore')\n#:::::::::::::::::::::::::::::::::::\nimport os\nimport gc\nimport glob\nimport random\nimport numpy as np \nimport pandas as pd\nimport seaborn as sns\nfrom tqdm import tqdm\nfrom scipy import stats\nfrom pathlib import Path\nfrom itertools import groupby\n#:::::::::::::::::::::::::::::::::::\nimport matplotlib.pyplot as plt\nimport plotly.figure_factory as ff\nimport plotly.express as px\n%matplotlib inline\n#:::::::::::::::::::::::::::::::::::\n!ls ../input/*","metadata":{"_kg_hide-input":false,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-12-28T23:02:43.390068Z","iopub.execute_input":"2023-12-28T23:02:43.390485Z","iopub.status.idle":"2023-12-28T23:02:44.479249Z","shell.execute_reply.started":"2023-12-28T23:02:43.390446Z","shell.execute_reply":"2023-12-28T23:02:44.47813Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_submission = pd.read_csv('../input/open-problems-single-cell-perturbations/sample_submission.csv', index_col='id')","metadata":{"execution":{"iopub.status.busy":"2023-12-28T23:02:44.480753Z","iopub.execute_input":"2023-12-28T23:02:44.481116Z","iopub.status.idle":"2023-12-28T23:02:49.069907Z","shell.execute_reply.started":"2023-12-28T23:02:44.481084Z","shell.execute_reply":"2023-12-28T23:02:49.068686Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##
Auxiliary Functions\n\n
","metadata":{}},{"cell_type":"code","source":"# :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 1\ndef df_creator(dfx, dfy, n):\n df_sub = pd.DataFrame(columns=['dfx','dfy'])\n \n df_sub['dfx'] = dfx.iloc[:, n].copy()\n df_sub['dfy'] = dfy.iloc[:, n].copy()\n \n return df_sub \n\n# :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 2 \ndef df_corr(df_sub): \n corr = df_sub.corr(numeric_only=True).round(3) \n \n corr_list = list(corr.iloc[0])[1:]\n return corr_list\n\n# :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 3\ndef heatmap(dfx, dfy):\n N = random.randrange(dfx.shape[1])\n \n print('\\n\\nHeatmap (For a random column)')\n print(':' *40)\n print('Column Name :', list(dfx.columns)[N])\n print('Column Number :', N)\n print(':' *40, '\\n')\n \n df_sub = df_creator(dfx, dfy, N)\n corr_matrix = df_sub.corr()\n fig = plt.figure(figsize=(4,3));\n\n cmap=sns.diverging_palette(240, 10, s=75, l=50, sep=1, n=6, center='light', as_cmap=False);\n sns.heatmap(corr_matrix, center=0, annot=True, cmap=cmap, linewidths=2);\n plt.suptitle(f'Heatmap (N={N})', y=0.95, fontsize=12, c='darkred');\n plt.show()\n\n# :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 4\ndef ensembling_histograms(submission, dfx, dfy):\n N = random.randrange(dfx.shape[1])\n \n print('\\n\\nEnsembling Histograms (For a random column)')\n print(':' *50)\n print('Column Name :', list(dfx.columns)[N])\n print('Column Number :', N)\n print(':' *50)\n \n hist_data = [submission.iloc[:, N], dfx.iloc[:, N], dfy.iloc[:, N]]\n group_labels = ['Generated', 'First Results (Main)', 'Second Results (Support)']\n \n fig = ff.create_distplot(hist_data, group_labels, bin_size=.2, show_hist=False, show_rug=False)\n fig.show() \n\n# :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 5\ndef info_corr(corr_limit, counter_x, counter_y, dfx, dfy):\n counter_z = dfx.shape[1] - (counter_x + counter_y)\n \n print('\\nCorrelation information (For all columns)')\n print(':' *70)\n print('A Percent =', round(counter_y / dfx.shape[1], 3)) \n print('A: Correlation less than Zero')\n print('The number of columns evaluated with only the second result =', counter_y) \n print('- ' *24)\n print('B Percent =', round(counter_z / dfx.shape[1], 3)) \n print('B: Correlation more than Zero and less than corr_limit')\n print('The number of columns evaluated by Ensembling =', counter_z)\n print('- ' *24)\n print('C Percent =', round(counter_x / dfx.shape[1], 3))\n print('C: Correlation more than corr_limit')\n print('The number of columns evaluated with only the first result =', counter_x)\n print('- ' *24) \n print('The correlation limit that was considered as the basis =', corr_limit)\n print(':' *70 ,'\\n\\n')\n \n columns = ['A: Correlation less than Zero','B: Correlation more than Zero and less than corr_limit','C: Correlation more than corr_limit']\n data = [[ round(counter_y / dfx.shape[1], 3), round(counter_z / dfx.shape[1], 3), round(counter_x / dfx.shape[1], 3)]]\n de_data = pd.DataFrame(data=data , columns=columns)\n \n sns.set()\n de_data.plot(kind='barh', stacked=True, figsize=(10,1), color=['pink','violet','purple'])\n plt.gca().set_facecolor('lightyellow')\n plt.legend(fontsize=10, loc=3, bbox_to_anchor=(0, 1))\n plt.show() \n\n# :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 6\ndef ensembling_scatter(submission, dfx, dfy): \n \n X = dfx # main\n Y1 = dfy # support\n Y2 = submission\n \n sns.set()\n plt.style.use('seaborn-whitegrid') \n plt.figure(figsize=(12, 6), facecolor='lightyellow')\n plt.title('Scatter Graph (For all columns)\\n', fontsize=12) \n\n plt.scatter(X, Y1, s=2.5, label='dfy - Support', c='darkcyan') \n plt.scatter(X, Y2, s=2.5, label='Generated', c='red')\n plt.scatter(X, X , s=4.0, label='dfx - Main(X=Y)', c='orange')\n \n plt.gca().set_facecolor('lightgreen')\n plt.legend(fontsize=10, loc=4)\n # plt.savefig('scatter101.png')\n plt.show() \n\n# :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 7\ndef plot_corr(tcorr): \n display(tcorr.iloc[:, :4].style.background_gradient(cmap='Pastel1', axis=None, vmin=0, vmax=1.0))\n\n sns.set()\n plt.style.use('seaborn-whitegrid') \n plt.figure(figsize=(12, 5), facecolor='lightyellow')\n plt.title('Percentage of States\\n', fontsize=12) \n plt.xlabel('Correlation Limit')\n plt.xticks(range(11), round(tcorr.iloc[:,0], 1))\n\n plt.plot(tcorr.iloc[:,1], color='orange', lw=2, label='A: Correlation less than Zero')\n plt.plot(tcorr.iloc[:,2], color='darkcyan', lw=2, label='B: Correlation more than Zero and less than corr_limit')\n plt.plot(tcorr.iloc[:,3], color='red', lw=2, label='C: Correlation more than corr_limit')\n \n plt.gca().set_facecolor('lightgreen')\n plt.legend(fontsize=10, loc=0)\n # plt.savefig('plot101.png')\n plt.show() \n \n# ::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::","metadata":{"execution":{"iopub.status.busy":"2023-12-28T23:02:49.144326Z","iopub.execute_input":"2023-12-28T23:02:49.14461Z","iopub.status.idle":"2023-12-28T23:02:49.153751Z","shell.execute_reply.started":"2023-12-28T23:02:49.144586Z","shell.execute_reply":"2023-12-28T23:02:49.152775Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##
Ensembling & Tuning Functions\n\n
","metadata":{}},{"cell_type":"markdown","source":"#
(1) generate_corr_coeff (dfx, dfy, corr_limit, coeff)
\n\n- This function takes the regression results and prediction of two notebooks, each of these results can have thousands of columns. The first result (dfx) is called \"Main\" and the second result (dfy) is called \"Support\".\n\n- This function ignores the column corresponding to \"Main\" when the correlation of two similar columns in \"Main\" and \"Support\" becomes negative. We can also set \"corr_limit\" to be less than one (and greater than zero). When the correlation of two similar columns in \"Main\" and \"Support\" is greater than \"corr_limit\", this function ignores the column corresponding to \"Support\".\n\n- This function performs Ensembling with a coefficient that we define as \"coeff\", only for columns whose correlation is between zero and \"corr_limit\".\n\n- It is obvious that if we change the places of \"Main\" and \"Support\", the result of Ensembling will change and the score may be better with the new setting of \"corr_limit\" and \"coeff\".","metadata":{}},{"cell_type":"code","source":"# Ensemble for two results by determining \"corr_limit @ coeff\"\ndef generate_corr_coeff(dfx, dfy, corr_limit, coeff):\n submission = sample_submission.copy()\n \n counter_x = 0\n counter_y = 0\n for n in range(dfx.shape[1]): \n df_sub = df_creator(dfx, dfy, n)\n corr_list = df_corr(df_sub)\n \n submission.iloc[:, n] = (dfx.iloc[:, n] * coeff) + (dfy.iloc[:, n] * (1.- coeff))\n \n if (corr_list[0] > corr_limit): \n submission.iloc[:, n] = dfx.iloc[:, n]\n counter_x += 1 \n \n if (corr_list[0] < 0):\n submission.iloc[:, n] = dfy.iloc[:, n]\n counter_y += 1\n \n print('\\n\\n', ':. ' *12, 'Ensembling (Different coefficients for different columns)', '.: ' *12)\n \n # heatmap(dfx, dfy)\n # ensembling_histograms(submission, dfx, dfy)\n \n info_corr(corr_limit, counter_x, counter_y, dfx, dfy)\n ensembling_scatter(submission, dfx, dfy)\n \n # display(submission)\n return submission","metadata":{"execution":{"iopub.status.busy":"2023-12-28T23:02:49.154914Z","iopub.execute_input":"2023-12-28T23:02:49.155646Z","iopub.status.idle":"2023-12-28T23:02:49.164643Z","shell.execute_reply.started":"2023-12-28T23:02:49.155616Z","shell.execute_reply":"2023-12-28T23:02:49.163583Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#
(2) generate_corr (dfx, dfy, corr_limit)
\n\n- This function is very similar to the previous function, except that you do not need to specify the \"coeff\". This function calculates a coefficient for Ensembling for both columns whose correlation is between zero and \"corr_limit\". That is, it considers \"coeff\" equal to the correlation value of two columns.\n\n- Certainly, the results of this function are not as good as the first function, but trying this function can be a good image to determine \"coeff\" and \"corr_limit\".","metadata":{}},{"cell_type":"code","source":"# Ensemble for two results by determining \"corr_limit\"\ndef generate_corr(dfx, dfy, corr_limit):\n submission = sample_submission.copy()\n \n counter_x = 0\n counter_y = 0\n for n in range(dfx.shape[1]): \n df_sub = df_creator(dfx, dfy, n)\n corr_list = df_corr(df_sub)\n \n submission.iloc[:, n] = (dfx.iloc[:, n] * corr_list[0]) + (dfy.iloc[:, n] * (1.- corr_list[0]))\n \n if (corr_list[0] > corr_limit): \n submission.iloc[:, n] = dfx.iloc[:, n]\n counter_x += 1 \n \n if (corr_list[0] < 0):\n submission.iloc[:, n] = dfy.iloc[:, n]\n counter_y += 1\n \n print('\\n\\n', ':. ' *12, 'Ensembling (Different coefficients for different columns)', '.: ' *12)\n \n # heatmap(dfx, dfy)\n # ensembling_histograms(submission, dfx, dfy)\n \n info_corr(corr_limit, counter_x, counter_y, dfx, dfy)\n ensembling_scatter(submission, dfx, dfy)\n \n # display(submission)\n return submission","metadata":{"execution":{"iopub.status.busy":"2023-12-28T23:02:49.165884Z","iopub.execute_input":"2023-12-28T23:02:49.166275Z","iopub.status.idle":"2023-12-28T23:02:49.177686Z","shell.execute_reply.started":"2023-12-28T23:02:49.166244Z","shell.execute_reply":"2023-12-28T23:02:49.17676Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#
(3) tuning_corr_limit (dfx, dfy)
\n\n- Before setting the parameters of the above functions, you can use the following function to see the correlation status of \"Main\" and \"Support\" in the table and graph.\n\n- If the number of prediction columns is thousands of numbers, this function must perform a lot of calculations and the execution of this function takes time.","metadata":{}},{"cell_type":"code","source":"def tuning_corr_limit(dfx, dfy): \n '''\n A : Less than Zero\n B : More than Zero and less than corr_limit\n C : more than corr_limit\n '''\n cname = ['corr_limit','Percentage of A','Percentage of B','Percentage of C','The number of A','The number of B','The number of C']\n \n clist = np.arange(0, 1.1, 0.1)\n cdata = np.zeros((len(clist), 7), dtype=int)\n tcorr = pd.DataFrame(data=cdata, columns=cname)\n \n for c in range(len(clist)):\n counter_a = 0\n counter_b = 0\n counter_c = 0\n \n for n in range(dfx.shape[1]): \n df_sub = df_creator(dfx, dfy, n)\n corr_list = df_corr(df_sub)\n \n if (corr_list[0] < 0): \n counter_a += 1 \n if (corr_list[0] > 0) and (corr_list[0] < clist[c]):\n counter_b += 1 \n if (corr_list[0] > clist[c]):\n counter_c += 1\n \n tcorr.iloc[c,0] = clist[c]\n tcorr.iloc[c,1] = round(counter_a / dfx.shape[1], 3)\n tcorr.iloc[c,2] = round(counter_b / dfx.shape[1], 3)\n tcorr.iloc[c,3] = round(counter_c / dfx.shape[1], 3)\n tcorr.iloc[c,4] = counter_a\n tcorr.iloc[c,5] = counter_b\n tcorr.iloc[c,6] = counter_c\n \n plot_corr(tcorr)\n display(tcorr)\n return tcorr","metadata":{"execution":{"iopub.status.busy":"2023-12-28T23:02:49.199781Z","iopub.execute_input":"2023-12-28T23:02:49.200173Z","iopub.status.idle":"2023-12-28T23:02:49.213418Z","shell.execute_reply.started":"2023-12-28T23:02:49.200137Z","shell.execute_reply":"2023-12-28T23:02:49.212671Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##
Ensembling with Correlation Guidance - ECG\n\n
\n
\n
\n\n#
EXAMPLE : 1 - Chain Ensembling
\n\n
\n\n- For the first example, we Ensembling the results of four notebooks in a row. We create the so-called \"Ensembling Chain\". The first one is our own notebook where we have done \"Feature Augmentation\". The following results are before any Ensembling.\n\n- https://www.kaggle.com/code/mehrankazeminia/80-3-op2-feature-augmentation/notebook\n\n-----\n\n","metadata":{}},{"cell_type":"code","source":"# Feature Augmentation\nsub_606 = pd.read_csv('../input/80-3-op2-feature-augmentation/prediction.csv', index_col='id')\n# ............................................................................................\n# Public Score: 0.606\n# Private Score: 0.809","metadata":{"execution":{"iopub.status.busy":"2023-12-28T23:02:49.214383Z","iopub.execute_input":"2023-12-28T23:02:49.214703Z","iopub.status.idle":"2023-12-28T23:02:54.384099Z","shell.execute_reply.started":"2023-12-28T23:02:49.214677Z","shell.execute_reply":"2023-12-28T23:02:54.382939Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n- The next notebook has effectively used \"Neural Network\".\n\n- https://www.kaggle.com/code/kishanvavdara/neural-network-regression\n\n-----\n\n\n","metadata":{}},{"cell_type":"code","source":"# Thanks to: @kishanvavdara - Neural Network\nsub_604 = pd.read_csv('../input/op2-604/submission_df.csv', index_col='id')\n# ..........................................................................\n# Public Score: 0.604\n# Private Score: 0.824","metadata":{"execution":{"iopub.status.busy":"2023-12-28T23:02:54.385886Z","iopub.execute_input":"2023-12-28T23:02:54.386342Z","iopub.status.idle":"2023-12-28T23:02:59.247289Z","shell.execute_reply.started":"2023-12-28T23:02:54.386302Z","shell.execute_reply":"2023-12-28T23:02:59.246182Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n- The next notebook has effectively used NLP (SMILES Embedding).\n\n- https://www.kaggle.com/code/kishanvavdara/nlp-regression?scriptVersionId=146928847\n\n-----\n\n","metadata":{}},{"cell_type":"code","source":"# Thanks to: @kishanvavdara - NLP(SMILES Embedding)\nsub_607 = pd.read_csv('../input/op2-607/OP2_607.csv', index_col='id')\n# ....................................................................\n# Public Score: 0.607\n# Private Score: 0.813","metadata":{"execution":{"iopub.status.busy":"2023-12-28T23:02:59.248855Z","iopub.execute_input":"2023-12-28T23:02:59.249156Z","iopub.status.idle":"2023-12-28T23:03:04.171892Z","shell.execute_reply.started":"2023-12-28T23:02:59.24913Z","shell.execute_reply":"2023-12-28T23:03:04.170897Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n- Before starting any Ensembling for parameter tuning, we can compare the correlation values of all the columns of the two results and see the details in the table and graph. (This function takes a long time)","metadata":{}},{"cell_type":"code","source":"# :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 1a\ntuning_corr_limit(sub_604, sub_607)\ngen_1a = generate_corr_coeff(sub_604, sub_607, 0.95, 0.50)\ngen_1a.to_csv('gen_1a.csv')\n# Public Score: 0.595\n# Private Score: 0.808\n\n# Classic method of Ensembling - 1a\nens_1a = (sub_604 *0.50) + (sub_607 *0.50)\nens_1a.to_csv('ens_1a.csv')\n# Public Score: 0.596\n# Private Score: 0.808\n# :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 1b\ntuning_corr_limit(gen_1a, sub_606)\ngen_1b = generate_corr_coeff(gen_1a, sub_606, 0.85, 0.60)\ngen_1b.to_csv('gen_1b.csv')\n# Public Score: 0.589\n# Private Score: 0.796\n\n# Classic method of Ensembling - 1b\nens_1b = (gen_1a *0.60) + (sub_606 *0.40)\nens_1b.to_csv('ens_1b.csv')\n# Public Score: 0.590\n# Private Score: 0.797\n# ::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::","metadata":{"execution":{"iopub.status.busy":"2023-12-28T23:03:04.173341Z","iopub.execute_input":"2023-12-28T23:03:04.173782Z","iopub.status.idle":"2023-12-28T23:03:04.179948Z","shell.execute_reply.started":"2023-12-28T23:03:04.173743Z","shell.execute_reply":"2023-12-28T23:03:04.178826Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n### Explanations for Example : 1\n\n- Only the correlation of 3.7% of sub_604 and sub_607 columns is negative, maybe because their scores are close to each other. However, if these columns are ignored during Ensembling, the Ensembling score will be better. Note that the values of 3.7% of the columns must be extracted from only one of the results. In the function we wrote, these values are extracted from the second result in the function, the sub_607 result. In cases where you see a negative correlation, you should definitely try which result should be written first and which result should be written second. If the number of columns with negative correlation is high, removing them from Ensembling is very important and will have a great impact on the final score. As mentioned earlier, **two columns with negative correlation have no chance for Ensembling**.\n\n- To calculate gen_1a, the value of 5% of the columns with a correlation close to one was ignored, and for this reason, the value of corr_limit is considered equal to 0.95. This 5% is extracted from the sub_604 result. But for the calculation of gen_1b, 15% of the columns with a correlation close to one were ignored, and for this reason, the value of corr_limit is considered equal to 0.85. This 15% is extracted from the gen_1a result.\n\n- The coeff value for gen_1a calculation is equal to 0.50 and the coeff value for gen_1b calculation is equal to 0.60. Meanwhile, Ensembling has been done separately in the classical way, with exactly the same coefficients with the names ens_1a and ens_1b, so that there is a possibility of comparison.\n","metadata":{}},{"cell_type":"markdown","source":"
\n
\n
\n\n#
EXAMPLE : 2 - Ensembling for challenge winners
\n\n
\n\n- The [\"Open Problems - Single-Cell Perturbations\"](https://www.kaggle.com/competitions/open-problems-single-cell-perturbations/overview) challenge has ended and the winners of this challenge were determined based on the best private score. For this reason, for the second example, we use the results of the competition winners and try to optimize the private score by Ensembling.\n\n- https://www.kaggle.com/competitions/open-problems-single-cell-perturbations/leaderboard\n\n-----\n\n","metadata":{}},{"cell_type":"markdown","source":"
\n\n- The results of the first and second teams have not been published or we could not find them. So we will check the results of the third and fourth teams.\n\n- https://www.kaggle.com/code/jankowalski2000/3rd-place-solution\n\n-----\n\n","metadata":{}},{"cell_type":"code","source":"# Thanks to: @jankowalski2000 (3rd place)\nsub_3rd = pd.read_csv('/kaggle/input/op2-548-3rd/OP2_548_3rd.csv', index_col='id')\n# ................................................................................\n# Private Score: 0.732","metadata":{"execution":{"iopub.status.busy":"2023-12-28T23:03:04.181475Z","iopub.execute_input":"2023-12-28T23:03:04.181909Z","iopub.status.idle":"2023-12-28T23:03:09.317828Z","shell.execute_reply.started":"2023-12-28T23:03:04.181871Z","shell.execute_reply":"2023-12-28T23:03:09.316804Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n- We noticed that after the end of the \"4th Place\" contest, he published a better version and a better private score than his Leaderboard score. So we also use their final version.\n\n- https://www.kaggle.com/code/raki21/4th-place-magic-postprocessing?scriptVersionId=153284875\n\n-----\n\n","metadata":{}},{"cell_type":"code","source":"# Thanks to: @raki21 (4th Place)\nsub_4th = pd.read_csv('../input/op2-565-4th/OP2_565_4th.csv', index_col='id')\n# ............................................................................\n# Private Score: 0.712","metadata":{"execution":{"iopub.status.busy":"2023-12-28T23:03:09.319092Z","iopub.execute_input":"2023-12-28T23:03:09.319375Z","iopub.status.idle":"2023-12-28T23:03:14.521122Z","shell.execute_reply.started":"2023-12-28T23:03:09.31935Z","shell.execute_reply":"2023-12-28T23:03:14.51972Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n- Before doing anything on the above two results, we can compare the correlation value of all their columns. (This function takes a long time)","metadata":{}},{"cell_type":"code","source":"# :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 2\ntuning_corr_limit(sub_4th, sub_3rd)\ngen_2 = generate_corr_coeff(sub_4th, sub_3rd, 0.90, 0.50)\ngen_2.to_csv('gen_2.csv')\n# Private Score: 0.707\n\n# Classic method of Ensembling - 2\nens_2 = (sub_4th *0.50) + (sub_3rd *0.50)\nens_2.to_csv('ens_2.csv')\n# Private Score: 0.713\n# ::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::","metadata":{"execution":{"iopub.status.busy":"2023-12-28T23:03:14.523039Z","iopub.execute_input":"2023-12-28T23:03:14.523464Z","iopub.status.idle":"2023-12-28T23:03:14.527986Z","shell.execute_reply.started":"2023-12-28T23:03:14.523431Z","shell.execute_reply":"2023-12-28T23:03:14.526975Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n### Explanations for Example : 2\n\n- So the correlation of two similar columns is never negative. In addition, if we consider \"corr_limit\" equal to 0.90, almost Ensembling is done for forty percent of the columns. Because the scores of the two results are close to each other, we consider the \"coeff\" value to be 0.50 so that they can correct each other if they can.\n\n- The private score for this calculation is 0.707, which is better than the score of sub_3rd and sub_4th results. In addition, it is much better than the challenge champion's private score, which is 0.729.\n\n- But as you can see above; If the ensembling is done with the classical method and with a coefficient of 0.50, the private score becomes 0.713, which is even worse than the sub_4th result. That is, at first glance and with simple Ensembling, we think that these two results cannot help each other. But by using correlation guidance, the score will improve.","metadata":{}},{"cell_type":"markdown","source":"
\n
\n
\n\n#
EXAMPLE : 3 - \"Results of impure golden\"
\n\n
\n\n- For the third example, let's go to a specific notebook, to make some things clearer. The public score of the following notebook is 0.720 and its private score is 0.960. These scores are not good at all, because they are even worse than \"sample_submission\" scores. If you try \"sample_submission\" (ie when all answers are zero), the public score is 0.666 and the private score is 0.902.\n\n- But at the beginning of the challenge, we realized that the solution of this notebook is very good, and for this reason, we Ensembled the results of this notebook with the results of our own notebooks, and our score was much better. We named this type of results \"Results of impure golden\". Of course, in [another challenge](https://www.kaggle.com/code/mehrankazeminia/2-tps22nov-results-of-impure-golden-eda),we have checked this type of results. \n\n- Next, we will try Ensembling our own notebook, i.e. sub_606 result, with this particular notebook, so that you can see the improvement in the score.\n\n- https://www.kaggle.com/code/vendekagonlabs/jax-autoencoder-quickstart?scriptVersionId=144671651\n\n-----\n\n","metadata":{}},{"cell_type":"code","source":"# Thanks to: @vendekagonlabs - \"Results of impure golden\"\nsub_720 = pd.read_csv('../input/op2-720/op2_720.csv', index_col='id')\n# ....................................................................\n# Public Score: 0.720\n# Private Score: 0.960","metadata":{"execution":{"iopub.status.busy":"2023-12-28T23:03:14.529377Z","iopub.execute_input":"2023-12-28T23:03:14.529704Z","iopub.status.idle":"2023-12-28T23:03:19.831153Z","shell.execute_reply.started":"2023-12-28T23:03:14.529677Z","shell.execute_reply":"2023-12-28T23:03:19.830025Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n- Before doing anything on the above two results, we can compare the correlation value of all their columns. (This function takes a long time)","metadata":{}},{"cell_type":"code","source":"# :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 3\ntuning_corr_limit(sub_720, sub_606)\ngen_3 = generate_corr_coeff(sub_720, sub_606, 1.00, 0.25)\ngen_3.to_csv('gen_3.csv')\n# Public Score: 0.601\n# Private Score: 0.792\n# ::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::\n!ls","metadata":{"execution":{"iopub.status.busy":"2023-12-28T23:03:19.832803Z","iopub.execute_input":"2023-12-28T23:03:19.833248Z","iopub.status.idle":"2023-12-28T23:10:59.848248Z","shell.execute_reply.started":"2023-12-28T23:03:19.833207Z","shell.execute_reply":"2023-12-28T23:10:59.846625Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n### Explanations for Example : 3\n\n- So the correlation of 8.4 percent of the columns is negative. These columns cannot be useful for Ensembling. We ignored them and Ensembled the rest of the sub_720 columns by a factor of 0.25 with our notebook columns. The general score of our notebook changed from 0.606 to 0.601 and the private score changed from 0.809 to 0.792. We see a significant improvement in the score, while the sub_720 scores were not good at all.\n\n- Please note that in all the above examples, you still need to use trial and error to get the optimal value for \"coeff\" and \"corr_limit\", but knowing the correlation of the columns can somewhat clarify the decision scene. It means that we **no longer need to work in absolute darkness**.\n","metadata":{}},{"cell_type":"markdown","source":"
\n
\n
\n: .","metadata":{}}]}