I found that current baseline notebooks include \"duplicate item\" in output CSV.
\nAt first, I checked the output submission of the following notebook which generates the highest score (0.577).
I checked the duplicate line as follows:
\ndf = pd.read_csv(f'{PATH}/submission.csv')\ndf['labels'] = df['labels'].str.split(' ')\ndf.head()\n session_type labels\n0 12899779_clicks [59625, 1253524, 737445, 438191, 731692, 17907...\n1 12899780_clicks [1142000, 736515, 973453, 582732, 889686, 4871...\n2 12899781_clicks [918667, 199008, 194067, 57315, 141736, 146057...\n3 12899782_clicks [834354, 740494, 987399, 889671, 779477, 12740...\n4 12899783_clicks [1817895, 607638, 1754419, 1216820, 1729553, 3...\nex_df = df.explode(\"labels\").reset_index(drop=True)\nex_df[ex_df.duplicated()]\n session_type labels\n839 12899820_clicks 986164\n1918 12899874_clicks 108125\n2298 12899893_clicks 108125\n4276 12899992_clicks 1460571\n6156 12900086_clicks 1460571\n... ... ...\n97088356 14410590_carts 554660\n97535437 14432944_carts 1460571\n98239751 14468160_carts 1116095\n98983436 14505344_carts 1022566\n98995955 14505970_carts 986164\n40944 rows × 2 columns\nObviously, these output file include the duplicate line which means that the same aid is shown in the same session.
\nI think this problem caused from the following line which is included in the most of the baseline notebooks:
\n # USE TOP20 TEST CLICKS\n return result + list(top_clicks)[:20-len(result)]\nI changed this line to
\n #return result + list(top_clicks)[:20-len(result)]\n # FIXED BY tetsuro731, remove duplicate\n set_result = set(result)\n return result + [i for i in top_clicks if i not in set_result][:20 - len(result)]\nMy revised notebook: https://www.kaggle.com/code/tetsuro731/duplicate-fix-otto-tuning-pipeline2-lb-0-577
\nStrangely, the score has not improved even though I fixed the duplicate bugs.
\nI'm not sure the reason.
\nHowever, I'm wondering the number of duplicate lines are so small that the score is not changed by this update.
I found that current baseline notebooks include \"duplicate item\" in output CSV.
\nAt first, I checked the output submission of the following notebook which generates the highest score (0.577).
I checked the duplicate line as follows:
\ndf = pd.read_csv(f'{PATH}/submission.csv')\ndf['labels'] = df['labels'].str.split(' ')\ndf.head()\n session_type labels\n0 12899779_clicks [59625, 1253524, 737445, 438191, 731692, 17907...\n1 12899780_clicks [1142000, 736515, 973453, 582732, 889686, 4871...\n2 12899781_clicks [918667, 199008, 194067, 57315, 141736, 146057...\n3 12899782_clicks [834354, 740494, 987399, 889671, 779477, 12740...\n4 12899783_clicks [1817895, 607638, 1754419, 1216820, 1729553, 3...\nex_df = df.explode(\"labels\").reset_index(drop=True)\nex_df[ex_df.duplicated()]\n session_type labels\n839 12899820_clicks 986164\n1918 12899874_clicks 108125\n2298 12899893_clicks 108125\n4276 12899992_clicks 1460571\n6156 12900086_clicks 1460571\n... ... ...\n97088356 14410590_carts 554660\n97535437 14432944_carts 1460571\n98239751 14468160_carts 1116095\n98983436 14505344_carts 1022566\n98995955 14505970_carts 986164\n40944 rows × 2 columns\nObviously, these output file include the duplicate line which means that the same aid is shown in the same session.
\nI think this problem caused from the following line which is included in the most of the baseline notebooks:
\n # USE TOP20 TEST CLICKS\n return result + list(top_clicks)[:20-len(result)]\nI changed this line to
\n #return result + list(top_clicks)[:20-len(result)]\n # FIXED BY tetsuro731, remove duplicate\n set_result = set(result)\n return result + [i for i in top_clicks if i not in set_result][:20 - len(result)]\nMy revised notebook: https://www.kaggle.com/code/tetsuro731/duplicate-fix-otto-tuning-pipeline2-lb-0-577
\nStrangely, the score has not improved even though I fixed the duplicate bugs.
\nI'm not sure the reason.
\nHowever, I'm wondering the number of duplicate lines are so small that the score is not changed by this update.
Submitting or removing duplicate aids for the same session wouldn't push the metric. Normann answered here
", "rawMarkdown": "Submitting or removing duplicate aids for the same session wouldn't push the metric. Normann answered [here](https://www.kaggle.com/competitions/otto-recommender-system/discussion/363973#2029245)", "votes": null }, { "id": "2086925", "postDate": "01/05/2023 06:51:36", "content": "Thank you for your nice information!
\nI understand removing duplicate aid itself doesn't improve the metric.
\nHowever, we can chose another item if we remove duplicated aid, which can improve the metrics.
\nFor example:
Case1: No duplicate
\nCase2: Duplicate (aid1 is shown 2 times)
\nTherefore, I think we should remove duplicated aid and we should chose distinct 20 aids for each session to get a good score.
", "rawMarkdown": "Thank you for your nice information!\nI understand removing duplicate `aid` itself doesn't improve the metric.\nHowever, we can chose another item if we remove duplicated aid, which can improve the metrics.\nFor example:\n\nCase1: No duplicate\n- session1_click, aid1, aid2, aid3, ..., aid19, aid20\n-> We can select 20 aids\n\nCase2: Duplicate (aid1 is shown 2 times)\n- session1_click, aid1, aid2, aid3, ..., aid19, aid1\n-> We can select 19 aids\n\nTherefore, I think we should remove duplicated aid and we should chose distinct 20 aids for each session to get a good score.", "votes": null }, { "id": "2086939", "postDate": "01/05/2023 07:08:29", "content": "I think of 2 possibilities, one is that the aids you added are not good enough to increase the metric, and the other is that the session < 20 aids/or is already covered by previous predicted items.
", "rawMarkdown": "I think of 2 possibilities, one is that the aids you added are not good enough to increase the metric, and the other is that the session < 20 aids/or is already covered by previous predicted items.", "votes": null }, { "id": "2086948", "postDate": "01/05/2023 07:24:38", "content": "Yes.
\nI totally agree with your ideas.
In addition, I'm also wondering we should be careful if we choose the candidate items by similar procedures which are used for the next ranking phase.
\nI think most of the people just copied these notebook without understanding the fact that the results include the duplicated aids.
Anyway, my correction have not improve the score so this issue does not cause the critical problem for now.
", "rawMarkdown": "Yes.\nI totally agree with your ideas.\n\nIn addition, I'm also wondering we should be careful if we choose the candidate items by similar procedures which are used for the next ranking phase.\nI think most of the people just copied these notebook without understanding the fact that the results include the duplicated aids.\n\nAnyway, my correction have not improve the score so this issue does not cause the critical problem for now.", "votes": null }, { "id": "2087226", "postDate": "01/05/2023 12:52:44", "content": "You are correct that removing duplicate aid entries and choosing distinct products for each session can improve the model's performance. This is because having duplicate aid entries in the same session can reduce the number of unique products that the model can choose from, which may limit its ability to accurately predict the correct products for that session.
\nBy removing duplicate aid entries and ensuring that each session has a set of distinct products, you can give the model a better chance to make accurate predictions. This can potentially improve the overall score, depending on the specific metric being used to evaluate the model's performance.
\nIt is always a good idea to ensure that the data used to train and evaluate a model is as clean and accurate as possible. Removing duplicate entries and addressing other issues with the data can help improve the model's performance and increase its reliability.
", "rawMarkdown": "You are correct that removing duplicate aid entries and choosing distinct products for each session can improve the model's performance. This is because having duplicate aid entries in the same session can reduce the number of unique products that the model can choose from, which may limit its ability to accurately predict the correct products for that session.\n\nBy removing duplicate aid entries and ensuring that each session has a set of distinct products, you can give the model a better chance to make accurate predictions. This can potentially improve the overall score, depending on the specific metric being used to evaluate the model's performance.\n\nIt is always a good idea to ensure that the data used to train and evaluate a model is as clean and accurate as possible. Removing duplicate entries and addressing other issues with the data can help improve the model's performance and increase its reliability.", "votes": null }, { "id": "2087409", "postDate": "01/05/2023 15:21:03", "content": "Thank you for your opinion.
\nThat is just what I wanted to say!
\nI think I have a deeper understanding of this idea now.
Submitting or removing duplicate aids for the same session wouldn't push the metric. Normann answered here
", "votes": null, "replies": [ { "id": 2086925, "author_name": "tetsuro731", "author_url": "", "post_date": "01/05/2023 06:51:36", "content": "Thank you for your nice information!
\nI understand removing duplicate aid itself doesn't improve the metric.
\nHowever, we can chose another item if we remove duplicated aid, which can improve the metrics.
\nFor example:
Case1: No duplicate
\nCase2: Duplicate (aid1 is shown 2 times)
\nTherefore, I think we should remove duplicated aid and we should chose distinct 20 aids for each session to get a good score.
", "votes": null, "replies": [ { "id": 2086939, "author_name": "bibanh", "author_url": "", "post_date": "01/05/2023 07:08:29", "content": "I think of 2 possibilities, one is that the aids you added are not good enough to increase the metric, and the other is that the session < 20 aids/or is already covered by previous predicted items.
", "votes": null, "replies": [ { "id": 2086948, "author_name": "tetsuro731", "author_url": "", "post_date": "01/05/2023 07:24:38", "content": "Yes.
\nI totally agree with your ideas.
In addition, I'm also wondering we should be careful if we choose the candidate items by similar procedures which are used for the next ranking phase.
\nI think most of the people just copied these notebook without understanding the fact that the results include the duplicated aids.
Anyway, my correction have not improve the score so this issue does not cause the critical problem for now.
", "votes": null, "replies": [] } ] }, { "id": 2087226, "author_name": "aviralmishra1998", "author_url": "", "post_date": "01/05/2023 12:52:44", "content": "You are correct that removing duplicate aid entries and choosing distinct products for each session can improve the model's performance. This is because having duplicate aid entries in the same session can reduce the number of unique products that the model can choose from, which may limit its ability to accurately predict the correct products for that session.
\nBy removing duplicate aid entries and ensuring that each session has a set of distinct products, you can give the model a better chance to make accurate predictions. This can potentially improve the overall score, depending on the specific metric being used to evaluate the model's performance.
\nIt is always a good idea to ensure that the data used to train and evaluate a model is as clean and accurate as possible. Removing duplicate entries and addressing other issues with the data can help improve the model's performance and increase its reliability.
", "votes": null, "replies": [ { "id": 2087409, "author_name": "tetsuro731", "author_url": "", "post_date": "01/05/2023 15:21:03", "content": "Thank you for your opinion.
\nThat is just what I wanted to say!
\nI think I have a deeper understanding of this idea now.