Hello everyone,
\nI have a few things I would like to share. Congratulations to the winners. And I want to thank my teammates for participation.
\nI have joined the competition for 2 purposes. I wanted to achieve high score (which did not happen). And I wanted to learn new things, approaches, methods (which did happen). I have made a big progress on my learning curve.
\nI have invested too much time in this competition, which I regret very much. I have learned many new things (Python, data preparation, model evaluation). But I have invested too much time in implementing ideas that did not work, it's part of the learning though. And in the end, all the work on feature engineering, tuning single models and creating sophisticated ensemble models was for nothing. Because a simple solution that utilizes the metric hack could beat that easily.
\nThe competition hosts failed BIG time @jetakow @tomasjeline2 @inversion. They have failed to create environment for fair and meaningful competition. a) As proposed by many participants the evaluation metric itself (even without the possibility of being cheated) was engineered poorly. The metric does not represent stability well. b) Competition hosts allowed for metric hacking. On the beginning they showed effort to make adjustment and said they will disqualify cheaters. Instead, in the end they have allowed the cheating, which totally messed up the competition. I am not the only one with this negative opinion. Many participants showed disapointment and disapproval, many criticized allowing for metric hacking.
\nAbout the result. We placed the 130th in the public leaderboard and the 1093th in the private leaderboard. That is disapointment. Our solution belong to many who massively overfitted the public leaderboard. The discrepancy between CV and LB did not help it.
\nIt is a valuable lesson. I burned myself on the stove and now I will remember to stay away from the stove. Next time I have to research much better before investing so much time.
", "messages": [ { "id": 2840258, "postDate": "2024-05-28T03:56:45.707Z", "content": "Hello everyone,
\nI have a few things I would like to share. Congratulations to the winners. And I want to thank my teammates for participation.
\nI have joined the competition for 2 purposes. I wanted to achieve high score (which did not happen). And I wanted to learn new things, approaches, methods (which did happen). I have made a big progress on my learning curve.
\nI have invested too much time in this competition, which I regret very much. I have learned many new things (Python, data preparation, model evaluation). But I have invested too much time in implementing ideas that did not work, it's part of the learning though. And in the end, all the work on feature engineering, tuning single models and creating sophisticated ensemble models was for nothing. Because a simple solution that utilizes the metric hack could beat that easily.
\nThe competition hosts failed BIG time @jetakow @tomasjeline2 @inversion. They have failed to create environment for fair and meaningful competition. a) As proposed by many participants the evaluation metric itself (even without the possibility of being cheated) was engineered poorly. The metric does not represent stability well. b) Competition hosts allowed for metric hacking. On the beginning they showed effort to make adjustment and said they will disqualify cheaters. Instead, in the end they have allowed the cheating, which totally messed up the competition. I am not the only one with this negative opinion. Many participants showed disapointment and disapproval, many criticized allowing for metric hacking.
\nAbout the result. We placed the 130th in the public leaderboard and the 1093th in the private leaderboard. That is disapointment. Our solution belong to many who massively overfitted the public leaderboard. The discrepancy between CV and LB did not help it.
\nIt is a valuable lesson. I burned myself on the stove and now I will remember to stay away from the stove. Next time I have to research much better before investing so much time.
", "rawMarkdown": "Hello everyone,\n\nI have a few things I would like to share. Congratulations to the winners. And I want to thank my teammates for participation.\n\nI have joined the competition for 2 purposes. I wanted to achieve high score (which did not happen). And I wanted to learn new things, approaches, methods (which did happen). I have made a big progress on my learning curve.\n\nI have invested too much time in this competition, which I regret very much. I have learned many new things (Python, data preparation, model evaluation). But I have invested too much time in implementing ideas that did not work, it's part of the learning though. And in the end, all the work on feature engineering, tuning single models and creating sophisticated ensemble models was for nothing. Because a simple solution that utilizes the metric hack could beat that easily.\n\nThe competition hosts failed BIG time @jetakow @tomasjeline2 @inversion. They have failed to create environment for fair and meaningful competition. a) As proposed by many participants the evaluation metric itself (even without the possibility of being cheated) was engineered poorly. The metric does not represent stability well. b) Competition hosts allowed for metric hacking. On the beginning they showed effort to make adjustment and said they will disqualify cheaters. Instead, in the end they have allowed the cheating, which totally messed up the competition. I am not the only one with this negative opinion. Many participants showed disapointment and disapproval, many criticized allowing for metric hacking.\n\nAbout the result. We placed the 130th in the public leaderboard and the 1093th in the private leaderboard. That is disapointment. Our solution belong to many who massively overfitted the public leaderboard. The discrepancy between CV and LB did not help it.\n\nIt is a valuable lesson. I burned myself on the stove and now I will remember to stay away from the stove. Next time I have to research much better before investing so much time.", "votes": 16 }, { "id": 2841095, "postDate": "2024-05-28T11:59:21.267Z", "content": "Well, it was an interesting one,
\nThe only competition that I have ever participated, thanks @matousfamera , I had fun.
I did have my doubts joining this specific competition but with @matousfamera and @raylaomo. I continued with this competition.
\n@matousfamera was a great leader in this competition and is the reason we were driven throughout the competition to do better.
\n@raylaomo came up with really really interesting articles from different Chinese sources and they were really insightful
\nThe Metric hacking was sad, but I think, I only did want to learn new stuff , and I did ( metric hacking of this sorts was unheard of for me š)
\nThanks people and Peace
\nāļø
\n\nThe competition hosts failed BIG time
\n
Even if the objective of the competition itself was interesting, the competition was set to fail due to the vulnerabilities of the metric. It was unexpected that a solution with the most basic feature engineering and metric hacking would outperform one with well-thought data preprocessing.
", "rawMarkdown": "> The competition hosts failed BIG time\n\nEven if the objective of the competition itself was interesting, the competition was set to fail due to the vulnerabilities of the metric. It was unexpected that a solution with the most basic feature engineering and metric hacking would outperform one with well-thought data preprocessing.", "votes": 2, "replies": [ { "id": 2840432, "postDate": "2024-05-28T05:08:28.873Z", "rawMarkdown": "", "votes": -2, "isDeleted": true, "replies": [ { "id": 2840481, "postDate": "2024-05-28T05:44:14.277Z", "content": "\n\nin fact the hosts were excellent here
\n
That was true until they said that the only action they would punish would be if we became Edward Snowden and hacked the Kaggle platform itself.
", "rawMarkdown": "> in fact the hosts were excellent here\n\nThat was true until they said that the only action they would punish would be if we became Edward Snowden and hacked the Kaggle platform itself. ", "votes": 2 }, { "id": 2840575, "postDate": "2024-05-28T06:33:06.343Z", "content": "\n\nI agree, in fact the hosts were excellent here, answering our questions promptly and developing a good working relation with the participants.
\n
The competition hosts might answer questions of participants. However, they are fully responsible for the rules they have set and the choice of the evaluation metric. Many people wasted their time by working on the conventional/actual solution (without metric hacking).
", "rawMarkdown": ">I agree, in fact the hosts were excellent here, answering our questions promptly and developing a good working relation with the participants.\n\nThe competition hosts might answer questions of participants. However, they are fully responsible for the rules they have set and the choice of the evaluation metric. Many people wasted their time by working on the conventional/actual solution (without metric hacking).", "votes": 3 } ] } ] }, { "id": 2843353, "postDate": "2024-05-29T14:28:26.430Z", "content": "@matousfamera I sympathize with you. Just as a host has the right to choose a metric that is truly terrible and end the competition, a competitor has the right to not participate in such a competition in the first place or to drop out midway.
", "rawMarkdown": "@matousfamera I sympathize with you. Just as a host has the right to choose a metric that is truly terrible and end the competition, a competitor has the right to not participate in such a competition in the first place or to drop out midway.", "votes": 1, "replies": [ { "id": 2843954, "postDate": "2024-05-29T19:32:46.693Z", "content": "I agree with you. And next time I will choose where I spend time more wisely. The thing is the competition hosts said they will not accept any metric hacking solutions. But about 1 month before the deadline they have changed their mind and said that they will allow the metric hacking. And that time I have already invested so much time into the competition.
", "rawMarkdown": "I agree with you. And next time I will choose where I spend time more wisely. The thing is the competition hosts said they will not accept any metric hacking solutions. But about 1 month before the deadline they have changed their mind and said that they will allow the metric hacking. And that time I have already invested so much time into the competition.", "votes": 1 } ] } ], "comments": [ { "id": 2841095, "author_name": "Lucifer_is_back_", "author_url": "", "post_date": "2024-05-28T11:59:21.267000", "content": "Well, it was an interesting one,
\nThe only competition that I have ever participated, thanks @matousfamera , I had fun.
I did have my doubts joining this specific competition but with @matousfamera and @raylaomo. I continued with this competition.
\n@matousfamera was a great leader in this competition and is the reason we were driven throughout the competition to do better.
\n@raylaomo came up with really really interesting articles from different Chinese sources and they were really insightful
\nThe Metric hacking was sad, but I think, I only did want to learn new stuff , and I did ( metric hacking of this sorts was unheard of for me š)
\nThanks people and Peace
\nāļø
\n\nThe competition hosts failed BIG time
\n
Even if the objective of the competition itself was interesting, the competition was set to fail due to the vulnerabilities of the metric. It was unexpected that a solution with the most basic feature engineering and metric hacking would outperform one with well-thought data preprocessing.
", "votes": 2, "replies": [ { "id": 2840432, "author_name": "", "author_url": "", "post_date": "2024-05-28T05:08:28.873000", "content": "", "votes": -2, "replies": [ { "id": 2840481, "author_name": "Andreas Bisiadis", "author_url": "", "post_date": "2024-05-28T05:44:14.277000", "content": "\n\nin fact the hosts were excellent here
\n
That was true until they said that the only action they would punish would be if we became Edward Snowden and hacked the Kaggle platform itself.
", "votes": 2, "replies": [] }, { "id": 2840575, "author_name": "Matous Famera", "author_url": "", "post_date": "2024-05-28T06:33:06.343000", "content": "\n\nI agree, in fact the hosts were excellent here, answering our questions promptly and developing a good working relation with the participants.
\n
The competition hosts might answer questions of participants. However, they are fully responsible for the rules they have set and the choice of the evaluation metric. Many people wasted their time by working on the conventional/actual solution (without metric hacking).
", "votes": 3, "replies": [] } ] } ] }, { "id": 2843353, "author_name": "mh", "author_url": "", "post_date": "2024-05-29T14:28:26.430000", "content": "@matousfamera I sympathize with you. Just as a host has the right to choose a metric that is truly terrible and end the competition, a competitor has the right to not participate in such a competition in the first place or to drop out midway.
", "votes": 1, "replies": [ { "id": 2843954, "author_name": "Matous Famera", "author_url": "", "post_date": "2024-05-29T19:32:46.693000", "content": "I agree with you. And next time I will choose where I spend time more wisely. The thing is the competition hosts said they will not accept any metric hacking solutions. But about 1 month before the deadline they have changed their mind and said that they will allow the metric hacking. And that time I have already invested so much time into the competition.
", "votes": 1, "replies": [] } ] } ], "raw_markdown_by_id": { "2840258": "Hello everyone,\n\nI have a few things I would like to share. Congratulations to the winners. And I want to thank my teammates for participation.\n\nI have joined the competition for 2 purposes. I wanted to achieve high score (which did not happen). And I wanted to learn new things, approaches, methods (which did happen). I have made a big progress on my learning curve.\n\nI have invested too much time in this competition, which I regret very much. I have learned many new things (Python, data preparation, model evaluation). But I have invested too much time in implementing ideas that did not work, it's part of the learning though. And in the end, all the work on feature engineering, tuning single models and creating sophisticated ensemble models was for nothing. Because a simple solution that utilizes the metric hack could beat that easily.\n\nThe competition hosts failed BIG time @jetakow @tomasjeline2 @inversion. They have failed to create environment for fair and meaningful competition. a) As proposed by many participants the evaluation metric itself (even without the possibility of being cheated) was engineered poorly. The metric does not represent stability well. b) Competition hosts allowed for metric hacking. On the beginning they showed effort to make adjustment and said they will disqualify cheaters. Instead, in the end they have allowed the cheating, which totally messed up the competition. I am not the only one with this negative opinion. Many participants showed disapointment and disapproval, many criticized allowing for metric hacking.\n\nAbout the result. We placed the 130th in the public leaderboard and the 1093th in the private leaderboard. That is disapointment. Our solution belong to many who massively overfitted the public leaderboard. The discrepancy between CV and LB did not help it.\n\nIt is a valuable lesson. I burned myself on the stove and now I will remember to stay away from the stove. Next time I have to research much better before investing so much time.", "2841095": "Well, it was an interesting one, \nThe only competition that I have ever participated, thanks @matousfamera , I had fun.\n\nI did have my doubts joining this specific competition but with @matousfamera and @raylaomo. I continued with this competition.\n\n@matousfamera was a great leader in this competition and is the reason we were driven throughout the competition to do better.\n\n@raylaomo came up with really really interesting articles from different Chinese sources and they were really insightful\n\nThe Metric hacking was sad, but I think, I only did want to learn new stuff , and I did ( metric hacking of this sorts was unheard of for me š)\n\nThanks people and Peace\nāļø", "2840267": "> The competition hosts failed BIG time\n\nEven if the objective of the competition itself was interesting, the competition was set to fail due to the vulnerabilities of the metric. It was unexpected that a solution with the most basic feature engineering and metric hacking would outperform one with well-thought data preprocessing.", "2843353": "@matousfamera I sympathize with you. Just as a host has the right to choose a metric that is truly terrible and end the competition, a competitor has the right to not participate in such a competition in the first place or to drop out midway." } }