Hi
\nSorry for another metric hacking post but I have a genuine concern as I am about to spend a huge number of hours into this competition in the next two months.
\nI have some thoughts below, please feel free to correct me if I am wrong as I am a beginner.
\n/The metric has not changed but the test data has been changed so its difficult to do metric hacking but not impossible. I appreciate the hosts must have their reasons to keep the current metric but it doesn't change the fact that there will solutions submitted with metric hacking. If something is possible, there will always be people who will do that for whatever reason.
\n/I see some people quoting organizers that top 100 solutions will be checked but I couldn't find an actual post from organizers, please can someone post a link to that.
\n/For me, even if its confirmed by organizers, top x solutions will checked and if found guilty, will be disqualified so that there will be new standings but then lets say people using hacking are more than x, then you have to check more solutions.
So my conclusion is, there is probably a guarantee that you can't win a prize if you use hacking but it seems close to certain
\n/all the medals will not be distributed 'as they should'
\n/there will ways be two groups of people on current and final LB, those who approach the competition honestly and those who do not
\nCan anyone disagree with this ?
Also, the solutions being submitted to the public LB right now, we don't know who is doing metric hacking and who is not so the LB can be misleading and I don't really know how much my solution is actually improving when I am working on this competition honestly.
\nIf you understand this issue and still decide to take part in this competition, I would like to know your reason, maybe I am missing something :)
\nOr am I am overthinking it ?
\nThanks all
", "messages": [ { "id": "2721733", "postDate": "03/29/2024 07:17:30", "content": "Hi
\nSorry for another metric hacking post but I have a genuine concern as I am about to spend a huge number of hours into this competition in the next two months.
\nI have some thoughts below, please feel free to correct me if I am wrong as I am a beginner.
\n/The metric has not changed but the test data has been changed so its difficult to do metric hacking but not impossible. I appreciate the hosts must have their reasons to keep the current metric but it doesn't change the fact that there will solutions submitted with metric hacking. If something is possible, there will always be people who will do that for whatever reason.
\n/I see some people quoting organizers that top 100 solutions will be checked but I couldn't find an actual post from organizers, please can someone post a link to that.
\n/For me, even if its confirmed by organizers, top x solutions will checked and if found guilty, will be disqualified so that there will be new standings but then lets say people using hacking are more than x, then you have to check more solutions.
So my conclusion is, there is probably a guarantee that you can't win a prize if you use hacking but it seems close to certain
\n/all the medals will not be distributed 'as they should'
\n/there will ways be two groups of people on current and final LB, those who approach the competition honestly and those who do not
\nCan anyone disagree with this ?
Also, the solutions being submitted to the public LB right now, we don't know who is doing metric hacking and who is not so the LB can be misleading and I don't really know how much my solution is actually improving when I am working on this competition honestly.
\nIf you understand this issue and still decide to take part in this competition, I would like to know your reason, maybe I am missing something :)
\nOr am I am overthinking it ?
\nThanks all
", "rawMarkdown": "Hi\n\nSorry for another metric hacking post but I have a genuine concern as I am about to spend a huge number of hours into this competition in the next two months. \n\nI have some thoughts below, please feel free to correct me if I am wrong as I am a beginner.\n\n/The metric has not changed but the test data has been changed so its difficult to do metric hacking but not impossible. I appreciate the hosts must have their reasons to keep the current metric but it doesn't change the fact that there will solutions submitted with metric hacking. If something is possible, there will always be people who will do that for whatever reason.\n\n/I see some people quoting organizers that top 100 solutions will be checked but I couldn't find an actual post from organizers, please can someone post a link to that. \n/For me, even if its confirmed by organizers, top x solutions will checked and if found guilty, will be disqualified so that there will be new standings but then lets say people using hacking are more than x, then you have to check more solutions.\n\nSo my conclusion is, there is probably a guarantee that you can't win a prize if you use hacking but it seems close to certain \n/all the medals will not be distributed 'as they should' \n/there will ways be two groups of people on current and final LB, those who approach the competition honestly and those who do not\nCan anyone disagree with this ?\n\nAlso, the solutions being submitted to the public LB right now, we don't know who is doing metric hacking and who is not so the LB can be misleading and I don't really know how much my solution is actually improving when I am working on this competition honestly.\n\nIf you understand this issue and still decide to take part in this competition, I would like to know your reason, maybe I am missing something :) \n\nOr am I am overthinking it ?\n\nThanks all", "votes": null }, { "id": "2722075", "postDate": "03/29/2024 11:36:55", "content": "I'll speak for myself - without any attempt of hacking pretty high score in LB is possible, so do your hardwork honestly and all will be fine.
", "rawMarkdown": "I'll speak for myself - without any attempt of hacking pretty high score in LB is possible, so do your hardwork honestly and all will be fine.", "votes": null }, { "id": "2722863", "postDate": "03/29/2024 20:01:53", "content": "Our score is es well without metric hacking. Most difficult thing ist find a good CV-Strategie
", "rawMarkdown": "Our score is es well without metric hacking. Most difficult thing ist find a good CV-Strategie", "votes": null }, { "id": "2722884", "postDate": "03/29/2024 20:35:01", "content": "The term hacking sed here is some what different from actual hacking.
\nFor example, lets say there is competition of training a neural network to produce output from 2 input, now some one somehow figures out that there exists a function that generates the output directly, like fist input square plus 2nd input gives the output. Now this contestor, instead of training the neural network, directly calculates the answer by sing this function, his score will 100 perfect. now is this hacking or being smart.
\nHere it was possible that by altering some score by looking at weeks, stability score could be improved a the cost of accuracy of model. so again is this hacking or being smart that you figured out this trend.
\nBut as this is not what competition wants, competition wants to use the better models in future (understandable). So to shift the focus more in making better models. this trick had to be removed, now as the trick depended on altering the score based on week number, I guess the week number for test has been hidden, not even for code to reach it (need to confirm on this). so ppl can not do the hack in the same way.
\nCan a new way be developed to hack or a new trick that can predict the output directly. I do not know if there is any. but if one artificially tries to get higher on public leaderboard, by using some logic, than that logic might backfire for private leaderboard, as it does not capture the actual underlying logic. If someone finds an underlying logic, well that is we all are trying to do by building models, if there exists some hidden trend and some one finds it, will it be a hack or he did great job.
", "rawMarkdown": "The term hacking sed here is some what different from actual hacking.\n\nFor example, lets say there is competition of training a neural network to produce output from 2 input, now some one somehow figures out that there exists a function that generates the output directly, like fist input square plus 2nd input gives the output. Now this contestor, instead of training the neural network, directly calculates the answer by sing this function, his score will 100 perfect. now is this hacking or being smart.\n\nHere it was possible that by altering some score by looking at weeks, stability score could be improved a the cost of accuracy of model. so again is this hacking or being smart that you figured out this trend.\n\nBut as this is not what competition wants, competition wants to use the better models in future (understandable). So to shift the focus more in making better models. this trick had to be removed, now as the trick depended on altering the score based on week number, I guess the week number for test has been hidden, not even for code to reach it (need to confirm on this). so ppl can not do the hack in the same way.\n \nCan a new way be developed to hack or a new trick that can predict the output directly. I do not know if there is any. but if one artificially tries to get higher on public leaderboard, by using some logic, than that logic might backfire for private leaderboard, as it does not capture the actual underlying logic. If someone finds an underlying logic, well that is we all are trying to do by building models, if there exists some hidden trend and some one finds it, will it be a hack or he did great job.", "votes": null }, { "id": "2723279", "postDate": "03/30/2024 05:12:46", "content": "Thanks for the detailed insight.
\n\n\nCan a new way be developed to hack or a new trick that can predict the output directly.
\n
Looking at some comments in other threads its probably possible, but the good thing is, if someone tries to do this, they risk getting disqualified if the 'hack' gets detected after their solution is audited
", "rawMarkdown": "Thanks for the detailed insight.\n\n>Can a new way be developed to hack or a new trick that can predict the output directly.\n\nLooking at some comments in other threads its probably possible, but the good thing is, if someone tries to do this, they risk getting disqualified if the 'hack' gets detected after their solution is audited", "votes": null } ], "comments": [ { "id": 2722075, "author_name": "eu1234", "author_url": "", "post_date": "03/29/2024 11:36:55", "content": "I'll speak for myself - without any attempt of hacking pretty high score in LB is possible, so do your hardwork honestly and all will be fine.
", "votes": null, "replies": [] }, { "id": 2722863, "author_name": "lucamtb", "author_url": "", "post_date": "03/29/2024 20:01:53", "content": "Our score is es well without metric hacking. Most difficult thing ist find a good CV-Strategie
", "votes": null, "replies": [] }, { "id": 2722884, "author_name": "shreyas9181", "author_url": "", "post_date": "03/29/2024 20:35:01", "content": "The term hacking sed here is some what different from actual hacking.
\nFor example, lets say there is competition of training a neural network to produce output from 2 input, now some one somehow figures out that there exists a function that generates the output directly, like fist input square plus 2nd input gives the output. Now this contestor, instead of training the neural network, directly calculates the answer by sing this function, his score will 100 perfect. now is this hacking or being smart.
\nHere it was possible that by altering some score by looking at weeks, stability score could be improved a the cost of accuracy of model. so again is this hacking or being smart that you figured out this trend.
\nBut as this is not what competition wants, competition wants to use the better models in future (understandable). So to shift the focus more in making better models. this trick had to be removed, now as the trick depended on altering the score based on week number, I guess the week number for test has been hidden, not even for code to reach it (need to confirm on this). so ppl can not do the hack in the same way.
\nCan a new way be developed to hack or a new trick that can predict the output directly. I do not know if there is any. but if one artificially tries to get higher on public leaderboard, by using some logic, than that logic might backfire for private leaderboard, as it does not capture the actual underlying logic. If someone finds an underlying logic, well that is we all are trying to do by building models, if there exists some hidden trend and some one finds it, will it be a hack or he did great job.
", "votes": null, "replies": [ { "id": 2723279, "author_name": "jabranzahid", "author_url": "", "post_date": "03/30/2024 05:12:46", "content": "Thanks for the detailed insight.
\n\n\nCan a new way be developed to hack or a new trick that can predict the output directly.
\n
Looking at some comments in other threads its probably possible, but the good thing is, if someone tries to do this, they risk getting disqualified if the 'hack' gets detected after their solution is audited
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2721733": "Hi\n\nSorry for another metric hacking post but I have a genuine concern as I am about to spend a huge number of hours into this competition in the next two months. \n\nI have some thoughts below, please feel free to correct me if I am wrong as I am a beginner.\n\n/The metric has not changed but the test data has been changed so its difficult to do metric hacking but not impossible. I appreciate the hosts must have their reasons to keep the current metric but it doesn't change the fact that there will solutions submitted with metric hacking. If something is possible, there will always be people who will do that for whatever reason.\n\n/I see some people quoting organizers that top 100 solutions will be checked but I couldn't find an actual post from organizers, please can someone post a link to that. \n/For me, even if its confirmed by organizers, top x solutions will checked and if found guilty, will be disqualified so that there will be new standings but then lets say people using hacking are more than x, then you have to check more solutions.\n\nSo my conclusion is, there is probably a guarantee that you can't win a prize if you use hacking but it seems close to certain \n/all the medals will not be distributed 'as they should' \n/there will ways be two groups of people on current and final LB, those who approach the competition honestly and those who do not\nCan anyone disagree with this ?\n\nAlso, the solutions being submitted to the public LB right now, we don't know who is doing metric hacking and who is not so the LB can be misleading and I don't really know how much my solution is actually improving when I am working on this competition honestly.\n\nIf you understand this issue and still decide to take part in this competition, I would like to know your reason, maybe I am missing something :) \n\nOr am I am overthinking it ?\n\nThanks all", "2722075": "I'll speak for myself - without any attempt of hacking pretty high score in LB is possible, so do your hardwork honestly and all will be fine.", "2722863": "Our score is es well without metric hacking. Most difficult thing ist find a good CV-Strategie", "2722884": "The term hacking sed here is some what different from actual hacking.\n\nFor example, lets say there is competition of training a neural network to produce output from 2 input, now some one somehow figures out that there exists a function that generates the output directly, like fist input square plus 2nd input gives the output. Now this contestor, instead of training the neural network, directly calculates the answer by sing this function, his score will 100 perfect. now is this hacking or being smart.\n\nHere it was possible that by altering some score by looking at weeks, stability score could be improved a the cost of accuracy of model. so again is this hacking or being smart that you figured out this trend.\n\nBut as this is not what competition wants, competition wants to use the better models in future (understandable). So to shift the focus more in making better models. this trick had to be removed, now as the trick depended on altering the score based on week number, I guess the week number for test has been hidden, not even for code to reach it (need to confirm on this). so ppl can not do the hack in the same way.\n \nCan a new way be developed to hack or a new trick that can predict the output directly. I do not know if there is any. but if one artificially tries to get higher on public leaderboard, by using some logic, than that logic might backfire for private leaderboard, as it does not capture the actual underlying logic. If someone finds an underlying logic, well that is we all are trying to do by building models, if there exists some hidden trend and some one finds it, will it be a hack or he did great job.", "2723279": "Thanks for the detailed insight.\n\n>Can a new way be developed to hack or a new trick that can predict the output directly.\n\nLooking at some comments in other threads its probably possible, but the good thing is, if someone tries to do this, they risk getting disqualified if the 'hack' gets detected after their solution is audited" }, "source": "meta" }