Something I have struggled with in the last couple of weeks of the competition is notebooks submissions failing randomly.
\nSame notebook, with no changed, submitted twice, one would get accepted and the other would fail. Not sure if it a timeout, or a bug or what, but it surely didn't make sense. Looking at the discussion, I found some people struggling with this problem, yet clueless on how to solve it.
\nMy guess was that it was timing out, but now seeing other kagglers solutions, I think not. Some ensembled multiple transformers successfully, meanwhile my model was relatively small (4 M parameters, 100 seq).
\nLet's not leave it a mystery please, does anyone knows why?
", "messages": [ { "id": "1144028", "postDate": "01/08/2021 07:29:32", "content": "Something I have struggled with in the last couple of weeks of the competition is notebooks submissions failing randomly.
\nSame notebook, with no changed, submitted twice, one would get accepted and the other would fail. Not sure if it a timeout, or a bug or what, but it surely didn't make sense. Looking at the discussion, I found some people struggling with this problem, yet clueless on how to solve it.
\nMy guess was that it was timing out, but now seeing other kagglers solutions, I think not. Some ensembled multiple transformers successfully, meanwhile my model was relatively small (4 M parameters, 100 seq).
\nLet's not leave it a mystery please, does anyone knows why?
", "rawMarkdown": "Something I have struggled with in the last couple of weeks of the competition is notebooks submissions failing randomly. \n\nSame notebook, with no changed, submitted twice, one would get accepted and the other would fail. Not sure if it a timeout, or a bug or what, but it surely didn't make sense. Looking at the discussion, I found some people struggling with this problem, yet clueless on how to solve it.\n\nMy guess was that it was timing out, but now seeing other kagglers solutions, I think not. Some ensembled multiple transformers successfully, meanwhile my model was relatively small (4 M parameters, 100 seq). \n\nLet's not leave it a mystery please, does anyone knows why?", "votes": null }, { "id": "1144050", "postDate": "01/08/2021 07:52:31", "content": "I experienced the same thing towards the end of the competition. I had a model submitted successfully and I tried submitting the same model but trained for one more epoch and it gave me an error. Not sure if I have a bug as well or something else is going on.
", "rawMarkdown": "I experienced the same thing towards the end of the competition. I had a model submitted successfully and I tried submitting the same model but trained for one more epoch and it gave me an error. Not sure if I have a bug as well or something else is going on.", "votes": null }, { "id": "1144061", "postDate": "01/08/2021 08:06:32", "content": "It’s runtime obfuscation, mentioned in the data page. (I also experienced it) This competition was lot of fun except this inconsistent runtime part.
", "rawMarkdown": "It’s runtime obfuscation, mentioned in the data page. (I also experienced it) This competition was lot of fun except this inconsistent runtime part.", "votes": null }, { "id": "1144073", "postDate": "01/08/2021 08:11:18", "content": "Run time obfuscation means that the score would be revealed eventually, but in my case it just fails with the most obscure error code ever: Submission failed.
", "rawMarkdown": "Run time obfuscation means that the score would be revealed eventually, but in my case it just fails with the most obscure error code ever: Submission failed.", "votes": null }, { "id": "1144076", "postDate": "01/08/2021 08:14:41", "content": "May be one submission took ~8 hrs 30 mins (and you got your score) another one went past 9hr and all we got was error. (I faced this problem)
", "rawMarkdown": "May be one submission took ~8 hrs 30 mins (and you got your score) another one went past 9hr and all we got was error. (I faced this problem)", "votes": null }, { "id": "1144082", "postDate": "01/08/2021 08:17:07", "content": "Yeah, but how likely would that be? Some of the winning solutions ensembled two d_model 256 without any problem. Mine was 128.
", "rawMarkdown": "Yeah, but how likely would that be? Some of the winning solutions ensembled two d_model 256 without any problem. Mine was 128.", "votes": null }, { "id": "1144086", "postDate": "01/08/2021 08:18:29", "content": "I hope the organizers say something about that. I literally could've gained three of four days of extra model refinement if it wasn't for it.
", "rawMarkdown": "I hope the organizers say something about that. I literally could've gained three of four days of extra model refinement if it wasn't for it.", "votes": null }, { "id": "1144359", "postDate": "01/08/2021 12:20:04", "content": "I am afraid we won't get the official response. Despite I am grateful for Kaggle's kind resource and effort to host competitions, I can't understand some of their decisions.
\nFor this competition, why we need runtime obfuscation? Is running time will leak the private dataset (other than the size)?
No running time showing in the submission page - I understand showing error message might leak information, but no running time shown?
The debate of not publishing high score notebook near the end - although it is subjective about the definition near the end, but why just making it a rule but just a suggestion ? What's the benefit of doing this?
No strong effort to prevent copy-fork-submit-(plagiarism) - maybe there is and I am wrong?
Not evaluate participant's progress and effort in a competition, but just their final results on private LB.
\nIf Kaggle think it is a competition + learning platform, why not taking into account, maybe as an extra ranking category, their progress in a competition, like each improvement in their submissions, and the lag time of each improvement etc.
I agree with much of what you said. If pushing research in the AI field is the goal for this competition, then why we waste more than half of our time debugging errors that would've been easily preventable, errors that you wouldn't face in a real life project.
", "rawMarkdown": "I agree with much of what you said. If pushing research in the AI field is the goal for this competition, then why we waste more than half of our time debugging errors that would've been easily preventable, errors that you wouldn't face in a real life project.", "votes": null }, { "id": "1144583", "postDate": "01/08/2021 14:45:15", "content": "I agree with most you said but we can discuss about point#1. I think we would need runtime obfuscation and hidden private data. I loved the API concept even if I agree it was a pain to troubleshoot. I think the main problem is probing, there are so many solutions to probe data including error codes, runtime and public LB score. Can you probe in a real life/production system? No. If kaggle can limit any probing then I think it's a good point.
", "rawMarkdown": "I agree with most you said but we can discuss about point#1. I think we would need runtime obfuscation and hidden private data. I loved the API concept even if I agree it was a pain to troubleshoot. I think the main problem is probing, there are so many solutions to probe data including error codes, runtime and public LB score. Can you probe in a real life/production system? No. If kaggle can limit any probing then I think it's a good point.", "votes": null }, { "id": "1144624", "postDate": "01/08/2021 15:14:51", "content": "@mpware I can only agree some of your points related to probing.
\nIt really depends. For example:
\nNo.)Actually, in order to design my pipeline, I do a lot of probing, like:
\ntest batch, once we regroup them by user_id, if they all start with (maybe empty) lectures, followed by a unique question bundle.I need this to design my CV / submission pipeline, and model input. Of course, I can ignore this information and go for another design. But what's the point to hide such information? Even in the real life/production, we need to define some format of the data input.
\nI would appreciate if you can give me some other viewpoints.
", "rawMarkdown": "mpware I can only agree some of your points related to probing.\n\nIt really depends. For example:\n\n- is test dataset size should be hidden (I think we all agree the answer is `No`.)\n- is the batch size should be hidden (personally, I don't think so. And in production, we also need an explicit batch size for system to be stable).\n- running time: could you justify why you think we should hide it?\n\nActually, in order to design my pipeline, I do a lot of probing, like:\n\n- in submission pipeline, for each `test batch`, once we regroup them by `user_id`, if they all start with (maybe empty) lectures, followed by a unique question bundle.\n\nI need this to design my CV / submission pipeline, and model input. Of course, I can ignore this information and go for another design. But what's the point to hide such information? Even in the real life/production, we need to define some format of the data input.\n\nI would appreciate if you can give me some other viewpoints.", "votes": null }, { "id": "1144637", "postDate": "01/08/2021 15:27:57", "content": "I agree with you on dataset size, batch size because it has nothing to do with probing. They should be known. I've lost a few submissions \"scoring error\" because of unknown batch size, my problem was OOM but I was able to reproduce it by modifying the Tito's simulator (so I realized that some batch size could be high). Runtime can be used to probe, that's the only reason.
Would you mind to give 1 or 2 examples of what kind of information that could be be probed from running time, that should remain secret for fair competition? I am curious
", "rawMarkdown": "Would you mind to give 1 or 2 examples of what kind of information that could be be probed from running time, that should remain secret for fair competition? I am curious", "votes": null }, { "id": "1144642", "postDate": "01/08/2021 15:33:51", "content": "And although I didn't check Tito's simulator (I made my own), I think without some kind of probing, the simulator won't be developed …
", "rawMarkdown": "And although I didn't check Tito's simulator (I made my own), I think without some kind of probing, the simulator won't be developed ...", "votes": null }, { "id": "1144827", "postDate": "01/08/2021 17:40:07", "content": "you could insert some sleep commands based on the \"prior answered correctly\" data, for example revealing the number of correct answers in each batch - you would need a lot of submissions, where each submission only scores a few of the batches, but with a lot of fake accounts would be doable
", "rawMarkdown": "you could insert some sleep commands based on the \"prior answered correctly\" data, for example revealing the number of correct answers in each batch - you would need a lot of submissions, where each submission only scores a few of the batches, but with a lot of fake accounts would be doable", "votes": null }, { "id": "1144891", "postDate": "01/08/2021 18:21:24", "content": "Make the runtime stop after a given time depending on some condition that would give you info.
\nLook at runtime and you've the info. If runtime is obfuscated you cannot know.
Well, you are right on this point.
", "rawMarkdown": "Well, you are right on this point.", "votes": null } ], "comments": [ { "id": 1144050, "author_name": "pinterbagz", "author_url": "", "post_date": "01/08/2021 07:52:31", "content": "I experienced the same thing towards the end of the competition. I had a model submitted successfully and I tried submitting the same model but trained for one more epoch and it gave me an error. Not sure if I have a bug as well or something else is going on.
", "votes": null, "replies": [ { "id": 1144086, "author_name": "abdessalemboukil", "author_url": "", "post_date": "01/08/2021 08:18:29", "content": "I hope the organizers say something about that. I literally could've gained three of four days of extra model refinement if it wasn't for it.
", "votes": null, "replies": [] } ] }, { "id": 1144061, "author_name": "rashmibanthia", "author_url": "", "post_date": "01/08/2021 08:06:32", "content": "It’s runtime obfuscation, mentioned in the data page. (I also experienced it) This competition was lot of fun except this inconsistent runtime part.
", "votes": null, "replies": [ { "id": 1144073, "author_name": "abdessalemboukil", "author_url": "", "post_date": "01/08/2021 08:11:18", "content": "Run time obfuscation means that the score would be revealed eventually, but in my case it just fails with the most obscure error code ever: Submission failed.
", "votes": null, "replies": [] }, { "id": 1144076, "author_name": "rashmibanthia", "author_url": "", "post_date": "01/08/2021 08:14:41", "content": "May be one submission took ~8 hrs 30 mins (and you got your score) another one went past 9hr and all we got was error. (I faced this problem)
", "votes": null, "replies": [] }, { "id": 1144082, "author_name": "abdessalemboukil", "author_url": "", "post_date": "01/08/2021 08:17:07", "content": "Yeah, but how likely would that be? Some of the winning solutions ensembled two d_model 256 without any problem. Mine was 128.
", "votes": null, "replies": [] } ] }, { "id": 1144359, "author_name": "yihdarshieh", "author_url": "", "post_date": "01/08/2021 12:20:04", "content": "I am afraid we won't get the official response. Despite I am grateful for Kaggle's kind resource and effort to host competitions, I can't understand some of their decisions.
\nFor this competition, why we need runtime obfuscation? Is running time will leak the private dataset (other than the size)?
No running time showing in the submission page - I understand showing error message might leak information, but no running time shown?
The debate of not publishing high score notebook near the end - although it is subjective about the definition near the end, but why just making it a rule but just a suggestion ? What's the benefit of doing this?
No strong effort to prevent copy-fork-submit-(plagiarism) - maybe there is and I am wrong?
Not evaluate participant's progress and effort in a competition, but just their final results on private LB.
\nIf Kaggle think it is a competition + learning platform, why not taking into account, maybe as an extra ranking category, their progress in a competition, like each improvement in their submissions, and the lag time of each improvement etc.
I agree with much of what you said. If pushing research in the AI field is the goal for this competition, then why we waste more than half of our time debugging errors that would've been easily preventable, errors that you wouldn't face in a real life project.
", "votes": null, "replies": [] }, { "id": 1144583, "author_name": "mpware", "author_url": "", "post_date": "01/08/2021 14:45:15", "content": "I agree with most you said but we can discuss about point#1. I think we would need runtime obfuscation and hidden private data. I loved the API concept even if I agree it was a pain to troubleshoot. I think the main problem is probing, there are so many solutions to probe data including error codes, runtime and public LB score. Can you probe in a real life/production system? No. If kaggle can limit any probing then I think it's a good point.
", "votes": null, "replies": [] }, { "id": 1144624, "author_name": "yihdarshieh", "author_url": "", "post_date": "01/08/2021 15:14:51", "content": "@mpware I can only agree some of your points related to probing.
\nIt really depends. For example:
\nNo.)Actually, in order to design my pipeline, I do a lot of probing, like:
\ntest batch, once we regroup them by user_id, if they all start with (maybe empty) lectures, followed by a unique question bundle.I need this to design my CV / submission pipeline, and model input. Of course, I can ignore this information and go for another design. But what's the point to hide such information? Even in the real life/production, we need to define some format of the data input.
\nI would appreciate if you can give me some other viewpoints.
", "votes": null, "replies": [] }, { "id": 1144637, "author_name": "mpware", "author_url": "", "post_date": "01/08/2021 15:27:57", "content": "I agree with you on dataset size, batch size because it has nothing to do with probing. They should be known. I've lost a few submissions \"scoring error\" because of unknown batch size, my problem was OOM but I was able to reproduce it by modifying the Tito's simulator (so I realized that some batch size could be high). Runtime can be used to probe, that's the only reason.
Would you mind to give 1 or 2 examples of what kind of information that could be be probed from running time, that should remain secret for fair competition? I am curious
", "votes": null, "replies": [ { "id": 1144891, "author_name": "mpware", "author_url": "", "post_date": "01/08/2021 18:21:24", "content": "Make the runtime stop after a given time depending on some condition that would give you info.
\nLook at runtime and you've the info. If runtime is obfuscated you cannot know.
And although I didn't check Tito's simulator (I made my own), I think without some kind of probing, the simulator won't be developed …
", "votes": null, "replies": [] }, { "id": 1144827, "author_name": "steelrose", "author_url": "", "post_date": "01/08/2021 17:40:07", "content": "you could insert some sleep commands based on the \"prior answered correctly\" data, for example revealing the number of correct answers in each batch - you would need a lot of submissions, where each submission only scores a few of the batches, but with a lot of fake accounts would be doable
", "votes": null, "replies": [] }, { "id": 1144898, "author_name": "yihdarshieh", "author_url": "", "post_date": "01/08/2021 18:28:29", "content": "Well, you are right on this point.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1144028": "Something I have struggled with in the last couple of weeks of the competition is notebooks submissions failing randomly. \n\nSame notebook, with no changed, submitted twice, one would get accepted and the other would fail. Not sure if it a timeout, or a bug or what, but it surely didn't make sense. Looking at the discussion, I found some people struggling with this problem, yet clueless on how to solve it.\n\nMy guess was that it was timing out, but now seeing other kagglers solutions, I think not. Some ensembled multiple transformers successfully, meanwhile my model was relatively small (4 M parameters, 100 seq). \n\nLet's not leave it a mystery please, does anyone knows why?", "1144050": "I experienced the same thing towards the end of the competition. I had a model submitted successfully and I tried submitting the same model but trained for one more epoch and it gave me an error. Not sure if I have a bug as well or something else is going on.", "1144061": "It’s runtime obfuscation, mentioned in the data page. (I also experienced it) This competition was lot of fun except this inconsistent runtime part.", "1144073": "Run time obfuscation means that the score would be revealed eventually, but in my case it just fails with the most obscure error code ever: Submission failed.", "1144076": "May be one submission took ~8 hrs 30 mins (and you got your score) another one went past 9hr and all we got was error. (I faced this problem)", "1144082": "Yeah, but how likely would that be? Some of the winning solutions ensembled two d_model 256 without any problem. Mine was 128.", "1144086": "I hope the organizers say something about that. I literally could've gained three of four days of extra model refinement if it wasn't for it.", "1144359": "I am afraid we won't get the official response. Despite I am grateful for Kaggle's kind resource and effort to host competitions, I can't understand some of their decisions.\n\n- For this competition, why we need runtime obfuscation? Is running time will leak the private dataset (other than the size)?\n\n- No running time showing in the submission page - I understand showing error message might leak information, but no running time shown?\n\n- The debate of not publishing high score notebook near the end - although it is subjective about the definition `near the end`, but why just making it a rule but just a suggestion ? What's the benefit of doing this?\n\n- No strong effort to prevent copy-fork-submit-(plagiarism) - maybe there is and I am wrong?\n\n- Not evaluate participant's progress and effort in a competition, but just their final results on private LB.\n If Kaggle think it is a competition + learning platform, why not taking into account, maybe as an extra ranking category, their progress in a competition, like each improvement in their submissions, and the lag time of each improvement etc.", "1144510": "I agree with much of what you said. If pushing research in the AI field is the goal for this competition, then why we waste more than half of our time debugging errors that would've been easily preventable, errors that you wouldn't face in a real life project.", "1144583": "I agree with most you said but we can discuss about point#1. I think we would need runtime obfuscation and hidden private data. I loved the API concept even if I agree it was a pain to troubleshoot. I think the main problem is probing, there are so many solutions to probe data including error codes, runtime and public LB score. Can you probe in a real life/production system? No. If kaggle can limit any probing then I think it's a good point.", "1144624": "mpware I can only agree some of your points related to probing.\n\nIt really depends. For example:\n\n- is test dataset size should be hidden (I think we all agree the answer is `No`.)\n- is the batch size should be hidden (personally, I don't think so. And in production, we also need an explicit batch size for system to be stable).\n- running time: could you justify why you think we should hide it?\n\nActually, in order to design my pipeline, I do a lot of probing, like:\n\n- in submission pipeline, for each `test batch`, once we regroup them by `user_id`, if they all start with (maybe empty) lectures, followed by a unique question bundle.\n\nI need this to design my CV / submission pipeline, and model input. Of course, I can ignore this information and go for another design. But what's the point to hide such information? Even in the real life/production, we need to define some format of the data input.\n\nI would appreciate if you can give me some other viewpoints.", "1144637": "I agree with you on `dataset size`, `batch size` because it has nothing to do with probing. They should be known. I've lost a few submissions \"scoring error\" because of unknown batch size, my problem was OOM but I was able to reproduce it by modifying the Tito's simulator (so I realized that some batch size could be high). `Runtime` can be used to probe, that's the only reason.", "1144639": "Would you mind to give 1 or 2 examples of what kind of information that could be be probed from running time, that should remain secret for fair competition? I am curious", "1144642": "And although I didn't check Tito's simulator (I made my own), I think without some kind of probing, the simulator won't be developed ...", "1144827": "you could insert some sleep commands based on the \"prior answered correctly\" data, for example revealing the number of correct answers in each batch - you would need a lot of submissions, where each submission only scores a few of the batches, but with a lot of fake accounts would be doable", "1144891": "Make the runtime stop after a given time depending on some condition that would give you info.\nLook at runtime and you've the info. If runtime is obfuscated you cannot know.", "1144898": "Well, you are right on this point." }, "source": "meta" }