Hi, everyone!
\n\nI would like to understand what I have wrong in this competition. \nSo I made public my final kernel, that on public LB worked CORRECT! Also I made all datasets for this kernel public too.
\n\nkernel:\nhttps://www.kaggle.com/shonenkov/final-kfold-inference-effb2
\n\nmodels:\nhttps://www.kaggle.com/shonenkov/kfolddeepfakeeffb2-flip\nhttps://www.kaggle.com/shonenkov/kdold-deepfake-effb2
\n\nmodificated face detecto:\nhttps://www.kaggle.com/shonenkov/face-detector
\n\nI don't have perfect solution, but it has 113th public place.
\n\nWhat do you think about reasons for crushing my solution on private LB?
\n\nThank you in advance everyone for help to understand
\n\n[UPDATE]:
\n\nSo! After some discussions I have got some arguments. I do try/expect everywhere besides one moment! I don't try/expect for checking input signature and correctness input data. But on public data I have correct kernel!
\n\nI think organizers must give data with input signature according to competition rules and public data
\n\nWhat do you think about whose fault is it?
", "messages": [ { "id": 820358, "postDate": "2020-04-25T11:28:59.333Z", "content": "Hi, everyone!
\n\nI would like to understand what I have wrong in this competition. \nSo I made public my final kernel, that on public LB worked CORRECT! Also I made all datasets for this kernel public too.
\n\nkernel:\nhttps://www.kaggle.com/shonenkov/final-kfold-inference-effb2
\n\nmodels:\nhttps://www.kaggle.com/shonenkov/kfolddeepfakeeffb2-flip\nhttps://www.kaggle.com/shonenkov/kdold-deepfake-effb2
\n\nmodificated face detecto:\nhttps://www.kaggle.com/shonenkov/face-detector
\n\nI don't have perfect solution, but it has 113th public place.
\n\nWhat do you think about reasons for crushing my solution on private LB?
\n\nThank you in advance everyone for help to understand
\n\n[UPDATE]:
\n\nSo! After some discussions I have got some arguments. I do try/expect everywhere besides one moment! I don't try/expect for checking input signature and correctness input data. But on public data I have correct kernel!
\n\nI think organizers must give data with input signature according to competition rules and public data
\n\nWhat do you think about whose fault is it?
", "rawMarkdown": "Hi, everyone!\n\nI would like to understand what I have wrong in this competition. \nSo I made public my final kernel, that on public LB worked CORRECT! Also I made all datasets for this kernel public too.\n\nkernel:\nhttps://www.kaggle.com/shonenkov/final-kfold-inference-effb2 \n\nmodels:\nhttps://www.kaggle.com/shonenkov/kfolddeepfakeeffb2-flip\nhttps://www.kaggle.com/shonenkov/kdold-deepfake-effb2\n\nmodificated face detecto:\nhttps://www.kaggle.com/shonenkov/face-detector\n\n\nI don't have perfect solution, but it has 113th public place. \n\nWhat do you think about reasons for crushing my solution on private LB?\n\nThank you in advance everyone for help to understand \n\n\n\n\n\n[UPDATE]:\n\nSo! After some discussions I have got some arguments. I do try/expect everywhere besides one moment! I don't try/expect for checking input signature and correctness input data. But on public data I have correct kernel! \n\nI think organizers must give data with input signature according to competition rules and public data\n\nWhat do you think about whose fault is it? \n", "votes": 9 }, { "id": 820374, "postDate": "2020-04-25T11:51:38.937Z", "content": "It looks like you create a dataset of videos before iterating of them, and only if predicting a video fails do you set its prediction to 0.5.\nIt therefore appears that if a video fails before this point e.g. when building your DatasetRetrieverthat that video will end up without a prediction. This means it would be an invalid submission.
Also, because you haven't try/except escaped this, I'm worried that an error when instantiating your dataset = DatasetRetriever() such as when using your FaceDetectorcould also be unhandled, but maybe that it escaped inside that class which we can't see as it's imported:
\ndef process_dfs(df, num_workers=2):\n def process_df(sub_df):\n dataset = DatasetRetriever(sub_df)\n result = deep_fake_predictor.predict(dataset)\n return result\n
Thank you for answer!
\n\nIn FaceDetector I used try except also:
\n\ninit.py\">https://www.kaggle.com/shonenkov/face-detector#init.py
\n\nBut I didn't try/except on checking video signature. I expect video (according to rules of competition)
", "rawMarkdown": "Thank you for answer!\n\nIn FaceDetector I used try except also:\n\nhttps://www.kaggle.com/shonenkov/face-detector#__init__.py\n\nBut I didn't try/except on checking video signature. I expect video (according to rules of competition)" } ] }, { "id": 826472, "postDate": "2020-04-29T17:01:31.533Z", "content": "Я не могу прокомментировать Ваш пост по сути, но хотел бы поддержать Вас... Я думаю, то что Вы делаете достаточно круто. Удачи!
", "rawMarkdown": "Я не могу прокомментировать Ваш пост по сути, но хотел бы поддержать Вас... Я думаю, то что Вы делаете достаточно круто. Удачи!", "votes": 1 }, { "id": 820365, "postDate": "2020-04-25T11:40:00.863Z", "content": "It's common for people to do well on the public LB and get much worse scores on the private LB, usually due to overfitting.
", "rawMarkdown": "It's common for people to do well on the public LB and get much worse scores on the private LB, usually due to overfitting.", "votes": -8, "replies": [ { "id": 820369, "postDate": "2020-04-25T11:44:13.277Z", "content": "thank you for statistics! I have used cv kfold and checkpoints.
\n\nDo you think It is overfitting?))
\n\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1920073%2F51c8319fc288d0d8c2a105bdf4d25a6b%2F2020-04-25%2014-41-58.png?generation=1587814980045473&alt=media\")
Well, we were 67th position public, but ended up very bad like 0.5 something, with even try except loops and all, we had way too many models with perfect datas, there must had been a fault in the data pipeline, the models were excellent, we were even expecting a high silver or a gold but who knows what ended up that bad...
", "rawMarkdown": "Well, we were 67th position public, but ended up very bad like 0.5 something, with even try except loops and all, we had way too many models with perfect datas, there must had been a fault in the data pipeline, the models were excellent, we were even expecting a high silver or a gold but who knows what ended up that bad..." }, { "id": 820410, "postDate": "2020-04-25T12:23:46.097Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 820374, "author_name": "James Howard", "author_url": "", "post_date": "2020-04-25T11:51:38.937000", "content": "It looks like you create a dataset of videos before iterating of them, and only if predicting a video fails do you set its prediction to 0.5.\nIt therefore appears that if a video fails before this point e.g. when building your DatasetRetrieverthat that video will end up without a prediction. This means it would be an invalid submission.
Also, because you haven't try/except escaped this, I'm worried that an error when instantiating your dataset = DatasetRetriever() such as when using your FaceDetectorcould also be unhandled, but maybe that it escaped inside that class which we can't see as it's imported:
\ndef process_dfs(df, num_workers=2):\n def process_df(sub_df):\n dataset = DatasetRetriever(sub_df)\n result = deep_fake_predictor.predict(dataset)\n return result\n
Thank you for answer!
\n\nIn FaceDetector I used try except also:
\n\ninit.py\">https://www.kaggle.com/shonenkov/face-detector#init.py
\n\nBut I didn't try/except on checking video signature. I expect video (according to rules of competition)
", "votes": 0, "replies": [] } ] }, { "id": 826472, "author_name": " Igor Krasovskiy", "author_url": "", "post_date": "2020-04-29T17:01:31.533000", "content": "Я не могу прокомментировать Ваш пост по сути, но хотел бы поддержать Вас... Я думаю, то что Вы делаете достаточно круто. Удачи!
", "votes": 1, "replies": [] }, { "id": 820365, "author_name": "Human Analog", "author_url": "", "post_date": "2020-04-25T11:40:00.863000", "content": "It's common for people to do well on the public LB and get much worse scores on the private LB, usually due to overfitting.
", "votes": -8, "replies": [ { "id": 820369, "author_name": "Alex Shonenkov", "author_url": "", "post_date": "2020-04-25T11:44:13.277000", "content": "thank you for statistics! I have used cv kfold and checkpoints.
\n\nDo you think It is overfitting?))
\n\n
Well, we were 67th position public, but ended up very bad like 0.5 something, with even try except loops and all, we had way too many models with perfect datas, there must had been a fault in the data pipeline, the models were excellent, we were even expecting a high silver or a gold but who knows what ended up that bad...
", "votes": 0, "replies": [] }, { "id": 820410, "author_name": "", "author_url": "", "post_date": "2020-04-25T12:23:46.097000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "820358": "Hi, everyone!\n\nI would like to understand what I have wrong in this competition. \nSo I made public my final kernel, that on public LB worked CORRECT! Also I made all datasets for this kernel public too.\n\nkernel:\nhttps://www.kaggle.com/shonenkov/final-kfold-inference-effb2 \n\nmodels:\nhttps://www.kaggle.com/shonenkov/kfolddeepfakeeffb2-flip\nhttps://www.kaggle.com/shonenkov/kdold-deepfake-effb2\n\nmodificated face detecto:\nhttps://www.kaggle.com/shonenkov/face-detector\n\n\nI don't have perfect solution, but it has 113th public place. \n\nWhat do you think about reasons for crushing my solution on private LB?\n\nThank you in advance everyone for help to understand \n\n\n\n\n\n[UPDATE]:\n\nSo! After some discussions I have got some arguments. I do try/expect everywhere besides one moment! I don't try/expect for checking input signature and correctness input data. But on public data I have correct kernel! \n\nI think organizers must give data with input signature according to competition rules and public data\n\nWhat do you think about whose fault is it? \n", "820374": "It looks like you create a dataset of videos before iterating of them, and only if predicting a video fails do you set its prediction to 0.5.\nIt therefore appears that if a video fails before this point e.g. when building your `DatasetRetriever `that that video will end up without a prediction. This means it would be an invalid submission.\n\nAlso, because you haven't try/except escaped this, I'm worried that an error when instantiating your `dataset = DatasetRetriever()` such as when using your `FaceDetector `could also be unhandled, but maybe that it escaped inside that class which we can't see as it's imported:\n\n```\ndef process_dfs(df, num_workers=2):\n def process_df(sub_df):\n dataset = DatasetRetriever(sub_df)\n result = deep_fake_predictor.predict(dataset)\n return result\n```", "826472": "Я не могу прокомментировать Ваш пост по сути, но хотел бы поддержать Вас... Я думаю, то что Вы делаете достаточно круто. Удачи!", "820365": "It's common for people to do well on the public LB and get much worse scores on the private LB, usually due to overfitting.", "833931": "Well, we were 67th position public, but ended up very bad like 0.5 something, with even try except loops and all, we had way too many models with perfect datas, there must had been a fault in the data pipeline, the models were excellent, we were even expecting a high silver or a gold but who knows what ended up that bad...", "820410": "" } }