Heavy ensemble helped us to keep silver (if nothing changes).
\n\nHere is our result:
\n\nBackup submission\n\nRank 96 (Silver): 19 Model Ensamble: Private: 0.50585, Public: 0.30284\n
Preferred submission\n\nEquivalent to Rank 118 (Bronze): 9 Model Ensemble: Private: 0.51518, Public: 0.29879\n
How was yours?
", "messages": [ { "id": "818568", "postDate": "04/24/2020 01:16:33", "content": "Heavy ensemble helped us to keep silver (if nothing changes).
\n\nHere is our result:
\n\nBackup submission\n\nRank 96 (Silver): 19 Model Ensamble: Private: 0.50585, Public: 0.30284\n
Preferred submission\n\nEquivalent to Rank 118 (Bronze): 9 Model Ensemble: Private: 0.51518, Public: 0.29879\n
How was yours?
", "rawMarkdown": "Heavy ensemble helped us to keep silver (if nothing changes).\n\nHere is our result:\n\nBackup submission\n```\nRank 96 (Silver): 19 Model Ensamble: Private: 0.50585, Public: 0.30284\n```\n\nPreferred submission\n```\nEquivalent to Rank 118 (Bronze): 9 Model Ensemble: Private: 0.51518, Public: 0.29879\n```\n\nHow was yours?", "votes": null }, { "id": "818572", "postDate": "04/24/2020 01:20:22", "content": "Backup sub: Rank 69 (silver): 7 model Ensemble, Public/Private: 0.36240/0.49641\nPreferred sub: Rank 105 (silver): 3 model Ensemble, Public/Private: 0.36044/0.50889
", "rawMarkdown": "Backup sub: Rank 69 (silver): 7 model Ensemble, Public/Private: 0.36240/0.49641\nPreferred sub: Rank 105 (silver): 3 model Ensemble, Public/Private: 0.36044/0.50889", "votes": null }, { "id": "818573", "postDate": "04/24/2020 01:21:16", "content": "That's impressive. Congrats!
", "rawMarkdown": "That's impressive. Congrats!", "votes": null }, { "id": "818690", "postDate": "04/24/2020 03:57:05", "content": "Congratulations!
", "rawMarkdown": "Congratulations!", "votes": null }, { "id": "818693", "postDate": "04/24/2020 04:01:46", "content": "Thanks๐
", "rawMarkdown": "Thanks๐", "votes": null }, { "id": "818785", "postDate": "04/24/2020 06:00:39", "content": "Models: Each ensemble has 5 EfficentNet-B5's for face classification. I also used a separate audio model but I do not wish to disclose those details.
\n\nEnsemble 1: Public LB: .192 Private LB: .577\nEnsemble 2: Public LB: .195 Private LB: .575\nEnsemble 3 (5 EfficentNet-B5's with no audio): Public LB: ~.20
\n\nThere's a large discrepancy between my public and private LB scores and I notice that there are quite a few other people with very large differences. My guess is that the audio was handled very differently in the private set but I'm not sure.
\n\nIn my submitted ensembles, audio has a large impact on the final prediction of my ensemble if the audio prediction is very confident in a fake, therefore, if the audio was handled differently or fundamentally changed in the private set then I would expect my performance to be a lot worse. I really regret not submitting an ensemble without audio.
\n\nI'm very curious to hear from others who have a large difference between public LB and private LB.
", "rawMarkdown": "Models: Each ensemble has 5 EfficentNet-B5's for face classification. I also used a separate audio model but I do not wish to disclose those details.\n\nEnsemble 1: Public LB: .192 Private LB: .577\nEnsemble 2: Public LB: .195 Private LB: .575\nEnsemble 3 (5 EfficentNet-B5's with no audio): Public LB: ~.20\n\nThere's a large discrepancy between my public and private LB scores and I notice that there are quite a few other people with very large differences. My guess is that the audio was handled very differently in the private set but I'm not sure.\n\nIn my submitted ensembles, audio has a large impact on the final prediction of my ensemble if the audio prediction is very confident in a fake, therefore, if the audio was handled differently or fundamentally changed in the private set then I would expect my performance to be a lot worse. I really regret not submitting an ensemble without audio.\n\nI'm very curious to hear from others who have a large difference between public LB and private LB.", "votes": null }, { "id": "818871", "postDate": "04/24/2020 07:12:42", "content": "You were cool, I think you were just out of luck. I was in 39th place and ended up without a rank at all
", "rawMarkdown": "You were cool, I think you were just out of luck. I was in 39th place and ended up without a rank at all", "votes": null }, { "id": "818948", "postDate": "04/24/2020 08:25:12", "content": "Preferred submission\nRank 8 (Gold): 8 model ensemble with equal weighting (writeup here)\nPublic LB: 0.25322 (rank 17 (was 20))\nPrivate LB: 0.43711
\n\nBackup submission\nEquivalent to rank 8 (Gold): 8 model ensemble with different weights. \nPublic LB: 0.25467\nPrivate LB: 0.43954
", "rawMarkdown": "Preferred submission\nRank 8 (Gold): 8 model ensemble with equal weighting (writeup [here](https://www.kaggle.com/c/deepfake-detection-challenge/discussion/140364#794027))\nPublic LB: 0.25322 (rank 17 (was 20))\nPrivate LB: 0.43711\n\nBackup submission\nEquivalent to rank 8 (Gold): 8 model ensemble with different weights. \nPublic LB: 0.25467\nPrivate LB: 0.43954", "votes": null }, { "id": "819051", "postDate": "04/24/2020 10:11:37", "content": "Looks like audio harmed. Public 0.2LB with 5 EffB5 is amazing. We got 0.301LB with vanilla 3 Efficientnet B3 ensemble (3 folds), no audio. Our preferred version was just running this version 3 times with some mild hyper parameters change and ensembling these 9 modelsโ that got us Rank 108.
", "rawMarkdown": "Looks like audio harmed. Public 0.2LB with 5 EffB5 is amazing. We got 0.301LB with vanilla 3 Efficientnet B3 ensemble (3 folds), no audio. Our preferred version was just running this version 3 times with some mild hyper parameters change and ensembling these 9 modelsโ that got us Rank 108.", "votes": null }, { "id": "819499", "postDate": "04/24/2020 16:26:18", "content": "@azamatk Ouch, that must be very frustrating.\n@debanga Thanks for sharing and congrats on the silver!
", "rawMarkdown": "azamatk Ouch, that must be very frustrating.\n@debanga Thanks for sharing and congrats on the silver!", "votes": null }, { "id": "820761", "postDate": "04/25/2020 17:27:39", "content": "private: 0.55950 public: 0.29529
", "rawMarkdown": "private: 0.55950 public: 0.29529", "votes": null }, { "id": "820822", "postDate": "04/25/2020 18:28:08", "content": "Private: 0.50640 (99th position), public: 0.37520 (180th position, was 210-ish before the final standings).
\n\nNothing particularly special about my kernel. I made an ensemble of my 3 best-performing models (all based on ResNeXt-50). Also used horizontal flips at test time.
", "rawMarkdown": "Private: 0.50640 (99th position), public: 0.37520 (180th position, was 210-ish before the final standings).\n\nNothing particularly special about my kernel. I made an ensemble of my 3 best-performing models (all based on ResNeXt-50). Also used horizontal flips at test time.", "votes": null }, { "id": "821394", "postDate": "04/26/2020 06:34:27", "content": "Actually in my opinion there should be a special prize for you. You helped many people here, I think many people forked your Kernel as well. You also answered the questions of many people. I hope that you would get the gold next time. Really learned a lot from your work.
", "rawMarkdown": "Actually in my opinion there should be a special prize for you. You helped many people here, I think many people forked your Kernel as well. You also answered the questions of many people. I hope that you would get the gold next time. Really learned a lot from your work.", "votes": null }, { "id": "821700", "postDate": "04/26/2020 10:58:35", "content": "@davidmilam we need to hug and cry together) our team also applies audio detection, but we don't notice a large impact of audio on the public score.
", "rawMarkdown": "davidmilam we need to hug and cry together) our team also applies audio detection, but we don't notice a large impact of audio on the public score.", "votes": null }, { "id": "822304", "postDate": "04/26/2020 20:12:52", "content": "@sorokin Ouch, so your team went from rank 22 to 1261, it is painful to go down in rank by so much. And yeah, audio barely helps on the local and public set. It is a shame that the audio does not seem very well thought out for this competition. Iโm curious as to what their reasoning was regarding certain decisions with the audio.
", "rawMarkdown": "sorokin Ouch, so your team went from rank 22 to 1261, it is painful to go down in rank by so much. And yeah, audio barely helps on the local and public set. It is a shame that the audio does not seem very well thought out for this competition. Iโm curious as to what their reasoning was regarding certain decisions with the audio.", "votes": null }, { "id": "822996", "postDate": "04/27/2020 10:08:06", "content": "@davidmilam Thanks for your sharing. sorry to hear your condition. you are so cool ๐ ๐ just use 5 EfficentNet-B5, got 1st in pub LB. Expectedly for your later sharing details.
", "rawMarkdown": "davidmilam Thanks for your sharing. sorry to hear your condition. you are so cool ๐ ๐ just use 5 EfficentNet-B5, got 1st in pub LB. Expectedly for your later sharing details.", "votes": null }, { "id": "823018", "postDate": "04/27/2020 10:36:56", "content": "Private: 0.49881 (77th)\nPublic: 0.32986 (100th, was 123 or something after submission, 118 the day after submissions were closed - looks like some teams were wiped even before their submissions were scored)\nA simple ensemble of 3 Efnets - b0 + b3 + b5, though I've used a small amount of faces to train on to avoid overfitting.\n(350000 faces, 1 epoch)
\n\nI've also started with @humananalog 's kernel, but by the end there was only the video reader left from the original notebook :P\nHuge thanks for your starter notebook, I wonder how many submissions I would've wasted just making any notebook run without errors in this weird (error-wise) competition. I also wanted to team up with you, but weren't confident enough to approach, lol.
", "rawMarkdown": "Private: 0.49881 (77th)\nPublic: 0.32986 (100th, was 123 or something after submission, 118 the day after submissions were closed - looks like some teams were wiped even before their submissions were scored)\nA simple ensemble of 3 Efnets - b0 + b3 + b5, though I've used a small amount of faces to train on to avoid overfitting.\n(350000 faces, 1 epoch)\n\nI've also started with @humananalog 's kernel, but by the end there was only the video reader left from the original notebook :P\nHuge thanks for your starter notebook, I wonder how many submissions I would've wasted just making any notebook run without errors in this weird (error-wise) competition. I also wanted to team up with you, but weren't confident enough to approach, lol.", "votes": null }, { "id": "823041", "postDate": "04/27/2020 11:01:18", "content": "\n\n\nI also wanted to team up with you, but weren't confident enough to approach, lol.
\n
Good thing you didn't because you scored higher than I did. ๐
", "rawMarkdown": "> I also wanted to team up with you, but weren't confident enough to approach, lol.\n\nGood thing you didn't because you scored higher than I did. ๐", "votes": null }, { "id": "824287", "postDate": "04/28/2020 09:27:00", "content": "@humananalog haha, it would've been an ensemble, so who knows ๐
", "rawMarkdown": "humananalog haha, it would've been an ensemble, so who knows ๐", "votes": null } ], "comments": [ { "id": 818572, "author_name": "greatgamedota", "author_url": "", "post_date": "04/24/2020 01:20:22", "content": "Backup sub: Rank 69 (silver): 7 model Ensemble, Public/Private: 0.36240/0.49641\nPreferred sub: Rank 105 (silver): 3 model Ensemble, Public/Private: 0.36044/0.50889
", "votes": null, "replies": [ { "id": 818573, "author_name": "debanga", "author_url": "", "post_date": "04/24/2020 01:21:16", "content": "That's impressive. Congrats!
", "votes": null, "replies": [] } ] }, { "id": 818690, "author_name": "abualabed", "author_url": "", "post_date": "04/24/2020 03:57:05", "content": "Congratulations!
", "votes": null, "replies": [ { "id": 818693, "author_name": "debanga", "author_url": "", "post_date": "04/24/2020 04:01:46", "content": "Thanks๐
", "votes": null, "replies": [] } ] }, { "id": 818785, "author_name": "davidmilam", "author_url": "", "post_date": "04/24/2020 06:00:39", "content": "Models: Each ensemble has 5 EfficentNet-B5's for face classification. I also used a separate audio model but I do not wish to disclose those details.
\n\nEnsemble 1: Public LB: .192 Private LB: .577\nEnsemble 2: Public LB: .195 Private LB: .575\nEnsemble 3 (5 EfficentNet-B5's with no audio): Public LB: ~.20
\n\nThere's a large discrepancy between my public and private LB scores and I notice that there are quite a few other people with very large differences. My guess is that the audio was handled very differently in the private set but I'm not sure.
\n\nIn my submitted ensembles, audio has a large impact on the final prediction of my ensemble if the audio prediction is very confident in a fake, therefore, if the audio was handled differently or fundamentally changed in the private set then I would expect my performance to be a lot worse. I really regret not submitting an ensemble without audio.
\n\nI'm very curious to hear from others who have a large difference between public LB and private LB.
", "votes": null, "replies": [ { "id": 818871, "author_name": "azamatk", "author_url": "", "post_date": "04/24/2020 07:12:42", "content": "You were cool, I think you were just out of luck. I was in 39th place and ended up without a rank at all
", "votes": null, "replies": [] }, { "id": 819051, "author_name": "debanga", "author_url": "", "post_date": "04/24/2020 10:11:37", "content": "Looks like audio harmed. Public 0.2LB with 5 EffB5 is amazing. We got 0.301LB with vanilla 3 Efficientnet B3 ensemble (3 folds), no audio. Our preferred version was just running this version 3 times with some mild hyper parameters change and ensembling these 9 modelsโ that got us Rank 108.
", "votes": null, "replies": [] }, { "id": 819499, "author_name": "davidmilam", "author_url": "", "post_date": "04/24/2020 16:26:18", "content": "@azamatk Ouch, that must be very frustrating.\n@debanga Thanks for sharing and congrats on the silver!
", "votes": null, "replies": [] }, { "id": 821700, "author_name": "sorokin", "author_url": "", "post_date": "04/26/2020 10:58:35", "content": "@davidmilam we need to hug and cry together) our team also applies audio detection, but we don't notice a large impact of audio on the public score.
", "votes": null, "replies": [] }, { "id": 822304, "author_name": "davidmilam", "author_url": "", "post_date": "04/26/2020 20:12:52", "content": "@sorokin Ouch, so your team went from rank 22 to 1261, it is painful to go down in rank by so much. And yeah, audio barely helps on the local and public set. It is a shame that the audio does not seem very well thought out for this competition. Iโm curious as to what their reasoning was regarding certain decisions with the audio.
", "votes": null, "replies": [] }, { "id": 822996, "author_name": "wangtao360", "author_url": "", "post_date": "04/27/2020 10:08:06", "content": "@davidmilam Thanks for your sharing. sorry to hear your condition. you are so cool ๐ ๐ just use 5 EfficentNet-B5, got 1st in pub LB. Expectedly for your later sharing details.
", "votes": null, "replies": [] } ] }, { "id": 818948, "author_name": "jamesphoward", "author_url": "", "post_date": "04/24/2020 08:25:12", "content": "Preferred submission\nRank 8 (Gold): 8 model ensemble with equal weighting (writeup here)\nPublic LB: 0.25322 (rank 17 (was 20))\nPrivate LB: 0.43711
\n\nBackup submission\nEquivalent to rank 8 (Gold): 8 model ensemble with different weights. \nPublic LB: 0.25467\nPrivate LB: 0.43954
", "votes": null, "replies": [] }, { "id": 820761, "author_name": "unkownhihi", "author_url": "", "post_date": "04/25/2020 17:27:39", "content": "private: 0.55950 public: 0.29529
", "votes": null, "replies": [] }, { "id": 820822, "author_name": "humananalog", "author_url": "", "post_date": "04/25/2020 18:28:08", "content": "Private: 0.50640 (99th position), public: 0.37520 (180th position, was 210-ish before the final standings).
\n\nNothing particularly special about my kernel. I made an ensemble of my 3 best-performing models (all based on ResNeXt-50). Also used horizontal flips at test time.
", "votes": null, "replies": [ { "id": 821394, "author_name": "abualabed", "author_url": "", "post_date": "04/26/2020 06:34:27", "content": "Actually in my opinion there should be a special prize for you. You helped many people here, I think many people forked your Kernel as well. You also answered the questions of many people. I hope that you would get the gold next time. Really learned a lot from your work.
", "votes": null, "replies": [] } ] }, { "id": 823018, "author_name": "defileroff", "author_url": "", "post_date": "04/27/2020 10:36:56", "content": "Private: 0.49881 (77th)\nPublic: 0.32986 (100th, was 123 or something after submission, 118 the day after submissions were closed - looks like some teams were wiped even before their submissions were scored)\nA simple ensemble of 3 Efnets - b0 + b3 + b5, though I've used a small amount of faces to train on to avoid overfitting.\n(350000 faces, 1 epoch)
\n\nI've also started with @humananalog 's kernel, but by the end there was only the video reader left from the original notebook :P\nHuge thanks for your starter notebook, I wonder how many submissions I would've wasted just making any notebook run without errors in this weird (error-wise) competition. I also wanted to team up with you, but weren't confident enough to approach, lol.
", "votes": null, "replies": [ { "id": 823041, "author_name": "humananalog", "author_url": "", "post_date": "04/27/2020 11:01:18", "content": "\n\n\nI also wanted to team up with you, but weren't confident enough to approach, lol.
\n
Good thing you didn't because you scored higher than I did. ๐
", "votes": null, "replies": [] }, { "id": 824287, "author_name": "defileroff", "author_url": "", "post_date": "04/28/2020 09:27:00", "content": "@humananalog haha, it would've been an ensemble, so who knows ๐
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "818568": "Heavy ensemble helped us to keep silver (if nothing changes).\n\nHere is our result:\n\nBackup submission\n```\nRank 96 (Silver): 19 Model Ensamble: Private: 0.50585, Public: 0.30284\n```\n\nPreferred submission\n```\nEquivalent to Rank 118 (Bronze): 9 Model Ensemble: Private: 0.51518, Public: 0.29879\n```\n\nHow was yours?", "818572": "Backup sub: Rank 69 (silver): 7 model Ensemble, Public/Private: 0.36240/0.49641\nPreferred sub: Rank 105 (silver): 3 model Ensemble, Public/Private: 0.36044/0.50889", "818573": "That's impressive. Congrats!", "818690": "Congratulations!", "818693": "Thanks๐", "818785": "Models: Each ensemble has 5 EfficentNet-B5's for face classification. I also used a separate audio model but I do not wish to disclose those details.\n\nEnsemble 1: Public LB: .192 Private LB: .577\nEnsemble 2: Public LB: .195 Private LB: .575\nEnsemble 3 (5 EfficentNet-B5's with no audio): Public LB: ~.20\n\nThere's a large discrepancy between my public and private LB scores and I notice that there are quite a few other people with very large differences. My guess is that the audio was handled very differently in the private set but I'm not sure.\n\nIn my submitted ensembles, audio has a large impact on the final prediction of my ensemble if the audio prediction is very confident in a fake, therefore, if the audio was handled differently or fundamentally changed in the private set then I would expect my performance to be a lot worse. I really regret not submitting an ensemble without audio.\n\nI'm very curious to hear from others who have a large difference between public LB and private LB.", "818871": "You were cool, I think you were just out of luck. I was in 39th place and ended up without a rank at all", "818948": "Preferred submission\nRank 8 (Gold): 8 model ensemble with equal weighting (writeup [here](https://www.kaggle.com/c/deepfake-detection-challenge/discussion/140364#794027))\nPublic LB: 0.25322 (rank 17 (was 20))\nPrivate LB: 0.43711\n\nBackup submission\nEquivalent to rank 8 (Gold): 8 model ensemble with different weights. \nPublic LB: 0.25467\nPrivate LB: 0.43954", "819051": "Looks like audio harmed. Public 0.2LB with 5 EffB5 is amazing. We got 0.301LB with vanilla 3 Efficientnet B3 ensemble (3 folds), no audio. Our preferred version was just running this version 3 times with some mild hyper parameters change and ensembling these 9 modelsโ that got us Rank 108.", "819499": "azamatk Ouch, that must be very frustrating.\n@debanga Thanks for sharing and congrats on the silver!", "820761": "private: 0.55950 public: 0.29529", "820822": "Private: 0.50640 (99th position), public: 0.37520 (180th position, was 210-ish before the final standings).\n\nNothing particularly special about my kernel. I made an ensemble of my 3 best-performing models (all based on ResNeXt-50). Also used horizontal flips at test time.", "821394": "Actually in my opinion there should be a special prize for you. You helped many people here, I think many people forked your Kernel as well. You also answered the questions of many people. I hope that you would get the gold next time. Really learned a lot from your work.", "821700": "davidmilam we need to hug and cry together) our team also applies audio detection, but we don't notice a large impact of audio on the public score.", "822304": "sorokin Ouch, so your team went from rank 22 to 1261, it is painful to go down in rank by so much. And yeah, audio barely helps on the local and public set. It is a shame that the audio does not seem very well thought out for this competition. Iโm curious as to what their reasoning was regarding certain decisions with the audio.", "822996": "davidmilam Thanks for your sharing. sorry to hear your condition. you are so cool ๐ ๐ just use 5 EfficentNet-B5, got 1st in pub LB. Expectedly for your later sharing details.", "823018": "Private: 0.49881 (77th)\nPublic: 0.32986 (100th, was 123 or something after submission, 118 the day after submissions were closed - looks like some teams were wiped even before their submissions were scored)\nA simple ensemble of 3 Efnets - b0 + b3 + b5, though I've used a small amount of faces to train on to avoid overfitting.\n(350000 faces, 1 epoch)\n\nI've also started with @humananalog 's kernel, but by the end there was only the video reader left from the original notebook :P\nHuge thanks for your starter notebook, I wonder how many submissions I would've wasted just making any notebook run without errors in this weird (error-wise) competition. I also wanted to team up with you, but weren't confident enough to approach, lol.", "823041": "> I also wanted to team up with you, but weren't confident enough to approach, lol.\n\nGood thing you didn't because you scored higher than I did. ๐", "824287": "humananalog haha, it would've been an ensemble, so who knows ๐" }, "source": "meta" }