![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1035002%2F42ed785e6460c04be97bceca5ec2922b%2FScreenshot%20from%202020-03-14%2022-59-42.png?generation=1584252006114093&alt=media\")
It's always interesting looking at the kaggle leaderboard stats and seeing how the distribution lands. Sometimes there is a gradual trailing off and everyone is clustering towards the top. In this competition, it has been quite a bit different. There are clear groupings of teams who have overcome various hurdles.
\n\n
These are just some hypotheses as to what might be the trick to get to these points. Looking at the top 250 teams we can see quite a lot are bunched up on the best public kernel around .43 and then a small group who have made some minor improvements beyond that, likely with ensembling or some model/data improvements. Beyond that, I hypothesize the next group has done something smart in terms of removing the noise in the labels, either removing frames from videos that were labeled fake that weren't really fake or removing poorly formed videos.
\n\nThe next step up is probably just some better method to do this than the previous grouping. They may have also found some way to make validation and test line up a bit better or make their model more robust.
\n\nBeyond that, I have to assume there are some more clever methods being used. Alternative representations of the data, additional features, non-standard model architectures or massive ensembles.
", "messages": [ { "id": 772177, "postDate": "2020-03-15T06:06:06.893Z", "content": "It's always interesting looking at the kaggle leaderboard stats and seeing how the distribution lands. Sometimes there is a gradual trailing off and everyone is clustering towards the top. In this competition, it has been quite a bit different. There are clear groupings of teams who have overcome various hurdles.
\n\n
These are just some hypotheses as to what might be the trick to get to these points. Looking at the top 250 teams we can see quite a lot are bunched up on the best public kernel around .43 and then a small group who have made some minor improvements beyond that, likely with ensembling or some model/data improvements. Beyond that, I hypothesize the next group has done something smart in terms of removing the noise in the labels, either removing frames from videos that were labeled fake that weren't really fake or removing poorly formed videos.
\n\nThe next step up is probably just some better method to do this than the previous grouping. They may have also found some way to make validation and test line up a bit better or make their model more robust.
\n\nBeyond that, I have to assume there are some more clever methods being used. Alternative representations of the data, additional features, non-standard model architectures or massive ensembles.
", "rawMarkdown": "It's always interesting looking at the kaggle leaderboard stats and seeing how the distribution lands. Sometimes there is a gradual trailing off and everyone is clustering towards the top. In this competition, it has been quite a bit different. There are clear groupings of teams who have overcome various hurdles. \n\n\n\nThese are just some hypotheses as to what might be the trick to get to these points. Looking at the top 250 teams we can see quite a lot are bunched up on the best public kernel around .43 and then a small group who have made some minor improvements beyond that, likely with ensembling or some model/data improvements. Beyond that, I hypothesize the next group has done something smart in terms of removing the noise in the labels, either removing frames from videos that were labeled fake that weren't really fake or removing poorly formed videos. \n\nThe next step up is probably just some better method to do this than the previous grouping. They may have also found some way to make validation and test line up a bit better or make their model more robust. \n\nBeyond that, I have to assume there are some more clever methods being used. Alternative representations of the data, additional features, non-standard model architectures or massive ensembles. \n", "votes": 21 }, { "id": 772713, "postDate": "2020-03-15T20:06:36.500Z", "content": "Our 0.30003LB: No cleaning (simple uniform N frame extraction from video), ensembles of several single CNN models, multi-frame inference, good folder-wise validation.
", "rawMarkdown": "Our 0.30003LB: No cleaning (simple uniform N frame extraction from video), ensembles of several single CNN models, multi-frame inference, good folder-wise validation.", "votes": 14, "replies": [ { "id": 784369, "postDate": "2020-03-24T06:53:54.310Z", "content": "How many frames per video?and how mang CNN models do you use to ensamble?
", "rawMarkdown": "How many frames per video?and how mang CNN models do you use to ensamble? \n" } ] }, { "id": 776667, "postDate": "2020-03-17T14:43:35.483Z", "content": "we got 0.31 with single image classifier\nnot use ensemble or cleaning
", "rawMarkdown": "we got 0.31 with single image classifier\nnot use ensemble or cleaning", "votes": 10 }, { "id": 776443, "postDate": "2020-03-17T11:56:42.017Z", "content": "I think leaders have focused on cropped models of eyes, nose and mouth. Reason being if we visualise the difference between real and fake bounding boxes the eyes, nose and mouth are the most frequently changed regions. Below are samples from easiest (top) to hardest/false positives (bottom).
\n\n.jpg?generation=1584445924227726&alt=media)
Good analysis, I want to know how this difference map is generated, can you share the processing code? Direct subtraction of pictures introduces a lot of background noise
", "rawMarkdown": "Good analysis, I want to know how this difference map is generated, can you share the processing code? Direct subtraction of pictures introduces a lot of background noise", "votes": 2 }, { "id": 784130, "postDate": "2020-03-24T00:34:00.327Z", "content": "Here is a code snippet.
\n\nfrom PIL import Image, ImageOps, ImageChops, ImageEnhance\nreal_img = Image.open(real_file)\nfake_img = Image.open(fake_file)\ndiff_img = ImageEnhance.Brightness(ImageChops.difference(real_img, fake_img)).enhance(4.0)
Thanks~
", "rawMarkdown": "Thanks~" } ] }, { "id": 772330, "postDate": "2020-03-15T11:00:58.397Z", "content": "@ryches Interesting analysis about this competition!
\n\nWe're in 0.30 group and we didn't do any real cleaning other than selecting videos with less than 3 faces. I mean if our face detector finds more than 3 faces we keep only the best 3. If your assumption is correct then we should start to do some labels cleaning. Any thread here with known bad labels? It would be heplful for everyone.
\n\nAlso for the 0.25 group, maybe they're using audio (we're not because there is an additional work to get labels), if they're using images only then they've found something, I would bet it's based on frames sequence as such methods have good results in State Of The Art. However their implementation and tuning is not obvious, we've tried a few models with sequences and it always fails (underfit or overfit).
", "rawMarkdown": "@ryches Interesting analysis about this competition!\n\nWe're in 0.30 group and we didn't do any real cleaning other than selecting videos with less than 3 faces. I mean if our face detector finds more than 3 faces we keep only the best 3. If your assumption is correct then we should start to do some labels cleaning. Any thread here with known bad labels? It would be heplful for everyone.\n\nAlso for the 0.25 group, maybe they're using audio (we're not because there is an additional work to get labels), if they're using images only then they've found something, I would bet it's based on frames sequence as such methods have good results in State Of The Art. However their implementation and tuning is not obvious, we've tried a few models with sequences and it always fails (underfit or overfit).\n", "votes": 6, "replies": [ { "id": 772613, "postDate": "2020-03-15T17:24:13.063Z", "content": "Very interesting. My attempts with audio have not been particularly useful as some others have mentioned. Maybe that is the case though.
\n\nAnd in terms of label cleaning I meant stuff like exactly what you did. Not using the labels and applying them across all frames, but rather making sure you only have relevant images associated with the label.
", "rawMarkdown": "Very interesting. My attempts with audio have not been particularly useful as some others have mentioned. Maybe that is the case though. \n\nAnd in terms of label cleaning I meant stuff like exactly what you did. Not using the labels and applying them across all frames, but rather making sure you only have relevant images associated with the label. ", "votes": 2 }, { "id": 786996, "postDate": "2020-03-26T12:39:30.627Z", "content": "We've tried to remove noisy labels (assuming that some REAL were labelled as FAKE) and it did not help, score was not really better. Noisy labels detected (with high confidence) was about 1%-2% of our training set. Even, we've watched a few videos to confirm such noisy labels and it looks there are not FAKE at all (all frames seems REAL). Maybe what we consider as bad video labels are just great fake even human's eye cannot detect such as GAN ones at: https://www.thispersondoesnotexist.com/\nOrganizers should have included such high fake quality videos in dataset otherwise 1 Million price to detect bad fakes would not be worth.
", "rawMarkdown": "We've tried to remove noisy labels (assuming that some REAL were labelled as FAKE) and it did not help, score was not really better. Noisy labels detected (with high confidence) was about 1%-2% of our training set. Even, we've watched a few videos to confirm such noisy labels and it looks there are not FAKE at all (all frames seems REAL). Maybe what we consider as bad video labels are just great fake even human's eye cannot detect such as GAN ones at: https://www.thispersondoesnotexist.com/\nOrganizers should have included such high fake quality videos in dataset otherwise 1 Million price to detect bad fakes would not be worth.\n" } ] }, { "id": 772706, "postDate": "2020-03-15T19:46:58.443Z", "content": "For us, no cleaning label involved to get to 0.31, just 4 ensemble good single models.
", "rawMarkdown": "For us, no cleaning label involved to get to 0.31, just 4 ensemble good single models.", "votes": 4, "replies": [ { "id": 775161, "postDate": "2020-03-16T11:21:37.133Z", "content": "only 3 models not 4 lol
", "rawMarkdown": "only 3 models not 4 lol" }, { "id": 779350, "postDate": "2020-03-19T08:47:51.137Z", "content": "I am just happy that finally @unkownhihi found a good name .
", "rawMarkdown": "I am just happy that finally @unkownhihi found a good name . ", "votes": 1 }, { "id": 779670, "postDate": "2020-03-19T15:26:00.300Z", "content": "@phoenix9032 I was trying to find a good nickname, but I decided that I'm going to use my real name instead of a nickname. I feel like kaggle is somewhere I can trust, to not leak/use/make money of my information.
", "rawMarkdown": "@phoenix9032 I was trying to find a good nickname, but I decided that I'm going to use my real name instead of a nickname. I feel like kaggle is somewhere I can trust, to not leak/use/make money of my information. ", "votes": 2 } ] }, { "id": 772336, "postDate": "2020-03-15T11:16:32.183Z", "content": "I'm really curious to see what methods the top 5 are using. Vladislav (currently #2) posted on his LinkedIn that he used \"a simple solution\". It's probably not a neural network for image classification?!
", "rawMarkdown": "I'm really curious to see what methods the top 5 are using. Vladislav (currently #2) posted on his LinkedIn that he used \"a simple solution\". It's probably not a neural network for image classification?!", "votes": 1, "replies": [ { "id": 772569, "postDate": "2020-03-15T16:31:37.247Z", "content": "I guess he meant a 4 layers cnn? DL is not necessary at all
", "rawMarkdown": "I guess he meant a 4 layers cnn? DL is not necessary at all" }, { "id": 772608, "postDate": "2020-03-15T17:21:45.443Z", "content": "I would be intrigued by a simple method, I think there are likely some simple features that could be calculated to check continuity between frames and also extract various things like histogram manipulation and noise patterns. My bet is that the primary score driver is going to be the data prep/cleaning and the representation that the model receives. I think the specific NN architecture is likely not as important.
", "rawMarkdown": "I would be intrigued by a simple method, I think there are likely some simple features that could be calculated to check continuity between frames and also extract various things like histogram manipulation and noise patterns. My bet is that the primary score driver is going to be the data prep/cleaning and the representation that the model receives. I think the specific NN architecture is likely not as important. ", "votes": 1 }, { "id": 772641, "postDate": "2020-03-15T18:04:52.657Z", "content": "\n\n\nI guess he meant a 4 layers cnn? DL is not necessary at all
\n
I don't know what he meant but simple may mean different things to different people. He appears to work with face data in his job, so he probably knows more about this stuff than I do.
\n\nWhether DL is not necessary or not... I'd love to see a solution that doesn't use it. (Technically speaking, a 4 layer CNN is also DL.)
", "rawMarkdown": "> I guess he meant a 4 layers cnn? DL is not necessary at all\n\nI don't know what he meant but simple may mean different things to different people. He appears to work with face data in his job, so he probably knows more about this stuff than I do.\n\nWhether DL is not necessary or not... I'd love to see a solution that doesn't use it. (Technically speaking, a 4 layer CNN is also DL.)", "votes": 1 }, { "id": 775654, "postDate": "2020-03-16T22:59:53.450Z", "content": "Maybe they turned it into a facial recognition problem. Firstly, they memorized the real characters (limited amount）with a vector. The rest is to detect which character it corresponds to and if the eyes, nose, mouth are modified by comparing the vectors.
\nFrom my experiments, the less params, the better the performance.
I utilized feature vectors created from the pretrained facenet neural network and wasnt able to get any measurable boost from using those features.
", "rawMarkdown": "I utilized feature vectors created from the pretrained facenet neural network and wasnt able to get any measurable boost from using those features. ", "votes": 2 }, { "id": 777542, "postDate": "2020-03-17T18:55:33.067Z", "content": "Did you try to reduce the dimension of the vectors, I think vector from lower level of facenet should be more helpful.\nAnother idea is studying the distribution of noise and noise pattern of camera. However, I didn't have my own dataset and enough knowledge to test them out...
", "rawMarkdown": "Did you try to reduce the dimension of the vectors, I think vector from lower level of facenet should be more helpful.\nAnother idea is studying the distribution of noise and noise pattern of camera. However, I didn't have my own dataset and enough knowledge to test them out..." } ] }, { "id": 786927, "postDate": "2020-03-26T10:57:57.543Z", "content": "We only have ~500 faces here.
\n\nThe obvious way to improve one's score is to get more faces (for example, download a bunch of YouTube videos) and apply forgery methods similar to what the organizers used (or any publicly available methods).
\n\nThis is questionable from the legality perspective and has little value as far as improving the art of forgery detection.
\n\nHowever, the organizers had many opportunities to state that this is not allowed and didn't take them.
", "rawMarkdown": "We only have ~500 faces here.\n\nThe obvious way to improve one's score is to get more faces (for example, download a bunch of YouTube videos) and apply forgery methods similar to what the organizers used (or any publicly available methods).\n\nThis is questionable from the legality perspective and has little value as far as improving the art of forgery detection.\n\nHowever, the organizers had many opportunities to state that this is not allowed and didn't take them.\n", "votes": 2, "replies": [ { "id": 786955, "postDate": "2020-03-26T11:40:58.517Z", "content": "Okay, Mr. but I did train a model with 8000 unique faces, more than tripling my training data, and no improvement, if you think generalization is a problem, well, guess what, it is not.
", "rawMarkdown": "Okay, Mr. but I did train a model with 8000 unique faces, more than tripling my training data, and no improvement, if you think generalization is a problem, well, guess what, it is not." }, { "id": 786972, "postDate": "2020-03-26T12:13:23.713Z", "content": "@harshitsheoran Where did you get the faces, and which forgery software did you use?
", "rawMarkdown": "@harshitsheoran Where did you get the faces, and which forgery software did you use?" } ] }, { "id": 775097, "postDate": "2020-03-16T09:47:43.387Z", "content": "![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2234817%2F75f2420699a5257913c4002bba92653c%2FHere.png?generation=1584352035205130&alt=media\")
Massive ensambles are not really feasible in this competition because of the time and space constraints
", "rawMarkdown": "Massive ensambles are not really feasible in this competition because of the time and space constraints ", "replies": [ { "id": 772206, "postDate": "2020-03-15T06:43:08.950Z", "content": "I've found that it is easily possible to fit 25+ models in 4 hours runtime since the bottleneck is the image loading and not the actual neural net inference
", "rawMarkdown": "I've found that it is easily possible to fit 25+ models in 4 hours runtime since the bottleneck is the image loading and not the actual neural net inference", "votes": 10 }, { "id": 775089, "postDate": "2020-03-16T09:41:06.583Z", "content": "@ryches This was quite important comment .
", "rawMarkdown": "@ryches This was quite important comment ." } ] }, { "id": 783775, "postDate": "2020-03-23T16:48:32.953Z", "rawMarkdown": "", "votes": 1, "isDeleted": true, "replies": [ { "id": 783960, "postDate": "2020-03-23T20:29:07.477Z", "content": "Pretrained model is always more robust and converges faster.
", "rawMarkdown": "Pretrained model is always more robust and converges faster. ", "votes": 2 }, { "id": 784804, "postDate": "2020-03-24T14:10:05.807Z", "rawMarkdown": "", "isDeleted": true } ] } ], "comments": [ { "id": 772713, "author_name": "Debanga Raj Neog", "author_url": "", "post_date": "2020-03-15T20:06:36.500000", "content": "Our 0.30003LB: No cleaning (simple uniform N frame extraction from video), ensembles of several single CNN models, multi-frame inference, good folder-wise validation.
", "votes": 14, "replies": [ { "id": 784369, "author_name": "BokingChen", "author_url": "", "post_date": "2020-03-24T06:53:54.310000", "content": "How many frames per video?and how mang CNN models do you use to ensamble?
", "votes": 0, "replies": [] } ] }, { "id": 776667, "author_name": "あほ", "author_url": "", "post_date": "2020-03-17T14:43:35.483000", "content": "we got 0.31 with single image classifier\nnot use ensemble or cleaning
", "votes": 10, "replies": [] }, { "id": 776443, "author_name": "maralski", "author_url": "", "post_date": "2020-03-17T11:56:42.017000", "content": "I think leaders have focused on cropped models of eyes, nose and mouth. Reason being if we visualise the difference between real and fake bounding boxes the eyes, nose and mouth are the most frequently changed regions. Below are samples from easiest (top) to hardest/false positives (bottom).
\n\n.jpg?generation=1584445924227726&alt=media)
Good analysis, I want to know how this difference map is generated, can you share the processing code? Direct subtraction of pictures introduces a lot of background noise
", "votes": 2, "replies": [] }, { "id": 784130, "author_name": "maralski", "author_url": "", "post_date": "2020-03-24T00:34:00.327000", "content": "Here is a code snippet.
\n\nfrom PIL import Image, ImageOps, ImageChops, ImageEnhance\nreal_img = Image.open(real_file)\nfake_img = Image.open(fake_file)\ndiff_img = ImageEnhance.Brightness(ImageChops.difference(real_img, fake_img)).enhance(4.0)
Thanks~
", "votes": 0, "replies": [] } ] }, { "id": 772330, "author_name": "MPWARE", "author_url": "", "post_date": "2020-03-15T11:00:58.397000", "content": "@ryches Interesting analysis about this competition!
\n\nWe're in 0.30 group and we didn't do any real cleaning other than selecting videos with less than 3 faces. I mean if our face detector finds more than 3 faces we keep only the best 3. If your assumption is correct then we should start to do some labels cleaning. Any thread here with known bad labels? It would be heplful for everyone.
\n\nAlso for the 0.25 group, maybe they're using audio (we're not because there is an additional work to get labels), if they're using images only then they've found something, I would bet it's based on frames sequence as such methods have good results in State Of The Art. However their implementation and tuning is not obvious, we've tried a few models with sequences and it always fails (underfit or overfit).
", "votes": 6, "replies": [ { "id": 772613, "author_name": "ryches", "author_url": "", "post_date": "2020-03-15T17:24:13.063000", "content": "Very interesting. My attempts with audio have not been particularly useful as some others have mentioned. Maybe that is the case though.
\n\nAnd in terms of label cleaning I meant stuff like exactly what you did. Not using the labels and applying them across all frames, but rather making sure you only have relevant images associated with the label.
", "votes": 2, "replies": [] }, { "id": 786996, "author_name": "MPWARE", "author_url": "", "post_date": "2020-03-26T12:39:30.627000", "content": "We've tried to remove noisy labels (assuming that some REAL were labelled as FAKE) and it did not help, score was not really better. Noisy labels detected (with high confidence) was about 1%-2% of our training set. Even, we've watched a few videos to confirm such noisy labels and it looks there are not FAKE at all (all frames seems REAL). Maybe what we consider as bad video labels are just great fake even human's eye cannot detect such as GAN ones at: https://www.thispersondoesnotexist.com/\nOrganizers should have included such high fake quality videos in dataset otherwise 1 Million price to detect bad fakes would not be worth.
", "votes": 0, "replies": [] } ] }, { "id": 772706, "author_name": "Shangqiu Li", "author_url": "", "post_date": "2020-03-15T19:46:58.443000", "content": "For us, no cleaning label involved to get to 0.31, just 4 ensemble good single models.
", "votes": 4, "replies": [ { "id": 775161, "author_name": "Harshit Sheoran", "author_url": "", "post_date": "2020-03-16T11:21:37.133000", "content": "only 3 models not 4 lol
", "votes": 0, "replies": [] }, { "id": 779350, "author_name": "Nirjhar Roy", "author_url": "", "post_date": "2020-03-19T08:47:51.137000", "content": "I am just happy that finally @unkownhihi found a good name .
", "votes": 1, "replies": [] }, { "id": 779670, "author_name": "Shangqiu Li", "author_url": "", "post_date": "2020-03-19T15:26:00.300000", "content": "@phoenix9032 I was trying to find a good nickname, but I decided that I'm going to use my real name instead of a nickname. I feel like kaggle is somewhere I can trust, to not leak/use/make money of my information.
", "votes": 2, "replies": [] } ] }, { "id": 772336, "author_name": "Human Analog", "author_url": "", "post_date": "2020-03-15T11:16:32.183000", "content": "I'm really curious to see what methods the top 5 are using. Vladislav (currently #2) posted on his LinkedIn that he used \"a simple solution\". It's probably not a neural network for image classification?!
", "votes": 1, "replies": [ { "id": 772569, "author_name": "yuanzhe zhou", "author_url": "", "post_date": "2020-03-15T16:31:37.247000", "content": "I guess he meant a 4 layers cnn? DL is not necessary at all
", "votes": 0, "replies": [] }, { "id": 772608, "author_name": "ryches", "author_url": "", "post_date": "2020-03-15T17:21:45.443000", "content": "I would be intrigued by a simple method, I think there are likely some simple features that could be calculated to check continuity between frames and also extract various things like histogram manipulation and noise patterns. My bet is that the primary score driver is going to be the data prep/cleaning and the representation that the model receives. I think the specific NN architecture is likely not as important.
", "votes": 1, "replies": [] }, { "id": 772641, "author_name": "Human Analog", "author_url": "", "post_date": "2020-03-15T18:04:52.657000", "content": "\n\n\nI guess he meant a 4 layers cnn? DL is not necessary at all
\n
I don't know what he meant but simple may mean different things to different people. He appears to work with face data in his job, so he probably knows more about this stuff than I do.
\n\nWhether DL is not necessary or not... I'd love to see a solution that doesn't use it. (Technically speaking, a 4 layer CNN is also DL.)
", "votes": 1, "replies": [] }, { "id": 775654, "author_name": "yuanzhe zhou", "author_url": "", "post_date": "2020-03-16T22:59:53.450000", "content": "Maybe they turned it into a facial recognition problem. Firstly, they memorized the real characters (limited amount）with a vector. The rest is to detect which character it corresponds to and if the eyes, nose, mouth are modified by comparing the vectors.
\nFrom my experiments, the less params, the better the performance.
I utilized feature vectors created from the pretrained facenet neural network and wasnt able to get any measurable boost from using those features.
", "votes": 2, "replies": [] }, { "id": 777542, "author_name": "yuanzhe zhou", "author_url": "", "post_date": "2020-03-17T18:55:33.067000", "content": "Did you try to reduce the dimension of the vectors, I think vector from lower level of facenet should be more helpful.\nAnother idea is studying the distribution of noise and noise pattern of camera. However, I didn't have my own dataset and enough knowledge to test them out...
", "votes": 0, "replies": [] } ] }, { "id": 786927, "author_name": "Oleg Trott", "author_url": "", "post_date": "2020-03-26T10:57:57.543000", "content": "We only have ~500 faces here.
\n\nThe obvious way to improve one's score is to get more faces (for example, download a bunch of YouTube videos) and apply forgery methods similar to what the organizers used (or any publicly available methods).
\n\nThis is questionable from the legality perspective and has little value as far as improving the art of forgery detection.
\n\nHowever, the organizers had many opportunities to state that this is not allowed and didn't take them.
", "votes": 2, "replies": [ { "id": 786955, "author_name": "Harshit Sheoran", "author_url": "", "post_date": "2020-03-26T11:40:58.517000", "content": "Okay, Mr. but I did train a model with 8000 unique faces, more than tripling my training data, and no improvement, if you think generalization is a problem, well, guess what, it is not.
", "votes": 0, "replies": [] }, { "id": 786972, "author_name": "Oleg Trott", "author_url": "", "post_date": "2020-03-26T12:13:23.713000", "content": "@harshitsheoran Where did you get the faces, and which forgery software did you use?
", "votes": 0, "replies": [] } ] }, { "id": 775097, "author_name": "Nirjhar Roy", "author_url": "", "post_date": "2020-03-16T09:47:43.387000", "content": "
Massive ensambles are not really feasible in this competition because of the time and space constraints
", "votes": 0, "replies": [ { "id": 772206, "author_name": "ryches", "author_url": "", "post_date": "2020-03-15T06:43:08.950000", "content": "I've found that it is easily possible to fit 25+ models in 4 hours runtime since the bottleneck is the image loading and not the actual neural net inference
", "votes": 10, "replies": [] }, { "id": 775089, "author_name": "Nirjhar Roy", "author_url": "", "post_date": "2020-03-16T09:41:06.583000", "content": "@ryches This was quite important comment .
", "votes": 0, "replies": [] } ] }, { "id": 783775, "author_name": "", "author_url": "", "post_date": "2020-03-23T16:48:32.953000", "content": "", "votes": 1, "replies": [ { "id": 783960, "author_name": "yuanzhe zhou", "author_url": "", "post_date": "2020-03-23T20:29:07.477000", "content": "Pretrained model is always more robust and converges faster.
", "votes": 2, "replies": [] }, { "id": 784804, "author_name": "", "author_url": "", "post_date": "2020-03-24T14:10:05.807000", "content": "", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "772177": "It's always interesting looking at the kaggle leaderboard stats and seeing how the distribution lands. Sometimes there is a gradual trailing off and everyone is clustering towards the top. In this competition, it has been quite a bit different. There are clear groupings of teams who have overcome various hurdles. \n\n\n\nThese are just some hypotheses as to what might be the trick to get to these points. Looking at the top 250 teams we can see quite a lot are bunched up on the best public kernel around .43 and then a small group who have made some minor improvements beyond that, likely with ensembling or some model/data improvements. Beyond that, I hypothesize the next group has done something smart in terms of removing the noise in the labels, either removing frames from videos that were labeled fake that weren't really fake or removing poorly formed videos. \n\nThe next step up is probably just some better method to do this than the previous grouping. They may have also found some way to make validation and test line up a bit better or make their model more robust. \n\nBeyond that, I have to assume there are some more clever methods being used. Alternative representations of the data, additional features, non-standard model architectures or massive ensembles. \n", "772713": "Our 0.30003LB: No cleaning (simple uniform N frame extraction from video), ensembles of several single CNN models, multi-frame inference, good folder-wise validation.", "776667": "we got 0.31 with single image classifier\nnot use ensemble or cleaning", "776443": "I think leaders have focused on cropped models of eyes, nose and mouth. Reason being if we visualise the difference between real and fake bounding boxes the eyes, nose and mouth are the most frequently changed regions. Below are samples from easiest (top) to hardest/false positives (bottom).\n\n.jpg?generation=1584445924227726&alt=media)\n", "772330": "@ryches Interesting analysis about this competition!\n\nWe're in 0.30 group and we didn't do any real cleaning other than selecting videos with less than 3 faces. I mean if our face detector finds more than 3 faces we keep only the best 3. If your assumption is correct then we should start to do some labels cleaning. Any thread here with known bad labels? It would be heplful for everyone.\n\nAlso for the 0.25 group, maybe they're using audio (we're not because there is an additional work to get labels), if they're using images only then they've found something, I would bet it's based on frames sequence as such methods have good results in State Of The Art. However their implementation and tuning is not obvious, we've tried a few models with sequences and it always fails (underfit or overfit).\n", "772706": "For us, no cleaning label involved to get to 0.31, just 4 ensemble good single models.", "772336": "I'm really curious to see what methods the top 5 are using. Vladislav (currently #2) posted on his LinkedIn that he used \"a simple solution\". It's probably not a neural network for image classification?!", "786927": "We only have ~500 faces here.\n\nThe obvious way to improve one's score is to get more faces (for example, download a bunch of YouTube videos) and apply forgery methods similar to what the organizers used (or any publicly available methods).\n\nThis is questionable from the legality perspective and has little value as far as improving the art of forgery detection.\n\nHowever, the organizers had many opportunities to state that this is not allowed and didn't take them.\n", "775097": "\n", "772203": "Massive ensambles are not really feasible in this competition because of the time and space constraints ", "783775": "" } }