![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F351627%2Fb1bf131960c78613401f55740349248c%2Ffake.png?generation=1581187572634792&alt=media\")
NeuralTextures look almost perceptually authentic to a human, let alone a machine.
\n\n
Suppose the test set contains fraction_real and fraction_neuraltextures, and your algorithm can't tell the difference[1].
Then when the algorithm sees something authentic-looking, and it doesn't predict
\n\nfraction_neuraltextures / (fraction_neuraltextures + fraction_real)\nit will lose to similar algorithms that do.
\n\nSince we don't know these numbers[2], a large part of this contest seems to be guessing them, making this contest something of a lottery.
\n\nThis must demotivate serious competitors: you could expend a lot of effort to submit a sound entry, but your odds of winning are lottery-like (Even less, if another competitor learned fraction_real and fraction_neuraltextures through proximity to the organizers, for example, or by asking them privately)
To my mind, stating these two numbers publicly would make this contest infinitely better.
\n\n[1] Even if your algorithm is not entirely useless on NeuralTextures, I hope it's easy to see that knowing these numbers is critically important to what predictions it should make.
\n\n[2] We know that fraction_real is 0.5 on the public test, but not on the private test set.
NeuralTextures look almost perceptually authentic to a human, let alone a machine.
\n\n
Suppose the test set contains fraction_real and fraction_neuraltextures, and your algorithm can't tell the difference[1].
Then when the algorithm sees something authentic-looking, and it doesn't predict
\n\nfraction_neuraltextures / (fraction_neuraltextures + fraction_real)\nit will lose to similar algorithms that do.
\n\nSince we don't know these numbers[2], a large part of this contest seems to be guessing them, making this contest something of a lottery.
\n\nThis must demotivate serious competitors: you could expend a lot of effort to submit a sound entry, but your odds of winning are lottery-like (Even less, if another competitor learned fraction_real and fraction_neuraltextures through proximity to the organizers, for example, or by asking them privately)
To my mind, stating these two numbers publicly would make this contest infinitely better.
\n\n[1] Even if your algorithm is not entirely useless on NeuralTextures, I hope it's easy to see that knowing these numbers is critically important to what predictions it should make.
\n\n[2] We know that fraction_real is 0.5 on the public test, but not on the private test set.
Is data distribution different in training and test sets? Isn't the basic assumption of ML that they are sampled from the same distribution?
", "rawMarkdown": "Is data distribution different in training and test sets? Isn't the basic assumption of ML that they are sampled from the same distribution?", "votes": null }, { "id": "740429", "postDate": "02/09/2020 12:13:50", "content": "\n\n\nIsn't the basic assumption of ML that they are sampled from the same distribution?
\n
Haha, not on Kaggle. ;-) Part of building a solution is to think of ways that the competition organizer may have changed the private test set to make it harder. This why people whose models don't generalize enough will get great scores on the public leaderboard but then do much worse on the private LB.
", "rawMarkdown": "> Isn't the basic assumption of ML that they are sampled from the same distribution?\n\nHaha, not on Kaggle. ;-) Part of building a solution is to think of ways that the competition organizer may have changed the private test set to make it harder. This why people whose models don't generalize enough will get great scores on the public leaderboard but then do much worse on the private LB.", "votes": null }, { "id": "740557", "postDate": "02/09/2020 14:27:03", "content": "She looks like minecraft steve though..
", "rawMarkdown": "She looks like minecraft steve though..", "votes": null }, { "id": "740682", "postDate": "02/09/2020 16:49:54", "content": "\n\n\nHuman Analog wrote:
\n \nHaha, not on Kaggle. ;-) Part of building a solution is to think of ways that the competition organizer may have changed the private test set to make it harder. This why people whose models don't generalize enough will get great scores on the public leaderboard but then do much worse on the private LB.
\n
Generalization is good performance on novel samples from the same distribution.
\n\nWhich competitions changed the distribution? I think it's rare.
", "rawMarkdown": "> **Human Analog wrote:**\n> \n> Haha, not on Kaggle. ;-) Part of building a solution is to think of ways that the competition organizer may have changed the private test set to make it harder. This why people whose models don't generalize enough will get great scores on the public leaderboard but then do much worse on the private LB.\n\nGeneralization is good performance on novel samples from the *same* distribution.\n\nWhich competitions changed the distribution? I think it's rare.", "votes": null }, { "id": "740705", "postDate": "02/09/2020 17:15:53", "content": "> Sumit Mishra wrote:\n> \n> She looks like minecraft steve though..
\n\nReally? Here is frame 35 from a real and forged video (face only) :
\n\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F351627%2F65a8d30537b376a8b1a810ea3194eb59%2F1.png?generation=1581272587789257&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F351627%2F6a5e584b7cd316f4e8ddd23119e786db%2F11.png?generation=1581272602552317&alt=media\")
Which one is fake?
", "rawMarkdown": "> **Sumit Mishra wrote:**\n> \n> She looks like minecraft steve though..\n\nReally? Here is frame 35 from a real and forged video (face only) :\n\n\n\n\n\n\n\nWhich one is fake?", "votes": null }, { "id": "740751", "postDate": "02/09/2020 18:31:07", "content": "\n\n\nWhich one is fake?
\n
Just because it's hard to see for a human doesn't mean it's hard for a computer. The classifier may detect patterns in the fake image that are not present in the real image (or vice versa).
", "rawMarkdown": "> Which one is fake?\n\nJust because it's hard to see for a human doesn't mean it's hard for a computer. The classifier may detect patterns in the fake image that are not present in the real image (or vice versa).", "votes": null }, { "id": "740752", "postDate": "02/09/2020 18:32:02", "content": "The same frame from a real and forged video:
\n\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F351627%2F97960a3b7b03650adf84446e538e44a1%2F3.png?generation=1581273034862445&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F351627%2F8d83c25be7f4e790f40aa39c010323b6%2F33.png?generation=1581273059942158&alt=media\")
> Human Analog wrote:\n> \n> Just because it's hard to see for a human doesn't mean it's hard for a computer.
\n\nMy point is that some forgery methods will be much harder to detect than others. The probabilities you predict must depend on the fractions of these methods in the test set. If the organizers changed these fractions, you are forced to guess what they did or didn't do (See the original post, which outlines a somewhat extreme example of when your algorithm is completely useless against some method, NeuralTextures. But similar considerations apply when it's not useless, but just performs worse than against other methods).
", "rawMarkdown": "> **Human Analog wrote:**\n> \n> Just because it's hard to see for a human doesn't mean it's hard for a computer.\n\nMy point is that some forgery methods will be much harder to detect than others. The probabilities you predict must depend on the fractions of these methods in the test set. If the organizers changed these fractions, you are forced to **guess** what they did or didn't do (See the original post, which outlines a somewhat extreme example of when your algorithm is completely useless against some method, NeuralTextures. But similar considerations apply when it's not useless, but just performs worse than against other methods).", "votes": null }, { "id": "740795", "postDate": "02/09/2020 19:55:59", "content": "\n\n\nyou are forced to guess what they did or didn't do
\n
This has been the case for all the competitions I've been in (which isn't that many so perhaps I just picked the ones where this happened).
", "rawMarkdown": "> you are forced to guess what they did or didn't do\n\nThis has been the case for all the competitions I've been in (which isn't that many so perhaps I just picked the ones where this happened).", "votes": null }, { "id": "741207", "postDate": "02/10/2020 10:15:58", "content": "Case in point: my best scoring model is one that I didn't spend any time tweaking the training process on. I just picked something reasonable and let it train.
\n\nOver the weekend I improved the training process (1-cycle learning policy, AdamW optimizer, fine-tuning the earlier layers with different LRs for different groups of layers, etc). The model now learned a lot better (according to the train & val losses), but its leaderboard score was worse.
\n\nI don't think the model was overfitting per se (the validation loss kept going down). Even though it was learning better than before, it was just learning the wrong things.
\n\nThe results is that the model that didn't learn as well, actually generalized better on the test set used for the leaderboard.
\n\nSo if your model does better on the training set, but worse on the leaderboard, it's either overfitting -- or the distribution of the leaderboard test set is different.
", "rawMarkdown": "Case in point: my best scoring model is one that I didn't spend any time tweaking the training process on. I just picked something reasonable and let it train. \n\nOver the weekend I improved the training process (1-cycle learning policy, AdamW optimizer, fine-tuning the earlier layers with different LRs for different groups of layers, etc). The model now learned a lot better (according to the train & val losses), but its leaderboard score was worse. \n\nI don't think the model was overfitting per se (the validation loss kept going down). Even though it was learning better than before, it was just learning the wrong things.\n\nThe results is that the model that didn't learn as well, actually generalized better on the test set used for the leaderboard.\n\nSo if your model does better on the training set, but worse on the leaderboard, it's either overfitting -- or the distribution of the leaderboard test set is different.", "votes": null }, { "id": "741634", "postDate": "02/10/2020 21:03:26", "content": "I think you are right. In order for the network to generalise we’ll cross-domain it must be free of biases towards a particular deepfake method. That is why I think 80% of the work is in training set selection to avoid such bias.
", "rawMarkdown": "I think you are right. In order for the network to generalise we’ll cross-domain it must be free of biases towards a particular deepfake method. That is why I think 80% of the work is in training set selection to avoid such bias.", "votes": null }, { "id": "741685", "postDate": "02/10/2020 22:04:39", "content": "I have the same problem. I have models that performed much better on validation (and i made sure there are no cross links as much as i could between train and validation set).\nMy conclusion is that the test set is not made the same way as the training set. Probably real world videos? And the models overfit to something specific to the training set that we can't easily comprehend. Having said that, i might still have some other stupid mistakes...
", "rawMarkdown": "I have the same problem. I have models that performed much better on validation (and i made sure there are no cross links as much as i could between train and validation set).\nMy conclusion is that the test set is not made the same way as the training set. Probably real world videos? And the models overfit to something specific to the training set that we can't easily comprehend. Having said that, i might still have some other stupid mistakes...", "votes": null } ], "comments": [ { "id": 740245, "author_name": "uraler", "author_url": "", "post_date": "02/09/2020 04:37:01", "content": "Is data distribution different in training and test sets? Isn't the basic assumption of ML that they are sampled from the same distribution?
", "votes": null, "replies": [ { "id": 740429, "author_name": "humananalog", "author_url": "", "post_date": "02/09/2020 12:13:50", "content": "\n\n\nIsn't the basic assumption of ML that they are sampled from the same distribution?
\n
Haha, not on Kaggle. ;-) Part of building a solution is to think of ways that the competition organizer may have changed the private test set to make it harder. This why people whose models don't generalize enough will get great scores on the public leaderboard but then do much worse on the private LB.
", "votes": null, "replies": [] }, { "id": 740682, "author_name": "olegtrott", "author_url": "", "post_date": "02/09/2020 16:49:54", "content": "\n\n\nHuman Analog wrote:
\n \nHaha, not on Kaggle. ;-) Part of building a solution is to think of ways that the competition organizer may have changed the private test set to make it harder. This why people whose models don't generalize enough will get great scores on the public leaderboard but then do much worse on the private LB.
\n
Generalization is good performance on novel samples from the same distribution.
\n\nWhich competitions changed the distribution? I think it's rare.
", "votes": null, "replies": [] }, { "id": 741634, "author_name": "maralski", "author_url": "", "post_date": "02/10/2020 21:03:26", "content": "I think you are right. In order for the network to generalise we’ll cross-domain it must be free of biases towards a particular deepfake method. That is why I think 80% of the work is in training set selection to avoid such bias.
", "votes": null, "replies": [] } ] }, { "id": 740557, "author_name": "sumitm004", "author_url": "", "post_date": "02/09/2020 14:27:03", "content": "She looks like minecraft steve though..
", "votes": null, "replies": [ { "id": 740705, "author_name": "olegtrott", "author_url": "", "post_date": "02/09/2020 17:15:53", "content": "> Sumit Mishra wrote:\n> \n> She looks like minecraft steve though..
\n\nReally? Here is frame 35 from a real and forged video (face only) :
\n\n
Which one is fake?
", "votes": null, "replies": [] }, { "id": 740751, "author_name": "humananalog", "author_url": "", "post_date": "02/09/2020 18:31:07", "content": "\n\n\nWhich one is fake?
\n
Just because it's hard to see for a human doesn't mean it's hard for a computer. The classifier may detect patterns in the fake image that are not present in the real image (or vice versa).
", "votes": null, "replies": [] }, { "id": 740752, "author_name": "olegtrott", "author_url": "", "post_date": "02/09/2020 18:32:02", "content": "The same frame from a real and forged video:
\n\n
> Human Analog wrote:\n> \n> Just because it's hard to see for a human doesn't mean it's hard for a computer.
\n\nMy point is that some forgery methods will be much harder to detect than others. The probabilities you predict must depend on the fractions of these methods in the test set. If the organizers changed these fractions, you are forced to guess what they did or didn't do (See the original post, which outlines a somewhat extreme example of when your algorithm is completely useless against some method, NeuralTextures. But similar considerations apply when it's not useless, but just performs worse than against other methods).
", "votes": null, "replies": [] }, { "id": 740795, "author_name": "humananalog", "author_url": "", "post_date": "02/09/2020 19:55:59", "content": "\n\n\nyou are forced to guess what they did or didn't do
\n
This has been the case for all the competitions I've been in (which isn't that many so perhaps I just picked the ones where this happened).
", "votes": null, "replies": [] }, { "id": 741207, "author_name": "humananalog", "author_url": "", "post_date": "02/10/2020 10:15:58", "content": "Case in point: my best scoring model is one that I didn't spend any time tweaking the training process on. I just picked something reasonable and let it train.
\n\nOver the weekend I improved the training process (1-cycle learning policy, AdamW optimizer, fine-tuning the earlier layers with different LRs for different groups of layers, etc). The model now learned a lot better (according to the train & val losses), but its leaderboard score was worse.
\n\nI don't think the model was overfitting per se (the validation loss kept going down). Even though it was learning better than before, it was just learning the wrong things.
\n\nThe results is that the model that didn't learn as well, actually generalized better on the test set used for the leaderboard.
\n\nSo if your model does better on the training set, but worse on the leaderboard, it's either overfitting -- or the distribution of the leaderboard test set is different.
", "votes": null, "replies": [] }, { "id": 741685, "author_name": "moshel", "author_url": "", "post_date": "02/10/2020 22:04:39", "content": "I have the same problem. I have models that performed much better on validation (and i made sure there are no cross links as much as i could between train and validation set).\nMy conclusion is that the test set is not made the same way as the training set. Probably real world videos? And the models overfit to something specific to the training set that we can't easily comprehend. Having said that, i might still have some other stupid mistakes...
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "740046": "NeuralTextures look almost perceptually authentic to a human, let alone a machine.\n\n\n\n\n\nSuppose the test set contains `fraction_real` and `fraction_neuraltextures`, and your algorithm can't tell the difference[1].\n\nThen when the algorithm sees something authentic-looking, and it doesn't predict\n\n fraction_neuraltextures / (fraction_neuraltextures + fraction_real)\n\nit will lose to similar algorithms that do.\n\nSince we don't know these numbers[2], a large part of this contest seems to be guessing them, making this contest something of a lottery.\n\nThis must demotivate serious competitors: you could expend a lot of effort to submit a sound entry, but your odds of winning are lottery-like (Even less, if another competitor learned `fraction_real` and `fraction_neuraltextures` through proximity to the organizers, for example, or by asking them privately)\n\nTo my mind, stating these two numbers publicly would make this contest infinitely better.\n\n---\n\n[1] Even if your algorithm is not entirely useless on NeuralTextures, I hope it's easy to see that knowing these numbers is critically important to what predictions it should make.\n\n[2] We know that `fraction_real` is 0.5 on the public test, but not on the private test set.", "740245": "Is data distribution different in training and test sets? Isn't the basic assumption of ML that they are sampled from the same distribution?", "740429": "> Isn't the basic assumption of ML that they are sampled from the same distribution?\n\nHaha, not on Kaggle. ;-) Part of building a solution is to think of ways that the competition organizer may have changed the private test set to make it harder. This why people whose models don't generalize enough will get great scores on the public leaderboard but then do much worse on the private LB.", "740557": "She looks like minecraft steve though..", "740682": "> **Human Analog wrote:**\n> \n> Haha, not on Kaggle. ;-) Part of building a solution is to think of ways that the competition organizer may have changed the private test set to make it harder. This why people whose models don't generalize enough will get great scores on the public leaderboard but then do much worse on the private LB.\n\nGeneralization is good performance on novel samples from the *same* distribution.\n\nWhich competitions changed the distribution? I think it's rare.", "740705": "> **Sumit Mishra wrote:**\n> \n> She looks like minecraft steve though..\n\nReally? Here is frame 35 from a real and forged video (face only) :\n\n\n\n\n\n\n\nWhich one is fake?", "740751": "> Which one is fake?\n\nJust because it's hard to see for a human doesn't mean it's hard for a computer. The classifier may detect patterns in the fake image that are not present in the real image (or vice versa).", "740752": "The same frame from a real and forged video:\n\n\n\n", "740758": "> **Human Analog wrote:**\n> \n> Just because it's hard to see for a human doesn't mean it's hard for a computer.\n\nMy point is that some forgery methods will be much harder to detect than others. The probabilities you predict must depend on the fractions of these methods in the test set. If the organizers changed these fractions, you are forced to **guess** what they did or didn't do (See the original post, which outlines a somewhat extreme example of when your algorithm is completely useless against some method, NeuralTextures. But similar considerations apply when it's not useless, but just performs worse than against other methods).", "740795": "> you are forced to guess what they did or didn't do\n\nThis has been the case for all the competitions I've been in (which isn't that many so perhaps I just picked the ones where this happened).", "741207": "Case in point: my best scoring model is one that I didn't spend any time tweaking the training process on. I just picked something reasonable and let it train. \n\nOver the weekend I improved the training process (1-cycle learning policy, AdamW optimizer, fine-tuning the earlier layers with different LRs for different groups of layers, etc). The model now learned a lot better (according to the train & val losses), but its leaderboard score was worse. \n\nI don't think the model was overfitting per se (the validation loss kept going down). Even though it was learning better than before, it was just learning the wrong things.\n\nThe results is that the model that didn't learn as well, actually generalized better on the test set used for the leaderboard.\n\nSo if your model does better on the training set, but worse on the leaderboard, it's either overfitting -- or the distribution of the leaderboard test set is different.", "741634": "I think you are right. In order for the network to generalise we’ll cross-domain it must be free of biases towards a particular deepfake method. That is why I think 80% of the work is in training set selection to avoid such bias.", "741685": "I have the same problem. I have models that performed much better on validation (and i made sure there are no cross links as much as i could between train and validation set).\nMy conclusion is that the test set is not made the same way as the training set. Probably real world videos? And the models overfit to something specific to the training set that we can't easily comprehend. Having said that, i might still have some other stupid mistakes..." }, "source": "meta" }