All 0.5 submission scores 0.69314, first row changed to 0.999 scores 0.69470. It means that the possible range is 3977 to 4002. And then submission with the following row passed successfully
assert (len(df) == 400) or (len(df) == 4000)
Additionally, there are exactly 2000 samples of each class.
\n\nInteresting, what is the private test size?
", "messages": [ { "id": 700199, "postDate": "2019-12-21T15:56:34.283Z", "content": "All 0.5 submission scores 0.69314, first row changed to 0.999 scores 0.69470. It means that the possible range is 3977 to 4002. And then submission with the following row passed successfully
assert (len(df) == 400) or (len(df) == 4000)
Additionally, there are exactly 2000 samples of each class.
\n\nInteresting, what is the private test size?
", "rawMarkdown": "All `0.5` submission scores `0.69314`, first row changed to `0.999` scores `0.69470`. It means that the possible range is `3977` to `4002`. And then submission with the following row passed successfully\n\n```assert (len(df) == 400) or (len(df) == 4000)```\n\nAdditionally, there are exactly 2000 samples of each class.\n\nInteresting, what is the private test size?", "votes": 30 }, { "id": 701037, "postDate": "2019-12-23T02:35:33.007Z", "content": "Looks like you are right. https://www.kaggle.com/c/deepfake-detection-challenge/overview/getting-started\n> You are limited to 9 hours of GPU compute time. This constraint is also imposed on the Public Test Set re-run. You should anticipate the Public Test set to be 10 times the size of the Public Validation Set and budget accordingly.
", "rawMarkdown": "Looks like you are right. https://www.kaggle.com/c/deepfake-detection-challenge/overview/getting-started\n> You are limited to 9 hours of GPU compute time. This constraint is also imposed on the Public Test Set re-run. You should anticipate the Public Test set to be 10 times the size of the Public Validation Set and budget accordingly. ", "votes": 3 }, { "id": 705418, "postDate": "2019-12-28T22:58:04.997Z", "content": "The full dataset has 119146 files. All are in mp4 format. Not sure if the private test sets will also have all files of same format and files size will be in the same range as in training and public test set.
", "rawMarkdown": "The full dataset has 119146 files. All are in mp4 format. Not sure if the private test sets will also have all files of same format and files size will be in the same range as in training and public test set." }, { "id": 702912, "postDate": "2019-12-25T10:41:29.650Z", "content": "You're a genius👍 @zaharch \nHow do you get 'there are exactly 2000 samples of each class'?
", "rawMarkdown": "You're a genius👍 @zaharch \nHow do you get 'there are exactly 2000 samples of each class'?", "replies": [ { "id": 702964, "postDate": "2019-12-25T12:12:35.050Z", "content": "Initially I saw somewhere that 0.49 and 0.51 gave the same score, but it can also be calculated from all ones or all zeros scores, or immediately concluded from the fact that all ones and all zeros give the same score.
Not related to the question, how come you have forked and re-published a variation of my kernel, but have never took time to upvote the original? :)
", "rawMarkdown": "Initially I saw somewhere that `0.49` and `0.51` gave the same score, but it can also be calculated from all ones or all zeros scores, or immediately concluded from the fact that all ones and all zeros give the same score.\n\nNot related to the question, how come you have forked and re-published a variation of my kernel, but have never took time to upvote the original? :)", "votes": 3 }, { "id": 702976, "postDate": "2019-12-25T12:32:27.820Z", "content": "sorry, I've upvoted it.\nYeah, you're right, count(fake) == count(real). @zaharch
", "rawMarkdown": "sorry, I've upvoted it.\nYeah, you're right, count(fake) == count(real). @zaharch ", "votes": 1 }, { "id": 702982, "postDate": "2019-12-25T12:44:22.897Z", "content": "Thanks, yes I agree with your formula, in case REAL and FAKE labels counts are equal. But if not it is in more general form -x ln(p) - (1-x) ln(1-p) where x is the ratio. If such scores are equal for p and 1-p, it means either p=0.5 or x=0.5.
For single sample, yes, it is a label, but the score is the average over 4000 samples, that is why I say ratio. Number of samples in one class divided by 4000.
Good question. As far as I can tell, there are 119948 total video clips in total. The private set can't be too huge or it would never finish within the time limit.
\n\nI am equally interested in their description of the private set:
\n\n\n\n\nPrivate Test Set: This dataset is privately held outside of Kaggle’s platform, and is used to compute the private leaderboard. It contains videos with a similar format and nature as the Training and Public Validation/Test Sets, but are real, organic videos with and without deepfakes.
\n
Does this mean the methods used for tampering with the videos in private test might not all be the same as those we see in train and public? These private test videos are not among the 120k videos we are given?
", "rawMarkdown": "Good question. As far as I can tell, there are 119948 total video clips in total. The private set can't be too huge or it would never finish within the time limit.\n\nI am equally interested in their description of the private set:\n\n> Private Test Set: This dataset is privately held outside of Kaggle’s platform, and is used to compute the private leaderboard. It contains videos with a *similar* format and nature as the Training and Public Validation/Test Sets, but are **real, organic** videos with and without deepfakes.\n\nDoes this mean the methods used for tampering with the videos in private test might not all be the same as those we see in train and public? These private test videos are not among the 120k videos we are given?\n\n", "replies": [ { "id": 700222, "postDate": "2019-12-21T16:38:45.697Z", "content": "I fear that it means exactly what your thinking - the final scoring may be on videos nothing like the current set. Which I think means we need to use some videos from the wild to train our models?
", "rawMarkdown": "I fear that it means exactly what your thinking - the final scoring may be on videos nothing like the current set. Which I think means we need to use some videos from the wild to train our models?", "votes": 1 }, { "id": 700939, "postDate": "2019-12-22T21:02:16.600Z", "content": "Hi, I'm just curious how did you come to the number of 119948 total? (afaik 119154 train and 400 public test videos)
", "rawMarkdown": "Hi, I'm just curious how did you come to the number of 119948 total? (afaik 119154 train and 400 public test videos)" }, { "id": 700970, "postDate": "2019-12-22T22:24:17.777Z", "content": "That is the length of the list that results from doing os.walk in the folder I unzipped everything after removing everything not ending in .mp4.
\n\nIt is certainly possible I have some duplicates in there for whatever reason, I have not looked that carefully yet.
", "rawMarkdown": "That is the length of the list that results from doing os.walk in the folder I unzipped everything after removing everything not ending in .mp4.\n\nIt is certainly possible I have some duplicates in there for whatever reason, I have not looked that carefully yet. " }, { "id": 700996, "postDate": "2019-12-23T00:21:19.433Z", "content": "119146 is my count of the full data set - I get your number if I include the train and test samples - but they would be duplicates.
", "rawMarkdown": "119146 is my count of the full data set - I get your number if I include the train and test samples - but they would be duplicates.", "votes": 2 }, { "id": 701029, "postDate": "2019-12-23T02:22:39.517Z", "content": "metadata says there are 8 more @pcjimmmy https://www.kaggle.com/aleksandradeis/deepfake-challenge-eda#700928
", "rawMarkdown": "metadata says there are 8 more @pcjimmmy https://www.kaggle.com/aleksandradeis/deepfake-challenge-eda#700928 " }, { "id": 701069, "postDate": "2019-12-23T03:35:38.813Z", "content": "Excellent, thanks for checking up on that!
", "rawMarkdown": "Excellent, thanks for checking up on that!" } ] }, { "id": 700969, "postDate": "2019-12-22T22:23:25.380Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 701037, "author_name": "hirviö", "author_url": "", "post_date": "2019-12-23T02:35:33.007000", "content": "Looks like you are right. https://www.kaggle.com/c/deepfake-detection-challenge/overview/getting-started\n> You are limited to 9 hours of GPU compute time. This constraint is also imposed on the Public Test Set re-run. You should anticipate the Public Test set to be 10 times the size of the Public Validation Set and budget accordingly.
", "votes": 3, "replies": [] }, { "id": 705418, "author_name": "Al Warrior", "author_url": "", "post_date": "2019-12-28T22:58:04.997000", "content": "The full dataset has 119146 files. All are in mp4 format. Not sure if the private test sets will also have all files of same format and files size will be in the same range as in training and public test set.
", "votes": 0, "replies": [] }, { "id": 702912, "author_name": "DiegoJohnson", "author_url": "", "post_date": "2019-12-25T10:41:29.650000", "content": "You're a genius👍 @zaharch \nHow do you get 'there are exactly 2000 samples of each class'?
", "votes": 0, "replies": [ { "id": 702964, "author_name": "nosound", "author_url": "", "post_date": "2019-12-25T12:12:35.050000", "content": "Initially I saw somewhere that 0.49 and 0.51 gave the same score, but it can also be calculated from all ones or all zeros scores, or immediately concluded from the fact that all ones and all zeros give the same score.
Not related to the question, how come you have forked and re-published a variation of my kernel, but have never took time to upvote the original? :)
", "votes": 3, "replies": [] }, { "id": 702976, "author_name": "DiegoJohnson", "author_url": "", "post_date": "2019-12-25T12:32:27.820000", "content": "sorry, I've upvoted it.\nYeah, you're right, count(fake) == count(real). @zaharch
", "votes": 1, "replies": [] }, { "id": 702982, "author_name": "nosound", "author_url": "", "post_date": "2019-12-25T12:44:22.897000", "content": "Thanks, yes I agree with your formula, in case REAL and FAKE labels counts are equal. But if not it is in more general form -x ln(p) - (1-x) ln(1-p) where x is the ratio. If such scores are equal for p and 1-p, it means either p=0.5 or x=0.5.
For single sample, yes, it is a label, but the score is the average over 4000 samples, that is why I say ratio. Number of samples in one class divided by 4000.
Good question. As far as I can tell, there are 119948 total video clips in total. The private set can't be too huge or it would never finish within the time limit.
\n\nI am equally interested in their description of the private set:
\n\n\n\n\nPrivate Test Set: This dataset is privately held outside of Kaggle’s platform, and is used to compute the private leaderboard. It contains videos with a similar format and nature as the Training and Public Validation/Test Sets, but are real, organic videos with and without deepfakes.
\n
Does this mean the methods used for tampering with the videos in private test might not all be the same as those we see in train and public? These private test videos are not among the 120k videos we are given?
", "votes": 0, "replies": [ { "id": 700222, "author_name": "PC Jimmmy", "author_url": "", "post_date": "2019-12-21T16:38:45.697000", "content": "I fear that it means exactly what your thinking - the final scoring may be on videos nothing like the current set. Which I think means we need to use some videos from the wild to train our models?
", "votes": 1, "replies": [] }, { "id": 700939, "author_name": "hirviö", "author_url": "", "post_date": "2019-12-22T21:02:16.600000", "content": "Hi, I'm just curious how did you come to the number of 119948 total? (afaik 119154 train and 400 public test videos)
", "votes": 0, "replies": [] }, { "id": 700970, "author_name": "interneuron", "author_url": "", "post_date": "2019-12-22T22:24:17.777000", "content": "That is the length of the list that results from doing os.walk in the folder I unzipped everything after removing everything not ending in .mp4.
\n\nIt is certainly possible I have some duplicates in there for whatever reason, I have not looked that carefully yet.
", "votes": 0, "replies": [] }, { "id": 700996, "author_name": "PC Jimmmy", "author_url": "", "post_date": "2019-12-23T00:21:19.433000", "content": "119146 is my count of the full data set - I get your number if I include the train and test samples - but they would be duplicates.
", "votes": 2, "replies": [] }, { "id": 701029, "author_name": "hirviö", "author_url": "", "post_date": "2019-12-23T02:22:39.517000", "content": "metadata says there are 8 more @pcjimmmy https://www.kaggle.com/aleksandradeis/deepfake-challenge-eda#700928
", "votes": 0, "replies": [] }, { "id": 701069, "author_name": "interneuron", "author_url": "", "post_date": "2019-12-23T03:35:38.813000", "content": "Excellent, thanks for checking up on that!
", "votes": 0, "replies": [] } ] }, { "id": 700969, "author_name": "", "author_url": "", "post_date": "2019-12-22T22:23:25.380000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "700199": "All `0.5` submission scores `0.69314`, first row changed to `0.999` scores `0.69470`. It means that the possible range is `3977` to `4002`. And then submission with the following row passed successfully\n\n```assert (len(df) == 400) or (len(df) == 4000)```\n\nAdditionally, there are exactly 2000 samples of each class.\n\nInteresting, what is the private test size?", "701037": "Looks like you are right. https://www.kaggle.com/c/deepfake-detection-challenge/overview/getting-started\n> You are limited to 9 hours of GPU compute time. This constraint is also imposed on the Public Test Set re-run. You should anticipate the Public Test set to be 10 times the size of the Public Validation Set and budget accordingly. ", "705418": "The full dataset has 119146 files. All are in mp4 format. Not sure if the private test sets will also have all files of same format and files size will be in the same range as in training and public test set.", "702912": "You're a genius👍 @zaharch \nHow do you get 'there are exactly 2000 samples of each class'?", "700210": "Good question. As far as I can tell, there are 119948 total video clips in total. The private set can't be too huge or it would never finish within the time limit.\n\nI am equally interested in their description of the private set:\n\n> Private Test Set: This dataset is privately held outside of Kaggle’s platform, and is used to compute the private leaderboard. It contains videos with a *similar* format and nature as the Training and Public Validation/Test Sets, but are **real, organic** videos with and without deepfakes.\n\nDoes this mean the methods used for tampering with the videos in private test might not all be the same as those we see in train and public? These private test videos are not among the 120k videos we are given?\n\n", "700969": "" } }