Hi Guys!\nAfter comparing frames from fake videos to those from respective original videos, I realized that some of the frames from fake videos are exactly same as the respective original frames.\nAny thoughts on this ?
", "messages": [ { "id": 720407, "postDate": "2020-01-16T11:46:25.353Z", "content": "Hi Guys!\nAfter comparing frames from fake videos to those from respective original videos, I realized that some of the frames from fake videos are exactly same as the respective original frames.\nAny thoughts on this ?
", "rawMarkdown": "Hi Guys!\nAfter comparing frames from fake videos to those from respective original videos, I realized that some of the frames from fake videos are exactly same as the respective original frames.\nAny thoughts on this ? ", "votes": 12 }, { "id": 721222, "postDate": "2020-01-17T07:31:49.860Z", "content": "Maybe some of the frames are real, but the voice is fake, so the video is flagged as fake
", "rawMarkdown": "Maybe some of the frames are real, but the voice is fake, so the video is flagged as fake", "votes": 1, "replies": [ { "id": 721578, "postDate": "2020-01-17T13:57:54.867Z", "content": "oh yes, I haven't explored audio yet. Let me check if that helps. Thanks!
", "rawMarkdown": "oh yes, I haven't explored audio yet. Let me check if that helps. Thanks!" } ] }, { "id": 721003, "postDate": "2020-01-16T23:37:57.257Z", "content": "some of them are really hard to see with the bare eye, I had to use a video diff tool to find the modified pixels
", "rawMarkdown": "some of them are really hard to see with the bare eye, I had to use a video diff tool to find the modified pixels ", "votes": 1, "replies": [ { "id": 721142, "postDate": "2020-01-17T05:08:53.177Z", "content": "Yes! Even i used the same tool and the error is zero for many of them
", "rawMarkdown": "Yes! Even i used the same tool and the error is zero for many of them " }, { "id": 735524, "postDate": "2020-02-03T06:37:45.553Z", "content": "Can you share the tool you used for this?
", "rawMarkdown": "Can you share the tool you used for this?" }, { "id": 735949, "postDate": "2020-02-03T16:30:19.090Z", "content": "https://www.kaggle.com/kmat2019/easy-way-to-detect-face
", "rawMarkdown": "https://www.kaggle.com/kmat2019/easy-way-to-detect-face", "votes": 1 } ] }, { "id": 722764, "postDate": "2020-01-19T03:49:46.863Z", "content": "If you've ever tried using deepfake lab etc to understand this challenge, sometimes frames skip, or don't get matched, causing blur/jitter when switches between faces when viewed. So this would be expected in the run of the mill deepfakes, that some of the frames might be the same. Common imperfections w/current generation tools and different level techniques, especially longer the segment being deep faked.
", "rawMarkdown": "If you've ever tried using deepfake lab etc to understand this challenge, sometimes frames skip, or don't get matched, causing blur/jitter when switches between faces when viewed. So this would be expected in the run of the mill deepfakes, that some of the frames might be the same. Common imperfections w/current generation tools and different level techniques, especially longer the segment being deep faked. ", "votes": 2 }, { "id": 735873, "postDate": "2020-02-03T14:49:48.150Z", "content": "Hi @mrunalnasery , do you think we can use the real fake frames as new train set?
", "rawMarkdown": "Hi @mrunalnasery , do you think we can use the real fake frames as new train set?" }, { "id": 721900, "postDate": "2020-01-17T21:06:18.393Z", "content": "I guess ideally, one extracts all frames and runs classification on each one, classifying the video as fake if any frame is fake. I don't know about others, but for me it is currently possible to do this in the time allowed by the kernel submission. Perhaps we can generate statistics of numbers of frames that are fake and do a sampling of the entire video based on the statistics. Of course, there is no guarantee that the test videos share the same statistics...
", "rawMarkdown": "I guess ideally, one extracts all frames and runs classification on each one, classifying the video as fake if any frame is fake. I don't know about others, but for me it is currently possible to do this in the time allowed by the kernel submission. Perhaps we can generate statistics of numbers of frames that are fake and do a sampling of the entire video based on the statistics. Of course, there is no guarantee that the test videos share the same statistics...", "replies": [ { "id": 722104, "postDate": "2020-01-18T05:27:54.250Z", "content": "Yes, that's how i woud like to go about it. Also, do you extract the entire frame or just the faces? I have extracted the frames as whole but that tends to consume loads of memory, around 600MB for just one video.
", "rawMarkdown": "Yes, that's how i woud like to go about it. Also, do you extract the entire frame or just the faces? I have extracted the frames as whole but that tends to consume loads of memory, around 600MB for just one video. " }, { "id": 722113, "postDate": "2020-01-18T05:50:54.973Z", "content": "I do the faces only.
", "rawMarkdown": "I do the faces only." } ] }, { "id": 720984, "postDate": "2020-01-16T23:05:33.333Z", "content": "There are good fakes and bad fakes, really you want to categorize them first, if possible to optimize work load.
", "rawMarkdown": "There are good fakes and bad fakes, really you want to categorize them first, if possible to optimize work load.", "replies": [ { "id": 721140, "postDate": "2020-01-17T05:05:44.883Z", "content": "Yes makes sense, and maybe discard the bad fakes ?
", "rawMarkdown": "Yes makes sense, and maybe discard the bad fakes ? " } ] }, { "id": 720617, "postDate": "2020-01-16T15:27:31.133Z", "content": "Yes, even imagenet has some mislabled images.
", "rawMarkdown": "Yes, even imagenet has some mislabled images.", "replies": [ { "id": 720829, "postDate": "2020-01-16T19:41:18.417Z", "content": "As @ims0rry mentioned, do you think it makes sense to ignore them as noise?
", "rawMarkdown": "As @ims0rry mentioned, do you think it makes sense to ignore them as noise?" } ] }, { "id": 720537, "postDate": "2020-01-16T14:10:36.540Z", "content": "Nice observation! Do you have any examples?
", "rawMarkdown": "Nice observation! Do you have any examples?", "replies": [ { "id": 720548, "postDate": "2020-01-16T14:16:24.893Z", "content": "Hey, yes. I picked up first fake video in the metadata and compared it with the respective original video. I then generated a metafile which goes something like this:
\n\n{\"index\": [38, 202, 204, 208, 281, 285], \"label\": [\"FAKE\", \"FAKE\", \"FAKE\", \"FAKE\", \"FAKE\", \"FAKE\"]}
\n\nIt means only frame no - 38,202,204,208,281 and 285 are fake and rest of them are exactly similar to original frames.
", "rawMarkdown": "Hey, yes. I picked up first fake video in the metadata and compared it with the respective original video. I then generated a metafile which goes something like this:\n\n{\"index\": [38, 202, 204, 208, 281, 285], \"label\": [\"FAKE\", \"FAKE\", \"FAKE\", \"FAKE\", \"FAKE\", \"FAKE\"]}\n\nIt means only frame no - 38,202,204,208,281 and 285 are fake and rest of them are exactly similar to original frames. ", "votes": 5 }, { "id": 722755, "postDate": "2020-01-19T03:29:18.197Z", "content": "I'm a little curious about how you figured out this
", "rawMarkdown": "I'm a little curious about how you figured out this " }, { "id": 735857, "postDate": "2020-02-03T14:30:03.820Z", "content": "Can you share the file name of this video? thanks!
", "rawMarkdown": "Can you share the file name of this video? thanks!", "votes": 1 }, { "id": 752459, "postDate": "2020-02-21T03:29:39.507Z", "content": "Can you name this video file?
", "rawMarkdown": "Can you name this video file?" } ] }, { "id": 720442, "postDate": "2020-01-16T12:47:48.450Z", "content": "That's why you need to extract frames from different parts of the video and generalize predictions
", "rawMarkdown": "That's why you need to extract frames from different parts of the video and generalize predictions", "replies": [ { "id": 720518, "postDate": "2020-01-16T13:53:34.383Z", "content": "Shouldn't we extract only the fake frames as we are anyway getting the original ones from original videos?
", "rawMarkdown": "Shouldn't we extract only the fake frames as we are anyway getting the original ones from original videos?" }, { "id": 720566, "postDate": "2020-01-16T14:28:45.190Z", "content": "Also, for some fake videos only a few frames (5 or 6 of them sometimes !) are fake, I am not sure if generalizing predictions by extracting frames from different parts of the video will work. What do you think?
", "rawMarkdown": "Also, for some fake videos only a few frames (5 or 6 of them sometimes !) are fake, I am not sure if generalizing predictions by extracting frames from different parts of the video will work. What do you think?", "votes": 1 }, { "id": 720595, "postDate": "2020-01-16T15:04:45.977Z", "content": "> Shouldn't we extract only the fake frames as we are anyway getting the original ones from original videos?
\n\nIt would be great, but I see only two ways here: use some pretrained model (like face forensics) to detect fake frames or to do it by hands.
\n\n> Also, for some fake videos only a few frames (5 or 6 of them sometimes !) are fake, I am not sure if generalizing predictions by extracting frames from different parts of the video will work. What do you think?
\n\nI'm considering videos like this as noise and don't know how to deal with them yet .
", "rawMarkdown": "> Shouldn't we extract only the fake frames as we are anyway getting the original ones from original videos?\n\nIt would be great, but I see only two ways here: use some pretrained model (like face forensics) to detect fake frames or to do it by hands. \n\n> Also, for some fake videos only a few frames (5 or 6 of them sometimes !) are fake, I am not sure if generalizing predictions by extracting frames from different parts of the video will work. What do you think?\n\nI'm considering videos like this as noise and don't know how to deal with them yet . ", "votes": 1 }, { "id": 720827, "postDate": "2020-01-16T19:36:58.900Z", "content": "Yes, considering them as noise does make sense. Or we can just extract fake frames out and label them accordingly before feeding them into image classification models.
", "rawMarkdown": "Yes, considering them as noise does make sense. Or we can just extract fake frames out and label them accordingly before feeding them into image classification models. " }, { "id": 720877, "postDate": "2020-01-16T20:30:17.580Z", "content": "If your model predicts real frames well, you can just lower the needed count of frames detected as fake to round prediction to fake. I actually haven't tested it yet, but I think it would help with such videos.
", "rawMarkdown": "If your model predicts real frames well, you can just lower the needed count of frames detected as fake to round prediction to fake. I actually haven't tested it yet, but I think it would help with such videos." }, { "id": 721973, "postDate": "2020-01-18T00:34:47.757Z", "content": "In the training process, you can identify the faked frames with a method like [this.] (https://www.kaggle.com/kmat2019/easy-way-to-detect-face)
\n\nThe tricky part, of course, is when you're applying your system on a test set and don't know where to sample. I haven't seen what the characteristics are of a video that only contains a handful of altered frames...does the face noticeably change?
\n\nI suspect there will definitely be needle in a haystack videos like this in the test set, so I would strongly recommend against only training your model on the altered frames. Instead of keying in only on the altered frames, your system might instead be trained to identify signs a video has been encoded more than once, or has been encoded via a certain process within the alteration pipeline.
", "rawMarkdown": "In the training process, you can identify the faked frames with a method like [this.] (https://www.kaggle.com/kmat2019/easy-way-to-detect-face) \n\nThe tricky part, of course, is when you're applying your system on a test set and don't know where to sample. I haven't seen what the characteristics are of a video that only contains a handful of altered frames...does the face noticeably change? \n\nI suspect there will definitely be needle in a haystack videos like this in the test set, so I would strongly recommend **against** only training your model on the altered frames. Instead of keying in only on the altered frames, your system might instead be trained to identify signs a video has been encoded more than once, or has been encoded via a certain process within the alteration pipeline.", "votes": 1 }, { "id": 734234, "postDate": "2020-02-01T06:13:31.817Z", "content": "hi @mrunalnasery thanks for posting this..\nhow u figured out the frames in the Fake video as Fake
", "rawMarkdown": "hi @mrunalnasery thanks for posting this..\nhow u figured out the frames in the Fake video as Fake " }, { "id": 753291, "postDate": "2020-02-22T01:33:45.693Z", "content": "@mrunalnasery Can you explain what video diff tool you used?
", "rawMarkdown": "@mrunalnasery Can you explain what video diff tool you used? " } ] }, { "id": 720660, "postDate": "2020-01-16T16:15:23.120Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 721222, "author_name": "beiweizai", "author_url": "", "post_date": "2020-01-17T07:31:49.860000", "content": "Maybe some of the frames are real, but the voice is fake, so the video is flagged as fake
", "votes": 1, "replies": [ { "id": 721578, "author_name": "Mrunal", "author_url": "", "post_date": "2020-01-17T13:57:54.867000", "content": "oh yes, I haven't explored audio yet. Let me check if that helps. Thanks!
", "votes": 0, "replies": [] } ] }, { "id": 721003, "author_name": "B. Allabadi", "author_url": "", "post_date": "2020-01-16T23:37:57.257000", "content": "some of them are really hard to see with the bare eye, I had to use a video diff tool to find the modified pixels
", "votes": 1, "replies": [ { "id": 721142, "author_name": "Mrunal", "author_url": "", "post_date": "2020-01-17T05:08:53.177000", "content": "Yes! Even i used the same tool and the error is zero for many of them
", "votes": 0, "replies": [] }, { "id": 735524, "author_name": "Aditya Bharadwaj", "author_url": "", "post_date": "2020-02-03T06:37:45.553000", "content": "Can you share the tool you used for this?
", "votes": 0, "replies": [] }, { "id": 735949, "author_name": "Ryan Kelly", "author_url": "", "post_date": "2020-02-03T16:30:19.090000", "content": "https://www.kaggle.com/kmat2019/easy-way-to-detect-face
", "votes": 1, "replies": [] } ] }, { "id": 722764, "author_name": "daniel lesser", "author_url": "", "post_date": "2020-01-19T03:49:46.863000", "content": "If you've ever tried using deepfake lab etc to understand this challenge, sometimes frames skip, or don't get matched, causing blur/jitter when switches between faces when viewed. So this would be expected in the run of the mill deepfakes, that some of the frames might be the same. Common imperfections w/current generation tools and different level techniques, especially longer the segment being deep faked.
", "votes": 2, "replies": [] }, { "id": 735873, "author_name": "grhuang", "author_url": "", "post_date": "2020-02-03T14:49:48.150000", "content": "Hi @mrunalnasery , do you think we can use the real fake frames as new train set?
", "votes": 0, "replies": [] }, { "id": 721900, "author_name": "pete", "author_url": "", "post_date": "2020-01-17T21:06:18.393000", "content": "I guess ideally, one extracts all frames and runs classification on each one, classifying the video as fake if any frame is fake. I don't know about others, but for me it is currently possible to do this in the time allowed by the kernel submission. Perhaps we can generate statistics of numbers of frames that are fake and do a sampling of the entire video based on the statistics. Of course, there is no guarantee that the test videos share the same statistics...
", "votes": 0, "replies": [ { "id": 722104, "author_name": "Mrunal", "author_url": "", "post_date": "2020-01-18T05:27:54.250000", "content": "Yes, that's how i woud like to go about it. Also, do you extract the entire frame or just the faces? I have extracted the frames as whole but that tends to consume loads of memory, around 600MB for just one video.
", "votes": 0, "replies": [] }, { "id": 722113, "author_name": "pete", "author_url": "", "post_date": "2020-01-18T05:50:54.973000", "content": "I do the faces only.
", "votes": 0, "replies": [] } ] }, { "id": 720984, "author_name": "Mark Eckdahl", "author_url": "", "post_date": "2020-01-16T23:05:33.333000", "content": "There are good fakes and bad fakes, really you want to categorize them first, if possible to optimize work load.
", "votes": 0, "replies": [ { "id": 721140, "author_name": "Mrunal", "author_url": "", "post_date": "2020-01-17T05:05:44.883000", "content": "Yes makes sense, and maybe discard the bad fakes ?
", "votes": 0, "replies": [] } ] }, { "id": 720617, "author_name": "FrazierLei", "author_url": "", "post_date": "2020-01-16T15:27:31.133000", "content": "Yes, even imagenet has some mislabled images.
", "votes": 0, "replies": [ { "id": 720829, "author_name": "Mrunal", "author_url": "", "post_date": "2020-01-16T19:41:18.417000", "content": "As @ims0rry mentioned, do you think it makes sense to ignore them as noise?
", "votes": 0, "replies": [] } ] }, { "id": 720537, "author_name": "Polychronis Charitidis", "author_url": "", "post_date": "2020-01-16T14:10:36.540000", "content": "Nice observation! Do you have any examples?
", "votes": 0, "replies": [ { "id": 720548, "author_name": "Mrunal", "author_url": "", "post_date": "2020-01-16T14:16:24.893000", "content": "Hey, yes. I picked up first fake video in the metadata and compared it with the respective original video. I then generated a metafile which goes something like this:
\n\n{\"index\": [38, 202, 204, 208, 281, 285], \"label\": [\"FAKE\", \"FAKE\", \"FAKE\", \"FAKE\", \"FAKE\", \"FAKE\"]}
\n\nIt means only frame no - 38,202,204,208,281 and 285 are fake and rest of them are exactly similar to original frames.
", "votes": 5, "replies": [] }, { "id": 722755, "author_name": "xxn-xx", "author_url": "", "post_date": "2020-01-19T03:29:18.197000", "content": "I'm a little curious about how you figured out this
", "votes": 0, "replies": [] }, { "id": 735857, "author_name": "beeaware (looking for job)", "author_url": "", "post_date": "2020-02-03T14:30:03.820000", "content": "Can you share the file name of this video? thanks!
", "votes": 1, "replies": [] }, { "id": 752459, "author_name": "tsaha108", "author_url": "", "post_date": "2020-02-21T03:29:39.507000", "content": "Can you name this video file?
", "votes": 0, "replies": [] } ] }, { "id": 720442, "author_name": "Atsamaz Gatsoev", "author_url": "", "post_date": "2020-01-16T12:47:48.450000", "content": "That's why you need to extract frames from different parts of the video and generalize predictions
", "votes": 0, "replies": [ { "id": 720518, "author_name": "Mrunal", "author_url": "", "post_date": "2020-01-16T13:53:34.383000", "content": "Shouldn't we extract only the fake frames as we are anyway getting the original ones from original videos?
", "votes": 0, "replies": [] }, { "id": 720566, "author_name": "Mrunal", "author_url": "", "post_date": "2020-01-16T14:28:45.190000", "content": "Also, for some fake videos only a few frames (5 or 6 of them sometimes !) are fake, I am not sure if generalizing predictions by extracting frames from different parts of the video will work. What do you think?
", "votes": 1, "replies": [] }, { "id": 720595, "author_name": "Atsamaz Gatsoev", "author_url": "", "post_date": "2020-01-16T15:04:45.977000", "content": "> Shouldn't we extract only the fake frames as we are anyway getting the original ones from original videos?
\n\nIt would be great, but I see only two ways here: use some pretrained model (like face forensics) to detect fake frames or to do it by hands.
\n\n> Also, for some fake videos only a few frames (5 or 6 of them sometimes !) are fake, I am not sure if generalizing predictions by extracting frames from different parts of the video will work. What do you think?
\n\nI'm considering videos like this as noise and don't know how to deal with them yet .
", "votes": 1, "replies": [] }, { "id": 720827, "author_name": "Mrunal", "author_url": "", "post_date": "2020-01-16T19:36:58.900000", "content": "Yes, considering them as noise does make sense. Or we can just extract fake frames out and label them accordingly before feeding them into image classification models.
", "votes": 0, "replies": [] }, { "id": 720877, "author_name": "Atsamaz Gatsoev", "author_url": "", "post_date": "2020-01-16T20:30:17.580000", "content": "If your model predicts real frames well, you can just lower the needed count of frames detected as fake to round prediction to fake. I actually haven't tested it yet, but I think it would help with such videos.
", "votes": 0, "replies": [] }, { "id": 721973, "author_name": "Ryan Kelly", "author_url": "", "post_date": "2020-01-18T00:34:47.757000", "content": "In the training process, you can identify the faked frames with a method like [this.] (https://www.kaggle.com/kmat2019/easy-way-to-detect-face)
\n\nThe tricky part, of course, is when you're applying your system on a test set and don't know where to sample. I haven't seen what the characteristics are of a video that only contains a handful of altered frames...does the face noticeably change?
\n\nI suspect there will definitely be needle in a haystack videos like this in the test set, so I would strongly recommend against only training your model on the altered frames. Instead of keying in only on the altered frames, your system might instead be trained to identify signs a video has been encoded more than once, or has been encoded via a certain process within the alteration pipeline.
", "votes": 1, "replies": [] }, { "id": 734234, "author_name": "Jaideep", "author_url": "", "post_date": "2020-02-01T06:13:31.817000", "content": "hi @mrunalnasery thanks for posting this..\nhow u figured out the frames in the Fake video as Fake
", "votes": 0, "replies": [] }, { "id": 753291, "author_name": "Hari Santhanam", "author_url": "", "post_date": "2020-02-22T01:33:45.693000", "content": "@mrunalnasery Can you explain what video diff tool you used?
", "votes": 0, "replies": [] } ] }, { "id": 720660, "author_name": "", "author_url": "", "post_date": "2020-01-16T16:15:23.120000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "720407": "Hi Guys!\nAfter comparing frames from fake videos to those from respective original videos, I realized that some of the frames from fake videos are exactly same as the respective original frames.\nAny thoughts on this ? ", "721222": "Maybe some of the frames are real, but the voice is fake, so the video is flagged as fake", "721003": "some of them are really hard to see with the bare eye, I had to use a video diff tool to find the modified pixels ", "722764": "If you've ever tried using deepfake lab etc to understand this challenge, sometimes frames skip, or don't get matched, causing blur/jitter when switches between faces when viewed. So this would be expected in the run of the mill deepfakes, that some of the frames might be the same. Common imperfections w/current generation tools and different level techniques, especially longer the segment being deep faked. ", "735873": "Hi @mrunalnasery , do you think we can use the real fake frames as new train set?", "721900": "I guess ideally, one extracts all frames and runs classification on each one, classifying the video as fake if any frame is fake. I don't know about others, but for me it is currently possible to do this in the time allowed by the kernel submission. Perhaps we can generate statistics of numbers of frames that are fake and do a sampling of the entire video based on the statistics. Of course, there is no guarantee that the test videos share the same statistics...", "720984": "There are good fakes and bad fakes, really you want to categorize them first, if possible to optimize work load.", "720617": "Yes, even imagenet has some mislabled images.", "720537": "Nice observation! Do you have any examples?", "720442": "That's why you need to extract frames from different parts of the video and generalize predictions", "720660": "" } }