I will update this list more tomorrow, but I audited some people's work after they made their kernels public after the submission deadline and I noticed a couple of subtle errors that concerned me.
\n\nwriting to file and not cleaning up between predictions
\n\n2.Not putting boundaries on input size.
\n\n...to be continued when it's not 2am.
", "messages": [ { "id": 818973, "postDate": "2020-04-24T08:53:58.533Z", "content": "I will update this list more tomorrow, but I audited some people's work after they made their kernels public after the submission deadline and I noticed a couple of subtle errors that concerned me.
\n\nwriting to file and not cleaning up between predictions
\n\n2.Not putting boundaries on input size.
\n\n...to be continued when it's not 2am.
", "rawMarkdown": "I will update this list more tomorrow, but I audited some people's work after they made their kernels public after the submission deadline and I noticed a couple of subtle errors that concerned me. \n\n1. writing to file and not cleaning up between predictions\n\n- In this kernel, the videos are rewritten, but then never cleared until the very end with the rm-r videos call. https://www.kaggle.com/unkownhihi/dfdc-lrcn-inference/data\nThis can accumulate in disk space and eventually past a certain number of videos it will fail. \n\n2.Not putting boundaries on input size. \n\n- Some kernels did such things as sampling frames from the expected 300. if you sample every 10 frames it is only 30 frames and not that large in memory or to push to gpu, but we were given no guarantee of the length of the test set videos. Since they were in the wild it's entirely possible they were 60 fps and a minute long resulting in your input being a much larger size and potentially OOM'ing the entire kernel and killing it. It is better to put some bounds on the max size recorded from the videos. \n\n...to be continued when it's not 2am. \n\n", "votes": 6 }, { "id": 819029, "postDate": "2020-04-24T09:46:01.657Z", "content": "Good points. I think also as @vaillant raised here people might not have been filling completely failed predictions with a default prediction like 0.5 if their inference pipeline had never failed to parse a video in the public dataset. This is relevant giving their phrasing:
\n\nThere was a ~6.5% failure rate on re-runs, due mostly to incomplete submissions (submissions which failed to create predictions for all samples)
That looks like a plausible reason. After checking I noticed not all my error paths made sure the skipped file received a 0.5 prediction. 🤕
", "rawMarkdown": "That looks like a plausible reason. After checking I noticed not all my error paths made sure the skipped file received a 0.5 prediction. 🤕 " }, { "id": 819401, "postDate": "2020-04-24T15:13:44.400Z", "content": "https://www.kaggle.com/c/deepfake-detection-challenge/discussion/122841#701781
\n\nI listed some of these previously. It is very unfortunate that these subtle gotchas were revealed on the test set. I am unsure of how kaggle will handle this. Some in the past have said it's the fault of the competitor for not properly hardening their code, but it's exceptionally difficult to make a perfect pipeline on completely unseen data. Don't think it should be a defensive programming competition, we're here for machine learning.
", "rawMarkdown": "https://www.kaggle.com/c/deepfake-detection-challenge/discussion/122841#701781\n\nI listed some of these previously. It is very unfortunate that these subtle gotchas were revealed on the test set. I am unsure of how kaggle will handle this. Some in the past have said it's the fault of the competitor for not properly hardening their code, but it's exceptionally difficult to make a perfect pipeline on completely unseen data. Don't think it should be a defensive programming competition, we're here for machine learning. " } ] }, { "id": 823338, "postDate": "2020-04-27T15:20:13.360Z", "content": "variable resolution and variable frame rate videos (as described in the paper)
", "rawMarkdown": "variable resolution and variable frame rate videos (as described in the paper)" }, { "id": 820303, "postDate": "2020-04-25T10:19:31.790Z", "content": "Interesting points! There sure was a lot that could go wrong in pipelines for this competition.
", "rawMarkdown": "Interesting points! There sure was a lot that could go wrong in pipelines for this competition." }, { "id": 819156, "postDate": "2020-04-24T11:47:40.353Z", "content": "Here is mine. Submission failed on private LB. Interested to see if you can spot anything out of the ordinary.
\n\nhttps://www.kaggle.com/maralski/fork-of-ensemble-of-5-networks?scriptVersionId=31169240
", "rawMarkdown": "Here is mine. Submission failed on private LB. Interested to see if you can spot anything out of the ordinary.\n\nhttps://www.kaggle.com/maralski/fork-of-ensemble-of-5-networks?scriptVersionId=31169240", "replies": [ { "id": 819429, "postDate": "2020-04-24T15:31:24.330Z", "content": "get_frameids = np.linspace(0, frame_count - 1, grab_frames, endpoint=True, dtype=np.int)
this is the line that concerns me. Your grab_frames is saying to grab every 60 frames, but your frame_count is just the number of frames in the video so you are potentially putting an unbounded amount of video there. Unless I am missing somewhere else where this is protected against or I am misunderstanding the code, I think it might fall under the second point I raised.
", "rawMarkdown": "`get_frameids = np.linspace(0, frame_count - 1, grab_frames, endpoint=True, dtype=np.int)`\n\nthis is the line that concerns me. Your grab_frames is saying to grab every 60 frames, but your frame_count is just the number of frames in the video so you are potentially putting an unbounded amount of video there. Unless I am missing somewhere else where this is protected against or I am misunderstanding the code, I think it might fall under the second point I raised. ", "votes": 1 }, { "id": 819983, "postDate": "2020-04-25T04:26:20.640Z", "content": "Thanks for looking. That just grabs a total of 60 frames across the frame count. So if frame count is 300 it will grab:
\n\n\n\n\n[ins] In [6]: np.linspace(0, 300 - 1, 60, endpoint=True, dtype=np.int)
\n
\n Out[6]: \n array([ 0, 5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60,\n 65, 70, 76, 81, 86, 91, 96, 101, 106, 111, 116, 121, 126,\n 131, 136, 141, 146, 152, 157, 162, 167, 172, 177, 182, 187, 192,\n 197, 202, 207, 212, 217, 222, 228, 233, 238, 243, 248, 253, 258,\n 263, 268, 273, 278, 283, 288, 293, 299])
If it's 600 it will still grab 60.
\n\n\n\n\n[ins] In [7]: np.linspace(0, 600 - 1, 60, endpoint=True, dtype=np.int)
\n
\n Out[7]: \n array([ 0, 10, 20, 30, 40, 50, 60, 71, 81, 91, 101, 111, 121,\n 131, 142, 152, 162, 172, 182, 192, 203, 213, 223, 233, 243, 253,\n 263, 274, 284, 294, 304, 314, 324, 335, 345, 355, 365, 375, 385,\n 395, 406, 416, 426, 436, 446, 456, 467, 477, 487, 497, 507, 517,\n 527, 538, 548, 558, 568, 578, 588, 599])
I'll keep looking.
", "rawMarkdown": "Thanks for looking. That just grabs a total of 60 frames across the frame count. So if frame count is 300 it will grab:\n\n> [ins] In [6]: np.linspace(0, 300 - 1, 60, endpoint=True, dtype=np.int) \nOut[6]: \narray([ 0, 5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60,\n 65, 70, 76, 81, 86, 91, 96, 101, 106, 111, 116, 121, 126,\n 131, 136, 141, 146, 152, 157, 162, 167, 172, 177, 182, 187, 192,\n 197, 202, 207, 212, 217, 222, 228, 233, 238, 243, 248, 253, 258,\n 263, 268, 273, 278, 283, 288, 293, 299])\n\nIf it's 600 it will still grab 60.\n\n> [ins] In [7]: np.linspace(0, 600 - 1, 60, endpoint=True, dtype=np.int) \nOut[7]: \narray([ 0, 10, 20, 30, 40, 50, 60, 71, 81, 91, 101, 111, 121,\n 131, 142, 152, 162, 172, 182, 192, 203, 213, 223, 233, 243, 253,\n 263, 274, 284, 294, 304, 314, 324, 335, 345, 355, 365, 375, 385,\n 395, 406, 416, 426, 436, 446, 456, 467, 477, 487, 497, 507, 517,\n 527, 538, 548, 558, 568, 578, 588, 599])\n\nI'll keep looking." }, { "id": 819989, "postDate": "2020-04-25T04:35:20.827Z", "content": "That's true, I was thinking of the range function. Linspace is doing what you described
", "rawMarkdown": "That's true, I was thinking of the range function. Linspace is doing what you described" }, { "id": 823350, "postDate": "2020-04-27T15:28:09.933Z", "content": "\nframe_count = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))\n if frame_count < 1:\n frame_count = 300\n
The paper said there will be degraded videos so fixing the framecount is not a good idea ... what happens if you get a 150 frames total frames (the paper specifically said that the private test set will have artificial downgrading of video quality both on resolution and frame rates, I believe the numbers that were mentioned were 480p and 15 fps) so if the meta is missing and it had -1 or 0 in the fps property and you sampled based on the 300 constant , you would overflow ...
", "rawMarkdown": "```\nframe_count = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))\n if frame_count < 1:\n frame_count = 300\n```\n\nThe paper said there will be degraded videos so fixing the framecount is not a good idea ... what happens if you get a 150 frames total frames (the paper specifically said that the private test set will have artificial downgrading of video quality both on resolution and frame rates, I believe the numbers that were mentioned were 480p and 15 fps) so if the meta is missing and it had -1 or 0 in the fps property and you sampled based on the 300 constant , you would overflow ..." }, { "id": 823413, "postDate": "2020-04-27T16:07:15.613Z", "content": "Thanks for looking @ma7moudkhalid
\n\nThe video processing is in a try/except block so even if it overflows it will just get skipped and receive the default 0.5 probability.
", "rawMarkdown": "Thanks for looking @ma7moudkhalid \n\nThe video processing is in a try/except block so even if it overflows it will just get skipped and receive the default 0.5 probability.", "votes": 1 } ] }, { "id": 820112, "postDate": "2020-04-25T06:31:53.213Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 819029, "author_name": "James Howard", "author_url": "", "post_date": "2020-04-24T09:46:01.657000", "content": "Good points. I think also as @vaillant raised here people might not have been filling completely failed predictions with a default prediction like 0.5 if their inference pipeline had never failed to parse a video in the public dataset. This is relevant giving their phrasing:
\n\nThere was a ~6.5% failure rate on re-runs, due mostly to incomplete submissions (submissions which failed to create predictions for all samples)
That looks like a plausible reason. After checking I noticed not all my error paths made sure the skipped file received a 0.5 prediction. 🤕
", "votes": 0, "replies": [] }, { "id": 819401, "author_name": "ryches", "author_url": "", "post_date": "2020-04-24T15:13:44.400000", "content": "https://www.kaggle.com/c/deepfake-detection-challenge/discussion/122841#701781
\n\nI listed some of these previously. It is very unfortunate that these subtle gotchas were revealed on the test set. I am unsure of how kaggle will handle this. Some in the past have said it's the fault of the competitor for not properly hardening their code, but it's exceptionally difficult to make a perfect pipeline on completely unseen data. Don't think it should be a defensive programming competition, we're here for machine learning.
", "votes": 0, "replies": [] } ] }, { "id": 823338, "author_name": "mjsML", "author_url": "", "post_date": "2020-04-27T15:20:13.360000", "content": "variable resolution and variable frame rate videos (as described in the paper)
", "votes": 0, "replies": [] }, { "id": 820303, "author_name": "Carlo", "author_url": "", "post_date": "2020-04-25T10:19:31.790000", "content": "Interesting points! There sure was a lot that could go wrong in pipelines for this competition.
", "votes": 0, "replies": [] }, { "id": 819156, "author_name": "maralski", "author_url": "", "post_date": "2020-04-24T11:47:40.353000", "content": "Here is mine. Submission failed on private LB. Interested to see if you can spot anything out of the ordinary.
\n\nhttps://www.kaggle.com/maralski/fork-of-ensemble-of-5-networks?scriptVersionId=31169240
", "votes": 0, "replies": [ { "id": 819429, "author_name": "ryches", "author_url": "", "post_date": "2020-04-24T15:31:24.330000", "content": "get_frameids = np.linspace(0, frame_count - 1, grab_frames, endpoint=True, dtype=np.int)
this is the line that concerns me. Your grab_frames is saying to grab every 60 frames, but your frame_count is just the number of frames in the video so you are potentially putting an unbounded amount of video there. Unless I am missing somewhere else where this is protected against or I am misunderstanding the code, I think it might fall under the second point I raised.
", "votes": 1, "replies": [] }, { "id": 819983, "author_name": "maralski", "author_url": "", "post_date": "2020-04-25T04:26:20.640000", "content": "Thanks for looking. That just grabs a total of 60 frames across the frame count. So if frame count is 300 it will grab:
\n\n\n\n\n[ins] In [6]: np.linspace(0, 300 - 1, 60, endpoint=True, dtype=np.int)
\n
\n Out[6]: \n array([ 0, 5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60,\n 65, 70, 76, 81, 86, 91, 96, 101, 106, 111, 116, 121, 126,\n 131, 136, 141, 146, 152, 157, 162, 167, 172, 177, 182, 187, 192,\n 197, 202, 207, 212, 217, 222, 228, 233, 238, 243, 248, 253, 258,\n 263, 268, 273, 278, 283, 288, 293, 299])
If it's 600 it will still grab 60.
\n\n\n\n\n[ins] In [7]: np.linspace(0, 600 - 1, 60, endpoint=True, dtype=np.int)
\n
\n Out[7]: \n array([ 0, 10, 20, 30, 40, 50, 60, 71, 81, 91, 101, 111, 121,\n 131, 142, 152, 162, 172, 182, 192, 203, 213, 223, 233, 243, 253,\n 263, 274, 284, 294, 304, 314, 324, 335, 345, 355, 365, 375, 385,\n 395, 406, 416, 426, 436, 446, 456, 467, 477, 487, 497, 507, 517,\n 527, 538, 548, 558, 568, 578, 588, 599])
I'll keep looking.
", "votes": 0, "replies": [] }, { "id": 819989, "author_name": "ryches", "author_url": "", "post_date": "2020-04-25T04:35:20.827000", "content": "That's true, I was thinking of the range function. Linspace is doing what you described
", "votes": 0, "replies": [] }, { "id": 823350, "author_name": "mjsML", "author_url": "", "post_date": "2020-04-27T15:28:09.933000", "content": "\nframe_count = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))\n if frame_count < 1:\n frame_count = 300\n
The paper said there will be degraded videos so fixing the framecount is not a good idea ... what happens if you get a 150 frames total frames (the paper specifically said that the private test set will have artificial downgrading of video quality both on resolution and frame rates, I believe the numbers that were mentioned were 480p and 15 fps) so if the meta is missing and it had -1 or 0 in the fps property and you sampled based on the 300 constant , you would overflow ...
", "votes": 0, "replies": [] }, { "id": 823413, "author_name": "maralski", "author_url": "", "post_date": "2020-04-27T16:07:15.613000", "content": "Thanks for looking @ma7moudkhalid
\n\nThe video processing is in a try/except block so even if it overflows it will just get skipped and receive the default 0.5 probability.
", "votes": 1, "replies": [] } ] }, { "id": 820112, "author_name": "", "author_url": "", "post_date": "2020-04-25T06:31:53.213000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "818973": "I will update this list more tomorrow, but I audited some people's work after they made their kernels public after the submission deadline and I noticed a couple of subtle errors that concerned me. \n\n1. writing to file and not cleaning up between predictions\n\n- In this kernel, the videos are rewritten, but then never cleared until the very end with the rm-r videos call. https://www.kaggle.com/unkownhihi/dfdc-lrcn-inference/data\nThis can accumulate in disk space and eventually past a certain number of videos it will fail. \n\n2.Not putting boundaries on input size. \n\n- Some kernels did such things as sampling frames from the expected 300. if you sample every 10 frames it is only 30 frames and not that large in memory or to push to gpu, but we were given no guarantee of the length of the test set videos. Since they were in the wild it's entirely possible they were 60 fps and a minute long resulting in your input being a much larger size and potentially OOM'ing the entire kernel and killing it. It is better to put some bounds on the max size recorded from the videos. \n\n...to be continued when it's not 2am. \n\n", "819029": "Good points. I think also as @vaillant raised [here ](https://www.kaggle.com/c/deepfake-detection-challenge/discussion/145643#818587)people might not have been filling completely failed predictions with a default prediction like 0.5 if their inference pipeline had never failed to parse a video in the public dataset. This is relevant giving their phrasing:\n\n`There was a ~6.5% failure rate on re-runs, due mostly to incomplete submissions (submissions which failed to create predictions for all samples)`", "823338": "variable resolution and variable frame rate videos (as described in the paper)", "820303": "Interesting points! There sure was a lot that could go wrong in pipelines for this competition.", "819156": "Here is mine. Submission failed on private LB. Interested to see if you can spot anything out of the ordinary.\n\nhttps://www.kaggle.com/maralski/fork-of-ensemble-of-5-networks?scriptVersionId=31169240", "820112": "" } }