![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1258264%2F5fd082c547fb14f262a040724e290d0f%2FQQ20200112160934.png?generation=1578816784560555&alt=media\")
Does it mean my program had run for more than 9 hours? \n
I once received this two error messages, and they had been solved.\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1258264%2Fa043adf0dc2bdd42a91243d1bd5d1824%2FQQ20200112161221.png?generation=1578816782875322&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1258264%2F8269635030e75353fed3d81302919061%2FQQ20200112161213.png?generation=1578816783047634&alt=media\")
Does it mean my program had run for more than 9 hours? \n
I once received this two error messages, and they had been solved.\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F3117946%2F61048ef3bfea675bc6e57f92011ce491%2Finbox_3117946_e2e21cd7c43deb1b90fe9f7be54fec1b_1578124856714.jpg?generation=1578877705803369&alt=media\")
I have a similar error. Can someone tell me why?
", "rawMarkdown": "\n\nI have a similar error. Can someone tell me why?", "votes": 1, "replies": [ { "id": 717489, "postDate": "2020-01-13T07:32:49.423Z", "content": "I think you should make sure everytime you to(device), the device should be cuda:0. Because we are only allowed to use one GPU.
> Shen Chen wrote:\n> \n> \n> \n> I have a similar error. Can someone tell me why?
In my experience, this is to do with exceeding one type of resource limit that is applied on the notebook. I have two situations of the same message so far, both time I was reading 10 frames for each video to do average prediction on multiple models. In both situations, I have managed rerun successfully, once by reducing the number of frame I read per video, and once by reducing number of models I use. I believe this is mostly related to RAM usage
", "rawMarkdown": "> **Shen Chen wrote:**\n> \n> \n> \n> I have a similar error. Can someone tell me why?\n\nIn my experience, this is to do with exceeding one type of resource limit that is applied on the notebook. I have two situations of the same message so far, both time I was reading 10 frames for each video to do average prediction on multiple models. In both situations, I have managed rerun successfully, once by reducing the number of frame I read per video, and once by reducing number of models I use. I believe this is mostly related to RAM usage \n" }, { "id": 717558, "postDate": "2020-01-13T09:30:18.040Z", "content": "@yifanxie I only use 4 frames for each video and one model to prediction. I think these settings are unlikely to exceed the RAM usage limit. In addition, I change the face detector from mtcnn to dlib, this error will not occur, but I still couldn't find out why.
", "rawMarkdown": "@yifanxie I only use 4 frames for each video and one model to prediction. I think these settings are unlikely to exceed the RAM usage limit. In addition, I change the face detector from mtcnn to dlib, this error will not occur, but I still couldn't find out why.\n\n" }, { "id": 717569, "postDate": "2020-01-13T09:49:17.200Z", "content": "sure, everyone's setting might be different - that was just my experience so may not be directly relevant in your situation.
\n\nI would definitely try to profile your RAM or Hardisk usage though - i.e. are you generate output file on the harddisk that might exceed the limitation?
\n\nThe message itself is quite clear in a sense that you are using too much of something when predicting on the 4000 private test set. You can always create a routine on the public test set of 400 videos for several repetitive runs to \"simulate\" the full run, and monitor your RAM/Hardisk/CPU usage on the editing mode of the notebook.
", "rawMarkdown": "sure, everyone's setting might be different - that was just my experience so may not be directly relevant in your situation.\n\nI would definitely try to profile your RAM or Hardisk usage though - i.e. are you generate output file on the harddisk that might exceed the limitation? \n\nThe message itself is quite clear in a sense that you are using too much of something when predicting on the 4000 private test set. You can always create a routine on the public test set of 400 videos for several repetitive runs to \"simulate\" the full run, and monitor your RAM/Hardisk/CPU usage on the editing mode of the notebook. " } ] }, { "id": 720901, "postDate": "2020-01-16T20:49:23.337Z", "content": "I also got this error. Only difference from my previous submission was the weights of my model. Really weird.
\n\nEdit: Just to add that my notebook ran to way less than 9 hours prior to giving the \"Kaggle Error\", so not the same case as @carlossouza described.
\n\nAlso, @carlossouza if you are using a GPU to do face detection, I'm not sure if you are just increasing the batch size or using a fixed batch size. If you are increasing the batch size it shouldn't take 3x the time, as it doesn't scale linearly because the GPU should process the batch in parallel. The same should be true when doing the predictions. I've experimented with different numbers of faces and going from a small number to a much bigger number increased the time by 1~3 minutes in total (for the 400 videos in public test set).
", "rawMarkdown": "I also got this error. Only difference from my previous submission was the weights of my model. Really weird. \n\nEdit: Just to add that my notebook ran to way less than 9 hours prior to giving the \"Kaggle Error\", so not the same case as @carlossouza described. \n\nAlso, @carlossouza if you are using a GPU to do face detection, I'm not sure if you are just increasing the batch size or using a fixed batch size. If you are increasing the batch size it shouldn't take 3x the time, as it doesn't scale linearly because the GPU should process the batch in parallel. The same should be true when doing the predictions. I've experimented with different numbers of faces and going from a small number to a much bigger number increased the time by 1~3 minutes in total (for the 400 videos in public test set).", "votes": 2, "replies": [ { "id": 721990, "postDate": "2020-01-18T01:02:33.690Z", "content": "Thanks @pedromb ! You are right: it doesn't scale linearly! \nFollowing @juliaelliott 's advice, I re-run it and now it worked.\nSo, my first conclusion was incorrect: \"Kaggle Error\" does NOT mean we blow up 9hs time.
", "rawMarkdown": "Thanks @pedromb ! You are right: it doesn't scale linearly! \nFollowing @juliaelliott 's advice, I re-run it and now it worked.\nSo, my first conclusion was incorrect: \"Kaggle Error\" does NOT mean we blow up 9hs time." } ] }, { "id": 719903, "postDate": "2020-01-16T01:00:16.050Z", "content": "“Kaggle Error” is a rare system error, and it’s recommended that you resubmit to resolve the issue. Please let us know if resubmission continues to fail.
", "rawMarkdown": "“Kaggle Error” is a rare system error, and it’s recommended that you resubmit to resolve the issue. Please let us know if resubmission continues to fail.", "votes": 2, "replies": [ { "id": 721989, "postDate": "2020-01-18T01:00:50.970Z", "content": "Thank you @juliaelliott ! Thanks to your message, I re-run the notebook, and now it worked!
", "rawMarkdown": "Thank you @juliaelliott ! Thanks to your message, I re-run the notebook, and now it worked!" }, { "id": 722556, "postDate": "2020-01-18T17:54:05.993Z", "content": "@juliaelliott It looks @carlossouza is not alone to have this error today. Same kernel run twice today, first attempt with \"Kaggle error\" after a few hours, second attempt \"Notebook Exceeded Allowed compute\" after a few minutes.
\n\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F698363%2Fcf3c0b5933cae3b49a96025a25a00d77%2Fkaggle_error.png?generation=1579369964893669&alt=media\")
Quite difficult to troubleshoot.
", "rawMarkdown": "@juliaelliott It looks @carlossouza is not alone to have this error today. Same kernel run twice today, first attempt with \"Kaggle error\" after a few hours, second attempt \"Notebook Exceeded Allowed compute\" after a few minutes.\n\n\n\nQuite difficult to troubleshoot.", "votes": 1 }, { "id": 722570, "postDate": "2020-01-18T18:28:27Z", "content": "Very difficult to troubleshoot. My last 10(!) submissions are errors, I am trying different things, but with 2 submissions a day it is proving too hard to pinpoint the problem.
\n\nIt can be super nice if someone from Kaggle looks at 10-20 of the common failures and write a short guide about the pitfalls that we need to be mindful about.
", "rawMarkdown": "Very difficult to troubleshoot. My last 10(!) submissions are errors, I am trying different things, but with 2 submissions a day it is proving too hard to pinpoint the problem. \n\nIt can be super nice if someone from Kaggle looks at 10-20 of the common failures and write a short guide about the pitfalls that we need to be mindful about. ", "votes": 3 }, { "id": 722582, "postDate": "2020-01-18T18:55:52.917Z", "content": "@zaharch To predict safely, I've surrounded model.predict for each video by try/except, I've added np.clip to avoid log loss computation issue but it looks not enough. I think most of my problems comes batch_size, frames buffer and face size, but I cannot understand why it works on commit. Maybe some faces extracted are too big (only on test dataset loaded on submit) that makes memory crash (either RAM and/or GPU).
\nfor idx, row in tqdm(submission_pd.iterrows(), total=submission_pd.shape[0]):\n filename = row[\"filename\"]\n try: \nAnd
\n # Extract faces from video frames and prepare for model (normalize)\n ds_test= VideoDataset(filename, label=None, subset='test', transform=tr)
\n # Run model\n dl_test = DataLoader(ds_test, batch_size=24, shuffle=False, num_workers=1)
\n clf_predict_probas, _ = run_model_fn(dl_test )
\n ds_test.cleanup()\n # Average mean (of all faces within all frames, other strategy to try out)\n filename_prob = np.nanmean(clf_predict_probas)\n submission_pd.loc[idx, \"label\"] = np.clip(filename_prob, 0.01, 0.99)\n torch.cuda.empty_cache() # If any GPU issue\n except Exception as ex:\n submission_pd.loc[idx, \"label\"] = 0.5\n print(\"Cannot predict on %s\" % filename) \nnp.fillna if any prediction totally fails (no error but no face or something else).
\nsubmission_pd.fillna(0.5, inplace=True)\nsubmission_pd.sort_values('filename').to_csv('submission.csv', index=False)\n
@juliaelliott I've read in another thread that we should read test_videos folder instead of sample_submission.csv, is it true? I would expect getting filenames from sample_submission.csv like in other competitions.
```\nTEST_HOME = \"/kaggle/input/deepfake-detection-challenge/test_videos/\"\nSAMPLE_SUBMISSION = \"/kaggle/input/deepfake-detection-challenge/sample_submission.csv\"
\n\nfilenames = glob.glob(TEST_HOME + \"*.mp4\")\nbasenames = [os.path.basename(filename) for filename in filenames]\nsubmission_pd = pd.DataFrame(basenames, columns=[\"filename\"])\n```
", "rawMarkdown": "@zaharch To predict safely, I've surrounded `model.predict` for each video by `try/except`, I've added `np.clip` to avoid log loss computation issue but it looks not enough. I think most of my problems comes batch_size, frames buffer and face size, but I cannot understand why it works on commit. Maybe some faces extracted are too big (only on test dataset loaded on submit) that makes memory crash (either RAM and/or GPU). \n\n```\nfor idx, row in tqdm(submission_pd.iterrows(), total=submission_pd.shape[0]):\n\tfilename = row[\"filename\"]\n\ttry: \n\t\t# Extract faces from video frames and prepare for model (normalize)\n\t\tds_test= VideoDataset(filename, label=None, subset='test', transform=tr) \n\t\t# Run model\n\t\tdl_test = DataLoader(ds_test, batch_size=24, shuffle=False, num_workers=1) \n\t\tclf_predict_probas, _ = run_model_fn(dl_test ) \n\t\tds_test.cleanup()\n\t\t# Average mean (of all faces within all frames, other strategy to try out)\n\t\tfilename_prob = np.nanmean(clf_predict_probas)\n\t\tsubmission_pd.loc[idx, \"label\"] = np.clip(filename_prob, 0.01, 0.99)\n\t\ttorch.cuda.empty_cache() # If any GPU issue\n\texcept Exception as ex:\n\t\tsubmission_pd.loc[idx, \"label\"] = 0.5\n\t\tprint(\"Cannot predict on %s\" % filename) \n```\nAnd `np.fillna` if any prediction totally fails (no error but no face or something else). \n\n```\nsubmission_pd.fillna(0.5, inplace=True)\nsubmission_pd.sort_values('filename').to_csv('submission.csv', index=False)\n```\n\n@juliaelliott I've read in another thread that we should read `test_videos` folder instead of `sample_submission.csv`, is it true? I would expect getting filenames from `sample_submission.csv` like in other competitions.\n\n```\nTEST_HOME = \"/kaggle/input/deepfake-detection-challenge/test_videos/\"\nSAMPLE_SUBMISSION = \"/kaggle/input/deepfake-detection-challenge/sample_submission.csv\"\n\n# I would expect this one to work\n# submission_pd = pd.read_csv(SAMPLE_SUBMISSION) \n\n# But this is what I do:\nfilenames = glob.glob(TEST_HOME + \"*.mp4\")\nbasenames = [os.path.basename(filename) for filename in filenames]\nsubmission_pd = pd.DataFrame(basenames, columns=[\"filename\"])\n```\n\n", "votes": 2 }, { "id": 722728, "postDate": "2020-01-19T02:11:08.503Z", "content": "Thank you, this is extremely helpful and should be part of the competition guidelines. I think the latter way is the right way because you're not dependent on a submission file with a priori labels. Note that this question has been raised in other discussion topics without followup from the organizers.
", "rawMarkdown": "Thank you, this is extremely helpful and should be part of the competition guidelines. I think the latter way is the right way because you're not dependent on a submission file with a priori labels. Note that this question has been raised in other discussion topics without followup from the organizers." }, { "id": 767973, "postDate": "2020-03-10T10:13:48.487Z", "content": "Hi, I have encountered the kaggle error twice with only ckpt changed. It wasted two times of submissions. Can you help me out with this? @juliaelliott
", "rawMarkdown": "Hi, I have encountered the kaggle error twice with only ckpt changed. It wasted two times of submissions. Can you help me out with this? @juliaelliott " } ] }, { "id": 717042, "postDate": "2020-01-12T16:36:12.710Z", "content": "I just received the 1st one (Kaggle Error). The only thing I've changed vs. a previously successful submission was the # of faces extracted from the test videos (3x vs. my baseline), to make the inference by averaging. I was actually expecting an error, because my baseline was taking 4hs to complete; therefore, 3x-ing the time to extract faces from videos should approx. 3x the time to complete, i.e. 12hs, above the 9hs threshold.
\n\nThis experiment confirmed 2 things for me:\n1. This \"Kaggle Error\" message is what the system outputs when we blow up 9hs\n2. I need a faster Face Detector to make inferences using more frames/video :)
\n\nHope it helps! Cheers! :)
", "rawMarkdown": "I just received the 1st one (Kaggle Error). The only thing I've changed vs. a previously successful submission was the # of faces extracted from the test videos (3x vs. my baseline), to make the inference by averaging. I was actually expecting an error, because my baseline was taking 4hs to complete; therefore, 3x-ing the time to extract faces from videos should approx. 3x the time to complete, i.e. 12hs, above the 9hs threshold. \n\nThis experiment confirmed 2 things for me:\n1. This \"Kaggle Error\" message is what the system outputs when we blow up 9hs\n2. I need a faster Face Detector to make inferences using more frames/video :)\n\nHope it helps! Cheers! :)", "votes": 2, "replies": [ { "id": 717490, "postDate": "2020-01-13T07:33:41.497Z", "content": "Thanks! It does make sense.
", "rawMarkdown": "Thanks! It does make sense.", "votes": 2 } ] }, { "id": 736629, "postDate": "2020-02-04T12:02:52.673Z", "content": "I also got \"Kaggle Error\" now. I'll try to resubmit...
", "rawMarkdown": "I also got \"Kaggle Error\" now. I'll try to resubmit..." }, { "id": 717590, "postDate": "2020-01-13T10:47:06.153Z", "content": "The Kaggle Error is the error on your training set usually
", "rawMarkdown": "The Kaggle Error is the error on your training set usually" }, { "id": 722726, "postDate": "2020-01-19T02:02:40.923Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 717287, "author_name": "Chason", "author_url": "", "post_date": "2020-01-13T01:09:41.360000", "content": "
I have a similar error. Can someone tell me why?
", "votes": 1, "replies": [ { "id": 717489, "author_name": "FrazierLei", "author_url": "", "post_date": "2020-01-13T07:32:49.423000", "content": "I think you should make sure everytime you to(device), the device should be cuda:0. Because we are only allowed to use one GPU.
> Shen Chen wrote:\n> \n> \n> \n> I have a similar error. Can someone tell me why?
In my experience, this is to do with exceeding one type of resource limit that is applied on the notebook. I have two situations of the same message so far, both time I was reading 10 frames for each video to do average prediction on multiple models. In both situations, I have managed rerun successfully, once by reducing the number of frame I read per video, and once by reducing number of models I use. I believe this is mostly related to RAM usage
", "votes": 0, "replies": [] }, { "id": 717558, "author_name": "Chason", "author_url": "", "post_date": "2020-01-13T09:30:18.040000", "content": "@yifanxie I only use 4 frames for each video and one model to prediction. I think these settings are unlikely to exceed the RAM usage limit. In addition, I change the face detector from mtcnn to dlib, this error will not occur, but I still couldn't find out why.
", "votes": 0, "replies": [] }, { "id": 717569, "author_name": "Yifan Xie", "author_url": "", "post_date": "2020-01-13T09:49:17.200000", "content": "sure, everyone's setting might be different - that was just my experience so may not be directly relevant in your situation.
\n\nI would definitely try to profile your RAM or Hardisk usage though - i.e. are you generate output file on the harddisk that might exceed the limitation?
\n\nThe message itself is quite clear in a sense that you are using too much of something when predicting on the 4000 private test set. You can always create a routine on the public test set of 400 videos for several repetitive runs to \"simulate\" the full run, and monitor your RAM/Hardisk/CPU usage on the editing mode of the notebook.
", "votes": 0, "replies": [] } ] }, { "id": 720901, "author_name": "Pedro Bernardo", "author_url": "", "post_date": "2020-01-16T20:49:23.337000", "content": "I also got this error. Only difference from my previous submission was the weights of my model. Really weird.
\n\nEdit: Just to add that my notebook ran to way less than 9 hours prior to giving the \"Kaggle Error\", so not the same case as @carlossouza described.
\n\nAlso, @carlossouza if you are using a GPU to do face detection, I'm not sure if you are just increasing the batch size or using a fixed batch size. If you are increasing the batch size it shouldn't take 3x the time, as it doesn't scale linearly because the GPU should process the batch in parallel. The same should be true when doing the predictions. I've experimented with different numbers of faces and going from a small number to a much bigger number increased the time by 1~3 minutes in total (for the 400 videos in public test set).
", "votes": 2, "replies": [ { "id": 721990, "author_name": "Carlos Souza", "author_url": "", "post_date": "2020-01-18T01:02:33.690000", "content": "Thanks @pedromb ! You are right: it doesn't scale linearly! \nFollowing @juliaelliott 's advice, I re-run it and now it worked.\nSo, my first conclusion was incorrect: \"Kaggle Error\" does NOT mean we blow up 9hs time.
", "votes": 0, "replies": [] } ] }, { "id": 719903, "author_name": "Julia Elliott", "author_url": "", "post_date": "2020-01-16T01:00:16.050000", "content": "“Kaggle Error” is a rare system error, and it’s recommended that you resubmit to resolve the issue. Please let us know if resubmission continues to fail.
", "votes": 2, "replies": [ { "id": 721989, "author_name": "Carlos Souza", "author_url": "", "post_date": "2020-01-18T01:00:50.970000", "content": "Thank you @juliaelliott ! Thanks to your message, I re-run the notebook, and now it worked!
", "votes": 0, "replies": [] }, { "id": 722556, "author_name": "MPWARE", "author_url": "", "post_date": "2020-01-18T17:54:05.993000", "content": "@juliaelliott It looks @carlossouza is not alone to have this error today. Same kernel run twice today, first attempt with \"Kaggle error\" after a few hours, second attempt \"Notebook Exceeded Allowed compute\" after a few minutes.
\n\n
Quite difficult to troubleshoot.
", "votes": 1, "replies": [] }, { "id": 722570, "author_name": "nosound", "author_url": "", "post_date": "2020-01-18T18:28:27", "content": "Very difficult to troubleshoot. My last 10(!) submissions are errors, I am trying different things, but with 2 submissions a day it is proving too hard to pinpoint the problem.
\n\nIt can be super nice if someone from Kaggle looks at 10-20 of the common failures and write a short guide about the pitfalls that we need to be mindful about.
", "votes": 3, "replies": [] }, { "id": 722582, "author_name": "MPWARE", "author_url": "", "post_date": "2020-01-18T18:55:52.917000", "content": "@zaharch To predict safely, I've surrounded model.predict for each video by try/except, I've added np.clip to avoid log loss computation issue but it looks not enough. I think most of my problems comes batch_size, frames buffer and face size, but I cannot understand why it works on commit. Maybe some faces extracted are too big (only on test dataset loaded on submit) that makes memory crash (either RAM and/or GPU).
\nfor idx, row in tqdm(submission_pd.iterrows(), total=submission_pd.shape[0]):\n filename = row[\"filename\"]\n try: \nAnd
\n # Extract faces from video frames and prepare for model (normalize)\n ds_test= VideoDataset(filename, label=None, subset='test', transform=tr)
\n # Run model\n dl_test = DataLoader(ds_test, batch_size=24, shuffle=False, num_workers=1)
\n clf_predict_probas, _ = run_model_fn(dl_test )
\n ds_test.cleanup()\n # Average mean (of all faces within all frames, other strategy to try out)\n filename_prob = np.nanmean(clf_predict_probas)\n submission_pd.loc[idx, \"label\"] = np.clip(filename_prob, 0.01, 0.99)\n torch.cuda.empty_cache() # If any GPU issue\n except Exception as ex:\n submission_pd.loc[idx, \"label\"] = 0.5\n print(\"Cannot predict on %s\" % filename) \nnp.fillna if any prediction totally fails (no error but no face or something else).
\nsubmission_pd.fillna(0.5, inplace=True)\nsubmission_pd.sort_values('filename').to_csv('submission.csv', index=False)\n
@juliaelliott I've read in another thread that we should read test_videos folder instead of sample_submission.csv, is it true? I would expect getting filenames from sample_submission.csv like in other competitions.
```\nTEST_HOME = \"/kaggle/input/deepfake-detection-challenge/test_videos/\"\nSAMPLE_SUBMISSION = \"/kaggle/input/deepfake-detection-challenge/sample_submission.csv\"
\n\nfilenames = glob.glob(TEST_HOME + \"*.mp4\")\nbasenames = [os.path.basename(filename) for filename in filenames]\nsubmission_pd = pd.DataFrame(basenames, columns=[\"filename\"])\n```
", "votes": 2, "replies": [] }, { "id": 722728, "author_name": "Mark Conway", "author_url": "", "post_date": "2020-01-19T02:11:08.503000", "content": "Thank you, this is extremely helpful and should be part of the competition guidelines. I think the latter way is the right way because you're not dependent on a submission file with a priori labels. Note that this question has been raised in other discussion topics without followup from the organizers.
", "votes": 0, "replies": [] }, { "id": 767973, "author_name": "xxn-xx", "author_url": "", "post_date": "2020-03-10T10:13:48.487000", "content": "Hi, I have encountered the kaggle error twice with only ckpt changed. It wasted two times of submissions. Can you help me out with this? @juliaelliott
", "votes": 0, "replies": [] } ] }, { "id": 717042, "author_name": "Carlos Souza", "author_url": "", "post_date": "2020-01-12T16:36:12.710000", "content": "I just received the 1st one (Kaggle Error). The only thing I've changed vs. a previously successful submission was the # of faces extracted from the test videos (3x vs. my baseline), to make the inference by averaging. I was actually expecting an error, because my baseline was taking 4hs to complete; therefore, 3x-ing the time to extract faces from videos should approx. 3x the time to complete, i.e. 12hs, above the 9hs threshold.
\n\nThis experiment confirmed 2 things for me:\n1. This \"Kaggle Error\" message is what the system outputs when we blow up 9hs\n2. I need a faster Face Detector to make inferences using more frames/video :)
\n\nHope it helps! Cheers! :)
", "votes": 2, "replies": [ { "id": 717490, "author_name": "FrazierLei", "author_url": "", "post_date": "2020-01-13T07:33:41.497000", "content": "Thanks! It does make sense.
", "votes": 2, "replies": [] } ] }, { "id": 736629, "author_name": "Felipe Loque", "author_url": "", "post_date": "2020-02-04T12:02:52.673000", "content": "I also got \"Kaggle Error\" now. I'll try to resubmit...
", "votes": 0, "replies": [] }, { "id": 717590, "author_name": "Oleg Gribanov", "author_url": "", "post_date": "2020-01-13T10:47:06.153000", "content": "The Kaggle Error is the error on your training set usually
", "votes": 0, "replies": [] }, { "id": 722726, "author_name": "", "author_url": "", "post_date": "2020-01-19T02:02:40.923000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "716766": "Does it mean my program had run for more than 9 hours? \n\n\nI once received this two error messages, and they had been solved.\n\n\n\n", "717287": "\n\nI have a similar error. Can someone tell me why?", "720901": "I also got this error. Only difference from my previous submission was the weights of my model. Really weird. \n\nEdit: Just to add that my notebook ran to way less than 9 hours prior to giving the \"Kaggle Error\", so not the same case as @carlossouza described. \n\nAlso, @carlossouza if you are using a GPU to do face detection, I'm not sure if you are just increasing the batch size or using a fixed batch size. If you are increasing the batch size it shouldn't take 3x the time, as it doesn't scale linearly because the GPU should process the batch in parallel. The same should be true when doing the predictions. I've experimented with different numbers of faces and going from a small number to a much bigger number increased the time by 1~3 minutes in total (for the 400 videos in public test set).", "719903": "“Kaggle Error” is a rare system error, and it’s recommended that you resubmit to resolve the issue. Please let us know if resubmission continues to fail.", "717042": "I just received the 1st one (Kaggle Error). The only thing I've changed vs. a previously successful submission was the # of faces extracted from the test videos (3x vs. my baseline), to make the inference by averaging. I was actually expecting an error, because my baseline was taking 4hs to complete; therefore, 3x-ing the time to extract faces from videos should approx. 3x the time to complete, i.e. 12hs, above the 9hs threshold. \n\nThis experiment confirmed 2 things for me:\n1. This \"Kaggle Error\" message is what the system outputs when we blow up 9hs\n2. I need a faster Face Detector to make inferences using more frames/video :)\n\nHope it helps! Cheers! :)", "736629": "I also got \"Kaggle Error\" now. I'll try to resubmit...", "717590": "The Kaggle Error is the error on your training set usually", "722726": "" } }