{ "id": 297774, "title": "Tips about \"Submission Scoring Error\"", "url": "/competitions/tensorflow-great-barrier-reef/discussion/297774", "author_name": "", "post_date": "2021-12-29T13:58:34.855662600Z", "votes": 4, "comment_count": 23, "views": 0, "content": "
\n

Note for someone who encounters the same error

\n

I’m sorry but this error is too general error, and I can’t say the reason of failure only by reading your submission code.

\n

What I can say generally is just said in the below thread:
\nhttps://www.kaggle.com/c/tensorflow-great-barrier-reef/discussion/297774#1637598

\n

To be short, just make sure your submission is O.K. on your local before submit your code.
\nHope you success in detecting error.

\n
\n

Original Contents

\n

Maybe almost all of you are not in the case, but, for some reason, I have to call test data providing API from python script like this:

\n
cd /tmp/my_project\npython /tmp/infer.py\n
\n

in this case, submission.csv is created in directory /tmp/my_project, so the evaluation fails after completed inference over whole test data.

\n

In this case, don't forget to copy submission.csv to /kaggle/working like this:

\n
cd /tmp/my_project\npython /tmp/infer.py\ncp submission.csv /kaggle/working/.\n
\n

It took me some hours to find out this. (At first time, I just copied sample_submission.csv as submission.csv. Clearly this is wrong. API creates the file submission.csv. Don't place the other file.)

\n

I guess it is as usual in In the kernel competition, but the crew why the notebook failed is too few. it is really frustrating.

\n

Hope this tips helps someone.

\n

BTW, I share a useful tips for debugging in kernel competition:
\nhttps://www.kaggle.com/c/nfl-health-and-safety-helmet-assignment/discussion/279046

", "messages": [ { "id": "1632331", "postDate": "12/29/2021 13:58:34", "content": "
\n

Note for someone who encounters the same error

\n

I’m sorry but this error is too general error, and I can’t say the reason of failure only by reading your submission code.

\n

What I can say generally is just said in the below thread:
\nhttps://www.kaggle.com/c/tensorflow-great-barrier-reef/discussion/297774#1637598

\n

To be short, just make sure your submission is O.K. on your local before submit your code.
\nHope you success in detecting error.

\n
\n

Original Contents

\n

Maybe almost all of you are not in the case, but, for some reason, I have to call test data providing API from python script like this:

\n
cd /tmp/my_project\npython /tmp/infer.py\n
\n

in this case, submission.csv is created in directory /tmp/my_project, so the evaluation fails after completed inference over whole test data.

\n

In this case, don't forget to copy submission.csv to /kaggle/working like this:

\n
cd /tmp/my_project\npython /tmp/infer.py\ncp submission.csv /kaggle/working/.\n
\n

It took me some hours to find out this. (At first time, I just copied sample_submission.csv as submission.csv. Clearly this is wrong. API creates the file submission.csv. Don't place the other file.)

\n

I guess it is as usual in In the kernel competition, but the crew why the notebook failed is too few. it is really frustrating.

\n

Hope this tips helps someone.

\n

BTW, I share a useful tips for debugging in kernel competition:
\nhttps://www.kaggle.com/c/nfl-health-and-safety-helmet-assignment/discussion/279046

", "rawMarkdown": "# Note for someone who encounters the same error\n\nI’m sorry but this error is too general error, and I can’t say the reason of failure only by reading your submission code.\n\nWhat I can say generally is just said in the below thread:\nhttps://www.kaggle.com/c/tensorflow-great-barrier-reef/discussion/297774#1637598\n\nTo be short, just make sure your submission is O.K. on your local before submit your code.\nHope you success in detecting error.\n\n----------------\n\n# Original Contents\n\nMaybe almost all of you are not in the case, but, for some reason, I have to call test data providing API from python script like this:\n\n```bash\ncd /tmp/my_project\npython /tmp/infer.py\n```\n\nin this case, `submission.csv` is created in directory `/tmp/my_project`, so the evaluation fails after completed inference over whole test data.\n\nIn this case, don't forget to copy `submission.csv` to `/kaggle/working` like this:\n\n```bash\ncd /tmp/my_project\npython /tmp/infer.py\ncp submission.csv /kaggle/working/.\n```\n\nIt took me some hours to find out this. (At first time, I just copied `sample_submission.csv` as `submission.csv`. Clearly this is wrong. API creates the file `submission.csv`. Don't place the other file.)\n\nI guess it is as usual in In the kernel competition, but the crew why the notebook failed is too few. it is really frustrating.\n\nHope this tips helps someone.\n\nBTW, I share a useful tips for debugging in kernel competition:\nhttps://www.kaggle.com/c/nfl-health-and-safety-helmet-assignment/discussion/279046", "votes": null }, { "id": "1637236", "postDate": "01/03/2022 17:11:33", "content": "

Do you mind sharing if your submission csv has NaN in it? My team is having difficulty getting it to submit.

", "rawMarkdown": "Do you mind sharing if your submission csv has NaN in it? My team is having difficulty getting it to submit.", "votes": null }, { "id": "1637471", "postDate": "01/03/2022 23:14:50", "content": "

@hkrsmk Do you mean the annotation column of submission.csv contains nan or not? In my case, no. My notebook outputs blank string when zero detection.

\n

But some public notebook[1], it seems to contain nan, so it might not be the cause of submission error, if you show one.
\n[1] https://www.kaggle.com/remekkinas/yolox-inference-on-kaggle-for-cots-lb-0-507

\n

Why don’t you check if a simple nan only submission pass the evaluation If you want to be sure? I think it will take only 10 minutes or so.

", "rawMarkdown": "hkrsmk Do you mean the annotation column of submission.csv contains nan or not? In my case, no. My notebook outputs blank string when zero detection.\n\nBut some public notebook[1], it seems to contain nan, so it might not be the cause of submission error, if you show one.\n[1] https://www.kaggle.com/remekkinas/yolox-inference-on-kaggle-for-cots-lb-0-507\n\nWhy don’t you check if a simple nan only submission pass the evaluation If you want to be sure? I think it will take only 10 minutes or so.", "votes": null }, { "id": "1637563", "postDate": "01/04/2022 02:51:33", "content": "

Yeah that's what my code does as well… I'll put in a string of '0 0 0 0 0' to see if that changes things.

\n

You'll notice that empty string is evaluated to NaN when we put it in a pandas dataframe from a csv, and I think that's where it comes from.

\n

Edit: 0 0 0 0 0 doesn't work either. No idea where the issue is.

", "rawMarkdown": "Yeah that's what my code does as well... I'll put in a string of '0 0 0 0 0' to see if that changes things.\n\nYou'll notice that empty string is evaluated to NaN when we put it in a pandas dataframe from a csv, and I think that's where it comes from.\n\nEdit: 0 0 0 0 0 doesn't work either. No idea where the issue is.", "votes": null }, { "id": "1637598", "postDate": "01/04/2022 04:03:44", "content": "

My basic idea is like this:

\n
    \n
  1. check the infer script runs over train set and verify if the result does’t contain any unusual data.
    2. \n
  2. If you can’t find anything wrong, probing hack like [1] may help to check where your script fails.
    3. \n
\n

Hope you success.

\n

[1] https://www.kaggle.com/c/nfl-health-and-safety-helmet-assignment/discussion/279046

", "rawMarkdown": "My basic idea is like this:\n\n1. check the infer script runs over train set and verify if the result does’t contain any unusual data.\n2. If you can’t find anything wrong, probing hack like [1] may help to check where your script fails.\n\nHope you success.\n\n[1] https://www.kaggle.com/c/nfl-health-and-safety-helmet-assignment/discussion/279046", "votes": null }, { "id": "1637605", "postDate": "01/04/2022 04:12:51", "content": "
    \n
  1. Yeah, maybe more extensive testing is needed…
    2. \n
\n

Would 2. work even for submission scoring error? Because for that case at least a score is coming back in the submission and they're using that to debug instead.

", "rawMarkdown": "1. Yeah, maybe more extensive testing is needed...\n\nWould 2. work even for submission scoring error? Because for that case at least a score is coming back in the submission and they're using that to debug instead.", "votes": null }, { "id": "1637664", "postDate": "01/04/2022 05:30:53", "content": "

(edited)

\n
\n

Because for that case at least a score is coming back in the submission and they're using that to debug instead.

\n
\n

I agree with that. In my understanding, this error usually come upon during evaluation step (but I'm not 100% sure).
\nIf you are confident on your script never fails, I think checking your submission result is easier.

\n

I don't think you need complicated test. In most of the cases, the cause is simple: format error, value error, invalid bbox, length of submissions is inconsistent, etc.

", "rawMarkdown": "(edited)\n\n> Because for that case at least a score is coming back in the submission and they're using that to debug instead.\n\nI agree with that. In my understanding, this error usually come upon during evaluation step (but I'm not 100% sure).\nIf you are confident on your script never fails, I think checking your submission result is easier.\n\nI don't think you need complicated test. In most of the cases, the cause is simple: format error, value error, invalid bbox, length of submissions is inconsistent, etc.", "votes": null }, { "id": "1637851", "postDate": "01/04/2022 08:58:00", "content": "

Hmm alright, thanks!

", "rawMarkdown": "Hmm alright, thanks!", "votes": null }, { "id": "1638408", "postDate": "01/04/2022 17:34:54", "content": "

I would strongly recommend changing your setup so you can run a traditional pipeline. Interacting directly with the submission gets you flagged for a cheat ban review at the end of the competition.

", "rawMarkdown": "I would strongly recommend changing your setup so you can run a traditional pipeline. Interacting directly with the submission gets you flagged for a cheat ban review at the end of the competition.", "votes": null }, { "id": "1638697", "postDate": "01/04/2022 23:54:09", "content": "

Hi. Thank you for commenting.

\n

But I don't get what traditional pipeline and Interacting directly with the submission mean.
\nDoes the latter mean modifying submission.csv directly?

", "rawMarkdown": "Hi. Thank you for commenting.\n\nBut I don't get what `traditional pipeline` and `Interacting directly with the submission` mean.\nDoes the latter mean modifying `submission.csv` directly?", "votes": null }, { "id": "1638706", "postDate": "01/05/2022 00:05:48", "content": "

Yes, I mean directly modifying the submission file.

", "rawMarkdown": "Yes, I mean directly modifying the submission file.", "votes": null }, { "id": "1638718", "postDate": "01/05/2022 00:19:48", "content": "

I see. Thank you for letting me know.

", "rawMarkdown": "I see. Thank you for letting me know.", "votes": null }, { "id": "1638719", "postDate": "01/05/2022 00:25:37", "content": "

By the way, my current configuration is not modifying submission.csv. At the first time I thought submission.csv is just a dummy file because we already have an interface to submit with API.
\nNow I understands it is not correct, I already modified my infer notebook.

", "rawMarkdown": "By the way, my current configuration is not modifying `submission.csv`. At the first time I thought `submission.csv` is just a dummy file because we already have an interface to submit with API.\nNow I understands it is not correct, I already modified my infer notebook.", "votes": null }, { "id": "1638722", "postDate": "01/05/2022 00:37:04", "content": "

Wait, do you mean just moving submission.csv touches the rule?

\n

My inference script have to run under the specific directory, and submission API seems to create submission.csv in the current directory. So I have to move the file to /kaggle/working to correctly scores. I don't touch any of the file contents.
\nDoes Kaggle considers this is against rule?

", "rawMarkdown": "Wait, do you mean just moving `submission.csv` touches the rule?\n\nMy inference script have to run under the specific directory, and submission API seems to create `submission.csv` in the current directory. So I have to move the file to `/kaggle/working` to correctly scores. I don't touch any of the file contents.\nDoes Kaggle considers this is against rule?", "votes": null }, { "id": "1675112", "postDate": "02/04/2022 03:44:11", "content": "

Great information. Thanks.

", "rawMarkdown": "Great information. Thanks.", "votes": null }, { "id": "1675420", "postDate": "02/04/2022 09:00:12", "content": "

Hi, i stand with same error
\ncan you give me advice?

\n
def format_prediction(bboxes):\n    annot = ''\n    bboxes=np.array(bboxes)\n    bboxes=bboxes[bboxes[:,4]>CONF]\n    if len(bboxes)>0:\n        for bb in bboxes:\n            annot += f'{bb[4]:.2f} {int(bb[0])} {int(bb[1])} {int(bb[2])} {int(bb[3])}'\n            annot +=' '\n        annot = annot.strip(' ')\n    return annot\n
\n
for idx, (img, pred_df) in tqdm(enumerate(iter_test)):\n    bb_pred = get_img_predict_multi(models,img, sizes, flips)\n    annot = format_prediction(bb_pred)\n    pred_df['annotations'] = annot\n    env.predict(pred_df)\n
", "rawMarkdown": "Hi, i stand with same error\ncan you give me advice? \n```\ndef format_prediction(bboxes):\n annot = ''\n bboxes=np.array(bboxes)\n bboxes=bboxes[bboxes[:,4]>CONF]\n if len(bboxes)>0:\n for bb in bboxes:\n annot += f'{bb[4]:.2f} {int(bb[0])} {int(bb[1])} {int(bb[2])} {int(bb[3])}'\n annot +=' '\n annot = annot.strip(' ')\n return annot\n```\n\n```\nfor idx, (img, pred_df) in tqdm(enumerate(iter_test)):\n bb_pred = get_img_predict_multi(models,img, sizes, flips)\n annot = format_prediction(bb_pred)\n pred_df['annotations'] = annot\n env.predict(pred_df)\n```", "votes": null }, { "id": "1675510", "postDate": "02/04/2022 10:11:08", "content": "

@nicksergievskiy I’m sorry but this error is too general error, and I can’t say the reason of failure only by reading your submission code.

\n

What I can say generally is just said in the below thread:
\nhttps://www.kaggle.com/c/tensorflow-great-barrier-reef/discussion/297774#1637598

\n

To be short, just make sure your submission is O.K. on your local before submit your code.

", "rawMarkdown": "nicksergievskiy I’m sorry but this error is too general error, and I can’t say the reason of failure only by reading your submission code.\n\nWhat I can say generally is just said in the below thread:\nhttps://www.kaggle.com/c/tensorflow-great-barrier-reef/discussion/297774#1637598\n\nTo be short, just make sure your submission is O.K. on your local before submit your code.", "votes": null }, { "id": "1675535", "postDate": "02/04/2022 10:38:16", "content": "

Thank you answering
\nFor me its ok and looks same as public code, And results formating is ok too

", "rawMarkdown": "Thank you answering\nFor me its ok and looks same as public code, And results formating is ok too", "votes": null }, { "id": "1675543", "postDate": "02/04/2022 10:45:10", "content": "

@tatamikenn
\nCan you look at non runeble code (not open private datases and code)?
\nhttps://www.kaggle.com/nicksergievskiy/cots-ens-yolov5-submission-scoring-error

", "rawMarkdown": "tatamikenn \nCan you look at non runeble code (not open private datases and code)?\nhttps://www.kaggle.com/nicksergievskiy/cots-ens-yolov5-submission-scoring-error", "votes": null }, { "id": "1675570", "postDate": "02/04/2022 11:15:46", "content": "

@nicksergievskiy Sorry, I don't have much time. Hope you find the error.

", "rawMarkdown": "nicksergievskiy Sorry, I don't have much time. Hope you find the error.", "votes": null }, { "id": "1675601", "postDate": "02/04/2022 11:27:17", "content": "

For quick look, your submission code looks O.K. Maybe some edge case in your prediction code might matters.
\ne.g. blank box - bbox==[], nan prediction etc.

", "rawMarkdown": "For quick look, your submission code looks O.K. Maybe some edge case in your prediction code might matters.\ne.g. blank box - `bbox==[]`, `nan` prediction etc.", "votes": null }, { "id": "1675606", "postDate": "02/04/2022 11:31:43", "content": "

For example, your check_result function doesn't work for blank array.
\nQuick fix is replacing blank array to zero-lengh numpy array.

\n
>>> check_result([], (100, 20)) # <-- this does't work\nTraceback (most recent call last):\n  File \"<stdin>\", line 1, in <module>\n  File \"<stdin>\", line 3, in check_result\nIndexError: too many indices for array: array is 1-dimensional, but 2 were indexed\n>>> check_result(np.zeros((0, 5)), (100, 20)) # <-- this works\narray([], shape=(0, 5), dtype=float64)\n
", "rawMarkdown": "For example, your check_result function doesn't work for blank array.\nQuick fix is replacing blank array to zero-lengh numpy array.\n\n```\n>>> check_result([], (100, 20)) # <-- this does't work\nTraceback (most recent call last):\n File \"\", line 1, in \n File \"\", line 3, in check_result\nIndexError: too many indices for array: array is 1-dimensional, but 2 were indexed\n>>> check_result(np.zeros((0, 5)), (100, 20)) # <-- this works\narray([], shape=(0, 5), dtype=float64)\n```", "votes": null }, { "id": "1675620", "postDate": "02/04/2022 11:38:35", "content": "

One more quick advice: make your code as robust as possible.
\nAvoid any possible failure of your code.

\n

e.g.

\n
    \n
  • use try-catch at your suspicious code and gently handle error
    • \n
  • replacing all possible irregular values(nan) into regular value (0) etc.
    • \n
", "rawMarkdown": "One more quick advice: make your code as robust as possible.\nAvoid any possible failure of your code.\n\ne.g.\n\n* use try-catch at your suspicious code and gently handle error\n* replacing all possible irregular values(`nan`) into regular value (0) etc.", "votes": null }, { "id": "1676470", "postDate": "02/05/2022 02:22:17", "content": "

Thank you @tatamikenn
\nboth of your advice is correct!
\nNow with skip errors my code is working

", "rawMarkdown": "Thank you @tatamikenn \nboth of your advice is correct! \nNow with skip errors my code is working", "votes": null } ], "comments": [ { "id": 1637236, "author_name": "hkrsmk", "author_url": "", "post_date": "01/03/2022 17:11:33", "content": "

Do you mind sharing if your submission csv has NaN in it? My team is having difficulty getting it to submit.

", "votes": null, "replies": [ { "id": 1637471, "author_name": "tatamikenn", "author_url": "", "post_date": "01/03/2022 23:14:50", "content": "

@hkrsmk Do you mean the annotation column of submission.csv contains nan or not? In my case, no. My notebook outputs blank string when zero detection.

\n

But some public notebook[1], it seems to contain nan, so it might not be the cause of submission error, if you show one.
\n[1] https://www.kaggle.com/remekkinas/yolox-inference-on-kaggle-for-cots-lb-0-507

\n

Why don’t you check if a simple nan only submission pass the evaluation If you want to be sure? I think it will take only 10 minutes or so.

", "votes": null, "replies": [] }, { "id": 1637563, "author_name": "hkrsmk", "author_url": "", "post_date": "01/04/2022 02:51:33", "content": "

Yeah that's what my code does as well… I'll put in a string of '0 0 0 0 0' to see if that changes things.

\n

You'll notice that empty string is evaluated to NaN when we put it in a pandas dataframe from a csv, and I think that's where it comes from.

\n

Edit: 0 0 0 0 0 doesn't work either. No idea where the issue is.

", "votes": null, "replies": [] }, { "id": 1637598, "author_name": "tatamikenn", "author_url": "", "post_date": "01/04/2022 04:03:44", "content": "

My basic idea is like this:

\n
    \n
  1. check the infer script runs over train set and verify if the result does’t contain any unusual data.
    2. \n
  2. If you can’t find anything wrong, probing hack like [1] may help to check where your script fails.
    3. \n
\n

Hope you success.

\n

[1] https://www.kaggle.com/c/nfl-health-and-safety-helmet-assignment/discussion/279046

", "votes": null, "replies": [] }, { "id": 1637605, "author_name": "hkrsmk", "author_url": "", "post_date": "01/04/2022 04:12:51", "content": "
    \n
  1. Yeah, maybe more extensive testing is needed…
    2. \n
\n

Would 2. work even for submission scoring error? Because for that case at least a score is coming back in the submission and they're using that to debug instead.

", "votes": null, "replies": [] }, { "id": 1637664, "author_name": "tatamikenn", "author_url": "", "post_date": "01/04/2022 05:30:53", "content": "

(edited)

\n
\n

Because for that case at least a score is coming back in the submission and they're using that to debug instead.

\n
\n

I agree with that. In my understanding, this error usually come upon during evaluation step (but I'm not 100% sure).
\nIf you are confident on your script never fails, I think checking your submission result is easier.

\n

I don't think you need complicated test. In most of the cases, the cause is simple: format error, value error, invalid bbox, length of submissions is inconsistent, etc.

", "votes": null, "replies": [] }, { "id": 1637851, "author_name": "hkrsmk", "author_url": "", "post_date": "01/04/2022 08:58:00", "content": "

Hmm alright, thanks!

", "votes": null, "replies": [] }, { "id": 1675112, "author_name": "eugeneryu", "author_url": "", "post_date": "02/04/2022 03:44:11", "content": "

Great information. Thanks.

", "votes": null, "replies": [] } ] }, { "id": 1638408, "author_name": "sohier", "author_url": "", "post_date": "01/04/2022 17:34:54", "content": "

I would strongly recommend changing your setup so you can run a traditional pipeline. Interacting directly with the submission gets you flagged for a cheat ban review at the end of the competition.

", "votes": null, "replies": [ { "id": 1638697, "author_name": "tatamikenn", "author_url": "", "post_date": "01/04/2022 23:54:09", "content": "

Hi. Thank you for commenting.

\n

But I don't get what traditional pipeline and Interacting directly with the submission mean.
\nDoes the latter mean modifying submission.csv directly?

", "votes": null, "replies": [] }, { "id": 1638706, "author_name": "sohier", "author_url": "", "post_date": "01/05/2022 00:05:48", "content": "

Yes, I mean directly modifying the submission file.

", "votes": null, "replies": [] }, { "id": 1638718, "author_name": "tatamikenn", "author_url": "", "post_date": "01/05/2022 00:19:48", "content": "

I see. Thank you for letting me know.

", "votes": null, "replies": [] }, { "id": 1638719, "author_name": "tatamikenn", "author_url": "", "post_date": "01/05/2022 00:25:37", "content": "

By the way, my current configuration is not modifying submission.csv. At the first time I thought submission.csv is just a dummy file because we already have an interface to submit with API.
\nNow I understands it is not correct, I already modified my infer notebook.

", "votes": null, "replies": [] }, { "id": 1638722, "author_name": "tatamikenn", "author_url": "", "post_date": "01/05/2022 00:37:04", "content": "

Wait, do you mean just moving submission.csv touches the rule?

\n

My inference script have to run under the specific directory, and submission API seems to create submission.csv in the current directory. So I have to move the file to /kaggle/working to correctly scores. I don't touch any of the file contents.
\nDoes Kaggle considers this is against rule?

", "votes": null, "replies": [] } ] }, { "id": 1675420, "author_name": "nicksergievskiy", "author_url": "", "post_date": "02/04/2022 09:00:12", "content": "

Hi, i stand with same error
\ncan you give me advice?

\n
def format_prediction(bboxes):\n    annot = ''\n    bboxes=np.array(bboxes)\n    bboxes=bboxes[bboxes[:,4]>CONF]\n    if len(bboxes)>0:\n        for bb in bboxes:\n            annot += f'{bb[4]:.2f} {int(bb[0])} {int(bb[1])} {int(bb[2])} {int(bb[3])}'\n            annot +=' '\n        annot = annot.strip(' ')\n    return annot\n
\n
for idx, (img, pred_df) in tqdm(enumerate(iter_test)):\n    bb_pred = get_img_predict_multi(models,img, sizes, flips)\n    annot = format_prediction(bb_pred)\n    pred_df['annotations'] = annot\n    env.predict(pred_df)\n
", "votes": null, "replies": [ { "id": 1675510, "author_name": "tatamikenn", "author_url": "", "post_date": "02/04/2022 10:11:08", "content": "

@nicksergievskiy I’m sorry but this error is too general error, and I can’t say the reason of failure only by reading your submission code.

\n

What I can say generally is just said in the below thread:
\nhttps://www.kaggle.com/c/tensorflow-great-barrier-reef/discussion/297774#1637598

\n

To be short, just make sure your submission is O.K. on your local before submit your code.

", "votes": null, "replies": [] }, { "id": 1675535, "author_name": "nicksergievskiy", "author_url": "", "post_date": "02/04/2022 10:38:16", "content": "

Thank you answering
\nFor me its ok and looks same as public code, And results formating is ok too

", "votes": null, "replies": [] }, { "id": 1675543, "author_name": "nicksergievskiy", "author_url": "", "post_date": "02/04/2022 10:45:10", "content": "

@tatamikenn
\nCan you look at non runeble code (not open private datases and code)?
\nhttps://www.kaggle.com/nicksergievskiy/cots-ens-yolov5-submission-scoring-error

", "votes": null, "replies": [] }, { "id": 1675570, "author_name": "tatamikenn", "author_url": "", "post_date": "02/04/2022 11:15:46", "content": "

@nicksergievskiy Sorry, I don't have much time. Hope you find the error.

", "votes": null, "replies": [] }, { "id": 1675601, "author_name": "tatamikenn", "author_url": "", "post_date": "02/04/2022 11:27:17", "content": "

For quick look, your submission code looks O.K. Maybe some edge case in your prediction code might matters.
\ne.g. blank box - bbox==[], nan prediction etc.

", "votes": null, "replies": [] }, { "id": 1675606, "author_name": "tatamikenn", "author_url": "", "post_date": "02/04/2022 11:31:43", "content": "

For example, your check_result function doesn't work for blank array.
\nQuick fix is replacing blank array to zero-lengh numpy array.

\n
>>> check_result([], (100, 20)) # <-- this does't work\nTraceback (most recent call last):\n  File \"<stdin>\", line 1, in <module>\n  File \"<stdin>\", line 3, in check_result\nIndexError: too many indices for array: array is 1-dimensional, but 2 were indexed\n>>> check_result(np.zeros((0, 5)), (100, 20)) # <-- this works\narray([], shape=(0, 5), dtype=float64)\n
", "votes": null, "replies": [] }, { "id": 1675620, "author_name": "tatamikenn", "author_url": "", "post_date": "02/04/2022 11:38:35", "content": "

One more quick advice: make your code as robust as possible.
\nAvoid any possible failure of your code.

\n

e.g.

\n
    \n
  • use try-catch at your suspicious code and gently handle error
    • \n
  • replacing all possible irregular values(nan) into regular value (0) etc.
    • \n
", "votes": null, "replies": [] }, { "id": 1676470, "author_name": "nicksergievskiy", "author_url": "", "post_date": "02/05/2022 02:22:17", "content": "

Thank you @tatamikenn
\nboth of your advice is correct!
\nNow with skip errors my code is working

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1632331": "# Note for someone who encounters the same error\n\nI’m sorry but this error is too general error, and I can’t say the reason of failure only by reading your submission code.\n\nWhat I can say generally is just said in the below thread:\nhttps://www.kaggle.com/c/tensorflow-great-barrier-reef/discussion/297774#1637598\n\nTo be short, just make sure your submission is O.K. on your local before submit your code.\nHope you success in detecting error.\n\n----------------\n\n# Original Contents\n\nMaybe almost all of you are not in the case, but, for some reason, I have to call test data providing API from python script like this:\n\n```bash\ncd /tmp/my_project\npython /tmp/infer.py\n```\n\nin this case, `submission.csv` is created in directory `/tmp/my_project`, so the evaluation fails after completed inference over whole test data.\n\nIn this case, don't forget to copy `submission.csv` to `/kaggle/working` like this:\n\n```bash\ncd /tmp/my_project\npython /tmp/infer.py\ncp submission.csv /kaggle/working/.\n```\n\nIt took me some hours to find out this. (At first time, I just copied `sample_submission.csv` as `submission.csv`. Clearly this is wrong. API creates the file `submission.csv`. Don't place the other file.)\n\nI guess it is as usual in In the kernel competition, but the crew why the notebook failed is too few. it is really frustrating.\n\nHope this tips helps someone.\n\nBTW, I share a useful tips for debugging in kernel competition:\nhttps://www.kaggle.com/c/nfl-health-and-safety-helmet-assignment/discussion/279046", "1637236": "Do you mind sharing if your submission csv has NaN in it? My team is having difficulty getting it to submit.", "1637471": "hkrsmk Do you mean the annotation column of submission.csv contains nan or not? In my case, no. My notebook outputs blank string when zero detection.\n\nBut some public notebook[1], it seems to contain nan, so it might not be the cause of submission error, if you show one.\n[1] https://www.kaggle.com/remekkinas/yolox-inference-on-kaggle-for-cots-lb-0-507\n\nWhy don’t you check if a simple nan only submission pass the evaluation If you want to be sure? I think it will take only 10 minutes or so.", "1637563": "Yeah that's what my code does as well... I'll put in a string of '0 0 0 0 0' to see if that changes things.\n\nYou'll notice that empty string is evaluated to NaN when we put it in a pandas dataframe from a csv, and I think that's where it comes from.\n\nEdit: 0 0 0 0 0 doesn't work either. No idea where the issue is.", "1637598": "My basic idea is like this:\n\n1. check the infer script runs over train set and verify if the result does’t contain any unusual data.\n2. If you can’t find anything wrong, probing hack like [1] may help to check where your script fails.\n\nHope you success.\n\n[1] https://www.kaggle.com/c/nfl-health-and-safety-helmet-assignment/discussion/279046", "1637605": "1. Yeah, maybe more extensive testing is needed...\n\nWould 2. work even for submission scoring error? Because for that case at least a score is coming back in the submission and they're using that to debug instead.", "1637664": "(edited)\n\n> Because for that case at least a score is coming back in the submission and they're using that to debug instead.\n\nI agree with that. In my understanding, this error usually come upon during evaluation step (but I'm not 100% sure).\nIf you are confident on your script never fails, I think checking your submission result is easier.\n\nI don't think you need complicated test. In most of the cases, the cause is simple: format error, value error, invalid bbox, length of submissions is inconsistent, etc.", "1637851": "Hmm alright, thanks!", "1638408": "I would strongly recommend changing your setup so you can run a traditional pipeline. Interacting directly with the submission gets you flagged for a cheat ban review at the end of the competition.", "1638697": "Hi. Thank you for commenting.\n\nBut I don't get what `traditional pipeline` and `Interacting directly with the submission` mean.\nDoes the latter mean modifying `submission.csv` directly?", "1638706": "Yes, I mean directly modifying the submission file.", "1638718": "I see. Thank you for letting me know.", "1638719": "By the way, my current configuration is not modifying `submission.csv`. At the first time I thought `submission.csv` is just a dummy file because we already have an interface to submit with API.\nNow I understands it is not correct, I already modified my infer notebook.", "1638722": "Wait, do you mean just moving `submission.csv` touches the rule?\n\nMy inference script have to run under the specific directory, and submission API seems to create `submission.csv` in the current directory. So I have to move the file to `/kaggle/working` to correctly scores. I don't touch any of the file contents.\nDoes Kaggle considers this is against rule?", "1675112": "Great information. Thanks.", "1675420": "Hi, i stand with same error\ncan you give me advice? \n```\ndef format_prediction(bboxes):\n annot = ''\n bboxes=np.array(bboxes)\n bboxes=bboxes[bboxes[:,4]>CONF]\n if len(bboxes)>0:\n for bb in bboxes:\n annot += f'{bb[4]:.2f} {int(bb[0])} {int(bb[1])} {int(bb[2])} {int(bb[3])}'\n annot +=' '\n annot = annot.strip(' ')\n return annot\n```\n\n```\nfor idx, (img, pred_df) in tqdm(enumerate(iter_test)):\n bb_pred = get_img_predict_multi(models,img, sizes, flips)\n annot = format_prediction(bb_pred)\n pred_df['annotations'] = annot\n env.predict(pred_df)\n```", "1675510": "nicksergievskiy I’m sorry but this error is too general error, and I can’t say the reason of failure only by reading your submission code.\n\nWhat I can say generally is just said in the below thread:\nhttps://www.kaggle.com/c/tensorflow-great-barrier-reef/discussion/297774#1637598\n\nTo be short, just make sure your submission is O.K. on your local before submit your code.", "1675535": "Thank you answering\nFor me its ok and looks same as public code, And results formating is ok too", "1675543": "tatamikenn \nCan you look at non runeble code (not open private datases and code)?\nhttps://www.kaggle.com/nicksergievskiy/cots-ens-yolov5-submission-scoring-error", "1675570": "nicksergievskiy Sorry, I don't have much time. Hope you find the error.", "1675601": "For quick look, your submission code looks O.K. Maybe some edge case in your prediction code might matters.\ne.g. blank box - `bbox==[]`, `nan` prediction etc.", "1675606": "For example, your check_result function doesn't work for blank array.\nQuick fix is replacing blank array to zero-lengh numpy array.\n\n```\n>>> check_result([], (100, 20)) # <-- this does't work\nTraceback (most recent call last):\n File \"\", line 1, in \n File \"\", line 3, in check_result\nIndexError: too many indices for array: array is 1-dimensional, but 2 were indexed\n>>> check_result(np.zeros((0, 5)), (100, 20)) # <-- this works\narray([], shape=(0, 5), dtype=float64)\n```", "1675620": "One more quick advice: make your code as robust as possible.\nAvoid any possible failure of your code.\n\ne.g.\n\n* use try-catch at your suspicious code and gently handle error\n* replacing all possible irregular values(`nan`) into regular value (0) etc.", "1676470": "Thank you @tatamikenn \nboth of your advice is correct! \nNow with skip errors my code is working" }, "source": "meta" }