I recently made my first successful submission. Before that I validated my model on local computer and received pretty nice score (above 0.60). However, my submission was scored extremely low (0.003). I expected some natural degradation of results, but this gap seems like a result of some mistake that I made in submission. It seems to me too weird for just over-fitting or under-fitting.
\nI run inference in kaggle on my validation set and it is inline with my local score.
\nWhile I think that I strictly followed the code submission requirements, I hope that the problem is in submission of annotations. My predict function returns the list of bboxes, where each entry contains:
\n[confidence, y, x, height, width]
\nThe submission code for each image is following:
\n`
annotations = \"\"\nbboxes, _ = detect_bboxes(model, pixel_array) # returns list of lists [[confidence, y, x, height, width]\n\nfor bbox in bboxes:\n annotations +=f\" {bbox[0]:.3f} {bbox[2]} {bbox[1]} {bbox[4]} {bbox[3]}\"\n\nsample_prediction_df['annotations'] = annotations\nenv.predict(sample_prediction_df) # register your predictions\n`
\nMy model also doesn't find any starfishes in testing set. Is that Ok, or some starfishes are there and the lack of detection signal that model is actually not good enough?
", "messages": [ { "id": "1657035", "postDate": "01/19/2022 20:47:36", "content": "I recently made my first successful submission. Before that I validated my model on local computer and received pretty nice score (above 0.60). However, my submission was scored extremely low (0.003). I expected some natural degradation of results, but this gap seems like a result of some mistake that I made in submission. It seems to me too weird for just over-fitting or under-fitting.
\nI run inference in kaggle on my validation set and it is inline with my local score.
\nWhile I think that I strictly followed the code submission requirements, I hope that the problem is in submission of annotations. My predict function returns the list of bboxes, where each entry contains:
\n[confidence, y, x, height, width]
\nThe submission code for each image is following:
\n`
annotations = \"\"\nbboxes, _ = detect_bboxes(model, pixel_array) # returns list of lists [[confidence, y, x, height, width]\n\nfor bbox in bboxes:\n annotations +=f\" {bbox[0]:.3f} {bbox[2]} {bbox[1]} {bbox[4]} {bbox[3]}\"\n\nsample_prediction_df['annotations'] = annotations\nenv.predict(sample_prediction_df) # register your predictions\n`
\nMy model also doesn't find any starfishes in testing set. Is that Ok, or some starfishes are there and the lack of detection signal that model is actually not good enough?
", "rawMarkdown": "I recently made my first successful submission. Before that I validated my model on local computer and received pretty nice score (above 0.60). However, my submission was scored extremely low (0.003). I expected some natural degradation of results, but this gap seems like a result of some mistake that I made in submission. It seems to me too weird for just over-fitting or under-fitting.\nI run inference in kaggle on my validation set and it is inline with my local score.\nWhile I think that I strictly followed the code submission requirements, I hope that the problem is in submission of annotations. My predict function returns the list of bboxes, where each entry contains:\n`[confidence, y, x, height, width]`\nThe submission code for each image is following:\n`\n \n annotations = \"\"\n bboxes, _ = detect_bboxes(model, pixel_array) # returns list of lists [[confidence, y, x, height, width]\n\n for bbox in bboxes:\n annotations +=f\" {bbox[0]:.3f} {bbox[2]} {bbox[1]} {bbox[4]} {bbox[3]}\"\n\n sample_prediction_df['annotations'] = annotations\n env.predict(sample_prediction_df) # register your predictions\n`\n\nMy model also doesn't find any starfishes in testing set. Is that Ok, or some starfishes are there and the lack of detection signal that model is actually not good enough?", "votes": null }, { "id": "1657080", "postDate": "01/19/2022 22:10:17", "content": "You can:
\nfor i in range(len(bboxes)):\n …\n predictions.append('{:.2f} {} {} {} {}'.format(score, x_min, y_min, bbox_width, bbox_height)) \n\nprediction_str = ' '.join(predictions)\nsample_prediction_df['annotations'] = prediction_str\nenv.predict(sample_prediction_df)\nCheck if model works on RGB or BGR. Kaggle API provides RGB images. When your model was trained on BGR images (eg. YoloX) your score drops. You have to convert it before prediction.
Check if bbox coordinates are in correct format - COCO (not Yolo - normalized)
I am almost sure that on 3 photos provided by API there is no starfishes (but I am not sure if Kaggle API give us the same photos everytime :) ).
Remek,
\nThanks a lot. It looks like I've occasionally swapped x and y values. The score raised now to 0.203, which is also disappointing, but makes much more sense relative to previous result.
\nIt is also good to get confirmation that test images don't contain starfishes.
\nI very appreciate your answer.
bingo. beside swap x,y, you may also check Remek's suggestion.
", "rawMarkdown": "bingo. beside swap x,y, you may also check Remek's suggestion.", "votes": null } ], "comments": [ { "id": 1657080, "author_name": "remekkinas", "author_url": "", "post_date": "01/19/2022 22:10:17", "content": "You can:
\nfor i in range(len(bboxes)):\n …\n predictions.append('{:.2f} {} {} {} {}'.format(score, x_min, y_min, bbox_width, bbox_height)) \n\nprediction_str = ' '.join(predictions)\nsample_prediction_df['annotations'] = prediction_str\nenv.predict(sample_prediction_df)\nCheck if model works on RGB or BGR. Kaggle API provides RGB images. When your model was trained on BGR images (eg. YoloX) your score drops. You have to convert it before prediction.
Check if bbox coordinates are in correct format - COCO (not Yolo - normalized)
I am almost sure that on 3 photos provided by API there is no starfishes (but I am not sure if Kaggle API give us the same photos everytime :) ).
Remek,
\nThanks a lot. It looks like I've occasionally swapped x and y values. The score raised now to 0.203, which is also disappointing, but makes much more sense relative to previous result.
\nIt is also good to get confirmation that test images don't contain starfishes.
\nI very appreciate your answer.
bingo. beside swap x,y, you may also check Remek's suggestion.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1657035": "I recently made my first successful submission. Before that I validated my model on local computer and received pretty nice score (above 0.60). However, my submission was scored extremely low (0.003). I expected some natural degradation of results, but this gap seems like a result of some mistake that I made in submission. It seems to me too weird for just over-fitting or under-fitting.\nI run inference in kaggle on my validation set and it is inline with my local score.\nWhile I think that I strictly followed the code submission requirements, I hope that the problem is in submission of annotations. My predict function returns the list of bboxes, where each entry contains:\n`[confidence, y, x, height, width]`\nThe submission code for each image is following:\n`\n \n annotations = \"\"\n bboxes, _ = detect_bboxes(model, pixel_array) # returns list of lists [[confidence, y, x, height, width]\n\n for bbox in bboxes:\n annotations +=f\" {bbox[0]:.3f} {bbox[2]} {bbox[1]} {bbox[4]} {bbox[3]}\"\n\n sample_prediction_df['annotations'] = annotations\n env.predict(sample_prediction_df) # register your predictions\n`\n\nMy model also doesn't find any starfishes in testing set. Is that Ok, or some starfishes are there and the lack of detection signal that model is actually not good enough?", "1657080": "You can:\n\n- Try my code:\n\n```python\nfor i in range(len(bboxes)):\n …\n predictions.append('{:.2f} {} {} {} {}'.format(score, x_min, y_min, bbox_width, bbox_height)) \n\nprediction_str = ' '.join(predictions)\nsample_prediction_df['annotations'] = prediction_str\nenv.predict(sample_prediction_df)\n```\n\n- Check if model works on RGB or BGR. Kaggle API provides RGB images. When your model was trained on BGR images (eg. YoloX) your score drops. You have to convert it before prediction.\n\n- Check if bbox coordinates are in correct format - COCO (not Yolo - normalized)\n\n- I am almost sure that on 3 photos provided by API there is no starfishes (but I am not sure if Kaggle API give us the same photos everytime :) ).", "1657086": "Remek,\nThanks a lot. It looks like I've occasionally swapped x and y values. The score raised now to 0.203, which is also disappointing, but makes much more sense relative to previous result.\nIt is also good to get confirmation that test images don't contain starfishes.\nI very appreciate your answer.", "1657472": "bingo. beside swap x,y, you may also check Remek's suggestion." }, "source": "meta" }