Overall I did not make a lot of experiments as I spent just one week for this challenge. I was demotivated by public LB scores, stopped submitting and decided to wait a few weeks for the shakeup.
\nWith my experiments using MMdetection the best performing model was ConvNeXt base CascadeRCNN.
\nValidation by video splits:
\n0.632 public LB / 0.718 private LB
\nDuring training the following modifications were made to standard mmdet training pipeline:
\nRandom Sized Crops around bboxes
\nAs boxes were not dense using simple random crop during training was not optimal and I used random sized crop around bounding box if image is not empty. The crops covered multiple scales (1x-4x) and also allowed to use quite a big batch size during training.
Balanced sampling from empty/nonempty images
\nTo reduce FP predictions I used all images during training with probability to get a non empty/empty crop 0.6/0.4 respectively.
Compared to full image training this approach improved CV by 3% and slightly improved LB scores.
\nBecause random sized crops covered 1x-4x scale range I could safely use high resolution. Originally the images had 3840x2160 scales and were downscaled to 1280x720 for competition purposes.
\nIt was quite safe to assume that 3840x2160 resolution is the best for inference. It also provided the best results on CV/Public/Private splits
Removed false positive predictions (boosted CV by 1-2%, LB by 0.3%) by leaving only big clusters of detections
\ndf = pd.read_csv(\"submission.csv\")\nvals = df.annotations.values\npos = np.array([isinstance(v, str) and len(v) > 2 for v in vals])\npos_dilated = binary_dilation(pos, iterations=1)\npos = remove_small_objects(measure.label(pos_dilated), min_size=10)\nvals[pos == 0] = np.nan\ndf[\"annotations\"] = vals\ndf.to_csv(\"submission.csv\", index=False)\nThe model is quite heavy so no TTA or ensembling, just simple thresholding based on validation.
\nI did not make a lot of submissions and it was a no-brainer to choose the ones with high CV and public LB
\nWas really disappointed to see that I was removed from the leaderboard.
\nNo notification, no reasons, just removed.
Input:
\nOutput:
\nWaiting for the answer from the compliance team and @addisonhoward
", "messages": [ { "id": "1697472", "postDate": "02/19/2022 16:38:38", "content": "Overall I did not make a lot of experiments as I spent just one week for this challenge. I was demotivated by public LB scores, stopped submitting and decided to wait a few weeks for the shakeup.
\nWith my experiments using MMdetection the best performing model was ConvNeXt base CascadeRCNN.
\nValidation by video splits:
\n0.632 public LB / 0.718 private LB
\nDuring training the following modifications were made to standard mmdet training pipeline:
\nRandom Sized Crops around bboxes
\nAs boxes were not dense using simple random crop during training was not optimal and I used random sized crop around bounding box if image is not empty. The crops covered multiple scales (1x-4x) and also allowed to use quite a big batch size during training.
Balanced sampling from empty/nonempty images
\nTo reduce FP predictions I used all images during training with probability to get a non empty/empty crop 0.6/0.4 respectively.
Compared to full image training this approach improved CV by 3% and slightly improved LB scores.
\nBecause random sized crops covered 1x-4x scale range I could safely use high resolution. Originally the images had 3840x2160 scales and were downscaled to 1280x720 for competition purposes.
\nIt was quite safe to assume that 3840x2160 resolution is the best for inference. It also provided the best results on CV/Public/Private splits
Removed false positive predictions (boosted CV by 1-2%, LB by 0.3%) by leaving only big clusters of detections
\ndf = pd.read_csv(\"submission.csv\")\nvals = df.annotations.values\npos = np.array([isinstance(v, str) and len(v) > 2 for v in vals])\npos_dilated = binary_dilation(pos, iterations=1)\npos = remove_small_objects(measure.label(pos_dilated), min_size=10)\nvals[pos == 0] = np.nan\ndf[\"annotations\"] = vals\ndf.to_csv(\"submission.csv\", index=False)\nThe model is quite heavy so no TTA or ensembling, just simple thresholding based on validation.
\nI did not make a lot of submissions and it was a no-brainer to choose the ones with high CV and public LB
\nWas really disappointed to see that I was removed from the leaderboard.
\nNo notification, no reasons, just removed.
Input:
\nOutput:
\nWaiting for the answer from the compliance team and @addisonhoward
", "rawMarkdown": "## Solution Overview\n\nOverall I did not make a lot of experiments as I spent just one week for this challenge. I was demotivated by public LB scores, stopped submitting and decided to wait a few weeks for the shakeup.\n\nWith my experiments using MMdetection the best performing model was ConvNeXt base CascadeRCNN.\n\nValidation by video splits:\n- video 0 - 0.71\n- video 1 - 0.67\n- video 2 - 0.8 (used this fold for submissions)\n\n**0.632** public LB / **0.718** private LB\n\n### Training \nDuring training the following modifications were made to standard mmdet training pipeline:\n\n**Random Sized Crops around bboxes**\nAs boxes were not dense using simple random crop during training was not optimal and I used random sized crop around bounding box if image is not empty. The crops covered multiple scales (1x-4x) and also allowed to use quite a big batch size during training.\n\n**Balanced sampling from empty/nonempty images**\nTo reduce FP predictions I used all images during training with probability to get a non empty/empty crop 0.6/0.4 respectively.\n\nCompared to full image training this approach improved CV by 3% and slightly improved LB scores. \n\n### Inference\nBecause random sized crops covered 1x-4x scale range I could safely use high resolution. Originally the images had 3840x2160 scales and were downscaled to 1280x720 for competition purposes.\nIt was quite safe to assume that 3840x2160 resolution is the best for inference. It also provided the best results on CV/Public/Private splits\n\n### Postprocessing \n\nRemoved false positive predictions (boosted CV by 1-2%, LB by 0.3%) by leaving only big clusters of detections\n```\ndf = pd.read_csv(\"submission.csv\")\nvals = df.annotations.values\npos = np.array([isinstance(v, str) and len(v) > 2 for v in vals])\npos_dilated = binary_dilation(pos, iterations=1)\npos = remove_small_objects(measure.label(pos_dilated), min_size=10)\nvals[pos == 0] = np.nan\ndf[\"annotations\"] = vals\ndf.to_csv(\"submission.csv\", index=False)\n```\n\nThe model is quite heavy so no TTA or ensembling, just simple thresholding based on validation.\n \nI did not make a lot of submissions and it was a no-brainer to choose the ones with high CV and public LB\n\n## Disqualification\nWas really disappointed to see that I was removed from the leaderboard.\nNo notification, no reasons, just removed. \n\nInput:\n- not using anything from public kernels\n- competing solo\n- did not create fake accounts. That would be really a blatant action for an experienced kaggler.\n- did not share anything with other competitors\n\nOutput:\n- **cheater**! \n\nWaiting for the answer from the compliance team and @addisonhoward", "votes": null }, { "id": "1697569", "postDate": "02/19/2022 17:34:38", "content": "Sorry to hear that this happened to you @selimsef!
\nI didn't take part in the competition but read a thread where someone mentioned a LARGE (50+?) Teams got removed this time from the LB? Maybe there were false +ves? I hope they revert the decision for your case.
\nQQ: Do you follow progressive learning or is it just training on a smaller resolution and infering on a larger res?
", "rawMarkdown": "Sorry to hear that this happened to you @selimsef! \n\nI didn't take part in the competition but read a thread where someone mentioned a LARGE (50+?) Teams got removed this time from the LB? Maybe there were false +ves? I hope they revert the decision for your case. \n\nQQ: Do you follow progressive learning or is it just training on a smaller resolution and infering on a larger res?", "votes": null }, { "id": "1697579", "postDate": "02/19/2022 17:40:20", "content": "The resolution is quite high, it is like to resize full images to 1280-5120 range and then make a crop 800x800.
\nFor fully convolutional networks it is often not a problem to train on small crops and make inference on large images, the key here is to keep similar scales on train/inference.
@selimsef My guess you cant touch sumbission.csv, competition is supposed to be real-time data simulation. Feel bad for you thou.
", "rawMarkdown": "selimsef My guess you cant touch sumbission.csv, competition is supposed to be real-time data simulation. Feel bad for you thou.", "votes": null }, { "id": "1697701", "postDate": "02/19/2022 19:39:57", "content": "It sounds as a possibility, but shouldn't in such a case the system show Submission Error already at the time of submission processing?
", "rawMarkdown": "It sounds as a possibility, but shouldn't in such a case the system show Submission Error already at the time of submission processing?", "votes": null }, { "id": "1697715", "postDate": "02/19/2022 19:49:01", "content": "I have another sub without postprocessing, so this doesn't seem to be the case.
\nupd: rechecked kernels code, yeah, unfortunately I've chosen the wrong kernel version.
\nStrange though that it did not fail straight away or before LB finalization.
\nBut that could be an explanation.
If it really is the case it will be quite scary for future competitions - I mean the fact, that you have no idea that your submissions are not valid till it is too late to do anything about it. It would be nice to see the system working better.
", "rawMarkdown": "If it really is the case it will be quite scary for future competitions - I mean the fact, that you have no idea that your submissions are not valid till it is too late to do anything about it. It would be nice to see the system working better.", "votes": null }, { "id": "1697883", "postDate": "02/19/2022 23:47:14", "content": "@selimsef Nice solution, in particular the random crops around bboxes! And sorry to hear about your hopefully only temporal disqualification! Do you think you could share the modification in mmdet / code with respect to the random crops? I am working on a similar problem and it would really help me!
", "rawMarkdown": "selimsef Nice solution, in particular the random crops around bboxes! And sorry to hear about your hopefully only temporal disqualification! Do you think you could share the modification in mmdet / code with respect to the random crops? I am working on a similar problem and it would really help me!", "votes": null }, { "id": "1698080", "postDate": "02/20/2022 05:56:26", "content": "I tried crops in lot of ways
\n1) 440 square crops no resize on original image 720/1280 YoloM- best score 55.6 public LB
\n2) 3200 img size , crops 1480*1856 result were not good .
From albumentation part you can implement something like this code just change rectangles param to bboxes and parse bbox as xmin, ymin, xmax, ymax
", "rawMarkdown": "From albumentation part you can implement something like [this code](https://github.com/selimsef/xview2_solution/blob/5d0caba9c7a9c2707565a189f1a091c86d26b546/augs.py#L160) just change rectangles param to bboxes and parse bbox as xmin, ymin, xmax, ymax", "votes": null }, { "id": "1698893", "postDate": "02/20/2022 18:40:45", "content": "This is interesting.
\nAccording to https://www.kaggle.com/c/tensorflow-great-barrier-reef/overview/evaluation :
\n\n\nYou must submit to this competition using the provided python time-series API, which ensures that models do not peek forward in time.
\n
Now I'm not sure whether your post-processing peeks forward in time, if it does that's probably be the reason why you got removed. However …
\nFrom https://www.kaggle.com/c/tensorflow-great-barrier-reef/rules, using ctrl+f and the word \"submission\", I found these :
\n\n\nYour Submissions must be made in the manner and format, and in compliance with all other requirements, stated on the Competition Website (the \"Requirements\")
\n
And :
\n\n\nYour competition submissions (\"Submissions\") must conform to the requirements stated on the Competition Website.
\n
Now as far as I understand, requirements are mentioned here : https://www.kaggle.com/c/tensorflow-great-barrier-reef/overview/code-requirements - and do not state that you cannot overwrite the submission.csv file. I agree that if overwriting is not allowed, the submission should fail immediately, and I think that's a fairly easy thing to check.
\nAnyways, we need clarification on that.
\n@addisonhoward @philculliton @inversion @sohier - I'm not sure who I should tag.
It was mentioned here that it is not allowed:
\nhttps://www.kaggle.com/c/tensorflow-great-barrier-reef/discussion/297774#1638408
But I agree that it shouldn't be even possible in the first place.
", "rawMarkdown": "It was mentioned here that it is not allowed:\nhttps://www.kaggle.com/c/tensorflow-great-barrier-reef/discussion/297774#1638408\n\nBut I agree that it shouldn't be even possible in the first place.", "votes": null }, { "id": "1698934", "postDate": "02/20/2022 19:00:08", "content": "Thanks for sharing, interesting that you got ConvNext to work well, we didn't have any luck with it and the inference runtime was really slow.
", "rawMarkdown": "Thanks for sharing, interesting that you got ConvNext to work well, we didn't have any luck with it and the inference runtime was really slow.", "votes": null }, { "id": "1698975", "postDate": "02/20/2022 19:49:58", "content": "Nice, thanks! That should work!
", "rawMarkdown": "Nice, thanks! That should work!", "votes": null }, { "id": "1699038", "postDate": "02/20/2022 21:15:43", "content": "@philippsinger Thanks for the info, that makes it clear! As I joined late I missed these discussions and suggested that once submission passed it should work.
\nStill a good achievement for me - was banned the first time 💪😄
It performs really great on other tasks as well. But it is very slow compared to original Yolo models, that's true.
\nFor this dataset one needs to tune loss weights, I looked at absolute loss values and added some multipliers, for RPN head it was something like 100 due to rare boxes.
\nIn general losses for RPN head are overwhelmed by negative samples which leads to bad recall if image is really big like 3k and it has just a single box.
\nPlayed around with loss balancing, hard negative mining, focal loss etc. but did not have any improvements.
\nSo overall this dataset is not very RCNN friendly.
Sorry to hear that this happened to you @selimsef!
\nI didn't take part in the competition but read a thread where someone mentioned a LARGE (50+?) Teams got removed this time from the LB? Maybe there were false +ves? I hope they revert the decision for your case.
\nQQ: Do you follow progressive learning or is it just training on a smaller resolution and infering on a larger res?
", "votes": null, "replies": [ { "id": 1697579, "author_name": "selimsef", "author_url": "", "post_date": "02/19/2022 17:40:20", "content": "The resolution is quite high, it is like to resize full images to 1280-5120 range and then make a crop 800x800.
\nFor fully convolutional networks it is often not a problem to train on small crops and make inference on large images, the key here is to keep similar scales on train/inference.
I tried crops in lot of ways
\n1) 440 square crops no resize on original image 720/1280 YoloM- best score 55.6 public LB
\n2) 3200 img size , crops 1480*1856 result were not good .
@selimsef My guess you cant touch sumbission.csv, competition is supposed to be real-time data simulation. Feel bad for you thou.
", "votes": null, "replies": [ { "id": 1697701, "author_name": "blankaf", "author_url": "", "post_date": "02/19/2022 19:39:57", "content": "It sounds as a possibility, but shouldn't in such a case the system show Submission Error already at the time of submission processing?
", "votes": null, "replies": [] }, { "id": 1697715, "author_name": "selimsef", "author_url": "", "post_date": "02/19/2022 19:49:01", "content": "I have another sub without postprocessing, so this doesn't seem to be the case.
\nupd: rechecked kernels code, yeah, unfortunately I've chosen the wrong kernel version.
\nStrange though that it did not fail straight away or before LB finalization.
\nBut that could be an explanation.
If it really is the case it will be quite scary for future competitions - I mean the fact, that you have no idea that your submissions are not valid till it is too late to do anything about it. It would be nice to see the system working better.
", "votes": null, "replies": [] }, { "id": 1698893, "author_name": "theoviel", "author_url": "", "post_date": "02/20/2022 18:40:45", "content": "This is interesting.
\nAccording to https://www.kaggle.com/c/tensorflow-great-barrier-reef/overview/evaluation :
\n\n\nYou must submit to this competition using the provided python time-series API, which ensures that models do not peek forward in time.
\n
Now I'm not sure whether your post-processing peeks forward in time, if it does that's probably be the reason why you got removed. However …
\nFrom https://www.kaggle.com/c/tensorflow-great-barrier-reef/rules, using ctrl+f and the word \"submission\", I found these :
\n\n\nYour Submissions must be made in the manner and format, and in compliance with all other requirements, stated on the Competition Website (the \"Requirements\")
\n
And :
\n\n\nYour competition submissions (\"Submissions\") must conform to the requirements stated on the Competition Website.
\n
Now as far as I understand, requirements are mentioned here : https://www.kaggle.com/c/tensorflow-great-barrier-reef/overview/code-requirements - and do not state that you cannot overwrite the submission.csv file. I agree that if overwriting is not allowed, the submission should fail immediately, and I think that's a fairly easy thing to check.
\nAnyways, we need clarification on that.
\n@addisonhoward @philculliton @inversion @sohier - I'm not sure who I should tag.
It was mentioned here that it is not allowed:
\nhttps://www.kaggle.com/c/tensorflow-great-barrier-reef/discussion/297774#1638408
But I agree that it shouldn't be even possible in the first place.
", "votes": null, "replies": [] }, { "id": 1699038, "author_name": "selimsef", "author_url": "", "post_date": "02/20/2022 21:15:43", "content": "@philippsinger Thanks for the info, that makes it clear! As I joined late I missed these discussions and suggested that once submission passed it should work.
\nStill a good achievement for me - was banned the first time 💪😄
@selimsef Nice solution, in particular the random crops around bboxes! And sorry to hear about your hopefully only temporal disqualification! Do you think you could share the modification in mmdet / code with respect to the random crops? I am working on a similar problem and it would really help me!
", "votes": null, "replies": [ { "id": 1698502, "author_name": "selimsef", "author_url": "", "post_date": "02/20/2022 12:41:27", "content": "From albumentation part you can implement something like this code just change rectangles param to bboxes and parse bbox as xmin, ymin, xmax, ymax
", "votes": null, "replies": [] }, { "id": 1698975, "author_name": "hannes82", "author_url": "", "post_date": "02/20/2022 19:49:58", "content": "Nice, thanks! That should work!
", "votes": null, "replies": [] } ] }, { "id": 1698934, "author_name": "philippsinger", "author_url": "", "post_date": "02/20/2022 19:00:08", "content": "Thanks for sharing, interesting that you got ConvNext to work well, we didn't have any luck with it and the inference runtime was really slow.
", "votes": null, "replies": [ { "id": 1699039, "author_name": "selimsef", "author_url": "", "post_date": "02/20/2022 21:17:55", "content": "It performs really great on other tasks as well. But it is very slow compared to original Yolo models, that's true.
\nFor this dataset one needs to tune loss weights, I looked at absolute loss values and added some multipliers, for RPN head it was something like 100 due to rare boxes.
\nIn general losses for RPN head are overwhelmed by negative samples which leads to bad recall if image is really big like 3k and it has just a single box.
\nPlayed around with loss balancing, hard negative mining, focal loss etc. but did not have any improvements.
\nSo overall this dataset is not very RCNN friendly.