![\"starfish\"](\"https://i.ibb.co/GWKsrnW/inline-image-preview.jpg\")
Proper training of yolov5 can give decent score, will release train code later.
\nhttps://www.kaggle.com/steamedsheep/yolov5-is-all-you-need π
I release this merely due to that I find modifying several parameters(resolution, vflip and cutmix) then I can get a score looks good. Higher resolution DO benefit small object's detection. you can try classifier like Heng's to boost score further.
\nUpdate:
\nTrain notebook: https://www.kaggle.com/steamedsheep/yolov5-high-resolution-training
\nSince I overestimated the performance of P100 and 2T cpu in kaggle, to reproduce the score, you need change back the size to 3600 and epochs to 15. This setup will yield a 0.33 mAP@.5:.95
Update 2:
\nI successfully kick myself out of the bronze zone, which is not my intentionπ
Proper training of yolov5 can give decent score, will release train code later.
\nhttps://www.kaggle.com/steamedsheep/yolov5-is-all-you-need π
I release this merely due to that I find modifying several parameters(resolution, vflip and cutmix) then I can get a score looks good. Higher resolution DO benefit small object's detection. you can try classifier like Heng's to boost score further.
\nUpdate:
\nTrain notebook: https://www.kaggle.com/steamedsheep/yolov5-high-resolution-training
\nSince I overestimated the performance of P100 and 2T cpu in kaggle, to reproduce the score, you need change back the size to 3600 and epochs to 15. This setup will yield a 0.33 mAP@.5:.95
Update 2:
\nI successfully kick myself out of the bronze zone, which is not my intentionπ
here is the magic, using your same model and same code (i just change the input size)
\npath='../input/reef-baseline-fold12/l6_3600_uflip_vm5_f12_up/f1/best.pt',\n3600 LB 0.579\n4000 (1.11x) LB 0.586\n4800 (1.33x) LB 0.590\n5600 (1.55x) LB 0.594\n6400 (1.77x) LB 0.596\n7200 (2.00x) LB 0.603 !!!!??\n9000 (2.50x) LB 0.613 !!!!???? why it never ends\n...\n...\n# optimun is between 2.50x to 3.50x, please feel free to explore. \n# i estimate we can get to 6.20\n...\n12600(3.50x) LB 0.557 \nbut i haven't used yolo5 yet and i don't know about your train data. etc.
\nso i cannot tell if the results is overfitting (#1) or having good generalisation.
\nhence interprete the results with care!
(#1) overfitting as in the with enlarged image, the model is producing many FP. BUT for this public test video,
\nthe improvement in TP outweighs that of FP. We can't tell for sure in the private set.
we need to calibrate FP of the models under different input setting, etc
\nmost top kagglers have tricks in their bags, accumulated from work and competition experiences.
\nThis is results from hard work after numerous experiments.
Now i let one cat of the bag:
\neven in past competitions, i have been using this upsizing trick.
\nit is interesting to note that when we enlarge input image, there is no addition information.
\nbecause the new pixels are interpolated (i.e. information is from the original pixels).
\nyet, i have been seeing consisently improvement in results, e.g. in classification, segmentation, object detection
Hi, I have a quick question:
\nDo we have to use the YOLO detector all the time?
\nWill other detection models(e.g, Faster RCNN) achieve better results?
it depends on the data (and your augmentation).
\nall datascience problem as three parts:
\nThank you for your prompt reply!
\nI don't think the training data is enough, even though there are 4919 labeled images, they are from videos and have similarities to each other. after reading your previous post, I decided to collect some additional data from the web.
\nThanks again for the advice! I've learned a lot from you!
I wonder if we resize a 720x1280 image into 6000px. the interpolation of so many pixels will reduce the quality of the image. Also, the image is in .jpg format(which is already compressed and some pixels already lost).
\nAm I correct? Such a long interpolation generates some noise pixels, isn't it?
GO GO GO!
\nLet's see the limits.
\n10000 ?
\n20000 ?
Good direction β¦. I see that soon we can copy the best solution from this competition: https://www.kaggle.com/c/sartorius-cell-instance-segmentation πππ
\n
for people with more GPUs
\n![\"s1\"](\"https://i.ibb.co/7pRJK9j/270x430p135x215.jpg\")
besides image upsize, what other test time augment can lead to more TP than FP?
", "rawMarkdown": "besides image upsize, what other test time augment can lead to more TP than FP?", "votes": 1 }, { "id": 1649672, "postDate": "2022-01-14T13:49:57.140Z", "content": "Very good question @hengck23 π
", "rawMarkdown": "Very good question @hengck23 π", "votes": 2 }, { "id": 1649697, "postDate": "2022-01-14T14:19:41.257Z", "content": "@remekkinas I bet it will be an Astro Cell Type :P
", "rawMarkdown": "@remekkinas I bet it will be an Astro Cell Type :P ", "votes": 1 }, { "id": 1649726, "postDate": "2022-01-14T14:45:27.337Z", "content": "@phoenix9032 this prooves that you have spent a lot of time in this competition. You deserve additional prize :) πͺπ€ππ
", "rawMarkdown": "@phoenix9032 this prooves that you have spent a lot of time in this competition. You deserve additional prize :) πͺπ€ππ", "votes": 2 }, { "id": 1649757, "postDate": "2022-01-14T15:07:15.117Z", "content": "@hengck23 You made the LB alive and competitive.
", "rawMarkdown": "@hengck23 You made the LB alive and competitive.", "votes": 1 }, { "id": 1649916, "postDate": "2022-01-14T17:16:17.550Z", "content": "just realize that we have a video and not single image, and hence:
\nvideo resolution or learning better upsizing filter!
(e.g. using consistency loss to distill 5x input smiliar to 1x input)
\nLearning to Resize Images for Computer Vision Tasks
\nhttps://arxiv.org/pdf/2003.08237.pdf
sooner or later we're bound to hit memory runtime limits in Kaggle environment after 9k too. I was able to get my current score using up sizing trick (to 4032) and some other tta inspired from yolo repo and hengck's augmentation for 0.56.
\n\n\nLearning to Resize Images for Computer Vision Tasks
\n
\n https://arxiv.org/pdf/2003.08237.pdf
this is a useful paper imo
\nUPD: looks like up-sizing doesn't work well if initial training size is low so there is indeed a limit for higher res training as well
", "rawMarkdown": "sooner or later we're bound to hit memory runtime limits in Kaggle environment after 9k too. I was able to get my current score using up sizing trick (to 4032) and some other tta inspired from yolo repo and hengck's augmentation for 0.56. \n\n> Learning to Resize Images for Computer Vision Tasks\nhttps://arxiv.org/pdf/2003.08237.pdf\n\nthis is a useful paper imo\n\nUPD: looks like up-sizing [doesn't work well](https://www.kaggle.com/sentrankim/only-yolov5-tracking-lb-52) if initial training size is low so there is indeed a limit for higher res training as well", "votes": 3 }, { "id": 1650319, "postDate": "2022-01-15T00:50:32.927Z", "content": "6800(LB0.601)
\n7600(LB0.610)
10200 (LB 0.602)
\nI guess it end here π
It finally reaches end. Maybe probing confidence threshold make it better.
\n10800(LB 0.602)
I just woke up and see everyone jump on the LB :D
", "rawMarkdown": "I just woke up and see everyone jump on the LB :D " }, { "id": 1650437, "postDate": "2022-01-15T03:38:09.723Z", "content": "It seems that using two stage detectors results in a performace trade off which won't allow for the upsizing of images as effectively⦠however it is always worth experimenting with. My best two stage detector reached 0.578 but currently is a 2 hour runtime.
", "rawMarkdown": "It seems that using two stage detectors results in a performace trade off which won't allow for the upsizing of images as effectively... however it is always worth experimenting with. My best two stage detector reached 0.578 but currently is a 2 hour runtime.", "votes": 1 }, { "id": 1650530, "postDate": "2022-01-15T05:17:19.597Z", "content": "Sorry, I have not foresee this, I should not release this. Apology to the turbulence of leaderboard
", "rawMarkdown": "Sorry, I have not foresee this, I should not release this. Apology to the turbulence of leaderboard", "votes": 6 }, { "id": 1650549, "postDate": "2022-01-15T05:36:50.510Z", "content": "My understanding is that the private LB will be run with 75% of the test dataset? In that case, any notebook that takes > 3 hours to complete will breach the 9 hr limit on the private LB?
", "rawMarkdown": "My understanding is that the private LB will be run with 75% of the test dataset? In that case, any notebook that takes > 3 hours to complete will breach the 9 hr limit on the private LB?", "votes": 1 }, { "id": 1650577, "postDate": "2022-01-15T06:12:10.853Z", "content": "Nope, it is already counted but hidden don't worry :)
", "rawMarkdown": "Nope, it is already counted but hidden don't worry :)", "votes": 3 }, { "id": 1650798, "postDate": "2022-01-15T10:32:35.233Z", "content": "Thanks, that's good to know. Does that mean any notebook that runs to completion can be counted for the final / private LB?
\n9 hrs is a ridiculous amount of time.
", "rawMarkdown": "Thanks, that's good to know. Does that mean any notebook that runs to completion can be counted for the final / private LB?\n\n9 hrs is a ridiculous amount of time.", "votes": 1 }, { "id": 1650882, "postDate": "2022-01-15T11:45:42.123Z", "content": "@steamedsheep actually this is an important contribution since even for public notebooks or well known frameworks, it is our job as participants to find the best hyperparams.
\nHowever if in the final solution we were to see a few gold/silver medals won solely due to a single parameter change we'll feel a bit defeated and maybe even call that participant's hard work luck or \"easy-work\". Your contribution helped normalise this parameter tuning discrepancy and now we can move on to implementing techniques on top of these hyperparams :)
", "rawMarkdown": "@steamedsheep actually this is an important contribution since even for public notebooks or well known frameworks, it is our job as participants to find the best hyperparams. \n\nHowever if in the final solution we were to see a few gold/silver medals won solely due to a single parameter change we'll feel a bit defeated and maybe even call that participant's hard work luck or \"easy-work\". Your contribution helped normalise this parameter tuning discrepancy and now we can move on to implementing techniques on top of these hyperparams :)", "votes": 3 }, { "id": 1650885, "postDate": "2022-01-15T11:51:58.240Z", "content": "@alexchwong yup you will have an option to select 4 notebooks at the end of the competition out of which best score on private holdout(which has already been calculated) will determine your private lb rank (you can select them now too but you'll probably get 4 better scores in the next month)
", "rawMarkdown": "@alexchwong yup you will have an option to select 4 notebooks at the end of the competition out of which best score on private holdout(which has already been calculated) will determine your private lb rank (you can select them now too but you'll probably get 4 better scores in the next month)", "votes": 1 }, { "id": 1650898, "postDate": "2022-01-15T12:01:56.743Z", "content": "great, kind of fractal enlargement.
", "rawMarkdown": "great, kind of fractal enlargement. " }, { "id": 1651176, "postDate": "2022-01-15T15:49:58.533Z", "content": "@alexchwong Please refer to this discussion
\nhttps://www.kaggle.com/c/tensorflow-great-barrier-reef/discussion/301020
Thanks guys thats very helpful
", "rawMarkdown": "Thanks guys thats very helpful" } ] }, { "id": 2506668, "postDate": "2023-10-31T13:18:15.977Z", "content": "thanks a lot, sheep big old and frog god
", "rawMarkdown": "thanks a lot, sheep big old and frog god", "votes": 1, "replies": [ { "id": 2558274, "postDate": "2023-12-12T03:28:53.543Z", "content": "you are so cool,my god
", "rawMarkdown": "you are so cool,my god" } ] }, { "id": 1656668, "postDate": "2022-01-19T14:32:45.487Z", "content": "i think i have solved the puzzle
\n![\"https://i.ibb.co/FHQsG3J/Selection-005.png\"](\"https://i.ibb.co/FHQsG3J/Selection-005.png\")
the safest bet is to use public LB to measure your recall only. I think there is insufficient images to measure FPrate and precision. Hence use another dataset for that
", "rawMarkdown": "i think i have solved the puzzle\n\n\n\nthe safest bet is to use public LB to measure your recall only. I think there is insufficient images to measure FPrate and precision. Hence use another dataset for that\n", "votes": 23, "replies": [ { "id": 1656688, "postDate": "2022-01-19T14:49:02.497Z", "content": "I am starting to think your tactic is to confuse everybody @hengck23 , you throwed so many ideas maybe with intention that somebody will test it. But your graph now proves that IMG_SIZE is not all you need :)
", "rawMarkdown": "I am starting to think your tactic is to confuse everybody @hengck23 , you throwed so many ideas maybe with intention that somebody will test it. But your graph now proves that IMG_SIZE is not all you need :)", "votes": 3 }, { "id": 1656701, "postDate": "2022-01-19T14:57:24.297Z", "content": "large image size improve recall. but if you visually inspect the results, you do see many FP.
\nif you measure local validation, there is improvement in the local cv f2 score but not to the extent of 2.5x times. (I am referring to the results on my experiment on sheep models, i.e train at 3600, inference at much higher resolution)
hence I conclude that the number of empty images in your local cv and public test set are different.
\nthere is clearly a huge difference in trend on local cv and public lb score.
if there is too many empty images in the private set, the gain in recall will be eaten away by too many FPs and you will have low f2 score. this will cause shakeup.
\nyou have to measure your fp rate of your solution (and keep it low enough). i don't think the public lb score reflect this well enough, due to the lack of sufficient images.
\nin short:
\n\n\nyour tactic is to confuse everybody
\n
I think @hengck23 is well-intentioned. All these are observations based on experiments, nothing to confuse others on purpose.
", "rawMarkdown": "> your tactic is to confuse everybody\n\nI think @hengck23 is well-intentioned. All these are observations based on experiments, nothing to confuse others on purpose.", "votes": 4 }, { "id": 1656709, "postDate": "2022-01-19T15:02:53.760Z", "content": "yes this is what we do everytime test with full fold (with empties also) and there is border where FP are less. Doesn't work :)
\n![\"\"](\"https://i.ibb.co/jLmCPf1/F2-curve.png\")
Our models got high confidence from 0.4 to 0.6 removing FP but as you see its not the case this model got LB 0.59
", "rawMarkdown": "yes this is what we do everytime test with full fold (with empties also) and there is border where FP are less. Doesn't work :)\n\n\n\n\nOur models got high confidence from 0.4 to 0.6 removing FP but as you see its not the case this model got LB 0.59", "votes": 3 }, { "id": 1656713, "postDate": "2022-01-19T15:04:51.870Z", "content": "@zzy990106 to bo accurate i said i am starting to think, i didnt claimed @hengck23 is confusing everybody
", "rawMarkdown": "@zzy990106 to bo accurate i said i am starting to think, i didnt claimed @hengck23 is confusing everybody", "votes": 1 }, { "id": 1656717, "postDate": "2022-01-19T15:11:04.437Z", "content": "You gave some hypotheses, but you have no evidence to prove their correctness. Usually people show a hypothesis and then evidence to prove it right, eg increase in the result of their model, score , etc. Here are theses, but only theses and no facts β¦ This picture just show that β¦. rescaling is finally bad way of winning (you will be punished in priv LB). Now I can see many people who took this way β¦ \"oh no β¦. I will be punished β¦.\".
\nI am giving one example of an experiment carried out on a selected fold (we conducted this experiment on many different configuration). Which model do you think is significantly better on the LB?
\n![\"Grafika\"](\"https://i.ibb.co/Q8PGsyC/exp1.jpg\")
the issue now is that we don't know how many truth objects and how many empty images are there in the public test set. my experiments show that when there are insufficient images (and truth objects) the f2 score varies quite greatly.
\nin order to avoid shakeup, our model must perform well against various combinations of number of objects and empty images. (we need to plot f2 curve for different combinations)
\nusing larger inference image obviously don't reduce FP, so the gain in public LB score must come from the recall. a stable results must come from both improving recall and precision. i think the public LB score only reflects recall. so one has to be careful to interpret the results to avoid shakeup.
", "rawMarkdown": "the issue now is that we don't know how many truth objects and how many empty images are there in the public test set. my experiments show that when there are insufficient images (and truth objects) the f2 score varies quite greatly.\n\nin order to avoid shakeup, our model must perform well against various combinations of number of objects and empty images. (we need to plot f2 curve for different combinations)\n\nusing larger inference image obviously don't reduce FP, so the gain in public LB score must come from the recall. a stable results must come from both improving recall and precision. i think the public LB score only reflects recall. so one has to be careful to interpret the results to avoid shakeup.\n\n\n\n\n", "votes": 5 }, { "id": 1656726, "postDate": "2022-01-19T15:21:23.730Z", "content": "@hengck23 you think why i implemented f2 curve and why Remek made TP FP FN analysis ?:) we doing it from long time. Did you see our submission number ? we test it with many ways, and got many ways still to test
", "rawMarkdown": "@hengck23 you think why i implemented f2 curve and why Remek made TP FP FN analysis ?:) we doing it from long time. Did you see our submission number ? we test it with many ways, and got many ways still to test" }, { "id": 1656729, "postDate": "2022-01-19T15:24:42.930Z", "content": "\"experiment carried out on a selected fold\"
\nthe fold is fixed, so I would interpret the results with care.
\nalso, public data is also fixed.
besides FP,FN,FP I would try also different num of empty images and no. of true objects in the table.
\ni want to know how sensitive the metrics are when the dataset changes.
", "rawMarkdown": "\"experiment carried out on a selected fold\"\n\nthe fold is fixed, so I would interpret the results with care.\nalso, public data is also fixed.\n\nbesides FP,FN,FP I would try also different num of empty images and no. of true objects in the table.\n\ni want to know how sensitive the metrics are when the dataset changes.", "votes": 3 }, { "id": 1656732, "postDate": "2022-01-19T15:27:53.947Z", "content": "\" This picture just show that β¦. rescaling is finally bad way of winning (you will be punished in priv LB). Now I can see many people who took this way β¦ \"oh no β¦. I will be punished β¦.\".\"
\nthere will be no problem if you can keep your curve straight (instead of falling down)
", "rawMarkdown": "\" This picture just show that β¦. rescaling is finally bad way of winning (you will be punished in priv LB). Now I can see many people who took this way β¦ \"oh no β¦. I will be punished β¦.\".\"\n\nthere will be no problem if you can keep your curve straight (instead of falling down)", "votes": 2 }, { "id": 1656734, "postDate": "2022-01-19T15:29:28.357Z", "content": "I have once trained a model. don't know if it cause because of my split. it was one of my initial experiments.
\ni trained a model which had a great recall and inferred it. Give me a bad score like 0.382 or something.
\nMy hypothesis is that, both public and private have the same density percentage of nonempty and empty images.
\nperhaps like
frame 1 -> Empty => private\nframe 2 -> Empty => private\nframe 3 -> Empty => private\nframe 4 -> Empty => private\nframe 5 -> Empty => public\nframe 6 -> Empty => public\nframe 7 -> Empty => public\nframe 8 ... frame m -> Empty => public\nframe m+1 -> Non Empty => private\nframe m+2 -> Non Empty => private\nframe m+3 -> Non Empty => private\nframe m+4 -> Non Empty => private\nframe m+5 -> Non Empty => public\nframe m+6 .. frame n ->Non Empty => public\nand so on\nThe reason why I say they into split first 4 consecutive frames is because why tracking working
", "rawMarkdown": "I have once trained a model. don't know if it cause because of my split. it was one of my initial experiments. \ni trained a model which had a great recall and inferred it. Give me a bad score like 0.382 or something.\nMy hypothesis is that, both public and private have the same density percentage of nonempty and empty images.\nperhaps like \n```\nframe 1 -> Empty => private\nframe 2 -> Empty => private\nframe 3 -> Empty => private\nframe 4 -> Empty => private\nframe 5 -> Empty => public\nframe 6 -> Empty => public\nframe 7 -> Empty => public\nframe 8 ... frame m -> Empty => public\nframe m+1 -> Non Empty => private\nframe m+2 -> Non Empty => private\nframe m+3 -> Non Empty => private\nframe m+4 -> Non Empty => private\nframe m+5 -> Non Empty => public\nframe m+6 .. frame n ->Non Empty => public\nand so on\n```\nThe reason why I say they into split first 4 consecutive frames is because why tracking working", "votes": 2 }, { "id": 1658038, "postDate": "2022-01-20T16:56:01.790Z", "content": "I think we are simply inferring sequentially on frames but public LB is a 'random' (or not) select of frames afterwards
", "rawMarkdown": "I think we are simply inferring sequentially on frames but public LB is a 'random' (or not) select of frames afterwards" }, { "id": 1658425, "postDate": "2022-01-21T02:55:42.767Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1658430, "postDate": "2022-01-21T02:58:50.103Z", "content": "\n\nusing larger inference image obviously don't reduce FP, so the gain in public LB score must come from the recall.
\n
My experiments supports this hypothesis for the case of my model.
\nI estimated recall and precision in the Leader board [1], and have result below.
\nIn this result, recall is higher when we infer larger scale than that in train, whereas precision is nearly the same.
\nOf course we need more sample because it depends on the model.
\nI don't confirm this result is generally applicable.
train scale: x2.00, infer scale: x2.00
\n R P\nmean: 0.557 0.534\nstd: 0.028 0.106\ntrain scale: x2.00, infer scale: x3.20 (x1.60 larger than train scale)
\n R P\nmean: 0.623 0.527\nstd: 0.007 0.020\n[1] https://www.kaggle.com/c/tensorflow-great-barrier-reef/discussion/302130#1658385
", "rawMarkdown": "> using larger inference image obviously don't reduce FP, so the gain in public LB score must come from the recall.\n\nMy experiments supports this hypothesis for the case of my model.\nI estimated recall and precision in the Leader board [1], and have result below.\nIn this result, recall is higher when we infer larger scale than that in train, whereas precision is nearly the same.\nOf course we need more sample because it depends on the model.\nI don't confirm this result is generally applicable.\n\ntrain scale: x2.00, infer scale: x2.00\n```\n R P\nmean: 0.557 0.534\nstd: 0.028 0.106\n```\n\ntrain scale: x2.00, infer scale: x3.20 (x1.60 larger than train scale)\n```\n R P\nmean: 0.623 0.527\nstd: 0.007 0.020\n```\n\n[1] https://www.kaggle.com/c/tensorflow-great-barrier-reef/discussion/302130#1658385", "votes": 5 } ] }, { "id": 1648542, "postDate": "2022-01-13T13:41:07.850Z", "content": "I should start with competitions one month before deadline, apparently it is the time people release good solutions.
", "rawMarkdown": "I should start with competitions one month before deadline, apparently it is the time people release good solutions.", "votes": 18, "replies": [ { "id": 1648598, "postDate": "2022-01-13T14:42:13.867Z", "content": "This is not a good solution, just a custom run of yolo5.
", "rawMarkdown": "This is not a good solution, just a custom run of yolo5. ", "votes": 4 }, { "id": 1649389, "postDate": "2022-01-14T08:40:20.480Z", "content": "@steamedsheep this is a brilliant notebook. After trying 100 methods, finally found one secret of this competition (although it is no longer a secret).
", "rawMarkdown": "@steamedsheep this is a brilliant notebook. After trying 100 methods, finally found one secret of this competition (although it is no longer a secret).", "votes": 1 }, { "id": 1649758, "postDate": "2022-01-14T15:08:06.843Z", "content": "\nThat's why I am entering nowβ¦
", "rawMarkdown": "@philippsinger \n\nThat's why I am entering now...", "votes": 4 }, { "id": 1649761, "postDate": "2022-01-14T15:11:49.023Z", "content": "good idea @cpmpml - the public lb is now as high as we got after 2 months of fighting hard
", "rawMarkdown": "good idea @cpmpml - the public lb is now as high as we got after 2 months of fighting hard", "votes": 5 }, { "id": 1649766, "postDate": "2022-01-14T15:16:26.523Z", "content": "To tell the truth I didn't plan to spend so much time on santa comp. I would have joined earlier otherwise.
", "rawMarkdown": "To tell the truth I didn't plan to spend so much time on santa comp. I would have joined earlier otherwise." }, { "id": 1649781, "postDate": "2022-01-14T15:24:27.730Z", "content": "New NLP one might be an alternative, more time :P
", "rawMarkdown": "New NLP one might be an alternative, more time :P", "votes": 1 }, { "id": 1649918, "postDate": "2022-01-14T17:17:16.703Z", "content": "The one Abishek just ruined?
", "rawMarkdown": "The one Abishek just ruined?", "votes": 8 }, { "id": 1649924, "postDate": "2022-01-14T17:21:36.427Z", "content": "Funny coincidence
", "rawMarkdown": "Funny coincidence", "votes": 1 }, { "id": 1649925, "postDate": "2022-01-14T17:22:56.863Z", "content": "You Only Longformer Twice
", "rawMarkdown": "You Only Longformer Twice" } ] }, { "id": 1657271, "postDate": "2022-01-20T04:23:20.987Z", "content": "@hengck23 I did some testing on the notebook after finding out exactly what augment=True does. According to the yolov5 code, it scales images at 1x, 0.83x (also flips horizontally), and 0.67x. Inferencing at higher resolutions can still pick up bboxes from previous and lower resolutions. Here is some data that might help with augment=False at the same resolutions you tested.
\nResoluton With Aug. W/O Aug.\n3600 (1.00x) LB 0.579 | LB 0.577\n4000 (1.11x) LB 0.586 | LB 0.583\n4800 (1.33x) LB 0.590 | LB 0.575\n5600 (1.55x) LB 0.594 | LB 0.586\n6400 (1.77x) LB 0.596 | LB 0.597\n7200 (2.00x) LB 0.603 | LB 0.610\n9000 (2.50x) LB 0.613 | LB 0.584\n12600(3.50x) LB 0.557 | ????\nCould it be that the model learned the sizes of small, medium, and large cots, was able to find more \"smaller\" ones with higher resolutions? The NMS looks like it helps but it there's still a lot of error. Some F2 testing is needed on the last video for more confirmation.
\nOne thing this does show: Augment is not a big factor in the public leaderboard score and maybe with a bigger model (Large or medium), we can inference with augment=False for 2/3 reduction in submission time. Hopefully, this helps!
", "rawMarkdown": "@hengck23 I did some testing on the notebook after finding out exactly what augment=True does. According to the yolov5 code, it scales images at 1x, 0.83x (also flips horizontally), and 0.67x. Inferencing at higher resolutions can still pick up bboxes from previous and lower resolutions. Here is some data that might help with augment=False at the same resolutions you tested.\n\n```\nResoluton With Aug. W/O Aug.\n3600 (1.00x) LB 0.579 | LB 0.577\n4000 (1.11x) LB 0.586 | LB 0.583\n4800 (1.33x) LB 0.590 | LB 0.575\n5600 (1.55x) LB 0.594 | LB 0.586\n6400 (1.77x) LB 0.596 | LB 0.597\n7200 (2.00x) LB 0.603 | LB 0.610\n9000 (2.50x) LB 0.613 | LB 0.584\n12600(3.50x) LB 0.557 | ????\n```\n\nCould it be that the model learned the sizes of small, medium, and large cots, was able to find more \"smaller\" ones with higher resolutions? The NMS looks like it helps but it there's still a lot of error. Some F2 testing is needed on the last video for more confirmation.\n\nOne thing this does show: Augment is not a big factor in the public leaderboard score and maybe with a bigger model (Large or medium), we can inference with augment=False for 2/3 reduction in submission time. Hopefully, this helps!", "votes": 13, "replies": [ { "id": 1657285, "postDate": "2022-01-20T04:40:02.673Z", "content": "@outwrest this is insightful. Have you noted exact time difference?
", "rawMarkdown": "@outwrest this is insightful. Have you noted exact time difference?" }, { "id": 1657313, "postDate": "2022-01-20T05:07:07.573Z", "content": "@sanchitvj Iβm not sure exactly, I know the 9000px one ran in under 3h (or just about, not 100% sure). The time should scale with resolution but with Augment=True you are basically running an inference 3 times on every image (albeit at a smaller resolution).
", "rawMarkdown": "@sanchitvj Iβm not sure exactly, I know the 9000px one ran in under 3h (or just about, not 100% sure). The time should scale with resolution but with Augment=True you are basically running an inference 3 times on every image (albeit at a smaller resolution)." }, { "id": 1657915, "postDate": "2022-01-20T15:24:08.423Z", "content": "Model which gave LB of 0.648 is giving LB of 0.384 at incresed image size and No TTA. Inference time is decreased by ~2.5 hrs but poor LB.
", "rawMarkdown": "Model which gave LB of 0.648 is giving LB of 0.384 at incresed image size and No TTA. Inference time is decreased by ~2.5 hrs but poor LB. " }, { "id": 1657942, "postDate": "2022-01-20T15:44:25.530Z", "content": "I haven't tried it on one of the models, I'll try it out later but it looks like the 2.5x sweet spot from the last model is causing too many abnormally small predictions and maybe the NMS was getting rid of a lot of them when using TTA at smaller resolutions? I'm not exactly sure if it helped, but I forgot to mention that I tried to mitigate that by adding this to the inference function:
\narea = (int(row.xmax) - int(row.xmin)) * (int(row.ymax) - int(row.ymin))\nif row.confidence > 0.15 and area > 250:\n@sanchitvj I don't think this can work on everything but I am going to try to experiment with other augmentations on the yolov5 function here:
\nhttps://github.com/ultralytics/yolov5/blob/9708cf56eaead29bce789a07cfe73ecc7d7d4838/models/yolo.py#L128
\nMaybe inferencing at the same high resolution is good with other TTA's.
In my experiment, inference with 2x resolution boosted LB score, but CV got worse.
\nTraining: 2560 (train: video0&2, valid: video1)
\nTest inference: 2560β5120
\nLB: 0.486->0.540
\nCV: 0.533->0.473
\nHow about your CV? Is it improved??
according to the cots dataset paper
\n\"We set the GoPro cameras to record videos continuously at 24 frames per second at 3840x2160 resolution and manually removed the periods of no activity between transects\"
\nit would be better if kaggle has released the original high resolution image
", "rawMarkdown": "according to the cots dataset paper \n\n\"We set the GoPro cameras to record videos continuously at 24 frames per second at 3840x2160 resolution and manually removed the periods of no activity between transects\"\n\nit would be better if kaggle has released the original high resolution image", "votes": 12, "replies": [ { "id": 1648603, "postDate": "2022-01-13T14:45:27.463Z", "content": "Yeah, I guess host worry that if they release original image this will become a hardware war. Not every guy here have tons of A100s and A6000s
", "rawMarkdown": "Yeah, I guess host worry that if they release original image this will become a hardware war. Not every guy here have tons of A100s and A6000s", "votes": 15 }, { "id": 1648641, "postDate": "2022-01-13T15:09:52.130Z", "content": "They probably got good solution, but they throw it on kaggle to find new path to implement on original reso :)
", "rawMarkdown": "They probably got good solution, but they throw it on kaggle to find new path to implement on original reso :)" }, { "id": 1649076, "postDate": "2022-01-14T00:15:47.073Z", "content": "the application is robotics. for good results (e.g. near zero FP), algorithm should fuse with the path planning or sensor of the robots. as an example:
\nrobotic approach is just to move the robot closer and take a bigger picture of the suspected object
\nrobotic approach is just to increase camera gain to take better picture.
", "rawMarkdown": "the application is robotics. for good results (e.g. near zero FP), algorithm should fuse with the path planning or sensor of the robots. as an example:\n\n1. for far objects, it is small and low resolution.\nagorithm apporach to improve results is to think of better model or increase resolution or improve quality of the images\n\nrobotic approach is just to move the robot closer and take a bigger picture of the suspected object \n\n2. as another exmple, if the environment is dark, object is not clear\nagain, agorithm apporach tries to improve model, etc\n\nrobotic approach is just to increase camera gain to take better picture.", "votes": 3 }, { "id": 1649123, "postDate": "2022-01-14T01:27:24.150Z", "content": "Hahahaβ¦ You have started this hardware war.π
", "rawMarkdown": "Hahaha... You have started this hardware war.π", "votes": 3 } ] }, { "id": 1651637, "postDate": "2022-01-15T23:09:24.283Z", "content": "Does this work on CV as well? I trained a YoloV5m with image size 3000:
\nCV= 0.428
\nLB=0.652 (inference imsize: 6000)
I had a bug in my F2 implementation, the new score:
\nCV=0.528 (inference imsize: 3000)
nice! is this CV score F2? with empty images as well maybe?
", "rawMarkdown": "nice! is this CV score F2? with empty images as well maybe? ", "votes": 1 }, { "id": 1653001, "postDate": "2022-01-17T06:55:52.593Z", "content": "@amin how many epochs you trained ?
", "rawMarkdown": "@amin how many epochs you trained ? ", "votes": 1 }, { "id": 1653296, "postDate": "2022-01-17T12:19:53.097Z", "content": "@imeintanis Yes it's F2 score, I am using all the images with the subsequence split from this notebook, csv file train-0.2.csv
\nThe F2 score I report was calculated only at the 0.5 thresh. I retrained Yolov5l6 and calculated F2 over the thresholds 0.3-0.8 and the new CV=0.53 (inf size=3000) LB=0.624 (inf size=6000).
@seshurajup For 15 epochs (best epoch: 8), I tried to replicate the pipeline shared by sheep to compare with the benchmark.
", "rawMarkdown": "@imeintanis Yes it's F2 score, I am using all the images with the subsequence split from [this notebook](https://www.kaggle.com/julian3833/reef-a-cv-strategy-subsequences), csv file train-0.2.csv\nThe F2 score I report was calculated only at the 0.5 thresh. I retrained Yolov5l6 and calculated F2 over the thresholds 0.3-0.8 and the new CV=0.53 (inf size=3000) LB=0.624 (inf size=6000).\n\n@seshurajup For 15 epochs (best epoch: 8), I tried to replicate the pipeline shared by sheep to compare with the benchmark.", "votes": 4 }, { "id": 1653465, "postDate": "2022-01-17T14:54:22.463Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1653476, "postDate": "2022-01-17T15:03:41.250Z", "content": "@amiiiney After 6-7 epochs mAP starts to saturates. Can you share which optimizer you are using and also if you using image shear and rotation with Yolov5m?
", "rawMarkdown": "@amiiiney After 6-7 epochs mAP starts to saturates. Can you share which optimizer you are using and also if you using image shear and rotation with Yolov5m?", "votes": 1 }, { "id": 1653502, "postDate": "2022-01-17T15:37:30.580Z", "content": "I am using SGD with lr=0.01 the defaults parameters. It seems that with this configuration there is no need to train for more than 8 epochs. And no, I am not using rotation/shear
", "rawMarkdown": "I am using SGD with lr=0.01 the defaults parameters. It seems that with this configuration there is no need to train for more than 8 epochs. And no, I am not using rotation/shear", "votes": 1 }, { "id": 1656751, "postDate": "2022-01-19T15:48:56.553Z", "content": "I've done the same as you said, but always failed at epoch 3, things didn't go wellπ
", "rawMarkdown": "I've done the same as you said, but always failed at epoch 3, things didn't go wellπ ", "votes": 1 }, { "id": 1658096, "postDate": "2022-01-20T18:23:59.100Z", "content": "@amiiiney which kernel you used to measure CV ? since i am getting CV as 0.522 for same split
", "rawMarkdown": "@amiiiney which kernel you used to measure CV ? since i am getting CV as 0.522 for same split ", "votes": 1 }, { "id": 1658343, "postDate": "2022-01-21T00:28:57.960Z", "content": "@w3579628328 It seems that many people couldn't reproduce sheep's notebook either, I am not sure why is that :)
\n@seshurajup I am currently getting the same CV as you for inference size 3000. By CV I mean the F2 validation score per epoch, I calculate F2 by adding this line to the ap_per_class function in utils/metrics.py
\nf2 = 5 * p * r / (4 * p + r + 1e-16)
\nAnd then pass it through the fitness function to compute F2 over the IoU 0.3-0.8 thresholds.
\n![\"\"](\"https://i.ibb.co/yk2f1p5/Screen-Shot-2022-01-21-at-01-21-27.png\")
@Amin Can you please explain what you mean by the fitness function?
\nThis is all I see in metrics.py for the fitness function:
\ndef fitness(x):
\n # Model fitness as a weighted combination of metrics
\n w = [0.0, 0.0, 0.1, 0.9] # weights for [P, R, mAP@0.5, mAP@0.5:0.95]
\n return (x[:, :4] * w).sum(1)
@ayu055 so the first step is to add the F2 line of code to ap_per_class function and return the F2 as well:
\nreturn tp, fp, p, r, f2, ap, unique_classes.astype(\"int32\")
STEP 2: Go to val.py and modify these lines by adding f2:
\ntp, fp, p, r, f2, ap, ap_class = ap_per_class(*stats, plot=plots, save_dir=save_dir, names=names)\nap50, ap, f2 = ap[:, 0], ap.mean(1), , f2.mean(1) # AP@0.5, AP@0.5:0.95, f2@0.3:0.8\nmp, mr, f2, map50, map = p.mean(), r.mean(), f2.mean(), ap50.mean(), ap.mean()\nAlso change this line:
\niouv = torch.linspace(0.5, 0.95, 10).to(device) # iou vector for mAP@0.5:0.95\nto
\niouv= torch.from_numpy(np.arange(0.3, 0.85, 0.05)).to(device)\nand return the f2 score (I return it in the 5th position):
\n return (mp, mr, map50, map, f2, *(loss.cpu() / len(dataloader)).tolist()), maps, t\nSTEP 3: In the fitness function in utils/metrics.py add another weights 1.0 in the 5th position (depends on where you put the f2 in the above return) and give the other metrics a weight of 0.0:
\ndef fitness(x):\n# Model fitness as a weighted combination of metrics\nw = [0.0, 0.0, 0.0, 0.0, 1.0] # weights for [P, R, mAP@0.5, mAP@0.5:0.95, F2@0.3:0.8]\n return (x[:, :5] * w).sum(1)\nTo log F2 scores in wandb, add this code in train.py
\n if loggers.wandb:\n loggers.wandb.log({\"F2\": fi}) # W&B\nYou'll probably have to change something else in train.py to avoid some size mismatch errors because of the additional f2 variable, you can avoid it by returning f2 instead of map50 to keep the same shape of the results.
", "rawMarkdown": "@ayu055 so the first step is to add the F2 line of code to ap_per_class function and return the F2 as well:\n```return tp, fp, p, r, f2, ap, unique_classes.astype(\"int32\")```\n\n**STEP 2**: Go to val.py and modify these lines by adding f2:\n```\ntp, fp, p, r, f2, ap, ap_class = ap_per_class(*stats, plot=plots, save_dir=save_dir, names=names)\nap50, ap, f2 = ap[:, 0], ap.mean(1), , f2.mean(1) # AP@0.5, AP@0.5:0.95, f2@0.3:0.8\nmp, mr, f2, map50, map = p.mean(), r.mean(), f2.mean(), ap50.mean(), ap.mean()\n```\n\nAlso change this line:\n```\niouv = torch.linspace(0.5, 0.95, 10).to(device) # iou vector for mAP@0.5:0.95\n\n```\nto \n```\n\niouv= torch.from_numpy(np.arange(0.3, 0.85, 0.05)).to(device)\n```\nand return the f2 score (I return it in the 5th position):\n```\n return (mp, mr, map50, map, f2, *(loss.cpu() / len(dataloader)).tolist()), maps, t\n\n```\n\n**STEP 3:** In the fitness function in utils/metrics.py add another weights 1.0 in the 5th position (depends on where you put the f2 in the above return) and give the other metrics a weight of 0.0:\n```\ndef fitness(x):\n# Model fitness as a weighted combination of metrics\nw = [0.0, 0.0, 0.0, 0.0, 1.0] # weights for [P, R, mAP@0.5, mAP@0.5:0.95, F2@0.3:0.8]\n return (x[:, :5] * w).sum(1)\n```\nTo log F2 scores in wandb, add this code in train.py\n```\n\n if loggers.wandb:\n loggers.wandb.log({\"F2\": fi}) # W&B\n```\nYou'll probably have to change something else in train.py to avoid some size mismatch errors because of the additional f2 variable, you can avoid it by returning f2 instead of map50 to keep the same shape of the results.", "votes": 14 }, { "id": 1658394, "postDate": "2022-01-21T01:49:09.617Z", "content": "Thanks, Got it what mistake i am doing
", "rawMarkdown": "Thanks, Got it what mistake i am doing ", "votes": 1 }, { "id": 1658837, "postDate": "2022-01-21T10:51:52.823Z", "content": "@amiiiney Can you tell me which batch size did you use with yolov5m?
", "rawMarkdown": "@amiiiney Can you tell me which batch size did you use with yolov5m?", "votes": 1 }, { "id": 1658864, "postDate": "2022-01-21T11:13:59.737Z", "content": "Hey, @amiiiney Impressive score. May I know what change you made to get CV 0.528? Are the above-mentioned changes in the yolov5 are correct or not?
", "rawMarkdown": "Hey, @amiiiney Impressive score. May I know what change you made to get CV 0.528? Are the above-mentioned changes in the yolov5 are correct or not?", "votes": 1 }, { "id": 1658910, "postDate": "2022-01-21T12:01:54.523Z", "content": "\ndo you use Yolo-s or L? (FYI sheep's is with s6, --weights yolov5s6.pt)
\n\nYou'll probably have to change something else in train.py to avoid some size mismatch errors because of the additional f2 variable
\n
as long as x, w mult is valid and fitness(x) returns a single value should work OK.
@amiiiney thanks for sharing this, solved my errors. @morizin the above modifications are correct. There are some more modifications which can be done in order to log and visualize F2 score metrics on wandb while training. I will be sharing that in a new postβ¦.maybe beneficial for everyone in this competition.
@amiiiney Thanks for such an in-depth explanation. I was able to modify the code. I think I just had to make some changes in one of the callbacks so that it would log all metrics.
", "rawMarkdown": "@amiiiney Thanks for such an in-depth explanation. I was able to modify the code. I think I just had to make some changes in one of the callbacks so that it would log all metrics. ", "votes": 1 }, { "id": 1658969, "postDate": "2022-01-21T12:47:51.397Z", "content": "@amiiiney can u plz tell where extact in train.py changes needed. I am getting error and I am not able to solve it
", "rawMarkdown": "@amiiiney can u plz tell where extact in train.py changes needed. I am getting error and I am not able to solve it" }, { "id": 1658971, "postDate": "2022-01-21T12:49:18.950Z", "content": "@sanchitvj hi, Which batch size did you use in training
", "rawMarkdown": "@sanchitvj hi, Which batch size did you use in training", "votes": 1 }, { "id": 1658986, "postDate": "2022-01-21T13:15:28.797Z", "content": "@anshulkhadse What error are you getting?
", "rawMarkdown": "@anshulkhadse What error are you getting?" }, { "id": 1659002, "postDate": "2022-01-21T13:31:46.940Z", "content": "@nochanged I used batch size of 2, batch size of 1 with higher resolution is too slow.
\n@morizin I hope this is correct and I am not misleading anybody :))) The bug I had before was related to computing F2 only at the 0.5 threshold, not averaging over 0.3:0.8 thresholds. The new CV looks reasonable now given that my best model would score 0.5x on LB with inference size of 3000 instead of 6000.
\n@imeintanis I am using yolov5m6 weights, the only thing I am doing different to sheep is saving the weights based on F2 instead of mAP
", "rawMarkdown": "@nochanged I used batch size of 2, batch size of 1 with higher resolution is too slow.\n\n@morizin I hope this is correct and I am not misleading anybody :))) The bug I had before was related to computing F2 only at the 0.5 threshold, not averaging over 0.3:0.8 thresholds. The new CV looks reasonable now given that my best model would score 0.5x on LB with inference size of 3000 instead of 6000.\n\n@imeintanis I am using yolov5m6 weights, the only thing I am doing different to sheep is saving the weights based on F2 instead of mAP", "votes": 3 }, { "id": 1659008, "postDate": "2022-01-21T13:40:11.087Z", "content": "@ayu055 and @amiiiney I am getting errors in train.py file while computing in val.py file at end on epoch 1
\n@amiiiney can u also tell change in train.py files
What is the difference between the five fold csv file and the 0.2 csv? Is there any advantage to using one over the other?
", "rawMarkdown": "What is the difference between the five fold csv file and the 0.2 csv? Is there any advantage to using one over the other?" }, { "id": 1659014, "postDate": "2022-01-21T13:45:18.477Z", "content": "@deptraicute I am using batch_size of 4 with small models and resolution of 3000.
\n@anshulkhadse solution for that errors on the way :)
@sanchitvj Thanks you!
", "rawMarkdown": "@sanchitvj Thanks you!" }, { "id": 1659027, "postDate": "2022-01-21T13:58:50.620Z", "content": "@anshulkhadse You have to update the callbacks to fit the new metrics, it seems @sanchitvj is preparing a solution for this :)
", "rawMarkdown": "@anshulkhadse You have to update the callbacks to fit the new metrics, it seems @sanchitvj is preparing a solution for this :) ", "votes": 2 }, { "id": 1659031, "postDate": "2022-01-21T14:02:32.963Z", "content": "@amiiiney I trained like 14 epochs or something but I still reach like 0.49x
", "rawMarkdown": "@amiiiney I trained like 14 epochs or something but I still reach like 0.49x", "votes": 1 }, { "id": 1659072, "postDate": "2022-01-21T14:30:07.037Z", "content": "@amiiiney for this model, what IOU and Confidence u used while inference?
\n CV=0.528 (inference imsize: 3000)
\nLB= 0.652 (inference imsize: 6000)
@anshulkhadse @deptraicute @amiiiney solution out here :))
", "rawMarkdown": "@anshulkhadse @deptraicute @amiiiney solution out [here](https://www.kaggle.com/c/tensorflow-great-barrier-reef/discussion/302241) :))", "votes": 2 }, { "id": 1659121, "postDate": "2022-01-21T15:07:39.370Z", "content": "@morizin Did you use the same CV split I mentioned above? I am retraining the yolov5m6 again, I will report the F2 scores again once training is done
\n@anshulkhadse I used the same inference notebook shared by sheep, I didn't modify it.
", "rawMarkdown": "@morizin Did you use the same CV split I mentioned above? I am retraining the yolov5m6 again, I will report the F2 scores again once training is done\n\n@anshulkhadse I used the same inference notebook shared by sheep, I didn't modify it.", "votes": 1 }, { "id": 1659150, "postDate": "2022-01-21T15:21:59.223Z", "content": "@amiiiney thanks to share the infoβ¦
\nits quite weird that we cant able to reproduce resultsπ€£
@amiiiney Thank you. yes, i am using the same split!
", "rawMarkdown": "@amiiiney Thank you. yes, i am using the same split!", "votes": 1 }, { "id": 1659254, "postDate": "2022-01-21T16:34:40.063Z", "content": "@amiiiney What is the difference between the five fold csv file and the 0.2 csv? Is there any reason you chose one over the other?
", "rawMarkdown": "@amiiiney What is the difference between the five fold csv file and the 0.2 csv? Is there any reason you chose one over the other?", "votes": 1 }, { "id": 1659291, "postDate": "2022-01-21T17:15:56.370Z", "content": "@ayu055 There is no specific reason I chose it :) I just picked it to have the same split as my teammates for benchmarking. The author of the split notebook used StratifiedKFold for both 0.2csv and 5folds, so there shouldn't be much difference between them.
", "rawMarkdown": "@ayu055 There is no specific reason I chose it :) I just picked it to have the same split as my teammates for benchmarking. The author of the split notebook used StratifiedKFold for both 0.2csv and 5folds, so there shouldn't be much difference between them." }, { "id": 1659723, "postDate": "2022-01-22T04:31:31.057Z", "content": "@amiiiney are u using only annotated images or with annotated all non-annotated images as well?
", "rawMarkdown": "@amiiiney are u using only annotated images or with annotated all non-annotated images as well?" }, { "id": 1659933, "postDate": "2022-01-22T08:23:44.050Z", "content": "@amiiiney Did you check your LB at size 3000 with yolov5m?
", "rawMarkdown": "@amiiiney Did you check your LB at size 3000 with yolov5m?" }, { "id": 1660399, "postDate": "2022-01-22T16:41:51.083Z", "content": "\n\nYes it's F2 score, I am using all the images with the subsequence split from this notebook, csv file train-0.2.csv
\n
@amiiiney Do you filter out the images with no bounding boxes before training?
\n\n\nI am using SGD with lr=0.01 the defaults parameters. It seems that with this configuration there is no need to train for more than 8 epochs. And no, I am not using rotation/shear
\n
Are the default parameters the ones found in hyp.scratch.yaml?
", "rawMarkdown": "> Yes it's F2 score, I am using all the images with the subsequence split from this notebook, csv file train-0.2.csv\n\n@amiiiney Do you filter out the images with no bounding boxes before training?\n\n> I am using SGD with lr=0.01 the defaults parameters. It seems that with this configuration there is no need to train for more than 8 epochs. And no, I am not using rotation/shear\n\nAre the default parameters the ones found in hyp.scratch.yaml?" } ] }, { "id": 1653346, "postDate": "2022-01-17T13:06:29.077Z", "content": "Any one can reproduce the training notebook of yolov5s6, 3600 with 15 epoch? I can only get 0.48 for lb, and cv(on video 1) is also much lower than sheep big brother.
", "rawMarkdown": "Any one can reproduce the training notebook of yolov5s6, 3600 with 15 epoch? I can only get 0.48 for lb, and cv(on video 1) is also much lower than sheep big brother. ", "votes": 10, "replies": [ { "id": 1653402, "postDate": "2022-01-17T13:53:53.820Z", "content": "Not sure why the downvote ,but yes , i could not reproduce it exactly although not as low as you ,but surely not .579 . Not sure what is going on . My CV was exactly same as sheep though .
", "rawMarkdown": "Not sure why the downvote ,but yes , i could not reproduce it exactly although not as low as you ,but surely not .579 . Not sure what is going on . My CV was exactly same as sheep though . ", "votes": 3 }, { "id": 1653522, "postDate": "2022-01-17T15:57:30.293Z", "content": "I believe the split used on the training notebook is different compared to what the 0.579 model was trained on. I speculate this based on the filename for the weights (it says fold 12). Maybe it was trained on the 12th fold of the 20 fold subseq split?
", "rawMarkdown": "I believe the split used on the training notebook is different compared to what the 0.579 model was trained on. I speculate this based on the filename for the weights (it says fold 12). Maybe it was trained on the 12th fold of the 20 fold subseq split?", "votes": 2 }, { "id": 1653544, "postDate": "2022-01-17T16:20:46.777Z", "content": "No when you run the same experiment multiple times in Yolo5 it will append a sequence number at the end of the experiment folder . e.g run1,run2, run3 etc so fold1 became fold12 in second run ..this is my theory
", "rawMarkdown": "No when you run the same experiment multiple times in Yolo5 it will append a sequence number at the end of the experiment folder . e.g run1,run2, run3 etc so fold1 became fold12 in second run ..this is my theory ", "votes": 3 }, { "id": 1653599, "postDate": "2022-01-17T17:15:46.690Z", "content": "I can't also get the same results. When I trained on the same train split and same hyberparameters I got only 0.48 LB
", "rawMarkdown": "I can't also get the same results. When I trained on the same train split and same hyberparameters I got only 0.48 LB" }, { "id": 1654080, "postDate": "2022-01-18T06:06:01.887Z", "content": "I use this cmd for local training
\npython train.py --img 3600 --batch 4 --epochs 15 --data reef_f1_naive.yaml --weights yolov5s6.pt --name l6_3600_uflip_vm5_f1 --hyp data/hyps/hyp.heavy.2.yaml\nI am not sure the cause of the reproduction issue.
", "rawMarkdown": "I use this cmd for local training\n```\npython train.py --img 3600 --batch 4 --epochs 15 --data reef_f1_naive.yaml --weights yolov5s6.pt --name l6_3600_uflip_vm5_f1 --hyp data/hyps/hyp.heavy.2.yaml\n```\nI am not sure the cause of the reproduction issue.", "votes": 3 }, { "id": 1654109, "postDate": "2022-01-18T06:49:43.427Z", "content": "failed to reproduce. +1
", "rawMarkdown": "failed to reproduce. +1", "votes": 1 }, { "id": 1654115, "postDate": "2022-01-18T07:05:12.523Z", "content": "Forget about @steamedsheep 's model , i cant reproduce my own models β¦lol
", "rawMarkdown": "Forget about @steamedsheep 's model , i cant reproduce my own models ...lol", "votes": 7 }, { "id": 1654117, "postDate": "2022-01-18T07:14:12.360Z", "content": "I got 0.567 with batch=1 due to GPU limit.
", "rawMarkdown": "I got 0.567 with batch=1 due to GPU limit." }, { "id": 1654119, "postDate": "2022-01-18T07:20:21.760Z", "content": "batch_size=1 won't give good results because Yolos use batch norm (sync BN).
", "rawMarkdown": "batch_size=1 won't give good results because Yolos use batch norm (sync BN).", "votes": 1 }, { "id": 1654183, "postDate": "2022-01-18T08:48:20.847Z", "content": "They also use Gradient Accumulation and adjusts weight decay based on batch size
", "rawMarkdown": "They also use Gradient Accumulation and adjusts weight decay based on batch size ", "votes": 4 }, { "id": 1654254, "postDate": "2022-01-18T10:50:24.717Z", "content": "batch_size=1 is not giving a good result and batch_size>1 is giving an error on GPU on kaggle. I am not sure if we need to move to another platform or I made a mistake.
", "rawMarkdown": "batch_size=1 is not giving a good result and batch_size>1 is giving an error on GPU on kaggle. I am not sure if we need to move to another platform or I made a mistake.\n" }, { "id": 1654266, "postDate": "2022-01-18T11:08:34.727Z", "content": "train on cropped images are better solution then accumulated gradient. you have variation from different images and your bn statistics will be more stable
", "rawMarkdown": "train on cropped images are better solution then accumulated gradient. you have variation from different images and your bn statistics will be more stable", "votes": 5 }, { "id": 1654322, "postDate": "2022-01-18T11:41:08.133Z", "content": "\"I am not sure the cause of the reproduction issue.\"
\ntraining objection detection can be unstable if the number of annotation is few and the data is difficult.
\nimagine you have a batch of 8 images with 8 truth bbox, verus another batch with 32 truth bbox.
loss may changes from batch to batch.
\na soultion is to modified your sampler os that num of bbox per batch is fairly constant. but this is difficult becuase augmentation also changes number of true bbox (e.g. when you scale image, boxes become too small are ignore)
\nthis is also why when you check the code of the loss for object detection, they are normalised by number of truth object
", "rawMarkdown": "\"I am not sure the cause of the reproduction issue.\"\n\ntraining objection detection can be unstable if the number of annotation is few and the data is difficult.\nimagine you have a batch of 8 images with 8 truth bbox, verus another batch with 32 truth bbox.\n\nloss may changes from batch to batch.\n\n\na soultion is to modified your sampler os that num of bbox per batch is fairly constant. but this is difficult becuase augmentation also changes number of true bbox (e.g. when you scale image, boxes become too small are ignore)\n\nthis is also why when you check the code of the loss for object detection, they are normalised by number of truth object", "votes": 6 }, { "id": 1654447, "postDate": "2022-01-18T14:20:48.557Z", "content": "@hengck23 So is it a better idea to train on smaller image sizes with larger batch sizes?
", "rawMarkdown": "@hengck23 So is it a better idea to train on smaller image sizes with larger batch sizes?" }, { "id": 1654642, "postDate": "2022-01-18T17:22:14.300Z", "content": "@steamedsheep. I think you did something on the source code, perhaps. because when I looked into your weights. you cropped all the information of epoch number and the best_fitness score and also a couple of values. I may ask: have you done it to reduce the file size? I guess it won't reduce much
", "rawMarkdown": "@steamedsheep. I think you did something on the source code, perhaps. because when I looked into your weights. you cropped all the information of epoch number and the best_fitness score and also a couple of values. I may ask: have you done it to reduce the file size? I guess it won't reduce much", "votes": 1 }, { "id": 1656024, "postDate": "2022-01-19T03:44:27.600Z", "content": "I did NOTHING on source code, It is yolov5's host who decide only to save the model weight on weights/best.pt, If you still have some question on this, It is better for you to run some demo code and check the weight first.
@morizin β¦. it's true that @steamedsheep doe not do anything with its weight file..and u are also correct at ur position is that why u are notable able to see model training details on the trained modelβ¦ the reason is that in YOLOv5 when u trained ur model at the end of training it does striping of the model weight file it means it deletes model optimizer states and along with that it also removes the information about for many epoch ur models is trained onβ¦.the reason for this is keeping optimizer states in model inferencing is no used and it unnecessarily increases the weight file size that's why this stripping is doneβ¦it will reduce the weight file size
\nStriping comes with some drawbacks also that u can't really much fine-tune the same weight file due to the absence of optimizer stateβ¦and due to stripping model weight file size gets changed with same initials pre-trained weights.
", "rawMarkdown": "@morizin .... it's true that @steamedsheep doe not do anything with its weight file..and u are also correct at ur position is that why u are notable able to see model training details on the trained model... the reason is that in YOLOv5 when u trained ur model at the end of training it does striping of the model weight file it means it deletes model optimizer states and along with that it also removes the information about for many epoch ur models is trained on....the reason for this is keeping optimizer states in model inferencing is no used and it unnecessarily increases the weight file size that's why this stripping is done...it will reduce the weight file size\n\nStriping comes with some drawbacks also that u can't really much fine-tune the same weight file due to the absence of optimizer state...and due to stripping model weight file size gets changed with same initials pre-trained weights.\n", "votes": 1 }, { "id": 1658617, "postDate": "2022-01-21T07:07:46.763Z", "content": "Thank you! @anshulkhadse and Sorry @steamedsheep
", "rawMarkdown": "Thank you! @anshulkhadse and Sorry @steamedsheep ", "votes": 1 } ] }, { "id": 1649263, "postDate": "2022-01-14T06:24:06.817Z", "content": "4k is always better than 1080p π
", "rawMarkdown": "4k is always better than 1080p π", "votes": 5 }, { "id": 1653223, "postDate": "2022-01-17T11:15:29.017Z", "content": "i want to study more about lb metric and the reasons and effects of doing inference at higher resolution.
\nanyone has local CV (validation set = video1) at 3600 for the given model '../input/reef-baseline-fold12/l6_3600_uflip_vm5_f12_up/f1/best.pt' ?
I am getting around 0.57 for local CV
", "rawMarkdown": "i want to study more about lb metric and the reasons and effects of doing inference at higher resolution.\nanyone has local CV (validation set = video1) at 3600 for the given model '../input/reef-baseline-fold12/l6_3600_uflip_vm5_f12_up/f1/best.pt' ?\n\nI am getting around 0.57 for local CV", "votes": 5, "replies": [ { "id": 1653451, "postDate": "2022-01-17T14:36:59.953Z", "content": "Your CV is very similiar to mine.
", "rawMarkdown": "Your CV is very similiar to mine.", "votes": 1 }, { "id": 1653580, "postDate": "2022-01-17T16:58:18.820Z", "content": "How do you calculate the CV? And this is f2 score right? I was getting 0.69 at image size =3600 for sheeps model
", "rawMarkdown": "How do you calculate the CV? And this is f2 score right? I was getting [0.69 at image size =3600 ](https://www.kaggle.com/ferlockx/f2-cv-0-77-subseq-split-and-higher-res#F2-Score-Helpers) for sheeps model " }, { "id": 1653683, "postDate": "2022-01-17T19:35:44.663Z", "content": "I got about the same on CV 0.69x at img size=3600, conf 0.1 (~2100 images with annot)
EDIT: maybe the 0.57 is for all images including empty ones (8232 images, 6133 empty) ?
IMPORTANT !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
\ni thinnk there is a bug in https://www.kaggle.com/ferlockx/f2-cv-0-77-subseq-split-and-higher-res/notebook
\n(it reported CV of 0.69)
\nhttps://www.kaggle.com/ferlockx/f2-cv-0-77-subseq-split-and-higher-res/comments
\n@ferlockx , I think there is a bug in your code:
\nboxAArea = (x12 - x11 + 1) * (y12 - y11 + 1)\nboxBArea = (x22 - x21 + 1) * (y22 - y21 + 1)\nwould the host evaluation code also have such bug ???
", "rawMarkdown": "IMPORTANT !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!\n\ni thinnk there is a bug in https://www.kaggle.com/ferlockx/f2-cv-0-77-subseq-split-and-higher-res/notebook\n\n(it reported CV of 0.69)\n\nhttps://www.kaggle.com/ferlockx/f2-cv-0-77-subseq-split-and-higher-res/comments\n\n@ferlockx , I think there is a bug in your code:\n\n- you use +1 in computation of the area\n\n```\nboxAArea = (x12 - x11 + 1) * (y12 - y11 + 1)\nboxBArea = (x22 - x21 + 1) * (y22 - y21 + 1)\n\n```\n\n- but you used gt_bboxs_list, prd_bboxs_list which are normalised coordinates in the range of 0,1\n\n---\n\nwould the host evaluation code also have such bug ???", "votes": 7 }, { "id": 1654033, "postDate": "2022-01-18T04:42:33.930Z", "content": "i spend a whole day catching the bug and has other interesting findings. let's denote
\n(a) https://www.kaggle.com/ferlockx/f2-cv-0-77-subseq-split-and-higher-res/notebook
\n(b) https://www.kaggle.com/steamedsheep/yolov5-is-all-you-need
in particular, (a) is based on val.py which use dataset and dataloader. there is padding of 114 for input images (i think the train also uses pad of 114?). it improves slightly detection of cots objects at the boundary. (b) uses image directly (or a pad of zero values).
(b) incudes test time augmentation in submission. (a) is evaluation without augmentation
both (a) and (b) uses different input size becuase of the padding of 114. the resizing function is also different. (a) uses opencv resize, (b) uses torch F.interpolate.
3600 is not multiple of 64 (max stride of yolov5ms model). the code uses another size actually
\nif you are using (a), you would get the best f2 score for the lowest confidence threshold. you can even go as low as 0.01. this is becuase the evaluation only include non-empty images (i.e. those without ground truth annotations only).
\ni think the public test set has few empty images. this is why as long as you improve recall, you have increase in lb score
", "rawMarkdown": "i spend a whole day catching the bug and has other interesting findings. let's denote\n\n(a) https://www.kaggle.com/ferlockx/f2-cv-0-77-subseq-split-and-higher-res/notebook\n(b) https://www.kaggle.com/steamedsheep/yolov5-is-all-you-need\n \n1. in particular, (a) is based on val.py which use dataset and dataloader. there is padding of 114 for input images (i think the train also uses pad of 114?). it improves slightly detection of cots objects at the boundary. (b) uses image directly (or a pad of zero values).\n\n2. (b) incudes test time augmentation in submission. (a) is evaluation without augmentation\n\n3. both (a) and (b) uses different input size becuase of the padding of 114. the resizing function is also different. (a) uses opencv resize, (b) uses torch F.interpolate.\n\n3600 is not multiple of 64 (max stride of yolov5ms model). the code uses another size actually\n\n4. the most important one! \n---\n\nif you are using (a), you would get the best f2 score for the lowest confidence threshold. you can even go as low as 0.01. this is becuase the evaluation only include non-empty images (i.e. those without ground truth annotations only).\n\ni think the public test set has few empty images. this is why as long as you improve recall, you have increase in lb score", "votes": 10 }, { "id": 1654086, "postDate": "2022-01-18T06:14:58.340Z", "content": "@hengck23 you should use val.py with full fold with empty images, otherwise its wrong.
", "rawMarkdown": "@hengck23 you should use val.py with full fold with empty images, otherwise its wrong.", "votes": 1 }, { "id": 1654101, "postDate": "2022-01-18T06:34:43.683Z", "content": "thanks for spending time on reviewing the CV notebook @hengck23 most of the helper functions were taken from the first reference and I did not review the bbox area calculation part. I'll correct it in the next version.
\nI think in one of runs using -augment call of yolo val script was not improving results so I did not include it in the saved versions.
\nI think that the 4th point you mention was only true for the models which have been made public by sheep. In my own runs with some models, upsizing and then using higher thresholds was giving significant CV boost (even though this was without empty images).
\nIn runs with 0 annotation images I think that we can discard all \"small area\"(arbitrarily chosen as 800 or 1000) boxes under threshold 0.4-0.5 so that we don't have lot's of false positive small bbox predictions. I tested this technique suggested by hengck on LB and it was indeed boosting by ~0.005-0.01.
\nThanks Lukas I'll do a run with all data in vid 1 as val to get the accurate CV. I was resisting doing on Kaggle since it's compute expensive to run on a lot of images and that too testing with different sizes so I'll try locally on Colab.
", "rawMarkdown": "thanks for spending time on reviewing the CV notebook @hengck23 most of the helper functions were taken from the first reference and I did not review the bbox area calculation part. I'll correct it in the next version. \n\nI think in one of runs using -augment call of yolo val script was not improving results so I did not include it in the saved versions. \n\nI think that the 4th point you mention was only true for the models which have been made public by sheep. In my own runs with some models, upsizing and then using higher thresholds was giving significant CV boost (even though this was without empty images). \n\nIn runs with 0 annotation images I think that we can discard all \"small area\"(arbitrarily chosen as 800 or 1000) boxes under threshold 0.4-0.5 so that we don't have lot's of false positive small bbox predictions. I tested this technique suggested by hengck on LB and it was indeed boosting by ~0.005-0.01.\n\nThanks Lukas I'll do a run with all data in vid 1 as val to get the accurate CV. I was resisting doing on Kaggle since it's compute expensive to run on a lot of images and that too testing with different sizes so I'll try locally on Colab. " }, { "id": 1654346, "postDate": "2022-01-18T11:49:25.203Z", "content": "@ferlockx @hengck23 Thanks for the notebook and your valuable input
\nAfter making the necessary suggested modification :
\nSheep's code CV : 0.574 LB :0.578 @3600
\nA Version of my Checkpoint CV : 0.536 LB : 0.563 @3600
\nSame version of my checkpoint CV : 0.582 LB: (yet to check ) @4000
I used val.py with --augment parameter
\nNot making the notebook public as I did minimal modification from the original published by Aditya.
", "rawMarkdown": "@ferlockx @hengck23 Thanks for the notebook and your valuable input \nAfter making the necessary suggested modification : \nSheep's code CV : 0.574 LB :0.578 @3600 \nA Version of my Checkpoint CV : 0.536 LB : 0.563 @3600 \nSame version of my checkpoint CV : 0.582 LB: (yet to check ) @4000\n\nI used val.py with --augment parameter \n\nNot making the notebook public as I did minimal modification from the original published by Aditya." }, { "id": 1654382, "postDate": "2022-01-18T12:59:22.607Z", "content": "For me, my yolov5 scores 0.66 with pure video_1 validation(including empty images)
", "rawMarkdown": "For me, my yolov5 scores 0.66 with pure video_1 validation(including empty images)" }, { "id": 1654396, "postDate": "2022-01-18T13:17:21.633Z", "content": "Sheep's code ? If not May be you should submit it :D
", "rawMarkdown": "Sheep's code ? If not May be you should submit it :D" }, { "id": 1654471, "postDate": "2022-01-18T14:53:26.717Z", "content": "Yaa, I would. My model doesn't love enlarging images. In img_size 4800, giving 0.5x or something strange. It doesn't detect much in larger image sizes.
", "rawMarkdown": "Yaa, I would. My model doesn't love enlarging images. In img_size 4800, giving 0.5x or something strange. It doesn't detect much in larger image sizes.", "votes": 1 }, { "id": 1654608, "postDate": "2022-01-18T16:59:08.980Z", "content": "@phoenix9032 When you say CV score, do you refer to the F2 score or the mAP across 0.3:0.8 IOU thresholds?
\nIf it is F2, do you calculate this in val.py using the mean precision and recall that is already calculated?
", "rawMarkdown": "@phoenix9032 When you say CV score, do you refer to the F2 score or the mAP across 0.3:0.8 IOU thresholds?\n\nIf it is F2, do you calculate this in val.py using the mean precision and recall that is already calculated?" }, { "id": 1654638, "postDate": "2022-01-18T17:17:25.593Z", "content": "Sorry for informing the false score. unfortunately i found a bug in my code and i am just updating the CV score here. its 0.602 F2 score
\n@ayu055 you can use this notebook . there are some few changes are required which is described in these comments
", "rawMarkdown": "Sorry for informing the false score. unfortunately i found a bug in my code and i am just updating the CV score here. its 0.602 F2 score\n\n@ayu055 you can use this [notebook](https://www.kaggle.com/ferlockx/f2-cv-0-77-subseq-split-and-higher-res/notebook) . there are some few changes are required which is described in these comments\n", "votes": 1 }, { "id": 1654645, "postDate": "2022-01-18T17:26:15.990Z", "content": "@morizin Still its quite good local CV for lower image size ..But i have not tested the local CV for many cases, may be once you submit you will know about your model too β¦ @ayu055 , Rizin is right ..
", "rawMarkdown": "@morizin Still its quite good local CV for lower image size ..But i have not tested the local CV for many cases, may be once you submit you will know about your model too ... @ayu055 , Rizin is right .." }, { "id": 1654748, "postDate": "2022-01-18T18:59:57.900Z", "content": "So do you not calculate F2 score while training and only at the end on validation?
", "rawMarkdown": "So do you not calculate F2 score while training and only at the end on validation?" }, { "id": 1654749, "postDate": "2022-01-18T19:01:08.390Z", "content": "Yes that is correct , for now.
", "rawMarkdown": "Yes that is correct , for now." }, { "id": 1656357, "postDate": "2022-01-19T09:41:26.253Z", "content": "Can someone explain the changes to be made in this notebook to calculate the proper cv score.
", "rawMarkdown": "Can someone explain the changes to be made in this [notebook](https://www.kaggle.com/ferlockx/f2-cv-0-77-subseq-split-and-higher-res/notebook) to calculate the proper cv score." }, { "id": 1657380, "postDate": "2022-01-20T06:27:32.460Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1665208, "postDate": "2022-01-26T16:45:07.527Z", "content": "my best score with video1 is 0.624, without TTA
", "rawMarkdown": "my best score with video1 is 0.624, without TTA" } ] }, { "id": 1654106, "postDate": "2022-01-18T06:45:38.463Z", "content": "In fact, there is nothing to be sorry about. The purpose of this community is to use AI technology to solve industrial problems. Sharing good ideas can improve the performance of final solutions, and they are not shared near the deadline. I don't think there's anything wrong. Although some people feel that the experience is getting worse, others feel that the experience is getting better, such as me.
", "rawMarkdown": "In fact, there is nothing to be sorry about. The purpose of this community is to use AI technology to solve industrial problems. Sharing good ideas can improve the performance of final solutions, and they are not shared near the deadline. I don't think there's anything wrong. Although some people feel that the experience is getting worse, others feel that the experience is getting better, such as me.", "votes": 3 }, { "id": 1649248, "postDate": "2022-01-14T05:47:58.817Z", "content": "thanks a lot, sheep big old and frog god
", "rawMarkdown": "thanks a lot, sheep big old and frog god", "votes": 3 }, { "id": 1654267, "postDate": "2022-01-18T11:08:58.040Z", "content": "Anyone tried submitting Yolov5X6 (largest model). My inference
\nWith img size:
unable to train due to limit of GPUπ
", "rawMarkdown": "unable to train due to limit of GPUπ", "votes": 1 }, { "id": 1657399, "postDate": "2022-01-20T06:43:22.760Z", "content": "@rainfalllove don't worry large models aren't performing good anyways. I am performing experiment with all the models and will be clearing this in my post by next week.
", "rawMarkdown": "@rainfalllove don't worry large models aren't performing good anyways. I am performing experiment with all the models and will be clearing this in my post by next week." }, { "id": 1657691, "postDate": "2022-01-20T11:54:49.143Z", "content": "okay, I'm looking forward to it .
", "rawMarkdown": "okay, I'm looking forward to it ." }, { "id": 1658939, "postDate": "2022-01-21T12:24:28.817Z", "content": "I timed out with img_size=5600 so you're not alone. But it might be due to my own fault, due to messed up model.
", "rawMarkdown": "I timed out with img_size=5600 so you're not alone. But it might be due to my own fault, due to messed up model. " } ] }, { "id": 1652502, "postDate": "2022-01-16T17:28:38.250Z", "content": "@lukaszborecki I tried my cross val notebook on sheeps splitting of data and respective models by video id=1 as val folder and that split did indeed give increase in CV with increasing image sizes. It's at 6k size rn and should complete in an hour will update changes to public then. However at initial size, score was ~0.69 much lower than ~0.77 of my models.
\nAlso for my own models split by sub-sequence had higher CV for increased image sizes when evaluated using higher threshold. This leads me to think that maybe an ideal setup would be using different models to infer at different threshold value.
", "rawMarkdown": "@lukaszborecki I tried my [cross val notebook](https://www.kaggle.com/ferlockx/f2-cv-0-77-subseq-split-and-higher-res) on sheeps splitting of data and respective models by video id=1 as val folder and that split did indeed give increase in CV with increasing image sizes. It's at 6k size rn and should complete in an hour will update changes to public then. However at initial size, score was ~0.69 much lower than ~0.77 of my models.\n\nAlso for my own models split by sub-sequence had higher CV for increased image sizes when evaluated using higher threshold. This leads me to think that maybe an ideal setup would be using different models to infer at different threshold value.", "votes": 1, "replies": [ { "id": 1652533, "postDate": "2022-01-16T18:10:06.787Z", "content": "I bet it will fall :) however i tried split vid0 1 vs 2 and it was the same. Do you validating on full set ? (with empty images too :) ?
", "rawMarkdown": "I bet it will fall :) however i tried split vid0 1 vs 2 and it was the same. Do you validating on full set ? (with empty images too :) ?", "votes": 1 }, { "id": 1652545, "postDate": "2022-01-16T18:18:30.993Z", "content": "yup it fell at 6k haha made public just a few mins before your comment. But still it was this shows it was increasing uptil ~x1.5 scaling.
\nOops I think that's an important detail I've not been considering π¬ So if the test set has roughly 3/4 of the data without annotations does that mean I should randomly sample val_fold_size*3 images with no annotations into my cross validation?
\nOh splitting by video ID means I just take the entire data with vid id 1 got it :p
@ferlockx seriously idk how to answer :) i made really beutiful models, predicts f2 score on validation 0.8 beutiful inference and then LB 0.5 0.49 :D i made once different training with same set same settings but early stop at epoch3 . Inference was good but worse from others , it didnt detect all starfishes from my validation images i used before and LB score was 0.62. Our best model was not our best but it scored good on LB. If you got such a pearl you want to tweak it. Imagine loading ckpt into yolo then start training it on really low LR for one epoch!, val box drops val obj drops and LB drops from 0.67 -> 0.52 is it normal ?
", "rawMarkdown": "@ferlockx seriously idk how to answer :) i made really beutiful models, predicts f2 score on validation 0.8 beutiful inference and then LB 0.5 0.49 :D i made once different training with same set same settings but early stop at epoch3 . Inference was good but worse from others , it didnt detect all starfishes from my validation images i used before and LB score was 0.62. Our best model was not our best but it scored good on LB. If you got such a pearl you want to tweak it. Imagine loading ckpt into yolo then start training it on really low LR for one epoch!, val box drops val obj drops and LB drops from 0.67 -> 0.52 is it normal ? ", "votes": 1 }, { "id": 1652556, "postDate": "2022-01-16T18:32:17.343Z", "content": "hmm thanks for the tips :) I think I should focus more on transfer learning and ensembles once I've got this cross validation part sorted :p
", "rawMarkdown": "hmm thanks for the tips :) I think I should focus more on transfer learning and ensembles once I've got this cross validation part sorted :p " }, { "id": 1654046, "postDate": "2022-01-18T05:07:55.707Z", "content": "Hi, does ensemble increase LB ? I tried ,but LB got worse.
", "rawMarkdown": "Hi, does ensemble increase LB ? I tried ,but LB got worse." }, { "id": 1654263, "postDate": "2022-01-18T11:07:52.897Z", "content": "[deleted message]
", "rawMarkdown": "[deleted message]" } ] }, { "id": 1649330, "postDate": "2022-01-14T07:49:01.020Z", "content": "For memory issue with larger image on kaggle, reducing batchsize is an option. According to the devs of yolov5, batch size should not influence results much. However, if you're using other (older) YOLO versions, do note that there's report on earlier YOLO that small batch size produces inferior results. Hardware shouldn't be a big issue (hopefully), so don't get discouraged.
", "rawMarkdown": "For memory issue with larger image on kaggle, reducing batchsize is an option. According to the devs of yolov5, batch size should not influence results much. However, if you're using other (older) YOLO versions, do note that there's report on earlier YOLO that small batch size produces inferior results. Hardware shouldn't be a big issue (hopefully), so don't get discouraged.", "votes": 1, "replies": [ { "id": 1649678, "postDate": "2022-01-14T13:55:35.473Z", "content": "If I remember correctly (from Yolov5 previous versions) the nominal batch size is 64, that results to grad accumul steps = 64/BS, where BS the chosen batch size.
", "rawMarkdown": "If I remember correctly (from Yolov5 previous versions) the nominal batch size is 64, that results to grad accumul steps = 64/BS, where BS the chosen batch size. ", "votes": 1 } ] }, { "id": 1649059, "postDate": "2022-01-13T23:24:47.713Z", "content": "How did you manage to make your model weight only 28MB? XD
", "rawMarkdown": "How did you manage to make your model weight only 28MB? XD", "votes": 1, "replies": [ { "id": 1649362, "postDate": "2022-01-14T08:20:36.243Z", "content": "@steamedsheep I have the same question . Any hint ?
", "rawMarkdown": "@steamedsheep I have the same question . Any hint ?" }, { "id": 1649376, "postDate": "2022-01-14T08:31:08.033Z", "content": "I think is normal due to pruner & optim stripper at the end of run.
\nFor me I see that yolov5-L is ~98MB so I guess for S 28MB sound normal
thanks .. got it .. at the end its stripping optimizer and reducing ..
", "rawMarkdown": "thanks .. got it .. at the end its stripping optimizer and reducing ..\n", "votes": 1 } ] }, { "id": 1649006, "postDate": "2022-01-13T22:04:12.630Z", "content": "thanks for tip @steamedsheep! May i ask what was the CV metrics for that chekpoint and if you like to say around at which epoch?
", "rawMarkdown": "thanks for tip @steamedsheep! May i ask what was the CV metrics for that chekpoint and if you like to say around at which epoch? ", "votes": 1 }, { "id": 1648646, "postDate": "2022-01-13T15:15:07.040Z", "content": "So this again is a hardware competition dominated?
\nbecause I try to increase to 1920x1920, over 9hours limit of kaggle. I tried on colab pro, not pro+, which I have to check once in a while, It trains and converge slowly, so I gave up try bigger image.
\nupvote, though.
", "rawMarkdown": "So this again is a hardware competition dominated?\n\nbecause I try to increase to 1920x1920, over 9hours limit of kaggle. I tried on colab pro, not pro+, which I have to check once in a while, It trains and converge slowly, so I gave up try bigger image.\n\nupvote, though.", "votes": 1, "replies": [ { "id": 1648661, "postDate": "2022-01-13T15:30:35.947Z", "content": "Don't worry our solution is on P100 :)
", "rawMarkdown": "Don't worry our solution is on P100 :)", "votes": 2 }, { "id": 1648691, "postDate": "2022-01-13T15:53:56.547Z", "content": "you guys have pro+, don't you? which notebook keep running without disconnection, over my budget.
", "rawMarkdown": "you guys have pro+, don't you? which notebook keep running without disconnection, over my budget." }, { "id": 1648701, "postDate": "2022-01-13T16:01:09.143Z", "content": "Yeah we got pro+ we got p100/v100/a100 at start but now only p100 and model was trained on p100. There is huge traffic on colab right now.
", "rawMarkdown": "Yeah we got pro+ we got p100/v100/a100 at start but now only p100 and model was trained on p100. There is huge traffic on colab right now. " }, { "id": 1648704, "postDate": "2022-01-13T16:02:39.147Z", "content": "To score LB 0.673 laptop is enough β¦ belive me β¦
", "rawMarkdown": "To score LB 0.673 laptop is enough ... belive me ...", "votes": 10 }, { "id": 1648711, "postDate": "2022-01-13T16:07:42.577Z", "content": "I believe that you guys find better strategy. my laptop only with 1050ti.
", "rawMarkdown": "I believe that you guys find better strategy. my laptop only with 1050ti. " }, { "id": 1648735, "postDate": "2022-01-13T16:37:57.823Z", "content": "I've been training all my models on 3090 but I've been doing many experiments with tf-2.0. I'll need to get back to using yolov5 to score higher on LB. But I'll tell you this, image tilling might be the next big to happen since the COTs aren't big why make the whole picture bigger when you can have multiple smaller pictures that make the COTs appear bigger to the model. I've had some good results with it. Worth a try!
", "rawMarkdown": "I've been training all my models on 3090 but I've been doing many experiments with tf-2.0. I'll need to get back to using yolov5 to score higher on LB. But I'll tell you this, image tilling might be the next big to happen since the COTs aren't big why make the whole picture bigger when you can have multiple smaller pictures that make the COTs appear bigger to the model. I've had some good results with it. Worth a try!", "votes": 2 }, { "id": 1649001, "postDate": "2022-01-13T21:53:45.677Z", "content": "I see β¦ but using Kaggle GPU or free Collab will be enough at this moment. Certainly the more power you have the faster you can experiment β¦ but β¦ sometimes this is not a good to have to much GPUs :)
", "rawMarkdown": "I see ... but using Kaggle GPU or free Collab will be enough at this moment. Certainly the more power you have the faster you can experiment ... but ... sometimes this is not a good to have to much GPUs :)", "votes": 1 }, { "id": 1649011, "postDate": "2022-01-13T22:15:16.617Z", "content": "@remekkinas I don't know if I made it clear in my previous comment but I meant that there are many strategies to get around the issues of high resolution. Here is this one for example that was useful to me. It is worth a try in constraint environments and with some tuning, it maybe can boost your models.
\nhttps://github.com/obss/sahi
Good luck!
", "rawMarkdown": "@remekkinas I don't know if I made it clear in my previous comment but I meant that there are many strategies to get around the issues of high resolution. Here is this one for example that was useful to me. It is worth a try in constraint environments and with some tuning, it maybe can boost your models.\nhttps://github.com/obss/sahi\n\nGood luck!", "votes": 3 }, { "id": 1649018, "postDate": "2022-01-13T22:23:04.803Z", "content": "Yes, absolutely β¦ SAHI is one of β¦ tiling (presented in notebook section) is another one, resizing showed here is next one β¦ This is beauty of this competition - you can experiment a lot. Dataset is demanding. As I can see on LB everybody β¦ are learning :) To be honest I am waiting for end β¦ why β¦ I am really interested in TOP solutions π We have performed a lot of different experiments β¦ I am sure some of them β¦ which have not worked for us probably would work for somebody else. This is really great learning.
", "rawMarkdown": "Yes, absolutely ... SAHI is one of ... tiling (presented in notebook section) is another one, resizing showed here is next one ... This is beauty of this competition - you can experiment a lot. Dataset is demanding. As I can see on LB everybody ... are learning :) To be honest I am waiting for end ... why ... I am really interested in TOP solutions π We have performed a lot of different experiments ... I am sure some of them ... which have not worked for us probably would work for somebody else. This is really great learning.", "votes": 2 }, { "id": 1649176, "postDate": "2022-01-14T03:30:16.860Z", "content": "If you don't mind me asking @remekkinas. I tried several times both on lb and cv to improve the performance of my Yolo model with SAHI.
\nDid SAHI affect your current score? and how much?
@morizin I know SAHI very, very well :) I use it since it was released ( @clwwlc will be probably suprised π ) To be honest we have nor tried SAHI so far β¦ In December we decided to take totally different attitude.
", "rawMarkdown": "@morizin I know SAHI very, very well :) I use it since it was released ( @clwwlc will be probably suprised π ) To be honest we have nor tried SAHI so far ... In December we decided to take totally different attitude. ", "votes": 1 } ] }, { "id": 1648549, "postDate": "2022-01-13T13:48:54.557Z", "content": "Higher resolution is all your need π€ ? Maybe not, right ?
", "rawMarkdown": "Higher resolution is all your need π€ ? Maybe not, right ?", "votes": 1, "replies": [ { "id": 1648593, "postDate": "2022-01-13T14:40:26.903Z", "content": "yes, indeed
", "rawMarkdown": "yes, indeed", "votes": 1 }, { "id": 1648595, "postDate": "2022-01-13T14:40:45.037Z", "content": "Yeah you need hugeillions of pixels and zillions of compute power. Not :)
", "rawMarkdown": "Yeah you need hugeillions of pixels and zillions of compute power. Not :)" }, { "id": 1648599, "postDate": "2022-01-13T14:42:59.930Z", "content": "Not really, I train this on a single 3090, I think 16GB ram card is also ok.
", "rawMarkdown": "Not really, I train this on a single 3090, I think 16GB ram card is also ok.", "votes": 2 }, { "id": 1648604, "postDate": "2022-01-13T14:45:43.200Z", "content": "I didn't say it doesn't helps but is not all you need :)
", "rawMarkdown": "I didn't say it doesn't helps but is not all you need :)" }, { "id": 1648605, "postDate": "2022-01-13T14:47:40.047Z", "content": "aha, topic title in kaggle usually misleading, this isn't the worst oneπ
", "rawMarkdown": "aha, topic title in kaggle usually misleading, this isn't the worst oneπ", "votes": 2 }, { "id": 1648772, "postDate": "2022-01-13T17:24:41.907Z", "content": "Seems resolution is not all you need, but damn you have to have it haha :)
", "rawMarkdown": "Seems resolution is not all you need, but damn you have to have it haha :)" } ] }, { "id": 1651340, "postDate": "2022-01-15T17:24:25.830Z", "content": "The LB is funny now!
\nanyone who ends up in the top 50 now will be placed above 100 after an hour
Public LB is merely a public lb.
", "rawMarkdown": "Public LB is merely a public lb.", "votes": 5 } ] }, { "id": 1651265, "postDate": "2022-01-15T16:44:37.640Z", "content": "The most funny thing is when you run val.py --img 2500 (3200, 4600, 7200) on colab or kaggle score goes down, what happens here it goes up is kind of irrational. Either metrics in yolo are wrong or something weird is happening. But when you submit on higher resolution you got gain on LB
", "rawMarkdown": "The most funny thing is when you run val.py --img 2500 (3200, 4600, 7200) on colab or kaggle score goes down, what happens here it goes up is kind of irrational. Either metrics in yolo are wrong or something weird is happening. But when you submit on higher resolution you got gain on LB", "votes": 2, "replies": [ { "id": 1651278, "postDate": "2022-01-15T16:50:19.430Z", "content": "that shouldn't really be happening cause on my runs P and R were both pretty much above 70% in each epoch (and close to 80-90 during best epoch for Precision) using up-sizing so that translates to ~0.7-0.8 on F2 metric. This gives 0.58-0.61 on public LB using different sizes so correlation is pretty stable.
\nI also recently modified
\n # Print results\n pf = '%20s' + '%11i' * 2 + '%11.3g' * 4 # print format\n f2 = 5*mp*mr/(4*mr+mp)\n LOGGER.info(pf % ('all', seen, nt.sum(), mp, mr, map50, f2)) #map))\nand again some modifications in the part where model saving is done
\nin my val.py in yolo folder so that f2 is logged instead of MAP (though I'm not sure if this is the best way to implement the metric). This gives the 0.7-0.8 I was talking about earlier.
\nHow did you get the f2 score while using val.py? By default it gives map if i'm not wrong. Might be giving lower CV locally due to some leak or maybe wrong implementation?
", "rawMarkdown": "that shouldn't really be happening cause on my runs P and R were both pretty much above 70% in each epoch (and close to 80-90 during best epoch for Precision) using up-sizing so that translates to ~0.7-0.8 on F2 metric. This gives 0.58-0.61 on public LB using different sizes so correlation is pretty stable. \n\nI also recently modified\n```\n\n # Print results\n pf = '%20s' + '%11i' * 2 + '%11.3g' * 4 # print format\n f2 = 5*mp*mr/(4*mr+mp)\n LOGGER.info(pf % ('all', seen, nt.sum(), mp, mr, map50, f2)) #map))\n\n```\nand again some modifications in the part where model saving is done\n\nin my val.py in yolo folder so that f2 is logged instead of MAP (though I'm not sure if this is the best way to implement the metric). This gives the 0.7-0.8 I was talking about earlier. \n\nHow did you get the f2 score while using val.py? By default it gives map if i'm not wrong. Might be giving lower CV locally due to some leak or maybe wrong implementation? ", "votes": 2 }, { "id": 1651285, "postDate": "2022-01-15T16:53:22.473Z", "content": "Aditya i am talking about what @hengck23 done in experiment. He increased inference size on same weight and got gain and gain. When you do it on val.py recall and precission drops when you increase --img parameter. (tested on colab). Even our #7 got much worse f2 score (on that size) than pure f2 score after training.
", "rawMarkdown": "Aditya i am talking about what @hengck23 done in experiment. He increased inference size on same weight and got gain and gain. When you do it on val.py recall and precission drops when you increase --img parameter. (tested on colab). Even our #7 got much worse f2 score (on that size) than pure f2 score after training." }, { "id": 1651292, "postDate": "2022-01-15T16:58:55.290Z", "content": "Aditya i made implementation which plots f2 and f1 score on PNG. That doesnt matter i run my model which has recall 0.75, and increased resolution 2.5 times and recall was almost ZERO! :) either yolo got something screwed or colab or idk even i was suspicious in comeptition metric but watchin bboxes it does it correctly ;)
", "rawMarkdown": "Aditya i made implementation which plots f2 and f1 score on PNG. That doesnt matter i run my model which has recall 0.75, and increased resolution 2.5 times and recall was almost ZERO! :) either yolo got something screwed or colab or idk even i was suspicious in comeptition metric but watchin bboxes it does it correctly ;)" }, { "id": 1651295, "postDate": "2022-01-15T17:00:39.487Z", "content": "I've not yet tried using val.py post training has completed and was trusting the per epoch calculation it does by default(i.e. img_size is also = the set higher size).
\n![\"Screenshot-from-2022-01-15-22-35-30\"](\"https://i.ibb.co/cT3J48n/Screenshot-from-2022-01-15-22-35-30.png\")
(P, R and f2 score I get during training after modifying the val.py file ^)
\nI'll try using val on one of my saved models to see if there's a dip in the P, R.
", "rawMarkdown": "I've not yet tried using val.py post training has completed and was trusting the per epoch calculation it does by default(i.e. img_size is also = the set higher size).\n\n\n(P, R and f2 score I get during training after modifying the val.py file ^)\n\nI'll try using val on one of my saved models to see if there's a dip in the P, R."
},
{
"id": 1651299,
"postDate": "2022-01-15T17:05:00.237Z",
"content": "Hmm, close to 0 recall is sus π€ usually when metric calculation was this messed up in my experiments with mmdet it turned out to be some inconsistent dependencies or something but i've not encountered such issues in yolo yet
", "rawMarkdown": "Hmm, close to 0 recall is sus π€ usually when metric calculation was this messed up in my experiments with mmdet it turned out to be some inconsistent dependencies or something but i've not encountered such issues in yolo yet" }, { "id": 1651309, "postDate": "2022-01-15T17:10:29.807Z", "content": "ADitya you must have some leak, AP95 over 0.8 ? our #7 model got 0.35 ;)
", "rawMarkdown": "ADitya you must have some leak, AP95 over 0.8 ? our #7 model got 0.35 ;)" }, { "id": 1651317, "postDate": "2022-01-15T17:12:47.907Z", "content": "yeah that's the part i've modified, it's printing f2 score instead of AP95 :P For me AP95 was around 0.34-0.36 as well in previous runs when I hadn't modified the logging code
", "rawMarkdown": "yeah that's the part i've modified, it's printing f2 score instead of AP95 :P For me AP95 was around 0.34-0.36 as well in previous runs when I hadn't modified the logging code", "votes": 1 }, { "id": 1651320, "postDate": "2022-01-15T17:13:26.307Z", "content": "Yes, i also noticed it. I also tried training the model with img_size 7000 but it didn't perform very well than my 3600 secret model. i guess yolo doesn't like losing so much quality due to enlarging the images. i wonder why competition metric likes
", "rawMarkdown": "Yes, i also noticed it. I also tried training the model with img_size 7000 but it didn't perform very well than my 3600 secret model. i guess yolo doesn't like losing so much quality due to enlarging the images. i wonder why competition metric likes", "votes": 1 }, { "id": 1651330, "postDate": "2022-01-15T17:17:21.153Z", "content": "@morizin yeah we were suspicious even if we were on #1 that something is wrong or we cannot explain it
", "rawMarkdown": "@morizin yeah we were suspicious even if we were on #1 that something is wrong or we cannot explain it", "votes": 2 }, { "id": 1651342, "postDate": "2022-01-15T17:29:48.867Z", "content": "I'll try using this code to see if my val scores are correct though I'm pretty sure it'll be different than my epoch calculations cause it's mean over IOUs from 0.3-0.8. But still I felt changing it was appropriate since it might give a better sense of model performance compared to normal AP95 metric
", "rawMarkdown": "I'll try using [this code](https://www.kaggle.com/hyunmingu/evaluate-f2-score-for-yolov5-model) to see if my val scores are correct though I'm pretty sure it'll be different than my epoch calculations cause it's mean over IOUs from 0.3-0.8. But still I felt changing it was appropriate since it might give a better sense of model performance compared to normal AP95 metric" }, { "id": 1651347, "postDate": "2022-01-15T17:34:15.590Z", "content": "@ferlockx try this with high img
\n!python val.py --img 4800 --batch 1 --data /gdrive/MyDrive/Datasets/Great_Coral_Reef/yolo5_cfg/data.yaml --weights /gdrive/MyDrive/Datasets/Great_Coral_Reef/yolov5/runs/train/exp29/weights/epoch7.pt --iou 0.3 --conf 0.001
and go with img higher. Replace yaml and weights file. conf = 0.001 is becuase it draws metrics from 0 to 1 in confidence thresholds
\n-- hint if you dont have f2 metric plots you can watch f1 score . How looks f2 score ? same just hit f1 score from right ;) f2 got peak before f1
\n-- hint2 in metrics.py tweak place where is f1 plotting and change it to (5pr/(4p+r)) :) i added some lines to generate the metric after validation
okay thanks i'll try this out :)
", "rawMarkdown": "okay thanks i'll try this out :) " }, { "id": 1651671, "postDate": "2022-01-16T01:01:50.710Z", "content": "hey @lukaszborecki , you can check here that CV was coming as 0.77 while post training validation.
\nWhile this was obviously a drop from mid 0.8s I was getting while training, this was expected since while training yolo was doing no metric sweeps over 0.3-0.8 and then taking their mean. I don't think higher res is hurting CV results. I'll try switching img up to 4k, 5k etc to see if those values reduce f2.
\nWhile there's no data leakage since splitting was done using same code, I would love your input on whether the metric implementation used in these public notebooks is correct :)
\nUPD: yup upsizing from img size value used during training does indeed decrease f2, updated my notebook to reflect the same. wonder why LB is immune to this π€
", "rawMarkdown": "hey @lukaszborecki , you can check [here](https://www.kaggle.com/ferlockx/f2-cv-0-77-subseq-split-and-higher-res/notebook) that CV was coming as 0.77 while post training validation. \n\nWhile this was obviously a drop from mid 0.8s I was getting while training, this was expected since while training yolo was doing no metric sweeps over 0.3-0.8 and then taking their mean. I don't think higher res is hurting CV results. I'll try switching img up to 4k, 5k etc to see if those values reduce f2. \n\nWhile there's no data leakage since splitting was done using same code, I would love your input on whether the metric implementation used in these public notebooks is correct :) \n\nUPD: yup upsizing from img size value used during training does indeed decrease f2, updated my notebook to reflect the same. wonder why LB is immune to this π€\n", "votes": 1 }, { "id": 1651867, "postDate": "2022-01-16T06:16:40.317Z", "content": "So your expirement prooved it is droping while increasing inference size :)
\n@ferlockx F1 score drops too because its correlated with f2. Either there are small starfishes in 25% or there is less populated video or cv metric is wrong either ours either theirs ;). But if you split F2 into TP FP FN it is clear f2 should drop
", "rawMarkdown": "So your expirement prooved it is droping while increasing inference size :)\n\n@ferlockx F1 score drops too because its correlated with f2. Either there are small starfishes in 25% or there is less populated video or cv metric is wrong either ours either theirs ;). But if you split F2 into TP FP FN it is clear f2 should drop" }, { "id": 1652005, "postDate": "2022-01-16T08:55:07.897Z", "content": "UPD2: @lukaszborecki it seems that the initial public metric implementation was a tiny bit wrong in that it didn't discard annotations under a confidence metric post inference (0.15 in most public notebooks). I've added that confidence in my latest version and the decrease is a bit less drastic but the trend is still indeed decreasing. It's now 0.77 -> 0.72 -> 0.64 β¦ instead of 0.77 -> 0.5 -> 0.4 β¦ :)
\nAlso the number of images in which there are no annotations keeps decreasing with increase in image size i.e. it does indeed help with small object detection as expected
\nI think there might be a few more things that are different in inference environment from our local metric calculation that our causing decreasing CV, but still we should consider more augmentation techniques just to be sure ;)
\nThanks Lukasz I'll try to correct the things you're mentioning
", "rawMarkdown": "UPD2: @lukaszborecki it seems that the initial public metric implementation was a tiny bit wrong in that it didn't discard annotations under a confidence metric post inference (0.15 in most public notebooks). I've added that confidence in my latest version and the decrease is a bit less drastic but the trend is still indeed decreasing. It's now 0.77 -> 0.72 -> 0.64 ... instead of 0.77 -> 0.5 -> 0.4 ... :) \n\nAlso the number of images in which there are no annotations keeps decreasing with increase in image size i.e. it does indeed help with small object detection as expected\n\nI think there might be a few more things that are different in inference environment from our local metric calculation that our causing decreasing CV, but still we should consider more augmentation techniques just to be sure ;)\n\nThanks Lukasz I'll try to correct the things you're mentioning", "votes": 1 }, { "id": 1652007, "postDate": "2022-01-16T08:57:30.290Z", "content": "I tried increase img size in inference with looking at images and it was also getting worse (@ some point). To implementation yes there is more errors, first you have to count np.nanmean and second you need loop with confidence :)
\n-- anyway still the trend is decreasing and LB is increasing
", "rawMarkdown": "I tried increase img size in inference with looking at images and it was also getting worse (@ some point). To implementation yes there is more errors, first you have to count np.nanmean and second you need loop with confidence :)\n\n-- anyway still the trend is decreasing and LB is increasing" }, { "id": 1652009, "postDate": "2022-01-16T09:01:01.027Z", "content": "how about asking the host to release their metric computation code?
", "rawMarkdown": "how about asking the host to release their metric computation code?", "votes": 6, "replies": [ { "id": 1652020, "postDate": "2022-01-16T09:10:44.337Z", "content": "\n\nhow about asking the host to release their metric computation code?
\n
@hengck23 do you think it is possible ? they probably will answer the same as in paper :)
", "rawMarkdown": "> how about asking the host to release their metric computation code?\n\n@hengck23 do you think it is possible ? they probably will answer the same as in paper :)", "votes": 1 } ] }, { "id": 1652011, "postDate": "2022-01-16T09:02:30.123Z", "content": "I think the drop while increasing happens because there are lots of false positive
\ni visualized with @steamedsheep weights conf 0.25
\nFYI => RED - GT Boxes
\n BLUE - PRED_BOXES
\n![\"\"](\"https://i.ibb.co/2WvCXJ9/image.png\")
another hypothesis:
\nif you look at f2 per image, there should be some images that increase f2 while there are some decreases. maybe the public LB resembles more of those that improve?
@morizin interesting so you set model.conf =0.25 directly instead of removing below 0.25 post inference?
", "rawMarkdown": "@morizin interesting so you set model.conf =0.25 directly instead of removing below 0.25 post inference?" }, { "id": 1652038, "postDate": "2022-01-16T09:30:28.060Z", "content": "Firstly I thnink that Australian Government won't be happy with current solution :) Maybe I am wrong (totally wrong) but I created video on validation dataset using our best LB solution - it is totally crap! ππ
\nSimultanieusly we created video with our best local (crap according to LB score) model and β¦ it absolutely outperfoms our \"best LB\" β¦ and people annotating images (I know that such extra bboxes were unfortunately associatetd with FP - but it does not influence on f2 score much).
No. model.conf = 0.01 and filter boxes under 0.25
", "rawMarkdown": "No. model.conf = 0.01 and filter boxes under 0.25", "votes": 1 }, { "id": 1652044, "postDate": "2022-01-16T09:34:19.790Z", "content": "i didn't really check does csiro or tensorflow release any git repo for this data/competition?
", "rawMarkdown": "i didn't really check does csiro or tensorflow release any git repo for this data/competition?" }, { "id": 1652062, "postDate": "2022-01-16T09:46:16.430Z", "content": "\nfor the image you showed, maybe f2 is still high. one correct TP is weighted 5x. if other frames don't have that many FP, then f2 is improved.
\non a side note: one should filter off those very small boxes
", "rawMarkdown": "@morizin \n\nfor the image you showed, maybe f2 is still high. one correct TP is weighted 5x. if other frames don't have that many FP, then f2 is improved.\n\n---\n\non a side note: one should filter off those very small boxes", "votes": 2, "replies": [ { "id": 1652385, "postDate": "2022-01-16T15:03:21.997Z", "content": "\n \n\nfor the image you showed, maybe f2 is still high. one correct TP is weighted 5x. if other frames don't have that many FP, then f2 is improved.
\n
\non a side note: one should filter off those very small boxes
\n
Perhaps you are right @hengck23
\n![\"\"](\"https://i.ibb.co/SNtwhfg/image.png\")
\nthis was one of my models, i like this model very much. Predictions are almost perfect. but scores 0.418 in LB
I calculated f2 on the validation video(whether num_bbox>0 or >=0) and found that larger scales will bring worse results, which makes me very worried about the current LB.
", "rawMarkdown": "I calculated f2 on the validation video(whether num_bbox>0 or >=0) and found that larger scales will bring worse results, which makes me very worried about the current LB.", "votes": 2 }, { "id": 1652251, "postDate": "2022-01-16T13:11:46.537Z", "content": "@zzy same observations here and same observations in expirement. And the better model you make it usually goes worse with LB
", "rawMarkdown": "@zzy same observations here and same observations in expirement. And the better model you make it usually goes worse with LB", "votes": 1 }, { "id": 1652257, "postDate": "2022-01-16T13:21:39.020Z", "content": "@zzy990106 I do not how you and your team but we with @lukaszborecki have problem with eveluation and chosing right model. It is kind of luck :)
", "rawMarkdown": "@zzy990106 I do not how you and your team but we with @lukaszborecki have problem with eveluation and chosing right model. It is kind of luck :)" }, { "id": 1652258, "postDate": "2022-01-16T13:21:40.627Z", "content": "maybe that is the reason for 4 submission
", "rawMarkdown": "maybe that is the reason for 4 submission", "votes": 2 }, { "id": 1652259, "postDate": "2022-01-16T13:24:57.373Z", "content": "@hengck23 ππππ yes, it could be a great explanation to this situation β¦. You have to take one crap model but good on LB, second best looking model for team and two β¦ between β¦.
\nI know that for most competition looking on LB score is not good idea β¦. but in this case is strage β¦ because you definitely can build better model (using some ways to decrease FP and increase TP) but it does not work at all β¦. it dramatically drops on LB.
", "rawMarkdown": "@hengck23 ππππ yes, it could be a great explanation to this situation .... You have to take one crap model but good on LB, second best looking model for team and two ... between .... \n\nI know that for most competition looking on LB score is not good idea .... but in this case is strage ... because you definitely can build better model (using some ways to decrease FP and increase TP) but it does not work at all .... it dramatically drops on LB. ", "votes": 3 }, { "id": 1652303, "postDate": "2022-01-16T13:55:35.883Z", "content": "if i am the host and i want to be evil I would put all non-empty images as public test and all empty images as private.
\nI think we need to probe the ratio of empty and non empty images in private and public. (note that we do know approximately how many empty images are there in the test, we know there are 14k test images with about 150 cots objects. we just don't know how they are split between public and private)
\nfor confidently predicted empty images, we add FP.
\nfor confidently predicted TP, we drop them.
\nthe change in score should give us some clue?
\n\nfor confidently predicted empty images, we add FP.
\n
\n for confidently predicted TP, we drop them.
\n the change in score should give us some clue?
We have made such research.
\nWe increasedTP, decreased FP/FN (using many hacks) β¦ and β¦ rised f2 β¦ (on many folds confs). Posted and β¦. ππ¨πππππππππ
@hengck23 split private and public by video or sequence is enough to direct a shake up
", "rawMarkdown": "@hengck23 split private and public by video or sequence is enough to direct a shake up", "votes": 1 }, { "id": 1652407, "postDate": "2022-01-16T15:35:20.153Z", "content": "\"split private and public by video or sequence is enough to direct a shake up\"
\nthe splits are usually by random. else it is difficult to come up with 25% and 75% ratios.
\nanother way is that they are split by order, e.g. first 25% of seq or video
\n\nthe splits are usually by random.
\n
Usually it is, but there are some impressive outliers like this https://www.kaggle.com/c/microsoft-malware-prediction
", "rawMarkdown": "> the splits are usually by random. \n\nUsually it is, but there are some impressive outliers like this https://www.kaggle.com/c/microsoft-malware-prediction\n", "votes": 2 }, { "id": 1652418, "postDate": "2022-01-16T15:45:39.963Z", "content": "from data info:
\n\n\nThis competition uses a hidden test set that will be served by an API to ensure you evaluate the images in the same order they were recorded within each video.
\n
maybe this means/ensures that they are split by video/seq order (?)
", "rawMarkdown": "from data info:\n> This competition uses a hidden test set that will be served by an API to ensure you evaluate the images in the same order they were recorded within each video. \n\nmaybe this means/ensures that they are split by video/seq order (?)", "votes": 1 }, { "id": 1652426, "postDate": "2022-01-16T15:54:44.187Z", "content": "Agree, at least part of the sequence.
", "rawMarkdown": "Agree, at least part of the sequence.", "votes": 1 }, { "id": 1652443, "postDate": "2022-01-16T16:09:58.733Z", "content": "\"This competition uses a hidden test set that will be served by an API to ensure you evaluate the images in the same order they were recorded within each video.\"
\nthe public score could be based on every 4-th frame (25%) evaluation
", "rawMarkdown": "\"This competition uses a hidden test set that will be served by an API to ensure you evaluate the images in the same order they were recorded within each video.\"\n\nthe public score could be based on every 4-th frame (25%) evaluation", "votes": 5, "replies": [ { "id": 1653054, "postDate": "2022-01-17T07:51:26.697Z", "content": "\n\n\nthe public score could be based on every 4-th frame (25%) evaluation
\n
If so, we should fully trust LB. π
", "rawMarkdown": "@hengck23\n> the public score could be based on every 4-th frame (25%) evaluation\n\nIf so, we should fully trust LB. π" } ] }, { "id": 1652998, "postDate": "2022-01-17T06:51:59.750Z", "content": "\n\n\nwe know there are 14k test images with about 150 cots objects
\n
Did i miss this info.. 150 cots in 13k images of test set ?
\nI like your 4th frame CV, didn't expected LB can be split in that way
", "rawMarkdown": "@hengck23 \n> we know there are 14k test images with about 150 cots objects\n\nDid i miss this info.. 150 cots in 13k images of test set ?\n\nI like your 4th frame CV, didn't expected LB can be split in that way " }, { "id": 1653664, "postDate": "2022-01-17T19:13:25.303Z", "content": "I don't see the public LB being based on every 4th frame, wouldn't this screw up any temporal based solutions such as the yolox + tracking?
\nPlus it would promote overfitting to the public LB which generally isn't the case on kaggle.
", "rawMarkdown": "I don't see the public LB being based on every 4th frame, wouldn't this screw up any temporal based solutions such as the yolox + tracking?\n\nPlus it would promote overfitting to the public LB which generally isn't the case on kaggle.", "votes": 3 } ] }, { "id": 1656565, "postDate": "2022-01-19T12:51:42.223Z", "content": "Thank you for sharing the good notebook :)
\nI wish you will be entering the gold zone.
\nAnd I wish I could be entering the top 20% of this competition π
it may be easier to analyze by considering FP rate, recall, precision per box size.
\ne.g. we divide bbox into few category by sizes:
\nbelow 20,
\nfrom 30 to 32 β¦.
if you cannot explain the results (by theory or massive and careful experiment), it would be subjected to big shakeup. after all, test performance = (public+private) performance
\nhigher public score may means lower private one
", "rawMarkdown": "it may be easier to analyze by considering FP rate, recall, precision per box size.\n\ne.g. we divide bbox into few category by sizes:\nbelow 20,\nfrom 30 to 32 ....\n\nif you cannot explain the results (by theory or massive and careful experiment), it would be subjected to big shakeup. after all, test performance = (public+private) performance\n\nhigher public score may means lower private one", "votes": 1, "replies": [ { "id": 1652887, "postDate": "2022-01-17T04:49:21.343Z", "content": "classify first ? and then detect?
", "rawMarkdown": "classify first ? and then detect?", "votes": 1 } ] }, { "id": 1648551, "postDate": "2022-01-13T13:50:35.333Z", "content": "wow, 3600π²
", "rawMarkdown": "wow, 3600π²", "votes": 1 }, { "id": 1668827, "postDate": "2022-01-30T04:42:15.777Z", "content": "Are you use the big batch size and high resolution to train your model, I can not repeat your result with the batch size = 8 and picture size = 720*1280.I can not take more big size because of the limitation of GPU.
\n I just confuse that if the GPU limit my model 's performance. If that's True, I think I have no idea to train model as good as yours.
Hey
\ncan u help me understand which images we take for validation ??
\nIs it 80 train-20 test from the 3 video images provided for test ??
\nIm a newbie so dint understand the submission format
Yes I used it too and am really impressed with the performance and results
", "rawMarkdown": "Yes I used it too and am really impressed with the performance and results" }, { "id": 1657856, "postDate": "2022-01-20T14:14:33.203Z", "content": "Good job! Why did you guys all use yolov5s but not v5x?
", "rawMarkdown": "Good job! Why did you guys all use yolov5s but not v5x?", "replies": [ { "id": 1657891, "postDate": "2022-01-20T14:52:57.053Z", "content": "I used v5m. That was enough
", "rawMarkdown": "I used v5m. That was enough", "votes": 2 } ] }, { "id": 1653264, "postDate": "2022-01-17T11:50:08.933Z", "content": "I'll try 9000 and see how it works :-)
", "rawMarkdown": "I'll try 9000 and see how it works :-)" }, { "id": 1653048, "postDate": "2022-01-17T07:42:05.030Z", "content": ":) rookie here
", "rawMarkdown": ":) rookie here" }, { "id": 1652121, "postDate": "2022-01-16T11:10:14.263Z", "content": "impressive though you could apply skimage as well
", "rawMarkdown": "impressive though you could apply skimage as well" }, { "id": 1651326, "postDate": "2022-01-15T17:16:06.830Z", "content": "Great kernel with a simple code.
\nBut i think we should think this method can be generalized well to private lb dataset
\nWe just have 9 hours for a kernel inference.
nice, work
", "rawMarkdown": "nice, work\n\n" }, { "id": 1650620, "postDate": "2022-01-15T07:19:33.370Z", "content": "TKS for your share
", "rawMarkdown": "TKS for your share" }, { "id": 1650382, "postDate": "2022-01-15T02:33:27.130Z", "content": "Thanks for sharing, your work is great!π
", "rawMarkdown": "Thanks for sharing, your work is great!π" }, { "id": 1649299, "postDate": "2022-01-14T07:06:04.113Z", "content": "Simply enlarging input size produce better results. It's amazing but I suspect that this indicates the model architecture of YOLO could be improved, at least for this competition data.
", "rawMarkdown": "Simply enlarging input size produce better results. It's amazing but I suspect that this indicates the model architecture of YOLO could be improved, at least for this competition data." }, { "id": 1649267, "postDate": "2022-01-14T06:26:16.533Z", "content": "it is time to you guysπͺ use 640 --> 3x or 4x or 5x training?
", "rawMarkdown": "it is time to you guysπͺ use 640 --> 3x or 4x or 5x training?" }, { "id": 1648563, "postDate": "2022-01-13T14:06:51.560Z", "content": "amazingοΌππ
", "rawMarkdown": "amazingοΌππ" }, { "id": 1648559, "postDate": "2022-01-13T14:04:36.070Z", "content": "Agree, Yolo is all you needπ
", "rawMarkdown": "Agree, Yolo is all you needπ" }, { "id": 1648537, "postDate": "2022-01-13T13:38:24.323Z", "content": "Look forward toβ¦
", "rawMarkdown": "Look forward to..." }, { "id": 1658429, "postDate": "2022-01-21T02:57:29.410Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1656354, "postDate": "2022-01-19T09:38:06.903Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1648561, "postDate": "2022-01-13T14:05:43.487Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1664878, "postDate": "2022-01-26T11:04:41Z", "content": "Thank you!
", "rawMarkdown": "Thank you!" }, { "id": 1660980, "postDate": "2022-01-23T06:15:47.760Z", "content": "Thank you!
", "rawMarkdown": "Thank you!" }, { "id": 1659857, "postDate": "2022-01-22T06:43:45.977Z", "content": "It can work, thanks!
", "rawMarkdown": "It can work, thanks!" }, { "id": 1652441, "postDate": "2022-01-16T16:07:15.440Z", "content": "Thanks! good code
", "rawMarkdown": "Thanks! good code" }, { "id": 1649512, "postDate": "2022-01-14T10:35:15.160Z", "content": "thanks for sharing
", "rawMarkdown": "thanks for sharing" } ], "comments": [ { "id": 1649181, "author_name": "hengck23", "author_url": "", "post_date": "2022-01-14T03:43:02.130000", "content": "here is the magic, using your same model and same code (i just change the input size)
\npath='../input/reef-baseline-fold12/l6_3600_uflip_vm5_f12_up/f1/best.pt',\n3600 LB 0.579\n4000 (1.11x) LB 0.586\n4800 (1.33x) LB 0.590\n5600 (1.55x) LB 0.594\n6400 (1.77x) LB 0.596\n7200 (2.00x) LB 0.603 !!!!??\n9000 (2.50x) LB 0.613 !!!!???? why it never ends\n...\n...\n# optimun is between 2.50x to 3.50x, please feel free to explore. \n# i estimate we can get to 6.20\n...\n12600(3.50x) LB 0.557 \nbut i haven't used yolo5 yet and i don't know about your train data. etc.
\nso i cannot tell if the results is overfitting (#1) or having good generalisation.
\nhence interprete the results with care!
(#1) overfitting as in the with enlarged image, the model is producing many FP. BUT for this public test video,
\nthe improvement in TP outweighs that of FP. We can't tell for sure in the private set.
we need to calibrate FP of the models under different input setting, etc
\nmost top kagglers have tricks in their bags, accumulated from work and competition experiences.
\nThis is results from hard work after numerous experiments.
Now i let one cat of the bag:
\neven in past competitions, i have been using this upsizing trick.
\nit is interesting to note that when we enlarge input image, there is no addition information.
\nbecause the new pixels are interpolated (i.e. information is from the original pixels).
\nyet, i have been seeing consisently improvement in results, e.g. in classification, segmentation, object detection
Hi, I have a quick question:
\nDo we have to use the YOLO detector all the time?
\nWill other detection models(e.g, Faster RCNN) achieve better results?
it depends on the data (and your augmentation).
\nall datascience problem as three parts:
\nThank you for your prompt reply!
\nI don't think the training data is enough, even though there are 4919 labeled images, they are from videos and have similarities to each other. after reading your previous post, I decided to collect some additional data from the web.
\nThanks again for the advice! I've learned a lot from you!
I wonder if we resize a 720x1280 image into 6000px. the interpolation of so many pixels will reduce the quality of the image. Also, the image is in .jpg format(which is already compressed and some pixels already lost).
\nAm I correct? Such a long interpolation generates some noise pixels, isn't it?
GO GO GO!
\nLet's see the limits.
\n10000 ?
\n20000 ?
Good direction β¦. I see that soon we can copy the best solution from this competition: https://www.kaggle.com/c/sartorius-cell-instance-segmentation πππ
\n
for people with more GPUs
\n
besides image upsize, what other test time augment can lead to more TP than FP?
", "votes": 1, "replies": [] }, { "id": 1649672, "author_name": "Remek Kinas", "author_url": "", "post_date": "2022-01-14T13:49:57.140000", "content": "Very good question @hengck23 π
", "votes": 2, "replies": [] }, { "id": 1649697, "author_name": "Nirjhar Roy", "author_url": "", "post_date": "2022-01-14T14:19:41.257000", "content": "@remekkinas I bet it will be an Astro Cell Type :P
", "votes": 1, "replies": [] }, { "id": 1649726, "author_name": "Remek Kinas", "author_url": "", "post_date": "2022-01-14T14:45:27.337000", "content": "@phoenix9032 this prooves that you have spent a lot of time in this competition. You deserve additional prize :) πͺπ€ππ
", "votes": 2, "replies": [] }, { "id": 1649757, "author_name": "Mohammed Rizin V K", "author_url": "", "post_date": "2022-01-14T15:07:15.117000", "content": "@hengck23 You made the LB alive and competitive.
", "votes": 1, "replies": [] }, { "id": 1649916, "author_name": "hengck23", "author_url": "", "post_date": "2022-01-14T17:16:17.550000", "content": "just realize that we have a video and not single image, and hence:
\nvideo resolution or learning better upsizing filter!
(e.g. using consistency loss to distill 5x input smiliar to 1x input)
\nLearning to Resize Images for Computer Vision Tasks
\nhttps://arxiv.org/pdf/2003.08237.pdf
sooner or later we're bound to hit memory runtime limits in Kaggle environment after 9k too. I was able to get my current score using up sizing trick (to 4032) and some other tta inspired from yolo repo and hengck's augmentation for 0.56.
\n\n\nLearning to Resize Images for Computer Vision Tasks
\n
\n https://arxiv.org/pdf/2003.08237.pdf
this is a useful paper imo
\nUPD: looks like up-sizing doesn't work well if initial training size is low so there is indeed a limit for higher res training as well
", "votes": 3, "replies": [] }, { "id": 1650319, "author_name": "liuzhangzhen", "author_url": "", "post_date": "2022-01-15T00:50:32.927000", "content": "6800(LB0.601)
\n7600(LB0.610)
10200 (LB 0.602)
\nI guess it end here π
It finally reaches end. Maybe probing confidence threshold make it better.
\n10800(LB 0.602)
I just woke up and see everyone jump on the LB :D
", "votes": 0, "replies": [] }, { "id": 1650437, "author_name": "Max van Dijck", "author_url": "", "post_date": "2022-01-15T03:38:09.723000", "content": "It seems that using two stage detectors results in a performace trade off which won't allow for the upsizing of images as effectively⦠however it is always worth experimenting with. My best two stage detector reached 0.578 but currently is a 2 hour runtime.
", "votes": 1, "replies": [] }, { "id": 1650530, "author_name": "sheep", "author_url": "", "post_date": "2022-01-15T05:17:19.597000", "content": "Sorry, I have not foresee this, I should not release this. Apology to the turbulence of leaderboard
", "votes": 6, "replies": [] }, { "id": 1650549, "author_name": "Alex Wong", "author_url": "", "post_date": "2022-01-15T05:36:50.510000", "content": "My understanding is that the private LB will be run with 75% of the test dataset? In that case, any notebook that takes > 3 hours to complete will breach the 9 hr limit on the private LB?
", "votes": 1, "replies": [] }, { "id": 1650577, "author_name": "Lukasz Borecki", "author_url": "", "post_date": "2022-01-15T06:12:10.853000", "content": "Nope, it is already counted but hidden don't worry :)
", "votes": 3, "replies": [] }, { "id": 1650798, "author_name": "Alex Wong", "author_url": "", "post_date": "2022-01-15T10:32:35.233000", "content": "Thanks, that's good to know. Does that mean any notebook that runs to completion can be counted for the final / private LB?
\n9 hrs is a ridiculous amount of time.
", "votes": 1, "replies": [] }, { "id": 1650882, "author_name": "Hannah B", "author_url": "", "post_date": "2022-01-15T11:45:42.123000", "content": "@steamedsheep actually this is an important contribution since even for public notebooks or well known frameworks, it is our job as participants to find the best hyperparams.
\nHowever if in the final solution we were to see a few gold/silver medals won solely due to a single parameter change we'll feel a bit defeated and maybe even call that participant's hard work luck or \"easy-work\". Your contribution helped normalise this parameter tuning discrepancy and now we can move on to implementing techniques on top of these hyperparams :)
", "votes": 3, "replies": [] }, { "id": 1650885, "author_name": "Hannah B", "author_url": "", "post_date": "2022-01-15T11:51:58.240000", "content": "@alexchwong yup you will have an option to select 4 notebooks at the end of the competition out of which best score on private holdout(which has already been calculated) will determine your private lb rank (you can select them now too but you'll probably get 4 better scores in the next month)
", "votes": 1, "replies": [] }, { "id": 1650898, "author_name": "dragon zhang", "author_url": "", "post_date": "2022-01-15T12:01:56.743000", "content": "great, kind of fractal enlargement.
", "votes": 0, "replies": [] }, { "id": 1651176, "author_name": "Garvit Garg", "author_url": "", "post_date": "2022-01-15T15:49:58.533000", "content": "@alexchwong Please refer to this discussion
\nhttps://www.kaggle.com/c/tensorflow-great-barrier-reef/discussion/301020
Thanks guys thats very helpful
", "votes": 0, "replies": [] } ] }, { "id": 2506668, "author_name": "zhiyue666", "author_url": "", "post_date": "2023-10-31T13:18:15.977000", "content": "thanks a lot, sheep big old and frog god
", "votes": 1, "replies": [ { "id": 2558274, "author_name": "secondcaiji", "author_url": "", "post_date": "2023-12-12T03:28:53.543000", "content": "you are so cool,my god
", "votes": 0, "replies": [] } ] }, { "id": 1656668, "author_name": "hengck23", "author_url": "", "post_date": "2022-01-19T14:32:45.487000", "content": "i think i have solved the puzzle
\n
the safest bet is to use public LB to measure your recall only. I think there is insufficient images to measure FPrate and precision. Hence use another dataset for that
", "votes": 23, "replies": [ { "id": 1656688, "author_name": "Lukasz Borecki", "author_url": "", "post_date": "2022-01-19T14:49:02.497000", "content": "I am starting to think your tactic is to confuse everybody @hengck23 , you throwed so many ideas maybe with intention that somebody will test it. But your graph now proves that IMG_SIZE is not all you need :)
", "votes": 3, "replies": [] }, { "id": 1656701, "author_name": "hengck23", "author_url": "", "post_date": "2022-01-19T14:57:24.297000", "content": "large image size improve recall. but if you visually inspect the results, you do see many FP.
\nif you measure local validation, there is improvement in the local cv f2 score but not to the extent of 2.5x times. (I am referring to the results on my experiment on sheep models, i.e train at 3600, inference at much higher resolution)
hence I conclude that the number of empty images in your local cv and public test set are different.
\nthere is clearly a huge difference in trend on local cv and public lb score.
if there is too many empty images in the private set, the gain in recall will be eaten away by too many FPs and you will have low f2 score. this will cause shakeup.
\nyou have to measure your fp rate of your solution (and keep it low enough). i don't think the public lb score reflect this well enough, due to the lack of sufficient images.
\nin short:
\n\n\nyour tactic is to confuse everybody
\n
I think @hengck23 is well-intentioned. All these are observations based on experiments, nothing to confuse others on purpose.
", "votes": 4, "replies": [] }, { "id": 1656709, "author_name": "Lukasz Borecki", "author_url": "", "post_date": "2022-01-19T15:02:53.760000", "content": "yes this is what we do everytime test with full fold (with empties also) and there is border where FP are less. Doesn't work :)
\n
Our models got high confidence from 0.4 to 0.6 removing FP but as you see its not the case this model got LB 0.59
", "votes": 3, "replies": [] }, { "id": 1656713, "author_name": "Lukasz Borecki", "author_url": "", "post_date": "2022-01-19T15:04:51.870000", "content": "@zzy990106 to bo accurate i said i am starting to think, i didnt claimed @hengck23 is confusing everybody
", "votes": 1, "replies": [] }, { "id": 1656717, "author_name": "Remek Kinas", "author_url": "", "post_date": "2022-01-19T15:11:04.437000", "content": "You gave some hypotheses, but you have no evidence to prove their correctness. Usually people show a hypothesis and then evidence to prove it right, eg increase in the result of their model, score , etc. Here are theses, but only theses and no facts β¦ This picture just show that β¦. rescaling is finally bad way of winning (you will be punished in priv LB). Now I can see many people who took this way β¦ \"oh no β¦. I will be punished β¦.\".
\nI am giving one example of an experiment carried out on a selected fold (we conducted this experiment on many different configuration). Which model do you think is significantly better on the LB?
\n
the issue now is that we don't know how many truth objects and how many empty images are there in the public test set. my experiments show that when there are insufficient images (and truth objects) the f2 score varies quite greatly.
\nin order to avoid shakeup, our model must perform well against various combinations of number of objects and empty images. (we need to plot f2 curve for different combinations)
\nusing larger inference image obviously don't reduce FP, so the gain in public LB score must come from the recall. a stable results must come from both improving recall and precision. i think the public LB score only reflects recall. so one has to be careful to interpret the results to avoid shakeup.
", "votes": 5, "replies": [] }, { "id": 1656726, "author_name": "Lukasz Borecki", "author_url": "", "post_date": "2022-01-19T15:21:23.730000", "content": "@hengck23 you think why i implemented f2 curve and why Remek made TP FP FN analysis ?:) we doing it from long time. Did you see our submission number ? we test it with many ways, and got many ways still to test
", "votes": 0, "replies": [] }, { "id": 1656729, "author_name": "hengck23", "author_url": "", "post_date": "2022-01-19T15:24:42.930000", "content": "\"experiment carried out on a selected fold\"
\nthe fold is fixed, so I would interpret the results with care.
\nalso, public data is also fixed.
besides FP,FN,FP I would try also different num of empty images and no. of true objects in the table.
\ni want to know how sensitive the metrics are when the dataset changes.
", "votes": 3, "replies": [] }, { "id": 1656732, "author_name": "hengck23", "author_url": "", "post_date": "2022-01-19T15:27:53.947000", "content": "\" This picture just show that β¦. rescaling is finally bad way of winning (you will be punished in priv LB). Now I can see many people who took this way β¦ \"oh no β¦. I will be punished β¦.\".\"
\nthere will be no problem if you can keep your curve straight (instead of falling down)
", "votes": 2, "replies": [] }, { "id": 1656734, "author_name": "Mohammed Rizin V K", "author_url": "", "post_date": "2022-01-19T15:29:28.357000", "content": "I have once trained a model. don't know if it cause because of my split. it was one of my initial experiments.
\ni trained a model which had a great recall and inferred it. Give me a bad score like 0.382 or something.
\nMy hypothesis is that, both public and private have the same density percentage of nonempty and empty images.
\nperhaps like
frame 1 -> Empty => private\nframe 2 -> Empty => private\nframe 3 -> Empty => private\nframe 4 -> Empty => private\nframe 5 -> Empty => public\nframe 6 -> Empty => public\nframe 7 -> Empty => public\nframe 8 ... frame m -> Empty => public\nframe m+1 -> Non Empty => private\nframe m+2 -> Non Empty => private\nframe m+3 -> Non Empty => private\nframe m+4 -> Non Empty => private\nframe m+5 -> Non Empty => public\nframe m+6 .. frame n ->Non Empty => public\nand so on\nThe reason why I say they into split first 4 consecutive frames is because why tracking working
", "votes": 2, "replies": [] }, { "id": 1658038, "author_name": "AnhMeow", "author_url": "", "post_date": "2022-01-20T16:56:01.790000", "content": "I think we are simply inferring sequentially on frames but public LB is a 'random' (or not) select of frames afterwards
", "votes": 0, "replies": [] }, { "id": 1658425, "author_name": "", "author_url": "", "post_date": "2022-01-21T02:55:42.767000", "content": "", "votes": 0, "replies": [] }, { "id": 1658430, "author_name": "Bilzard", "author_url": "", "post_date": "2022-01-21T02:58:50.103000", "content": "\n\nusing larger inference image obviously don't reduce FP, so the gain in public LB score must come from the recall.
\n
My experiments supports this hypothesis for the case of my model.
\nI estimated recall and precision in the Leader board [1], and have result below.
\nIn this result, recall is higher when we infer larger scale than that in train, whereas precision is nearly the same.
\nOf course we need more sample because it depends on the model.
\nI don't confirm this result is generally applicable.
train scale: x2.00, infer scale: x2.00
\n R P\nmean: 0.557 0.534\nstd: 0.028 0.106\ntrain scale: x2.00, infer scale: x3.20 (x1.60 larger than train scale)
\n R P\nmean: 0.623 0.527\nstd: 0.007 0.020\n[1] https://www.kaggle.com/c/tensorflow-great-barrier-reef/discussion/302130#1658385
", "votes": 5, "replies": [] } ] }, { "id": 1648542, "author_name": "Psi", "author_url": "", "post_date": "2022-01-13T13:41:07.850000", "content": "I should start with competitions one month before deadline, apparently it is the time people release good solutions.
", "votes": 18, "replies": [ { "id": 1648598, "author_name": "sheep", "author_url": "", "post_date": "2022-01-13T14:42:13.867000", "content": "This is not a good solution, just a custom run of yolo5.
", "votes": 4, "replies": [] }, { "id": 1649389, "author_name": "Mohammed Rizin V K", "author_url": "", "post_date": "2022-01-14T08:40:20.480000", "content": "@steamedsheep this is a brilliant notebook. After trying 100 methods, finally found one secret of this competition (although it is no longer a secret).
", "votes": 1, "replies": [] }, { "id": 1649758, "author_name": "CPMP", "author_url": "", "post_date": "2022-01-14T15:08:06.843000", "content": "\nThat's why I am entering nowβ¦
", "votes": 4, "replies": [] }, { "id": 1649761, "author_name": "Psi", "author_url": "", "post_date": "2022-01-14T15:11:49.023000", "content": "good idea @cpmpml - the public lb is now as high as we got after 2 months of fighting hard
", "votes": 5, "replies": [] }, { "id": 1649766, "author_name": "CPMP", "author_url": "", "post_date": "2022-01-14T15:16:26.523000", "content": "To tell the truth I didn't plan to spend so much time on santa comp. I would have joined earlier otherwise.
", "votes": 0, "replies": [] }, { "id": 1649781, "author_name": "Psi", "author_url": "", "post_date": "2022-01-14T15:24:27.730000", "content": "New NLP one might be an alternative, more time :P
", "votes": 1, "replies": [] }, { "id": 1649918, "author_name": "CPMP", "author_url": "", "post_date": "2022-01-14T17:17:16.703000", "content": "The one Abishek just ruined?
", "votes": 8, "replies": [] }, { "id": 1649924, "author_name": "Chun Ming Lee", "author_url": "", "post_date": "2022-01-14T17:21:36.427000", "content": "Funny coincidence
", "votes": 1, "replies": [] }, { "id": 1649925, "author_name": "Nirjhar Roy", "author_url": "", "post_date": "2022-01-14T17:22:56.863000", "content": "You Only Longformer Twice
", "votes": 0, "replies": [] } ] }, { "id": 1657271, "author_name": "outwrest", "author_url": "", "post_date": "2022-01-20T04:23:20.987000", "content": "@hengck23 I did some testing on the notebook after finding out exactly what augment=True does. According to the yolov5 code, it scales images at 1x, 0.83x (also flips horizontally), and 0.67x. Inferencing at higher resolutions can still pick up bboxes from previous and lower resolutions. Here is some data that might help with augment=False at the same resolutions you tested.
\nResoluton With Aug. W/O Aug.\n3600 (1.00x) LB 0.579 | LB 0.577\n4000 (1.11x) LB 0.586 | LB 0.583\n4800 (1.33x) LB 0.590 | LB 0.575\n5600 (1.55x) LB 0.594 | LB 0.586\n6400 (1.77x) LB 0.596 | LB 0.597\n7200 (2.00x) LB 0.603 | LB 0.610\n9000 (2.50x) LB 0.613 | LB 0.584\n12600(3.50x) LB 0.557 | ????\nCould it be that the model learned the sizes of small, medium, and large cots, was able to find more \"smaller\" ones with higher resolutions? The NMS looks like it helps but it there's still a lot of error. Some F2 testing is needed on the last video for more confirmation.
\nOne thing this does show: Augment is not a big factor in the public leaderboard score and maybe with a bigger model (Large or medium), we can inference with augment=False for 2/3 reduction in submission time. Hopefully, this helps!
", "votes": 13, "replies": [ { "id": 1657285, "author_name": "Sanchit Vijay", "author_url": "", "post_date": "2022-01-20T04:40:02.673000", "content": "@outwrest this is insightful. Have you noted exact time difference?
", "votes": 0, "replies": [] }, { "id": 1657313, "author_name": "outwrest", "author_url": "", "post_date": "2022-01-20T05:07:07.573000", "content": "@sanchitvj Iβm not sure exactly, I know the 9000px one ran in under 3h (or just about, not 100% sure). The time should scale with resolution but with Augment=True you are basically running an inference 3 times on every image (albeit at a smaller resolution).
", "votes": 0, "replies": [] }, { "id": 1657915, "author_name": "Sanchit Vijay", "author_url": "", "post_date": "2022-01-20T15:24:08.423000", "content": "Model which gave LB of 0.648 is giving LB of 0.384 at incresed image size and No TTA. Inference time is decreased by ~2.5 hrs but poor LB.
", "votes": 0, "replies": [] }, { "id": 1657942, "author_name": "outwrest", "author_url": "", "post_date": "2022-01-20T15:44:25.530000", "content": "I haven't tried it on one of the models, I'll try it out later but it looks like the 2.5x sweet spot from the last model is causing too many abnormally small predictions and maybe the NMS was getting rid of a lot of them when using TTA at smaller resolutions? I'm not exactly sure if it helped, but I forgot to mention that I tried to mitigate that by adding this to the inference function:
\narea = (int(row.xmax) - int(row.xmin)) * (int(row.ymax) - int(row.ymin))\nif row.confidence > 0.15 and area > 250:\n@sanchitvj I don't think this can work on everything but I am going to try to experiment with other augmentations on the yolov5 function here:
\nhttps://github.com/ultralytics/yolov5/blob/9708cf56eaead29bce789a07cfe73ecc7d7d4838/models/yolo.py#L128
\nMaybe inferencing at the same high resolution is good with other TTA's.
In my experiment, inference with 2x resolution boosted LB score, but CV got worse.
\nTraining: 2560 (train: video0&2, valid: video1)
\nTest inference: 2560β5120
\nLB: 0.486->0.540
\nCV: 0.533->0.473
\nHow about your CV? Is it improved??
according to the cots dataset paper
\n\"We set the GoPro cameras to record videos continuously at 24 frames per second at 3840x2160 resolution and manually removed the periods of no activity between transects\"
\nit would be better if kaggle has released the original high resolution image
", "votes": 12, "replies": [ { "id": 1648603, "author_name": "sheep", "author_url": "", "post_date": "2022-01-13T14:45:27.463000", "content": "Yeah, I guess host worry that if they release original image this will become a hardware war. Not every guy here have tons of A100s and A6000s
", "votes": 15, "replies": [] }, { "id": 1648641, "author_name": "Lukasz Borecki", "author_url": "", "post_date": "2022-01-13T15:09:52.130000", "content": "They probably got good solution, but they throw it on kaggle to find new path to implement on original reso :)
", "votes": 0, "replies": [] }, { "id": 1649076, "author_name": "hengck23", "author_url": "", "post_date": "2022-01-14T00:15:47.073000", "content": "the application is robotics. for good results (e.g. near zero FP), algorithm should fuse with the path planning or sensor of the robots. as an example:
\nrobotic approach is just to move the robot closer and take a bigger picture of the suspected object
\nrobotic approach is just to increase camera gain to take better picture.
", "votes": 3, "replies": [] }, { "id": 1649123, "author_name": "Roc", "author_url": "", "post_date": "2022-01-14T01:27:24.150000", "content": "Hahahaβ¦ You have started this hardware war.π
", "votes": 3, "replies": [] } ] }, { "id": 1651637, "author_name": "Amin", "author_url": "", "post_date": "2022-01-15T23:09:24.283000", "content": "Does this work on CV as well? I trained a YoloV5m with image size 3000:
\nCV= 0.428
\nLB=0.652 (inference imsize: 6000)
I had a bug in my F2 implementation, the new score:
\nCV=0.528 (inference imsize: 3000)
nice! is this CV score F2? with empty images as well maybe?
", "votes": 1, "replies": [] }, { "id": 1653001, "author_name": "SeshuRaju π§ββοΈ", "author_url": "", "post_date": "2022-01-17T06:55:52.593000", "content": "@amin how many epochs you trained ?
", "votes": 1, "replies": [] }, { "id": 1653296, "author_name": "Amin", "author_url": "", "post_date": "2022-01-17T12:19:53.097000", "content": "@imeintanis Yes it's F2 score, I am using all the images with the subsequence split from this notebook, csv file train-0.2.csv
\nThe F2 score I report was calculated only at the 0.5 thresh. I retrained Yolov5l6 and calculated F2 over the thresholds 0.3-0.8 and the new CV=0.53 (inf size=3000) LB=0.624 (inf size=6000).
@seshurajup For 15 epochs (best epoch: 8), I tried to replicate the pipeline shared by sheep to compare with the benchmark.
", "votes": 4, "replies": [] }, { "id": 1653465, "author_name": "", "author_url": "", "post_date": "2022-01-17T14:54:22.463000", "content": "", "votes": 0, "replies": [] }, { "id": 1653476, "author_name": "Sanchit Vijay", "author_url": "", "post_date": "2022-01-17T15:03:41.250000", "content": "@amiiiney After 6-7 epochs mAP starts to saturates. Can you share which optimizer you are using and also if you using image shear and rotation with Yolov5m?
", "votes": 1, "replies": [] }, { "id": 1653502, "author_name": "Amin", "author_url": "", "post_date": "2022-01-17T15:37:30.580000", "content": "I am using SGD with lr=0.01 the defaults parameters. It seems that with this configuration there is no need to train for more than 8 epochs. And no, I am not using rotation/shear
", "votes": 1, "replies": [] }, { "id": 1656751, "author_name": "DiamondH", "author_url": "", "post_date": "2022-01-19T15:48:56.553000", "content": "I've done the same as you said, but always failed at epoch 3, things didn't go wellπ
", "votes": 1, "replies": [] }, { "id": 1658096, "author_name": "SeshuRaju π§ββοΈ", "author_url": "", "post_date": "2022-01-20T18:23:59.100000", "content": "@amiiiney which kernel you used to measure CV ? since i am getting CV as 0.522 for same split
", "votes": 1, "replies": [] }, { "id": 1658343, "author_name": "Amin", "author_url": "", "post_date": "2022-01-21T00:28:57.960000", "content": "@w3579628328 It seems that many people couldn't reproduce sheep's notebook either, I am not sure why is that :)
\n@seshurajup I am currently getting the same CV as you for inference size 3000. By CV I mean the F2 validation score per epoch, I calculate F2 by adding this line to the ap_per_class function in utils/metrics.py
\nf2 = 5 * p * r / (4 * p + r + 1e-16)
\nAnd then pass it through the fitness function to compute F2 over the IoU 0.3-0.8 thresholds.
\n
@Amin Can you please explain what you mean by the fitness function?
\nThis is all I see in metrics.py for the fitness function:
\ndef fitness(x):
\n # Model fitness as a weighted combination of metrics
\n w = [0.0, 0.0, 0.1, 0.9] # weights for [P, R, mAP@0.5, mAP@0.5:0.95]
\n return (x[:, :4] * w).sum(1)
@ayu055 so the first step is to add the F2 line of code to ap_per_class function and return the F2 as well:
\nreturn tp, fp, p, r, f2, ap, unique_classes.astype(\"int32\")
STEP 2: Go to val.py and modify these lines by adding f2:
\ntp, fp, p, r, f2, ap, ap_class = ap_per_class(*stats, plot=plots, save_dir=save_dir, names=names)\nap50, ap, f2 = ap[:, 0], ap.mean(1), , f2.mean(1) # AP@0.5, AP@0.5:0.95, f2@0.3:0.8\nmp, mr, f2, map50, map = p.mean(), r.mean(), f2.mean(), ap50.mean(), ap.mean()\nAlso change this line:
\niouv = torch.linspace(0.5, 0.95, 10).to(device) # iou vector for mAP@0.5:0.95\nto
\niouv= torch.from_numpy(np.arange(0.3, 0.85, 0.05)).to(device)\nand return the f2 score (I return it in the 5th position):
\n return (mp, mr, map50, map, f2, *(loss.cpu() / len(dataloader)).tolist()), maps, t\nSTEP 3: In the fitness function in utils/metrics.py add another weights 1.0 in the 5th position (depends on where you put the f2 in the above return) and give the other metrics a weight of 0.0:
\ndef fitness(x):\n# Model fitness as a weighted combination of metrics\nw = [0.0, 0.0, 0.0, 0.0, 1.0] # weights for [P, R, mAP@0.5, mAP@0.5:0.95, F2@0.3:0.8]\n return (x[:, :5] * w).sum(1)\nTo log F2 scores in wandb, add this code in train.py
\n if loggers.wandb:\n loggers.wandb.log({\"F2\": fi}) # W&B\nYou'll probably have to change something else in train.py to avoid some size mismatch errors because of the additional f2 variable, you can avoid it by returning f2 instead of map50 to keep the same shape of the results.
", "votes": 14, "replies": [] }, { "id": 1658394, "author_name": "SeshuRaju π§ββοΈ", "author_url": "", "post_date": "2022-01-21T01:49:09.617000", "content": "Thanks, Got it what mistake i am doing
", "votes": 1, "replies": [] }, { "id": 1658837, "author_name": "NoChanged", "author_url": "", "post_date": "2022-01-21T10:51:52.823000", "content": "@amiiiney Can you tell me which batch size did you use with yolov5m?
", "votes": 1, "replies": [] }, { "id": 1658864, "author_name": "Mohammed Rizin V K", "author_url": "", "post_date": "2022-01-21T11:13:59.737000", "content": "Hey, @amiiiney Impressive score. May I know what change you made to get CV 0.528? Are the above-mentioned changes in the yolov5 are correct or not?
", "votes": 1, "replies": [] }, { "id": 1658910, "author_name": "Ioannis M", "author_url": "", "post_date": "2022-01-21T12:01:54.523000", "content": "\ndo you use Yolo-s or L? (FYI sheep's is with s6, --weights yolov5s6.pt)
\n\nYou'll probably have to change something else in train.py to avoid some size mismatch errors because of the additional f2 variable
\n
as long as x, w mult is valid and fitness(x) returns a single value should work OK.
@amiiiney thanks for sharing this, solved my errors. @morizin the above modifications are correct. There are some more modifications which can be done in order to log and visualize F2 score metrics on wandb while training. I will be sharing that in a new postβ¦.maybe beneficial for everyone in this competition.
@amiiiney Thanks for such an in-depth explanation. I was able to modify the code. I think I just had to make some changes in one of the callbacks so that it would log all metrics.
", "votes": 1, "replies": [] }, { "id": 1658969, "author_name": "Anshul Khadse", "author_url": "", "post_date": "2022-01-21T12:47:51.397000", "content": "@amiiiney can u plz tell where extact in train.py changes needed. I am getting error and I am not able to solve it
", "votes": 0, "replies": [] }, { "id": 1658971, "author_name": "Ζ con lα»«a!", "author_url": "", "post_date": "2022-01-21T12:49:18.950000", "content": "@sanchitvj hi, Which batch size did you use in training
", "votes": 1, "replies": [] }, { "id": 1658986, "author_name": "ayu055", "author_url": "", "post_date": "2022-01-21T13:15:28.797000", "content": "@anshulkhadse What error are you getting?
", "votes": 0, "replies": [] }, { "id": 1659002, "author_name": "Amin", "author_url": "", "post_date": "2022-01-21T13:31:46.940000", "content": "@nochanged I used batch size of 2, batch size of 1 with higher resolution is too slow.
\n@morizin I hope this is correct and I am not misleading anybody :))) The bug I had before was related to computing F2 only at the 0.5 threshold, not averaging over 0.3:0.8 thresholds. The new CV looks reasonable now given that my best model would score 0.5x on LB with inference size of 3000 instead of 6000.
\n@imeintanis I am using yolov5m6 weights, the only thing I am doing different to sheep is saving the weights based on F2 instead of mAP
", "votes": 3, "replies": [] }, { "id": 1659008, "author_name": "Anshul Khadse", "author_url": "", "post_date": "2022-01-21T13:40:11.087000", "content": "@ayu055 and @amiiiney I am getting errors in train.py file while computing in val.py file at end on epoch 1
\n@amiiiney can u also tell change in train.py files
What is the difference between the five fold csv file and the 0.2 csv? Is there any advantage to using one over the other?
", "votes": 0, "replies": [] }, { "id": 1659014, "author_name": "Sanchit Vijay", "author_url": "", "post_date": "2022-01-21T13:45:18.477000", "content": "@deptraicute I am using batch_size of 4 with small models and resolution of 3000.
\n@anshulkhadse solution for that errors on the way :)
@sanchitvj Thanks you!
", "votes": 0, "replies": [] }, { "id": 1659027, "author_name": "Amin", "author_url": "", "post_date": "2022-01-21T13:58:50.620000", "content": "@anshulkhadse You have to update the callbacks to fit the new metrics, it seems @sanchitvj is preparing a solution for this :)
", "votes": 2, "replies": [] }, { "id": 1659031, "author_name": "Mohammed Rizin V K", "author_url": "", "post_date": "2022-01-21T14:02:32.963000", "content": "@amiiiney I trained like 14 epochs or something but I still reach like 0.49x
", "votes": 1, "replies": [] }, { "id": 1659072, "author_name": "Anshul Khadse", "author_url": "", "post_date": "2022-01-21T14:30:07.037000", "content": "@amiiiney for this model, what IOU and Confidence u used while inference?
\n CV=0.528 (inference imsize: 3000)
\nLB= 0.652 (inference imsize: 6000)
@anshulkhadse @deptraicute @amiiiney solution out here :))
", "votes": 2, "replies": [] }, { "id": 1659121, "author_name": "Amin", "author_url": "", "post_date": "2022-01-21T15:07:39.370000", "content": "@morizin Did you use the same CV split I mentioned above? I am retraining the yolov5m6 again, I will report the F2 scores again once training is done
\n@anshulkhadse I used the same inference notebook shared by sheep, I didn't modify it.
", "votes": 1, "replies": [] }, { "id": 1659150, "author_name": "Anshul Khadse", "author_url": "", "post_date": "2022-01-21T15:21:59.223000", "content": "@amiiiney thanks to share the infoβ¦
\nits quite weird that we cant able to reproduce resultsπ€£
@amiiiney Thank you. yes, i am using the same split!
", "votes": 1, "replies": [] }, { "id": 1659254, "author_name": "ayu055", "author_url": "", "post_date": "2022-01-21T16:34:40.063000", "content": "@amiiiney What is the difference between the five fold csv file and the 0.2 csv? Is there any reason you chose one over the other?
", "votes": 1, "replies": [] }, { "id": 1659291, "author_name": "Amin", "author_url": "", "post_date": "2022-01-21T17:15:56.370000", "content": "@ayu055 There is no specific reason I chose it :) I just picked it to have the same split as my teammates for benchmarking. The author of the split notebook used StratifiedKFold for both 0.2csv and 5folds, so there shouldn't be much difference between them.
", "votes": 0, "replies": [] }, { "id": 1659723, "author_name": "Anshul Khadse", "author_url": "", "post_date": "2022-01-22T04:31:31.057000", "content": "@amiiiney are u using only annotated images or with annotated all non-annotated images as well?
", "votes": 0, "replies": [] }, { "id": 1659933, "author_name": "HiαΊΏu PhαΊ‘m TrαΊ§n Minh", "author_url": "", "post_date": "2022-01-22T08:23:44.050000", "content": "@amiiiney Did you check your LB at size 3000 with yolov5m?
", "votes": 0, "replies": [] }, { "id": 1660399, "author_name": "ayu055", "author_url": "", "post_date": "2022-01-22T16:41:51.083000", "content": "\n\nYes it's F2 score, I am using all the images with the subsequence split from this notebook, csv file train-0.2.csv
\n
@amiiiney Do you filter out the images with no bounding boxes before training?
\n\n\nI am using SGD with lr=0.01 the defaults parameters. It seems that with this configuration there is no need to train for more than 8 epochs. And no, I am not using rotation/shear
\n
Are the default parameters the ones found in hyp.scratch.yaml?
", "votes": 0, "replies": [] } ] }, { "id": 1653346, "author_name": "clwclw", "author_url": "", "post_date": "2022-01-17T13:06:29.077000", "content": "Any one can reproduce the training notebook of yolov5s6, 3600 with 15 epoch? I can only get 0.48 for lb, and cv(on video 1) is also much lower than sheep big brother.
", "votes": 10, "replies": [ { "id": 1653402, "author_name": "Nirjhar Roy", "author_url": "", "post_date": "2022-01-17T13:53:53.820000", "content": "Not sure why the downvote ,but yes , i could not reproduce it exactly although not as low as you ,but surely not .579 . Not sure what is going on . My CV was exactly same as sheep though .
", "votes": 3, "replies": [] }, { "id": 1653522, "author_name": "Sarvagya Malaviya", "author_url": "", "post_date": "2022-01-17T15:57:30.293000", "content": "I believe the split used on the training notebook is different compared to what the 0.579 model was trained on. I speculate this based on the filename for the weights (it says fold 12). Maybe it was trained on the 12th fold of the 20 fold subseq split?
", "votes": 2, "replies": [] }, { "id": 1653544, "author_name": "Nirjhar Roy", "author_url": "", "post_date": "2022-01-17T16:20:46.777000", "content": "No when you run the same experiment multiple times in Yolo5 it will append a sequence number at the end of the experiment folder . e.g run1,run2, run3 etc so fold1 became fold12 in second run ..this is my theory
", "votes": 3, "replies": [] }, { "id": 1653599, "author_name": "omarsamir", "author_url": "", "post_date": "2022-01-17T17:15:46.690000", "content": "I can't also get the same results. When I trained on the same train split and same hyberparameters I got only 0.48 LB
", "votes": 0, "replies": [] }, { "id": 1654080, "author_name": "sheep", "author_url": "", "post_date": "2022-01-18T06:06:01.887000", "content": "I use this cmd for local training
\npython train.py --img 3600 --batch 4 --epochs 15 --data reef_f1_naive.yaml --weights yolov5s6.pt --name l6_3600_uflip_vm5_f1 --hyp data/hyps/hyp.heavy.2.yaml\nI am not sure the cause of the reproduction issue.
", "votes": 3, "replies": [] }, { "id": 1654109, "author_name": "Tian", "author_url": "", "post_date": "2022-01-18T06:49:43.427000", "content": "failed to reproduce. +1
", "votes": 1, "replies": [] }, { "id": 1654115, "author_name": "Nirjhar Roy", "author_url": "", "post_date": "2022-01-18T07:05:12.523000", "content": "Forget about @steamedsheep 's model , i cant reproduce my own models β¦lol
", "votes": 7, "replies": [] }, { "id": 1654117, "author_name": "dragon zhang", "author_url": "", "post_date": "2022-01-18T07:14:12.360000", "content": "I got 0.567 with batch=1 due to GPU limit.
", "votes": 0, "replies": [] }, { "id": 1654119, "author_name": "Sanchit Vijay", "author_url": "", "post_date": "2022-01-18T07:20:21.760000", "content": "batch_size=1 won't give good results because Yolos use batch norm (sync BN).
", "votes": 1, "replies": [] }, { "id": 1654183, "author_name": "Nirjhar Roy", "author_url": "", "post_date": "2022-01-18T08:48:20.847000", "content": "They also use Gradient Accumulation and adjusts weight decay based on batch size
", "votes": 4, "replies": [] }, { "id": 1654254, "author_name": "Hoda", "author_url": "", "post_date": "2022-01-18T10:50:24.717000", "content": "batch_size=1 is not giving a good result and batch_size>1 is giving an error on GPU on kaggle. I am not sure if we need to move to another platform or I made a mistake.
", "votes": 0, "replies": [] }, { "id": 1654266, "author_name": "hengck23", "author_url": "", "post_date": "2022-01-18T11:08:34.727000", "content": "train on cropped images are better solution then accumulated gradient. you have variation from different images and your bn statistics will be more stable
", "votes": 5, "replies": [] }, { "id": 1654322, "author_name": "hengck23", "author_url": "", "post_date": "2022-01-18T11:41:08.133000", "content": "\"I am not sure the cause of the reproduction issue.\"
\ntraining objection detection can be unstable if the number of annotation is few and the data is difficult.
\nimagine you have a batch of 8 images with 8 truth bbox, verus another batch with 32 truth bbox.
loss may changes from batch to batch.
\na soultion is to modified your sampler os that num of bbox per batch is fairly constant. but this is difficult becuase augmentation also changes number of true bbox (e.g. when you scale image, boxes become too small are ignore)
\nthis is also why when you check the code of the loss for object detection, they are normalised by number of truth object
", "votes": 6, "replies": [] }, { "id": 1654447, "author_name": "Dwight Foster", "author_url": "", "post_date": "2022-01-18T14:20:48.557000", "content": "@hengck23 So is it a better idea to train on smaller image sizes with larger batch sizes?
", "votes": 0, "replies": [] }, { "id": 1654642, "author_name": "Mohammed Rizin V K", "author_url": "", "post_date": "2022-01-18T17:22:14.300000", "content": "@steamedsheep. I think you did something on the source code, perhaps. because when I looked into your weights. you cropped all the information of epoch number and the best_fitness score and also a couple of values. I may ask: have you done it to reduce the file size? I guess it won't reduce much
", "votes": 1, "replies": [] }, { "id": 1656024, "author_name": "sheep", "author_url": "", "post_date": "2022-01-19T03:44:27.600000", "content": "I did NOTHING on source code, It is yolov5's host who decide only to save the model weight on weights/best.pt, If you still have some question on this, It is better for you to run some demo code and check the weight first.
@morizin β¦. it's true that @steamedsheep doe not do anything with its weight file..and u are also correct at ur position is that why u are notable able to see model training details on the trained modelβ¦ the reason is that in YOLOv5 when u trained ur model at the end of training it does striping of the model weight file it means it deletes model optimizer states and along with that it also removes the information about for many epoch ur models is trained onβ¦.the reason for this is keeping optimizer states in model inferencing is no used and it unnecessarily increases the weight file size that's why this stripping is doneβ¦it will reduce the weight file size
\nStriping comes with some drawbacks also that u can't really much fine-tune the same weight file due to the absence of optimizer stateβ¦and due to stripping model weight file size gets changed with same initials pre-trained weights.
", "votes": 1, "replies": [] }, { "id": 1658617, "author_name": "Mohammed Rizin V K", "author_url": "", "post_date": "2022-01-21T07:07:46.763000", "content": "Thank you! @anshulkhadse and Sorry @steamedsheep
", "votes": 1, "replies": [] } ] }, { "id": 1649263, "author_name": "Hannah B", "author_url": "", "post_date": "2022-01-14T06:24:06.817000", "content": "4k is always better than 1080p π
", "votes": 5, "replies": [] }, { "id": 1653223, "author_name": "hengck23", "author_url": "", "post_date": "2022-01-17T11:15:29.017000", "content": "i want to study more about lb metric and the reasons and effects of doing inference at higher resolution.
\nanyone has local CV (validation set = video1) at 3600 for the given model '../input/reef-baseline-fold12/l6_3600_uflip_vm5_f12_up/f1/best.pt' ?
I am getting around 0.57 for local CV
", "votes": 5, "replies": [ { "id": 1653451, "author_name": "sheep", "author_url": "", "post_date": "2022-01-17T14:36:59.953000", "content": "Your CV is very similiar to mine.
", "votes": 1, "replies": [] }, { "id": 1653580, "author_name": "Hannah B", "author_url": "", "post_date": "2022-01-17T16:58:18.820000", "content": "How do you calculate the CV? And this is f2 score right? I was getting 0.69 at image size =3600 for sheeps model
", "votes": 0, "replies": [] }, { "id": 1653683, "author_name": "Ioannis M", "author_url": "", "post_date": "2022-01-17T19:35:44.663000", "content": "I got about the same on CV 0.69x at img size=3600, conf 0.1 (~2100 images with annot)
EDIT: maybe the 0.57 is for all images including empty ones (8232 images, 6133 empty) ?
IMPORTANT !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
\ni thinnk there is a bug in https://www.kaggle.com/ferlockx/f2-cv-0-77-subseq-split-and-higher-res/notebook
\n(it reported CV of 0.69)
\nhttps://www.kaggle.com/ferlockx/f2-cv-0-77-subseq-split-and-higher-res/comments
\n@ferlockx , I think there is a bug in your code:
\nboxAArea = (x12 - x11 + 1) * (y12 - y11 + 1)\nboxBArea = (x22 - x21 + 1) * (y22 - y21 + 1)\nwould the host evaluation code also have such bug ???
", "votes": 7, "replies": [] }, { "id": 1654033, "author_name": "hengck23", "author_url": "", "post_date": "2022-01-18T04:42:33.930000", "content": "i spend a whole day catching the bug and has other interesting findings. let's denote
\n(a) https://www.kaggle.com/ferlockx/f2-cv-0-77-subseq-split-and-higher-res/notebook
\n(b) https://www.kaggle.com/steamedsheep/yolov5-is-all-you-need
in particular, (a) is based on val.py which use dataset and dataloader. there is padding of 114 for input images (i think the train also uses pad of 114?). it improves slightly detection of cots objects at the boundary. (b) uses image directly (or a pad of zero values).
(b) incudes test time augmentation in submission. (a) is evaluation without augmentation
both (a) and (b) uses different input size becuase of the padding of 114. the resizing function is also different. (a) uses opencv resize, (b) uses torch F.interpolate.
3600 is not multiple of 64 (max stride of yolov5ms model). the code uses another size actually
\nif you are using (a), you would get the best f2 score for the lowest confidence threshold. you can even go as low as 0.01. this is becuase the evaluation only include non-empty images (i.e. those without ground truth annotations only).
\ni think the public test set has few empty images. this is why as long as you improve recall, you have increase in lb score
", "votes": 10, "replies": [] }, { "id": 1654086, "author_name": "Lukasz Borecki", "author_url": "", "post_date": "2022-01-18T06:14:58.340000", "content": "@hengck23 you should use val.py with full fold with empty images, otherwise its wrong.
", "votes": 1, "replies": [] }, { "id": 1654101, "author_name": "Hannah B", "author_url": "", "post_date": "2022-01-18T06:34:43.683000", "content": "thanks for spending time on reviewing the CV notebook @hengck23 most of the helper functions were taken from the first reference and I did not review the bbox area calculation part. I'll correct it in the next version.
\nI think in one of runs using -augment call of yolo val script was not improving results so I did not include it in the saved versions.
\nI think that the 4th point you mention was only true for the models which have been made public by sheep. In my own runs with some models, upsizing and then using higher thresholds was giving significant CV boost (even though this was without empty images).
\nIn runs with 0 annotation images I think that we can discard all \"small area\"(arbitrarily chosen as 800 or 1000) boxes under threshold 0.4-0.5 so that we don't have lot's of false positive small bbox predictions. I tested this technique suggested by hengck on LB and it was indeed boosting by ~0.005-0.01.
\nThanks Lukas I'll do a run with all data in vid 1 as val to get the accurate CV. I was resisting doing on Kaggle since it's compute expensive to run on a lot of images and that too testing with different sizes so I'll try locally on Colab.
", "votes": 0, "replies": [] }, { "id": 1654346, "author_name": "Nirjhar Roy", "author_url": "", "post_date": "2022-01-18T11:49:25.203000", "content": "@ferlockx @hengck23 Thanks for the notebook and your valuable input
\nAfter making the necessary suggested modification :
\nSheep's code CV : 0.574 LB :0.578 @3600
\nA Version of my Checkpoint CV : 0.536 LB : 0.563 @3600
\nSame version of my checkpoint CV : 0.582 LB: (yet to check ) @4000
I used val.py with --augment parameter
\nNot making the notebook public as I did minimal modification from the original published by Aditya.
", "votes": 0, "replies": [] }, { "id": 1654382, "author_name": "Mohammed Rizin V K", "author_url": "", "post_date": "2022-01-18T12:59:22.607000", "content": "For me, my yolov5 scores 0.66 with pure video_1 validation(including empty images)
", "votes": 0, "replies": [] }, { "id": 1654396, "author_name": "Nirjhar Roy", "author_url": "", "post_date": "2022-01-18T13:17:21.633000", "content": "Sheep's code ? If not May be you should submit it :D
", "votes": 0, "replies": [] }, { "id": 1654471, "author_name": "Mohammed Rizin V K", "author_url": "", "post_date": "2022-01-18T14:53:26.717000", "content": "Yaa, I would. My model doesn't love enlarging images. In img_size 4800, giving 0.5x or something strange. It doesn't detect much in larger image sizes.
", "votes": 1, "replies": [] }, { "id": 1654608, "author_name": "ayu055", "author_url": "", "post_date": "2022-01-18T16:59:08.980000", "content": "@phoenix9032 When you say CV score, do you refer to the F2 score or the mAP across 0.3:0.8 IOU thresholds?
\nIf it is F2, do you calculate this in val.py using the mean precision and recall that is already calculated?
", "votes": 0, "replies": [] }, { "id": 1654638, "author_name": "Mohammed Rizin V K", "author_url": "", "post_date": "2022-01-18T17:17:25.593000", "content": "Sorry for informing the false score. unfortunately i found a bug in my code and i am just updating the CV score here. its 0.602 F2 score
\n@ayu055 you can use this notebook . there are some few changes are required which is described in these comments
", "votes": 1, "replies": [] }, { "id": 1654645, "author_name": "Nirjhar Roy", "author_url": "", "post_date": "2022-01-18T17:26:15.990000", "content": "@morizin Still its quite good local CV for lower image size ..But i have not tested the local CV for many cases, may be once you submit you will know about your model too β¦ @ayu055 , Rizin is right ..
", "votes": 0, "replies": [] }, { "id": 1654748, "author_name": "ayu055", "author_url": "", "post_date": "2022-01-18T18:59:57.900000", "content": "So do you not calculate F2 score while training and only at the end on validation?
", "votes": 0, "replies": [] }, { "id": 1654749, "author_name": "Nirjhar Roy", "author_url": "", "post_date": "2022-01-18T19:01:08.390000", "content": "Yes that is correct , for now.
", "votes": 0, "replies": [] }, { "id": 1656357, "author_name": "Bavesh123", "author_url": "", "post_date": "2022-01-19T09:41:26.253000", "content": "Can someone explain the changes to be made in this notebook to calculate the proper cv score.
", "votes": 0, "replies": [] }, { "id": 1657380, "author_name": "", "author_url": "", "post_date": "2022-01-20T06:27:32.460000", "content": "", "votes": 0, "replies": [] }, { "id": 1665208, "author_name": "Chenglu", "author_url": "", "post_date": "2022-01-26T16:45:07.527000", "content": "my best score with video1 is 0.624, without TTA
", "votes": 0, "replies": [] } ] }, { "id": 1654106, "author_name": "Tian", "author_url": "", "post_date": "2022-01-18T06:45:38.463000", "content": "In fact, there is nothing to be sorry about. The purpose of this community is to use AI technology to solve industrial problems. Sharing good ideas can improve the performance of final solutions, and they are not shared near the deadline. I don't think there's anything wrong. Although some people feel that the experience is getting worse, others feel that the experience is getting better, such as me.
", "votes": 3, "replies": [] }, { "id": 1649248, "author_name": "clwclw", "author_url": "", "post_date": "2022-01-14T05:47:58.817000", "content": "thanks a lot, sheep big old and frog god
", "votes": 3, "replies": [] }, { "id": 1654267, "author_name": "Sanchit Vijay", "author_url": "", "post_date": "2022-01-18T11:08:58.040000", "content": "Anyone tried submitting Yolov5X6 (largest model). My inference
\nWith img size:
unable to train due to limit of GPUπ
", "votes": 1, "replies": [] }, { "id": 1657399, "author_name": "Sanchit Vijay", "author_url": "", "post_date": "2022-01-20T06:43:22.760000", "content": "@rainfalllove don't worry large models aren't performing good anyways. I am performing experiment with all the models and will be clearing this in my post by next week.
", "votes": 0, "replies": [] }, { "id": 1657691, "author_name": "Time Master", "author_url": "", "post_date": "2022-01-20T11:54:49.143000", "content": "okay, I'm looking forward to it .
", "votes": 0, "replies": [] }, { "id": 1658939, "author_name": "Lime-Cake", "author_url": "", "post_date": "2022-01-21T12:24:28.817000", "content": "I timed out with img_size=5600 so you're not alone. But it might be due to my own fault, due to messed up model.
", "votes": 0, "replies": [] } ] }, { "id": 1652502, "author_name": "Hannah B", "author_url": "", "post_date": "2022-01-16T17:28:38.250000", "content": "@lukaszborecki I tried my cross val notebook on sheeps splitting of data and respective models by video id=1 as val folder and that split did indeed give increase in CV with increasing image sizes. It's at 6k size rn and should complete in an hour will update changes to public then. However at initial size, score was ~0.69 much lower than ~0.77 of my models.
\nAlso for my own models split by sub-sequence had higher CV for increased image sizes when evaluated using higher threshold. This leads me to think that maybe an ideal setup would be using different models to infer at different threshold value.
", "votes": 1, "replies": [ { "id": 1652533, "author_name": "Lukasz Borecki", "author_url": "", "post_date": "2022-01-16T18:10:06.787000", "content": "I bet it will fall :) however i tried split vid0 1 vs 2 and it was the same. Do you validating on full set ? (with empty images too :) ?
", "votes": 1, "replies": [] }, { "id": 1652545, "author_name": "Hannah B", "author_url": "", "post_date": "2022-01-16T18:18:30.993000", "content": "yup it fell at 6k haha made public just a few mins before your comment. But still it was this shows it was increasing uptil ~x1.5 scaling.
\nOops I think that's an important detail I've not been considering π¬ So if the test set has roughly 3/4 of the data without annotations does that mean I should randomly sample val_fold_size*3 images with no annotations into my cross validation?
\nOh splitting by video ID means I just take the entire data with vid id 1 got it :p
@ferlockx seriously idk how to answer :) i made really beutiful models, predicts f2 score on validation 0.8 beutiful inference and then LB 0.5 0.49 :D i made once different training with same set same settings but early stop at epoch3 . Inference was good but worse from others , it didnt detect all starfishes from my validation images i used before and LB score was 0.62. Our best model was not our best but it scored good on LB. If you got such a pearl you want to tweak it. Imagine loading ckpt into yolo then start training it on really low LR for one epoch!, val box drops val obj drops and LB drops from 0.67 -> 0.52 is it normal ?
", "votes": 1, "replies": [] }, { "id": 1652556, "author_name": "Hannah B", "author_url": "", "post_date": "2022-01-16T18:32:17.343000", "content": "hmm thanks for the tips :) I think I should focus more on transfer learning and ensembles once I've got this cross validation part sorted :p
", "votes": 0, "replies": [] }, { "id": 1654046, "author_name": "Time Master", "author_url": "", "post_date": "2022-01-18T05:07:55.707000", "content": "Hi, does ensemble increase LB ? I tried ,but LB got worse.
", "votes": 0, "replies": [] }, { "id": 1654263, "author_name": "hengck23", "author_url": "", "post_date": "2022-01-18T11:07:52.897000", "content": "[deleted message]
", "votes": 0, "replies": [] } ] }, { "id": 1649330, "author_name": "Lime-Cake", "author_url": "", "post_date": "2022-01-14T07:49:01.020000", "content": "For memory issue with larger image on kaggle, reducing batchsize is an option. According to the devs of yolov5, batch size should not influence results much. However, if you're using other (older) YOLO versions, do note that there's report on earlier YOLO that small batch size produces inferior results. Hardware shouldn't be a big issue (hopefully), so don't get discouraged.
", "votes": 1, "replies": [ { "id": 1649678, "author_name": "Ioannis M", "author_url": "", "post_date": "2022-01-14T13:55:35.473000", "content": "If I remember correctly (from Yolov5 previous versions) the nominal batch size is 64, that results to grad accumul steps = 64/BS, where BS the chosen batch size.
", "votes": 1, "replies": [] } ] }, { "id": 1649059, "author_name": "Vinicio Castillo", "author_url": "", "post_date": "2022-01-13T23:24:47.713000", "content": "How did you manage to make your model weight only 28MB? XD
", "votes": 1, "replies": [ { "id": 1649362, "author_name": "Nirjhar Roy", "author_url": "", "post_date": "2022-01-14T08:20:36.243000", "content": "@steamedsheep I have the same question . Any hint ?
", "votes": 0, "replies": [] }, { "id": 1649376, "author_name": "Ioannis M", "author_url": "", "post_date": "2022-01-14T08:31:08.033000", "content": "I think is normal due to pruner & optim stripper at the end of run.
\nFor me I see that yolov5-L is ~98MB so I guess for S 28MB sound normal
thanks .. got it .. at the end its stripping optimizer and reducing ..
", "votes": 1, "replies": [] } ] }, { "id": 1649006, "author_name": "Ioannis M", "author_url": "", "post_date": "2022-01-13T22:04:12.630000", "content": "thanks for tip @steamedsheep! May i ask what was the CV metrics for that chekpoint and if you like to say around at which epoch?
", "votes": 1, "replies": [] }, { "id": 1648646, "author_name": "dragon zhang", "author_url": "", "post_date": "2022-01-13T15:15:07.040000", "content": "So this again is a hardware competition dominated?
\nbecause I try to increase to 1920x1920, over 9hours limit of kaggle. I tried on colab pro, not pro+, which I have to check once in a while, It trains and converge slowly, so I gave up try bigger image.
\nupvote, though.
", "votes": 1, "replies": [ { "id": 1648661, "author_name": "Lukasz Borecki", "author_url": "", "post_date": "2022-01-13T15:30:35.947000", "content": "Don't worry our solution is on P100 :)
", "votes": 2, "replies": [] }, { "id": 1648691, "author_name": "dragon zhang", "author_url": "", "post_date": "2022-01-13T15:53:56.547000", "content": "you guys have pro+, don't you? which notebook keep running without disconnection, over my budget.
", "votes": 0, "replies": [] }, { "id": 1648701, "author_name": "Lukasz Borecki", "author_url": "", "post_date": "2022-01-13T16:01:09.143000", "content": "Yeah we got pro+ we got p100/v100/a100 at start but now only p100 and model was trained on p100. There is huge traffic on colab right now.
", "votes": 0, "replies": [] }, { "id": 1648704, "author_name": "Remek Kinas", "author_url": "", "post_date": "2022-01-13T16:02:39.147000", "content": "To score LB 0.673 laptop is enough β¦ belive me β¦
", "votes": 10, "replies": [] }, { "id": 1648711, "author_name": "dragon zhang", "author_url": "", "post_date": "2022-01-13T16:07:42.577000", "content": "I believe that you guys find better strategy. my laptop only with 1050ti.
", "votes": 0, "replies": [] }, { "id": 1648735, "author_name": "outwrest", "author_url": "", "post_date": "2022-01-13T16:37:57.823000", "content": "I've been training all my models on 3090 but I've been doing many experiments with tf-2.0. I'll need to get back to using yolov5 to score higher on LB. But I'll tell you this, image tilling might be the next big to happen since the COTs aren't big why make the whole picture bigger when you can have multiple smaller pictures that make the COTs appear bigger to the model. I've had some good results with it. Worth a try!
", "votes": 2, "replies": [] }, { "id": 1649001, "author_name": "Remek Kinas", "author_url": "", "post_date": "2022-01-13T21:53:45.677000", "content": "I see β¦ but using Kaggle GPU or free Collab will be enough at this moment. Certainly the more power you have the faster you can experiment β¦ but β¦ sometimes this is not a good to have to much GPUs :)
", "votes": 1, "replies": [] }, { "id": 1649011, "author_name": "outwrest", "author_url": "", "post_date": "2022-01-13T22:15:16.617000", "content": "@remekkinas I don't know if I made it clear in my previous comment but I meant that there are many strategies to get around the issues of high resolution. Here is this one for example that was useful to me. It is worth a try in constraint environments and with some tuning, it maybe can boost your models.
\nhttps://github.com/obss/sahi
Good luck!
", "votes": 3, "replies": [] }, { "id": 1649018, "author_name": "Remek Kinas", "author_url": "", "post_date": "2022-01-13T22:23:04.803000", "content": "Yes, absolutely β¦ SAHI is one of β¦ tiling (presented in notebook section) is another one, resizing showed here is next one β¦ This is beauty of this competition - you can experiment a lot. Dataset is demanding. As I can see on LB everybody β¦ are learning :) To be honest I am waiting for end β¦ why β¦ I am really interested in TOP solutions π We have performed a lot of different experiments β¦ I am sure some of them β¦ which have not worked for us probably would work for somebody else. This is really great learning.
", "votes": 2, "replies": [] }, { "id": 1649176, "author_name": "Mohammed Rizin V K", "author_url": "", "post_date": "2022-01-14T03:30:16.860000", "content": "If you don't mind me asking @remekkinas. I tried several times both on lb and cv to improve the performance of my Yolo model with SAHI.
\nDid SAHI affect your current score? and how much?
@morizin I know SAHI very, very well :) I use it since it was released ( @clwwlc will be probably suprised π ) To be honest we have nor tried SAHI so far β¦ In December we decided to take totally different attitude.
", "votes": 1, "replies": [] } ] }, { "id": 1648549, "author_name": "Liam Nguyen", "author_url": "", "post_date": "2022-01-13T13:48:54.557000", "content": "Higher resolution is all your need π€ ? Maybe not, right ?
", "votes": 1, "replies": [ { "id": 1648593, "author_name": "sheep", "author_url": "", "post_date": "2022-01-13T14:40:26.903000", "content": "yes, indeed
", "votes": 1, "replies": [] }, { "id": 1648595, "author_name": "Lukasz Borecki", "author_url": "", "post_date": "2022-01-13T14:40:45.037000", "content": "Yeah you need hugeillions of pixels and zillions of compute power. Not :)
", "votes": 0, "replies": [] }, { "id": 1648599, "author_name": "sheep", "author_url": "", "post_date": "2022-01-13T14:42:59.930000", "content": "Not really, I train this on a single 3090, I think 16GB ram card is also ok.
", "votes": 2, "replies": [] }, { "id": 1648604, "author_name": "Lukasz Borecki", "author_url": "", "post_date": "2022-01-13T14:45:43.200000", "content": "I didn't say it doesn't helps but is not all you need :)
", "votes": 0, "replies": [] }, { "id": 1648605, "author_name": "sheep", "author_url": "", "post_date": "2022-01-13T14:47:40.047000", "content": "aha, topic title in kaggle usually misleading, this isn't the worst oneπ
", "votes": 2, "replies": [] }, { "id": 1648772, "author_name": "Lukasz Borecki", "author_url": "", "post_date": "2022-01-13T17:24:41.907000", "content": "Seems resolution is not all you need, but damn you have to have it haha :)
", "votes": 0, "replies": [] } ] }, { "id": 1651340, "author_name": "Mohammed Rizin V K", "author_url": "", "post_date": "2022-01-15T17:24:25.830000", "content": "The LB is funny now!
\nanyone who ends up in the top 50 now will be placed above 100 after an hour
Public LB is merely a public lb.
", "votes": 5, "replies": [] } ] }, { "id": 1651265, "author_name": "Lukasz Borecki", "author_url": "", "post_date": "2022-01-15T16:44:37.640000", "content": "The most funny thing is when you run val.py --img 2500 (3200, 4600, 7200) on colab or kaggle score goes down, what happens here it goes up is kind of irrational. Either metrics in yolo are wrong or something weird is happening. But when you submit on higher resolution you got gain on LB
", "votes": 2, "replies": [ { "id": 1651278, "author_name": "Hannah B", "author_url": "", "post_date": "2022-01-15T16:50:19.430000", "content": "that shouldn't really be happening cause on my runs P and R were both pretty much above 70% in each epoch (and close to 80-90 during best epoch for Precision) using up-sizing so that translates to ~0.7-0.8 on F2 metric. This gives 0.58-0.61 on public LB using different sizes so correlation is pretty stable.
\nI also recently modified
\n # Print results\n pf = '%20s' + '%11i' * 2 + '%11.3g' * 4 # print format\n f2 = 5*mp*mr/(4*mr+mp)\n LOGGER.info(pf % ('all', seen, nt.sum(), mp, mr, map50, f2)) #map))\nand again some modifications in the part where model saving is done
\nin my val.py in yolo folder so that f2 is logged instead of MAP (though I'm not sure if this is the best way to implement the metric). This gives the 0.7-0.8 I was talking about earlier.
\nHow did you get the f2 score while using val.py? By default it gives map if i'm not wrong. Might be giving lower CV locally due to some leak or maybe wrong implementation?
", "votes": 2, "replies": [] }, { "id": 1651285, "author_name": "Lukasz Borecki", "author_url": "", "post_date": "2022-01-15T16:53:22.473000", "content": "Aditya i am talking about what @hengck23 done in experiment. He increased inference size on same weight and got gain and gain. When you do it on val.py recall and precission drops when you increase --img parameter. (tested on colab). Even our #7 got much worse f2 score (on that size) than pure f2 score after training.
", "votes": 0, "replies": [] }, { "id": 1651292, "author_name": "Lukasz Borecki", "author_url": "", "post_date": "2022-01-15T16:58:55.290000", "content": "Aditya i made implementation which plots f2 and f1 score on PNG. That doesnt matter i run my model which has recall 0.75, and increased resolution 2.5 times and recall was almost ZERO! :) either yolo got something screwed or colab or idk even i was suspicious in comeptition metric but watchin bboxes it does it correctly ;)
", "votes": 0, "replies": [] }, { "id": 1651295, "author_name": "Hannah B", "author_url": "", "post_date": "2022-01-15T17:00:39.487000", "content": "I've not yet tried using val.py post training has completed and was trusting the per epoch calculation it does by default(i.e. img_size is also = the set higher size).
\n
(P, R and f2 score I get during training after modifying the val.py file ^)
\nI'll try using val on one of my saved models to see if there's a dip in the P, R.
", "votes": 0, "replies": [] }, { "id": 1651299, "author_name": "Hannah B", "author_url": "", "post_date": "2022-01-15T17:05:00.237000", "content": "Hmm, close to 0 recall is sus π€ usually when metric calculation was this messed up in my experiments with mmdet it turned out to be some inconsistent dependencies or something but i've not encountered such issues in yolo yet
", "votes": 0, "replies": [] }, { "id": 1651309, "author_name": "Lukasz Borecki", "author_url": "", "post_date": "2022-01-15T17:10:29.807000", "content": "ADitya you must have some leak, AP95 over 0.8 ? our #7 model got 0.35 ;)
", "votes": 0, "replies": [] }, { "id": 1651317, "author_name": "Hannah B", "author_url": "", "post_date": "2022-01-15T17:12:47.907000", "content": "yeah that's the part i've modified, it's printing f2 score instead of AP95 :P For me AP95 was around 0.34-0.36 as well in previous runs when I hadn't modified the logging code
", "votes": 1, "replies": [] }, { "id": 1651320, "author_name": "Mohammed Rizin V K", "author_url": "", "post_date": "2022-01-15T17:13:26.307000", "content": "Yes, i also noticed it. I also tried training the model with img_size 7000 but it didn't perform very well than my 3600 secret model. i guess yolo doesn't like losing so much quality due to enlarging the images. i wonder why competition metric likes
", "votes": 1, "replies": [] }, { "id": 1651330, "author_name": "Lukasz Borecki", "author_url": "", "post_date": "2022-01-15T17:17:21.153000", "content": "@morizin yeah we were suspicious even if we were on #1 that something is wrong or we cannot explain it
", "votes": 2, "replies": [] }, { "id": 1651342, "author_name": "Hannah B", "author_url": "", "post_date": "2022-01-15T17:29:48.867000", "content": "I'll try using this code to see if my val scores are correct though I'm pretty sure it'll be different than my epoch calculations cause it's mean over IOUs from 0.3-0.8. But still I felt changing it was appropriate since it might give a better sense of model performance compared to normal AP95 metric
", "votes": 0, "replies": [] }, { "id": 1651347, "author_name": "Lukasz Borecki", "author_url": "", "post_date": "2022-01-15T17:34:15.590000", "content": "@ferlockx try this with high img
\n!python val.py --img 4800 --batch 1 --data /gdrive/MyDrive/Datasets/Great_Coral_Reef/yolo5_cfg/data.yaml --weights /gdrive/MyDrive/Datasets/Great_Coral_Reef/yolov5/runs/train/exp29/weights/epoch7.pt --iou 0.3 --conf 0.001
and go with img higher. Replace yaml and weights file. conf = 0.001 is becuase it draws metrics from 0 to 1 in confidence thresholds
\n-- hint if you dont have f2 metric plots you can watch f1 score . How looks f2 score ? same just hit f1 score from right ;) f2 got peak before f1
\n-- hint2 in metrics.py tweak place where is f1 plotting and change it to (5pr/(4p+r)) :) i added some lines to generate the metric after validation
okay thanks i'll try this out :)
", "votes": 0, "replies": [] }, { "id": 1651671, "author_name": "Hannah B", "author_url": "", "post_date": "2022-01-16T01:01:50.710000", "content": "hey @lukaszborecki , you can check here that CV was coming as 0.77 while post training validation.
\nWhile this was obviously a drop from mid 0.8s I was getting while training, this was expected since while training yolo was doing no metric sweeps over 0.3-0.8 and then taking their mean. I don't think higher res is hurting CV results. I'll try switching img up to 4k, 5k etc to see if those values reduce f2.
\nWhile there's no data leakage since splitting was done using same code, I would love your input on whether the metric implementation used in these public notebooks is correct :)
\nUPD: yup upsizing from img size value used during training does indeed decrease f2, updated my notebook to reflect the same. wonder why LB is immune to this π€
", "votes": 1, "replies": [] }, { "id": 1651867, "author_name": "Lukasz Borecki", "author_url": "", "post_date": "2022-01-16T06:16:40.317000", "content": "So your expirement prooved it is droping while increasing inference size :)
\n@ferlockx F1 score drops too because its correlated with f2. Either there are small starfishes in 25% or there is less populated video or cv metric is wrong either ours either theirs ;). But if you split F2 into TP FP FN it is clear f2 should drop
", "votes": 0, "replies": [] }, { "id": 1652005, "author_name": "Hannah B", "author_url": "", "post_date": "2022-01-16T08:55:07.897000", "content": "UPD2: @lukaszborecki it seems that the initial public metric implementation was a tiny bit wrong in that it didn't discard annotations under a confidence metric post inference (0.15 in most public notebooks). I've added that confidence in my latest version and the decrease is a bit less drastic but the trend is still indeed decreasing. It's now 0.77 -> 0.72 -> 0.64 β¦ instead of 0.77 -> 0.5 -> 0.4 β¦ :)
\nAlso the number of images in which there are no annotations keeps decreasing with increase in image size i.e. it does indeed help with small object detection as expected
\nI think there might be a few more things that are different in inference environment from our local metric calculation that our causing decreasing CV, but still we should consider more augmentation techniques just to be sure ;)
\nThanks Lukasz I'll try to correct the things you're mentioning
", "votes": 1, "replies": [] }, { "id": 1652007, "author_name": "Lukasz Borecki", "author_url": "", "post_date": "2022-01-16T08:57:30.290000", "content": "I tried increase img size in inference with looking at images and it was also getting worse (@ some point). To implementation yes there is more errors, first you have to count np.nanmean and second you need loop with confidence :)
\n-- anyway still the trend is decreasing and LB is increasing
", "votes": 0, "replies": [] }, { "id": 1652009, "author_name": "hengck23", "author_url": "", "post_date": "2022-01-16T09:01:01.027000", "content": "how about asking the host to release their metric computation code?
", "votes": 6, "replies": [ { "id": 1652020, "author_name": "Lukasz Borecki", "author_url": "", "post_date": "2022-01-16T09:10:44.337000", "content": "\n\nhow about asking the host to release their metric computation code?
\n
@hengck23 do you think it is possible ? they probably will answer the same as in paper :)
", "votes": 1, "replies": [] } ] }, { "id": 1652011, "author_name": "Mohammed Rizin V K", "author_url": "", "post_date": "2022-01-16T09:02:30.123000", "content": "I think the drop while increasing happens because there are lots of false positive
\ni visualized with @steamedsheep weights conf 0.25
\nFYI => RED - GT Boxes
\n BLUE - PRED_BOXES
\n
another hypothesis:
\nif you look at f2 per image, there should be some images that increase f2 while there are some decreases. maybe the public LB resembles more of those that improve?
@morizin interesting so you set model.conf =0.25 directly instead of removing below 0.25 post inference?
", "votes": 0, "replies": [] }, { "id": 1652038, "author_name": "Remek Kinas", "author_url": "", "post_date": "2022-01-16T09:30:28.060000", "content": "Firstly I thnink that Australian Government won't be happy with current solution :) Maybe I am wrong (totally wrong) but I created video on validation dataset using our best LB solution - it is totally crap! ππ
\nSimultanieusly we created video with our best local (crap according to LB score) model and β¦ it absolutely outperfoms our \"best LB\" β¦ and people annotating images (I know that such extra bboxes were unfortunately associatetd with FP - but it does not influence on f2 score much).
No. model.conf = 0.01 and filter boxes under 0.25
", "votes": 1, "replies": [] }, { "id": 1652044, "author_name": "Hannah B", "author_url": "", "post_date": "2022-01-16T09:34:19.790000", "content": "i didn't really check does csiro or tensorflow release any git repo for this data/competition?
", "votes": 0, "replies": [] }, { "id": 1652062, "author_name": "hengck23", "author_url": "", "post_date": "2022-01-16T09:46:16.430000", "content": "\nfor the image you showed, maybe f2 is still high. one correct TP is weighted 5x. if other frames don't have that many FP, then f2 is improved.
\non a side note: one should filter off those very small boxes
", "votes": 2, "replies": [ { "id": 1652385, "author_name": "Mohammed Rizin V K", "author_url": "", "post_date": "2022-01-16T15:03:21.997000", "content": "\n \n\nfor the image you showed, maybe f2 is still high. one correct TP is weighted 5x. if other frames don't have that many FP, then f2 is improved.
\n
\non a side note: one should filter off those very small boxes
\n
Perhaps you are right @hengck23
\n
\nthis was one of my models, i like this model very much. Predictions are almost perfect. but scores 0.418 in LB
I calculated f2 on the validation video(whether num_bbox>0 or >=0) and found that larger scales will bring worse results, which makes me very worried about the current LB.
", "votes": 2, "replies": [] }, { "id": 1652251, "author_name": "Lukasz Borecki", "author_url": "", "post_date": "2022-01-16T13:11:46.537000", "content": "@zzy same observations here and same observations in expirement. And the better model you make it usually goes worse with LB
", "votes": 1, "replies": [] }, { "id": 1652257, "author_name": "Remek Kinas", "author_url": "", "post_date": "2022-01-16T13:21:39.020000", "content": "@zzy990106 I do not how you and your team but we with @lukaszborecki have problem with eveluation and chosing right model. It is kind of luck :)
", "votes": 0, "replies": [] }, { "id": 1652258, "author_name": "hengck23", "author_url": "", "post_date": "2022-01-16T13:21:40.627000", "content": "maybe that is the reason for 4 submission
", "votes": 2, "replies": [] }, { "id": 1652259, "author_name": "Remek Kinas", "author_url": "", "post_date": "2022-01-16T13:24:57.373000", "content": "@hengck23 ππππ yes, it could be a great explanation to this situation β¦. You have to take one crap model but good on LB, second best looking model for team and two β¦ between β¦.
\nI know that for most competition looking on LB score is not good idea β¦. but in this case is strage β¦ because you definitely can build better model (using some ways to decrease FP and increase TP) but it does not work at all β¦. it dramatically drops on LB.
", "votes": 3, "replies": [] }, { "id": 1652303, "author_name": "hengck23", "author_url": "", "post_date": "2022-01-16T13:55:35.883000", "content": "if i am the host and i want to be evil I would put all non-empty images as public test and all empty images as private.
\nI think we need to probe the ratio of empty and non empty images in private and public. (note that we do know approximately how many empty images are there in the test, we know there are 14k test images with about 150 cots objects. we just don't know how they are split between public and private)
\nfor confidently predicted empty images, we add FP.
\nfor confidently predicted TP, we drop them.
\nthe change in score should give us some clue?
\n\nfor confidently predicted empty images, we add FP.
\n
\n for confidently predicted TP, we drop them.
\n the change in score should give us some clue?
We have made such research.
\nWe increasedTP, decreased FP/FN (using many hacks) β¦ and β¦ rised f2 β¦ (on many folds confs). Posted and β¦. ππ¨πππππππππ
@hengck23 split private and public by video or sequence is enough to direct a shake up
", "votes": 1, "replies": [] }, { "id": 1652407, "author_name": "hengck23", "author_url": "", "post_date": "2022-01-16T15:35:20.153000", "content": "\"split private and public by video or sequence is enough to direct a shake up\"
\nthe splits are usually by random. else it is difficult to come up with 25% and 75% ratios.
\nanother way is that they are split by order, e.g. first 25% of seq or video
\n\nthe splits are usually by random.
\n
Usually it is, but there are some impressive outliers like this https://www.kaggle.com/c/microsoft-malware-prediction
", "votes": 2, "replies": [] }, { "id": 1652418, "author_name": "Ioannis M", "author_url": "", "post_date": "2022-01-16T15:45:39.963000", "content": "from data info:
\n\n\nThis competition uses a hidden test set that will be served by an API to ensure you evaluate the images in the same order they were recorded within each video.
\n
maybe this means/ensures that they are split by video/seq order (?)
", "votes": 1, "replies": [] }, { "id": 1652426, "author_name": "sheep", "author_url": "", "post_date": "2022-01-16T15:54:44.187000", "content": "Agree, at least part of the sequence.
", "votes": 1, "replies": [] }, { "id": 1652443, "author_name": "hengck23", "author_url": "", "post_date": "2022-01-16T16:09:58.733000", "content": "\"This competition uses a hidden test set that will be served by an API to ensure you evaluate the images in the same order they were recorded within each video.\"
\nthe public score could be based on every 4-th frame (25%) evaluation
", "votes": 5, "replies": [ { "id": 1653054, "author_name": "Leon", "author_url": "", "post_date": "2022-01-17T07:51:26.697000", "content": "\n\n\nthe public score could be based on every 4-th frame (25%) evaluation
\n
If so, we should fully trust LB. π
", "votes": 0, "replies": [] } ] }, { "id": 1652998, "author_name": "SeshuRaju π§ββοΈ", "author_url": "", "post_date": "2022-01-17T06:51:59.750000", "content": "\n\n\nwe know there are 14k test images with about 150 cots objects
\n
Did i miss this info.. 150 cots in 13k images of test set ?
\nI like your 4th frame CV, didn't expected LB can be split in that way
", "votes": 0, "replies": [] }, { "id": 1653664, "author_name": "Max van Dijck", "author_url": "", "post_date": "2022-01-17T19:13:25.303000", "content": "I don't see the public LB being based on every 4th frame, wouldn't this screw up any temporal based solutions such as the yolox + tracking?
\nPlus it would promote overfitting to the public LB which generally isn't the case on kaggle.
", "votes": 3, "replies": [] } ] }, { "id": 1656565, "author_name": "Eugene J. Ryu", "author_url": "", "post_date": "2022-01-19T12:51:42.223000", "content": "Thank you for sharing the good notebook :)
\nI wish you will be entering the gold zone.
\nAnd I wish I could be entering the top 20% of this competition π
it may be easier to analyze by considering FP rate, recall, precision per box size.
\ne.g. we divide bbox into few category by sizes:
\nbelow 20,
\nfrom 30 to 32 β¦.
if you cannot explain the results (by theory or massive and careful experiment), it would be subjected to big shakeup. after all, test performance = (public+private) performance
\nhigher public score may means lower private one
", "votes": 1, "replies": [ { "id": 1652887, "author_name": "dragon zhang", "author_url": "", "post_date": "2022-01-17T04:49:21.343000", "content": "classify first ? and then detect?
", "votes": 1, "replies": [] } ] }, { "id": 1648551, "author_name": "Leon", "author_url": "", "post_date": "2022-01-13T13:50:35.333000", "content": "wow, 3600π²
", "votes": 1, "replies": [] }, { "id": 1668827, "author_name": "Good Moon", "author_url": "", "post_date": "2022-01-30T04:42:15.777000", "content": "Are you use the big batch size and high resolution to train your model, I can not repeat your result with the batch size = 8 and picture size = 720*1280.I can not take more big size because of the limitation of GPU.
\n I just confuse that if the GPU limit my model 's performance. If that's True, I think I have no idea to train model as good as yours.
Hey
\ncan u help me understand which images we take for validation ??
\nIs it 80 train-20 test from the 3 video images provided for test ??
\nIm a newbie so dint understand the submission format
Yes I used it too and am really impressed with the performance and results
", "votes": 0, "replies": [] }, { "id": 1657856, "author_name": "Sean", "author_url": "", "post_date": "2022-01-20T14:14:33.203000", "content": "Good job! Why did you guys all use yolov5s but not v5x?
", "votes": 0, "replies": [ { "id": 1657891, "author_name": "Nitya", "author_url": "", "post_date": "2022-01-20T14:52:57.053000", "content": "I used v5m. That was enough
", "votes": 2, "replies": [] } ] }, { "id": 1653264, "author_name": "Yi.S386", "author_url": "", "post_date": "2022-01-17T11:50:08.933000", "content": "I'll try 9000 and see how it works :-)
", "votes": 0, "replies": [] }, { "id": 1653048, "author_name": "", "author_url": "", "post_date": "2022-01-17T07:42:05.030000", "content": "", "votes": 0, "replies": [] }, { "id": 1652121, "author_name": "", "author_url": "", "post_date": "2022-01-16T11:10:14.263000", "content": "", "votes": 0, "replies": [] }, { "id": 1651326, "author_name": "", "author_url": "", "post_date": "2022-01-15T17:16:06.830000", "content": "", "votes": 0, "replies": [] }, { "id": 1650674, "author_name": "", "author_url": "", "post_date": "2022-01-15T08:09:15.080000", "content": "", "votes": 0, "replies": [] }, { "id": 1650620, "author_name": "", "author_url": "", "post_date": "2022-01-15T07:19:33.370000", "content": "", "votes": 0, "replies": [] }, { "id": 1650382, "author_name": "", "author_url": "", "post_date": "2022-01-15T02:33:27.130000", "content": "", "votes": 0, "replies": [] }, { "id": 1649299, "author_name": "", "author_url": "", "post_date": "2022-01-14T07:06:04.113000", "content": "", "votes": 0, "replies": [] }, { "id": 1649267, "author_name": "", "author_url": "", "post_date": "2022-01-14T06:26:16.533000", "content": "", "votes": 0, "replies": [] }, { "id": 1648563, "author_name": "", "author_url": "", "post_date": "2022-01-13T14:06:51.560000", "content": "", "votes": 0, "replies": [] }, { "id": 1648559, "author_name": "", "author_url": "", "post_date": "2022-01-13T14:04:36.070000", "content": "", "votes": 0, "replies": [] }, { "id": 1648537, "author_name": "", "author_url": "", "post_date": "2022-01-13T13:38:24.323000", "content": "", "votes": 0, "replies": [] }, { "id": 1658429, "author_name": "", "author_url": "", "post_date": "2022-01-21T02:57:29.410000", "content": "", "votes": 0, "replies": [] }, { "id": 1656354, "author_name": "", "author_url": "", "post_date": "2022-01-19T09:38:06.903000", "content": "", "votes": 0, "replies": [] }, { "id": 1648561, "author_name": "", "author_url": "", "post_date": "2022-01-13T14:05:43.487000", "content": "", "votes": 0, "replies": [] }, { "id": 1664878, "author_name": "", "author_url": "", "post_date": "2022-01-26T11:04:41", "content": "", "votes": 0, "replies": [] }, { "id": 1660980, "author_name": "", "author_url": "", "post_date": "2022-01-23T06:15:47.760000", "content": "", "votes": 0, "replies": [] }, { "id": 1659857, "author_name": "", "author_url": "", "post_date": "2022-01-22T06:43:45.977000", "content": "", "votes": 0, "replies": [] }, { "id": 1652441, "author_name": "", "author_url": "", "post_date": "2022-01-16T16:07:15.440000", "content": "", "votes": 0, "replies": [] }, { "id": 1649512, "author_name": "", "author_url": "", "post_date": "2022-01-14T10:35:15.160000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1648529": "Proper training of yolov5 can give decent score, will release train code later.\nhttps://www.kaggle.com/steamedsheep/yolov5-is-all-you-need π\n\nI release this merely due to that I find modifying several parameters(resolution, vflip and cutmix) then I can get a score looks good. Higher resolution DO benefit small object's detection. you can try classifier like Heng's to boost score further.\n\nUpdate:\nTrain notebook: https://www.kaggle.com/steamedsheep/yolov5-high-resolution-training\nSince I overestimated the performance of P100 and 2T cpu in kaggle, to reproduce the score, you need change back the size to `3600` and epochs to 15. This setup will yield a 0.33 mAP@.5:.95 \n\nUpdate 2:\nI successfully kick myself out of the bronze zone, which is not my intentionπ", "1649181": "here is the magic, using your same model and same code (i just change the input size)\n```\npath='../input/reef-baseline-fold12/l6_3600_uflip_vm5_f12_up/f1/best.pt',\n3600 LB 0.579\n4000 (1.11x) LB 0.586\n4800 (1.33x) LB 0.590\n5600 (1.55x) LB 0.594\n6400 (1.77x) LB 0.596\n7200 (2.00x) LB 0.603 !!!!??\n9000 (2.50x) LB 0.613 !!!!???? why it never ends\n...\n...\n# optimun is between 2.50x to 3.50x, please feel free to explore. \n# i estimate we can get to 6.20\n...\n12600(3.50x) LB 0.557 \n```\n\nbut i haven't used yolo5 yet and i don't know about your train data. etc.\nso i cannot tell if the results is overfitting (#1) or having good generalisation.\nhence interprete the results with care!\n\n\n(#1) overfitting as in the with enlarged image, the model is producing many FP. BUT for this public test video,\nthe improvement in TP outweighs that of FP. We can't tell for sure in the private set.\n\nwe need to calibrate FP of the models under different input setting, etc\n\n---\n\nmost top kagglers have tricks in their bags, accumulated from work and competition experiences.\nThis is results from hard work after numerous experiments.\n\nNow i let one cat of the bag:\n\neven in past competitions, i have been using this upsizing trick.\nit is interesting to note that when we enlarge input image, there is no addition information.\nbecause the new pixels are interpolated (i.e. information is from the original pixels).\nyet, i have been seeing consisently improvement in results, e.g. in classification, segmentation, object detection\n", "2506668": "thanks a lot, sheep big old and frog god", "1656668": "i think i have solved the puzzle\n\n\n\nthe safest bet is to use public LB to measure your recall only. I think there is insufficient images to measure FPrate and precision. Hence use another dataset for that\n", "1648542": "I should start with competitions one month before deadline, apparently it is the time people release good solutions.", "1657271": "@hengck23 I did some testing on the notebook after finding out exactly what augment=True does. According to the yolov5 code, it scales images at 1x, 0.83x (also flips horizontally), and 0.67x. Inferencing at higher resolutions can still pick up bboxes from previous and lower resolutions. Here is some data that might help with augment=False at the same resolutions you tested.\n\n```\nResoluton With Aug. W/O Aug.\n3600 (1.00x) LB 0.579 | LB 0.577\n4000 (1.11x) LB 0.586 | LB 0.583\n4800 (1.33x) LB 0.590 | LB 0.575\n5600 (1.55x) LB 0.594 | LB 0.586\n6400 (1.77x) LB 0.596 | LB 0.597\n7200 (2.00x) LB 0.603 | LB 0.610\n9000 (2.50x) LB 0.613 | LB 0.584\n12600(3.50x) LB 0.557 | ????\n```\n\nCould it be that the model learned the sizes of small, medium, and large cots, was able to find more \"smaller\" ones with higher resolutions? The NMS looks like it helps but it there's still a lot of error. Some F2 testing is needed on the last video for more confirmation.\n\nOne thing this does show: Augment is not a big factor in the public leaderboard score and maybe with a bigger model (Large or medium), we can inference with augment=False for 2/3 reduction in submission time. Hopefully, this helps!", "1648571": "according to the cots dataset paper \n\n\"We set the GoPro cameras to record videos continuously at 24 frames per second at 3840x2160 resolution and manually removed the periods of no activity between transects\"\n\nit would be better if kaggle has released the original high resolution image", "1651637": "Does this work on CV as well? I trained a YoloV5m with image size 3000:\nCV= 0.428\nLB=0.652 (inference imsize: 6000)\n\n\n##### UPDATE:\nI had a bug in my F2 implementation, the new score:\nCV=0.528 (inference imsize: 3000)", "1653346": "Any one can reproduce the training notebook of yolov5s6, 3600 with 15 epoch? I can only get 0.48 for lb, and cv(on video 1) is also much lower than sheep big brother. ", "1649263": "4k is always better than 1080p π", "1653223": "i want to study more about lb metric and the reasons and effects of doing inference at higher resolution.\nanyone has local CV (validation set = video1) at 3600 for the given model '../input/reef-baseline-fold12/l6_3600_uflip_vm5_f12_up/f1/best.pt' ?\n\nI am getting around 0.57 for local CV", "1654106": "In fact, there is nothing to be sorry about. The purpose of this community is to use AI technology to solve industrial problems. Sharing good ideas can improve the performance of final solutions, and they are not shared near the deadline. I don't think there's anything wrong. Although some people feel that the experience is getting worse, others feel that the experience is getting better, such as me.", "1649248": "thanks a lot, sheep big old and frog god", "1654267": "Anyone tried submitting Yolov5X6 (largest model). My inference \nWith img size:\n- 6000 -> timeout\n- 8400 -> timeout", "1652502": "@lukaszborecki I tried my [cross val notebook](https://www.kaggle.com/ferlockx/f2-cv-0-77-subseq-split-and-higher-res) on sheeps splitting of data and respective models by video id=1 as val folder and that split did indeed give increase in CV with increasing image sizes. It's at 6k size rn and should complete in an hour will update changes to public then. However at initial size, score was ~0.69 much lower than ~0.77 of my models.\n\nAlso for my own models split by sub-sequence had higher CV for increased image sizes when evaluated using higher threshold. This leads me to think that maybe an ideal setup would be using different models to infer at different threshold value.", "1649330": "For memory issue with larger image on kaggle, reducing batchsize is an option. According to the devs of yolov5, batch size should not influence results much. However, if you're using other (older) YOLO versions, do note that there's report on earlier YOLO that small batch size produces inferior results. Hardware shouldn't be a big issue (hopefully), so don't get discouraged.", "1649059": "How did you manage to make your model weight only 28MB? XD", "1649006": "thanks for tip @steamedsheep! May i ask what was the CV metrics for that chekpoint and if you like to say around at which epoch? ", "1648646": "So this again is a hardware competition dominated?\n\nbecause I try to increase to 1920x1920, over 9hours limit of kaggle. I tried on colab pro, not pro+, which I have to check once in a while, It trains and converge slowly, so I gave up try bigger image.\n\nupvote, though.", "1648549": "Higher resolution is all your need π€ ? Maybe not, right ?", "1651340": "The LB is funny now!\nanyone who ends up in the top 50 now will be placed above 100 after an hour ", "1651265": "The most funny thing is when you run val.py --img 2500 (3200, 4600, 7200) on colab or kaggle score goes down, what happens here it goes up is kind of irrational. Either metrics in yolo are wrong or something weird is happening. But when you submit on higher resolution you got gain on LB", "1656565": "Thank you for sharing the good notebook :) \nI wish you will be entering the gold zone. \nAnd I wish I could be entering the top 20% of this competition π", "1651476": "it may be easier to analyze by considering FP rate, recall, precision per box size.\n\ne.g. we divide bbox into few category by sizes:\nbelow 20,\nfrom 30 to 32 ....\n\nif you cannot explain the results (by theory or massive and careful experiment), it would be subjected to big shakeup. after all, test performance = (public+private) performance\n\nhigher public score may means lower private one", "1648551": "wow, 3600π²", "1668827": "Are you use the big batch size and high resolution to train your model, I can not repeat your result with the batch size = 8 and picture size = 720*1280.I can not take more big size because of the limitation of GPU.\n I just confuse that if the GPU limit my model 's performance. If that's True, I think I have no idea to train model as good as yours.", "1664157": "Hey\ncan u help me understand which images we take for validation ??\nIs it 80 train-20 test from the 3 video images provided for test ??\nIm a newbie so dint understand the submission format ", "1657890": "Yes I used it too and am really impressed with the performance and results", "1657856": "Good job! Why did you guys all use yolov5s but not v5x?", "1653264": "I'll try 9000 and see how it works :-)", "1653048": ":) rookie here", "1652121": "impressive though you could apply skimage as well", "1651326": "Great kernel with a simple code.\nBut i think we should think this method can be generalized well to private lb dataset\nWe just have 9 hours for a kernel inference. ", "1650674": "nice, work\n\n", "1650620": "TKS for your share", "1650382": "Thanks for sharing, your work is great!π", "1649299": "Simply enlarging input size produce better results. It's amazing but I suspect that this indicates the model architecture of YOLO could be improved, at least for this competition data.", "1649267": "it is time to you guysπͺ use 640 --> 3x or 4x or 5x training?", "1648563": "amazingοΌππ", "1648559": "Agree, Yolo is all you needπ", "1648537": "Look forward to...", "1658429": "", "1656354": "", "1648561": "", "1664878": "Thank you!", "1660980": "Thank you!", "1659857": "It can work, thanks!", "1652441": "Thanks! good code", "1649512": "thanks for sharing" } }