![\"original](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F548996%2Fe3051f624e3709486e98740c80fc7dd9%2Forig.png?generation=1578227744391797&alt=media\")
Hi all, since only 9% of testing data will be calculated for public score, so I guess the local cv will be much worth to be reference. Just wondering the best local CV of you guys.\nSince I kinda I hope that I can get at least a bronze medal in this competition(for competition expert), so it will be nice that you can share the local CV to me for reference.
\n\nMine is 0.153 with 0.071 LB\nAnd all my submissions which can get 0.07+ LB, the local CV is swing around 0.12 ~ 0.153(...quite huge different)
", "messages": [ { "id": 710175, "postDate": "2020-01-04T12:08:13.350Z", "content": "Hi all, since only 9% of testing data will be calculated for public score, so I guess the local cv will be much worth to be reference. Just wondering the best local CV of you guys.\nSince I kinda I hope that I can get at least a bronze medal in this competition(for competition expert), so it will be nice that you can share the local CV to me for reference.
\n\nMine is 0.153 with 0.071 LB\nAnd all my submissions which can get 0.07+ LB, the local CV is swing around 0.12 ~ 0.153(...quite huge different)
", "rawMarkdown": "Hi all, since only 9% of testing data will be calculated for public score, so I guess the local cv will be much worth to be reference. Just wondering the best local CV of you guys.\nSince I kinda I hope that I can get at least a bronze medal in this competition(for competition expert), so it will be nice that you can share the local CV to me for reference. \n\nMine is 0.153 with 0.071 LB\nAnd all my submissions which can get 0.07+ LB, the local CV is swing around 0.12 ~ 0.153(...quite huge different)", "votes": 14 }, { "id": 713381, "postDate": "2020-01-08T08:12:29.970Z", "content": "Update the new result of my submission:
\n\n0.156/0.154 local CV with corresponding confidence values\n0.126/0.127 local CV with random confidence values\n0.084/0.078 LB
\n\nThe conclusion is : \nThe value is much stable and much consistent with LB by using random confidence values.\nThanks again for the share @its7171.
", "rawMarkdown": "Update the new result of my submission:\n\n0.156/0.154 local CV with corresponding confidence values\n0.126/0.127 local CV with random confidence values\n0.084/0.078 LB\n\nThe conclusion is : \nThe value is much stable and much consistent with LB by using random confidence values.\nThanks again for the share @its7171.\n\n\n", "votes": 5 }, { "id": 710647, "postDate": "2020-01-05T03:12:55.550Z", "content": "Trust CV, I tried predicting with and without the test image masks and found LB score dropped a bit while CV increased a lot.
\nHere are my CV vs LB scores (not very good but):\n0.089 -> 0.051\n0.0619 -> 0.030
Thanks for the share, you score relation between local CV and LB is quite similar to mine.
", "rawMarkdown": "Thanks for the share, you score relation between local CV and LB is quite similar to mine." }, { "id": 710702, "postDate": "2020-01-05T05:03:56.950Z", "content": "I have gotten CV 0.17-0.18 LB 0.07.\nThe train test split was 20%.. not very good so far but I wanted to share and hear about others.
", "rawMarkdown": "I have gotten CV 0.17-0.18 LB 0.07.\nThe train test split was 20%.. not very good so far but I wanted to share and hear about others.", "votes": 2 }, { "id": 710704, "postDate": "2020-01-05T05:06:59.730Z", "content": "Nice CV score! I prefer to trust the CV, so you might will get better place with this score.
", "rawMarkdown": "Nice CV score! I prefer to trust the CV, so you might will get better place with this score." }, { "id": 710857, "postDate": "2020-01-05T10:41:51.697Z", "content": "i am not trusting the CV calculation from titos kernel anymore. By playing with the thresholds you can increase the score by a lot, at the same time the LB score drops a lot. So there is definetly something different in the metric used here.
", "rawMarkdown": "i am not trusting the CV calculation from titos kernel anymore. By playing with the thresholds you can increase the score by a lot, at the same time the LB score drops a lot. So there is definetly something different in the metric used here. ", "votes": 4 }, { "id": 710894, "postDate": "2020-01-05T12:00:47.523Z", "content": "When I set the random value to confidence value, my CV score was relatively consistent with LB.
\n\na) original precision-recall curve\n
b) random confidence precision-recall curve\n![\"random](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F548996%2F49959fd41c7853fff92049bb1997eaf7%2Frandom_confidence.png?generation=1578227812442656&alt=media\")
I guess that predictions are not sorted by confidence values in the LB mAP calculation.\nIn fact, I replaced confidence with 1-confidence and got same LB score.
@ilu000 Thanks a lot for the share, I didn't play with threshold yet, so I didn't know that. (My threshold is 0.1 right now) But I think the evaluation method provided by sponsor is not very clear, and the way @its7171 calculate the score is quite similar to my idea. So I think it is still quite valuable for me to reference with.(At least for me). But still, I might select my final submissions with two metrics, one with best LB and one with best local CV. Like I always did in previous competition.
\n\n@its7171 Again, thanks a lot for your kernel sharing. But if the prediction is not sorted by confidence values in LB score calculation, then the LB score should be different when you replace the confidence values with 1.0-confidence, right?
", "rawMarkdown": "@ilu000 Thanks a lot for the share, I didn't play with threshold yet, so I didn't know that. (My threshold is 0.1 right now) But I think the evaluation method provided by sponsor is not very clear, and the way @its7171 calculate the score is quite similar to my idea. So I think it is still quite valuable for me to reference with.(At least for me). But still, I might select my final submissions with two metrics, one with best LB and one with best local CV. Like I always did in previous competition.\n\n@its7171 Again, thanks a lot for your kernel sharing. But if the prediction is not sorted by confidence values in LB score calculation, then the LB score should be different when you replace the confidence values with 1.0-confidence, right? ", "votes": 1 }, { "id": 710973, "postDate": "2020-01-05T13:50:59.743Z", "content": "My guessing is that LB score calculation is similar to area under random confidence precision-recall curve which I posted above.\nSo LB score became same or similar value regardless of confidence even original confidence or 1.0-confidence.
Anyway, this is just my guess from LB observation.\nI hope more information from admin!
", "rawMarkdown": "My guessing is that LB score calculation is similar to area under `random confidence precision-recall curve` which I posted above.\nSo LB score became same or similar value regardless of confidence even original confidence or 1.0-confidence.\n\nAnyway, this is just my guess from LB observation.\nI hope more information from admin!", "votes": 3 }, { "id": 711000, "postDate": "2020-01-05T14:34:32.883Z", "content": "Thanks for the explanation! \nDon't know why sponsor seems give up this competition...maybe the score is far away from their expectation..
", "rawMarkdown": "Thanks for the explanation! \nDon't know why sponsor seems give up this competition...maybe the score is far away from their expectation..", "votes": 1 }, { "id": 711148, "postDate": "2020-01-05T18:36:15.457Z", "content": "@its7171 you very well poised on lb .. wishing for in and around same position in final lb.
\n\nI read new thing in your write up \" random value to confidence value\" \nwhat do u mean by that, how to do that.
", "rawMarkdown": "@its7171 you very well poised on lb .. wishing for in and around same position in final lb.\n\nI read new thing in your write up \" random value to confidence value\" \nwhat do u mean by that, how to do that. \n", "votes": 2 }, { "id": 711149, "postDate": "2020-01-05T18:37:08.350Z", "content": "@xiejialun when you say threshold are u talking about logit threshold with or without sigmoid
", "rawMarkdown": "@xiejialun when you say threshold are u talking about logit threshold with or without sigmoid" }, { "id": 711245, "postDate": "2020-01-05T20:52:03.990Z", "content": "He means ranking by the confidence score that is in your submission file. \nGood find, @its7171
", "rawMarkdown": "He means ranking by the confidence score that is in your submission file. \nGood find, @its7171", "votes": 2 }, { "id": 711339, "postDate": "2020-01-06T00:29:02.127Z", "content": "@jaideepvalani Without sigmoid, perform on raw prediction of network
", "rawMarkdown": "@jaideepvalani Without sigmoid, perform on raw prediction of network" }, { "id": 711350, "postDate": "2020-01-06T00:48:57.303Z", "content": "@jaideepvalani \nmaybe by substituting confidence by random values (e.g. np.random.rand(1))
", "rawMarkdown": "@jaideepvalani \nmaybe by substituting confidence by random values (e.g. np.random.rand(1))", "votes": 1 }, { "id": 711362, "postDate": "2020-01-06T01:32:02.837Z", "content": "Ilu, arutema47, thanks for comment,
\n\nJaideep,\nI replaced score -> np.random.rand(len(result_flg)) in following lines of my notebook
ap = average_precision_score(result_flg, scores)*recall\nprint_pr_curve(result_flg, scores, recall)\n", "rawMarkdown": "Ilu, arutema47, thanks for comment,\n\nJaideep,\nI replaced `score -> np.random.rand(len(result_flg))` in following lines of [my notebook](https://www.kaggle.com/its7171/metrics-evaluation-script)\n\n
ap = average_precision_score(result_flg, scores)*recall\nprint_pr_curve(result_flg, scores, recall)\n", "votes": 1 }, { "id": 711408, "postDate": "2020-01-06T03:34:05.653Z", "content": "
@its7171 \nThanks for the insights! Your comments and notebooks have helped a lot in this competition.
\n\nIntuitively, random confidence will mean that submitting high-confidence predictions will give higher scores (or having higher prediction thresholds) than submitting lots of predictions with low confidences.
\n\nIs this thinking right or do you have other comments?
", "rawMarkdown": "@its7171 \nThanks for the insights! Your comments and notebooks have helped a lot in this competition.\n\nIntuitively, random confidence will mean that submitting high-confidence predictions will give higher scores (or having higher prediction thresholds) than submitting lots of predictions with low confidences.\n\nIs this thinking right or do you have other comments?", "votes": 1 }, { "id": 711586, "postDate": "2020-01-06T09:30:22.060Z", "content": "Thank you arutema47,\nI have same understanding as you.
\n\nI added 2 additional images c), d) which have smaller threshold than image a), b).
\n\nLooking at a) and c), we can understand intuitively the fact that the lower threshold we use, the higher mAP we can get.
\n\nLooking at b) and d), we can see that AP of d) is decreased as you mentioned.
\n\nc) original precision-recall curve with small threshold\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F548996%2F0d6e81c80c6f2ef632bd0962d75ab89f%2Forig_smallThre.png?generation=1578302727575314&alt=media\")
d) random confidence precision-recall curve with small threshold\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F548996%2Fb7824f1eb2be2afba2139c6b754332f2%2Frandom_confidence_smallThre.png?generation=1578302774496137&alt=media\")
@its7171 what's the threshold you used for your present best model?
", "rawMarkdown": "@its7171 what's the threshold you used for your present best model? " }, { "id": 711603, "postDate": "2020-01-06T09:50:59.670Z", "content": "It's 0.3.
", "rawMarkdown": "It's 0.3.", "votes": 1 }, { "id": 712288, "postDate": "2020-01-07T03:19:09.493Z", "content": "I suspect that the score is calculated as \"recall * precision\" using whole results without thresholding; in other words, \"recall * precision\" at threshold = -infinity.
", "rawMarkdown": "I suspect that the score is calculated as \"recall * precision\" using whole results without thresholding; in other words, \"recall * precision\" at threshold = -infinity.", "votes": 5 }, { "id": 712313, "postDate": "2020-01-07T04:29:38.287Z", "content": "Hi yu4u,
\n\nThresholding is done before submitting.(I submitted the predictions which filtered with threshold 0.3.)\nSo I agree with that the score is calculated using whole results without thresholding in LB.
\n\nAnd I agree with you too that the score is calculated as \"recall * precision\", which is almost equal as AP with random confidence as you can see b) and d).\nI just thought random confidence AP since it is said that metrics is mAP officially.
I replaced all predictions confidence values with 1) random uniform numbers in (0,1), 2) with ones, 3) no replacement. All 3 submissions gave the same PL score (at least to the rounding accuracy we observe on PL)
", "rawMarkdown": "I replaced all predictions confidence values with 1) random uniform numbers in (0,1), 2) with ones, 3) no replacement. All 3 submissions gave the same PL score (at least to the rounding accuracy we observe on PL)", "votes": 4 }, { "id": 713217, "postDate": "2020-01-08T03:06:02.957Z", "content": "@its7171 by 0.3 threshold you mean sigmoid ? \nIf so I find that at low th points start dilating in mask image ,and more than required car start getting depicted in val set img but surprisingly in lb we might get better score, can low threshold turn out to be an over fit?
", "rawMarkdown": "@its7171 by 0.3 threshold you mean sigmoid ? \nIf so I find that at low th points start dilating in mask image ,and more than required car start getting depicted in val set img but surprisingly in lb we might get better score, can low threshold turn out to be an over fit? \n", "votes": 1 }, { "id": 713241, "postDate": "2020-01-08T04:15:26.353Z", "content": "It's sigmoid but I think it is better to compare the number of predicted cars that are filtered by the threshold instead of the threshold it self.
\n\nIt's 16K cars for my best submission(8 cars per images).\nIt's bit smaller than training data(12 cars per Images).
", "rawMarkdown": "It's sigmoid but I think it is better to compare the number of predicted cars that are filtered by the threshold instead of the threshold it self.\n\nIt's 16K cars for my best submission(8 cars per images).\nIt's bit smaller than training data(12 cars per Images).", "votes": 2 } ] }, { "id": 716361, "postDate": "2020-01-11T15:34:00.693Z", "content": "Today i got Map = 0.16068738799186721 (calculated using 20% of training data)\nbut got public lb score = 0.056 :(
\n\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2034058%2F1d27de061c0ef0802a85d5f9cf4980ee%2Fmap0point16068738799186721.png?generation=1578756744078017&alt=media\")
If there's nothing wrong, this may be a good model. \n9% test data can't represent final score. @mobassir
", "rawMarkdown": "If there's nothing wrong, this may be a good model. \n9% test data can't represent final score. @mobassir " }, { "id": 717072, "postDate": "2020-01-12T17:29:20.027Z", "content": "i am able to touch map of .186 but still fail to replicate max of it to lb...\nNot sure what to trust what to not..
", "rawMarkdown": "i am able to touch map of .186 but still fail to replicate max of it to lb...\nNot sure what to trust what to not..\n" } ] }, { "id": 711146, "postDate": "2020-01-05T18:32:30.770Z", "content": "here is my 0.142 to 0.146 + puts me to .064
", "rawMarkdown": "here is my 0.142 to 0.146 + puts me to .064 \n", "votes": 1, "replies": [ { "id": 711346, "postDate": "2020-01-06T00:38:29.090Z", "content": "Thanks for the share.
", "rawMarkdown": "Thanks for the share." } ] }, { "id": 711064, "postDate": "2020-01-05T16:26:24.450Z", "content": "I get the best LB when the logit is -0.2. But you are 0.1 right? I'm thinking is there any way to find the best logit?
", "rawMarkdown": "I get the best LB when the logit is -0.2. But you are 0.1 right? I'm thinking is there any way to find the best logit?", "votes": 1, "replies": [ { "id": 711344, "postDate": "2020-01-06T00:37:19.233Z", "content": "I think it is quite hard to answer that question, this might depends on the model you train and what penalty you design on the loss. Normally I don't play with this parameter unless in the very end stage. Since I like to keep thing simple when I still trying to improve the model architecture or loss/metric.
", "rawMarkdown": "I think it is quite hard to answer that question, this might depends on the model you train and what penalty you design on the loss. Normally I don't play with this parameter unless in the very end stage. Since I like to keep thing simple when I still trying to improve the model architecture or loss/metric.", "votes": 2 } ] }, { "id": 710392, "postDate": "2020-01-04T17:20:28.193Z", "content": "I met a different situation... I am wonder which code do you compute the local CV?
", "rawMarkdown": "I met a different situation... I am wonder which code do you compute the local CV?", "replies": [ { "id": 710605, "postDate": "2020-01-05T00:51:00.240Z", "content": "I used this kernel to calculate the local cv. (with false negative included)\nhttps://www.kaggle.com/its7171/metrics-evaluation-script
", "rawMarkdown": "I used this kernel to calculate the local cv. (with false negative included)\nhttps://www.kaggle.com/its7171/metrics-evaluation-script" }, { "id": 711047, "postDate": "2020-01-05T15:43:37.983Z", "content": "Thank you!
", "rawMarkdown": "Thank you!" } ] }, { "id": 710195, "postDate": "2020-01-04T12:45:25.230Z", "content": "how much data you are using for calculating local cv? are you using your own gpu or kaggle kernels for this competition?
", "rawMarkdown": "how much data you are using for calculating local cv? are you using your own gpu or kaggle kernels for this competition?", "replies": [ { "id": 710295, "postDate": "2020-01-04T14:54:15.240Z", "content": "My validation data is 20% of training data, which is all the data I used to calculate local cv.\nAnd since I don't have my own gpu, I use kaggle kernels and colab to do this competition.
", "rawMarkdown": "My validation data is 20% of training data, which is all the data I used to calculate local cv.\nAnd since I don't have my own gpu, I use kaggle kernels and colab to do this competition.", "votes": 3 }, { "id": 710311, "postDate": "2020-01-04T15:22:14.307Z", "content": "thats great,,,are you using tensorflow? i was trying ruslan's centernet baseline kernel but it is not working well for me,i tried 10% data for validation,,,now it is time to try 20%\nare you changing learning during training?
", "rawMarkdown": "thats great,,,are you using tensorflow? i was trying ruslan's centernet baseline kernel but it is not working well for me,i tried 10% data for validation,,,now it is time to try 20%\nare you changing learning during training?", "votes": 1 }, { "id": 710608, "postDate": "2020-01-05T00:55:25.670Z", "content": "Yes, I'm using keras/tensorflow for this competition which the code refer to centernet repo, so I didn't try public kernel.\nAnd yes, I had implemented a learning rate scheduler.
", "rawMarkdown": "Yes, I'm using keras/tensorflow for this competition which the code refer to centernet repo, so I didn't try public kernel.\nAnd yes, I had implemented a learning rate scheduler.", "votes": 1 }, { "id": 710881, "postDate": "2020-01-05T11:27:09.483Z", "rawMarkdown": "" }, { "id": 710952, "postDate": "2020-01-05T13:15:16.920Z", "content": "@hekaikai Thanks a lot for your kindly invite! But I tend to finish this competition solo, so I might have no plan to team up right now. But good luck in the competition!
", "rawMarkdown": "@hekaikai Thanks a lot for your kindly invite! But I tend to finish this competition solo, so I might have no plan to team up right now. But good luck in the competition!" } ] }, { "id": 710700, "postDate": "2020-01-05T05:03:26.207Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 713381, "author_name": "Tsai29", "author_url": "", "post_date": "2020-01-08T08:12:29.970000", "content": "Update the new result of my submission:
\n\n0.156/0.154 local CV with corresponding confidence values\n0.126/0.127 local CV with random confidence values\n0.084/0.078 LB
\n\nThe conclusion is : \nThe value is much stable and much consistent with LB by using random confidence values.\nThanks again for the share @its7171.
", "votes": 5, "replies": [] }, { "id": 710647, "author_name": "GreatGameDota", "author_url": "", "post_date": "2020-01-05T03:12:55.550000", "content": "Trust CV, I tried predicting with and without the test image masks and found LB score dropped a bit while CV increased a lot.
\nHere are my CV vs LB scores (not very good but):\n0.089 -> 0.051\n0.0619 -> 0.030
Thanks for the share, you score relation between local CV and LB is quite similar to mine.
", "votes": 0, "replies": [] }, { "id": 710702, "author_name": "arutema47", "author_url": "", "post_date": "2020-01-05T05:03:56.950000", "content": "I have gotten CV 0.17-0.18 LB 0.07.\nThe train test split was 20%.. not very good so far but I wanted to share and hear about others.
", "votes": 2, "replies": [] }, { "id": 710704, "author_name": "Tsai29", "author_url": "", "post_date": "2020-01-05T05:06:59.730000", "content": "Nice CV score! I prefer to trust the CV, so you might will get better place with this score.
", "votes": 0, "replies": [] }, { "id": 710857, "author_name": "Pascal Pfeiffer", "author_url": "", "post_date": "2020-01-05T10:41:51.697000", "content": "i am not trusting the CV calculation from titos kernel anymore. By playing with the thresholds you can increase the score by a lot, at the same time the LB score drops a lot. So there is definetly something different in the metric used here.
", "votes": 4, "replies": [] }, { "id": 710894, "author_name": "tito", "author_url": "", "post_date": "2020-01-05T12:00:47.523000", "content": "When I set the random value to confidence value, my CV score was relatively consistent with LB.
\n\na) original precision-recall curve\n
b) random confidence precision-recall curve\n
I guess that predictions are not sorted by confidence values in the LB mAP calculation.\nIn fact, I replaced confidence with 1-confidence and got same LB score.
@ilu000 Thanks a lot for the share, I didn't play with threshold yet, so I didn't know that. (My threshold is 0.1 right now) But I think the evaluation method provided by sponsor is not very clear, and the way @its7171 calculate the score is quite similar to my idea. So I think it is still quite valuable for me to reference with.(At least for me). But still, I might select my final submissions with two metrics, one with best LB and one with best local CV. Like I always did in previous competition.
\n\n@its7171 Again, thanks a lot for your kernel sharing. But if the prediction is not sorted by confidence values in LB score calculation, then the LB score should be different when you replace the confidence values with 1.0-confidence, right?
", "votes": 1, "replies": [] }, { "id": 710973, "author_name": "tito", "author_url": "", "post_date": "2020-01-05T13:50:59.743000", "content": "My guessing is that LB score calculation is similar to area under random confidence precision-recall curve which I posted above.\nSo LB score became same or similar value regardless of confidence even original confidence or 1.0-confidence.
Anyway, this is just my guess from LB observation.\nI hope more information from admin!
", "votes": 3, "replies": [] }, { "id": 711000, "author_name": "Tsai29", "author_url": "", "post_date": "2020-01-05T14:34:32.883000", "content": "Thanks for the explanation! \nDon't know why sponsor seems give up this competition...maybe the score is far away from their expectation..
", "votes": 1, "replies": [] }, { "id": 711148, "author_name": "Jaideep", "author_url": "", "post_date": "2020-01-05T18:36:15.457000", "content": "@its7171 you very well poised on lb .. wishing for in and around same position in final lb.
\n\nI read new thing in your write up \" random value to confidence value\" \nwhat do u mean by that, how to do that.
", "votes": 2, "replies": [] }, { "id": 711149, "author_name": "Jaideep", "author_url": "", "post_date": "2020-01-05T18:37:08.350000", "content": "@xiejialun when you say threshold are u talking about logit threshold with or without sigmoid
", "votes": 0, "replies": [] }, { "id": 711245, "author_name": "Pascal Pfeiffer", "author_url": "", "post_date": "2020-01-05T20:52:03.990000", "content": "He means ranking by the confidence score that is in your submission file. \nGood find, @its7171
", "votes": 2, "replies": [] }, { "id": 711339, "author_name": "Tsai29", "author_url": "", "post_date": "2020-01-06T00:29:02.127000", "content": "@jaideepvalani Without sigmoid, perform on raw prediction of network
", "votes": 0, "replies": [] }, { "id": 711350, "author_name": "arutema47", "author_url": "", "post_date": "2020-01-06T00:48:57.303000", "content": "@jaideepvalani \nmaybe by substituting confidence by random values (e.g. np.random.rand(1))
", "votes": 1, "replies": [] }, { "id": 711362, "author_name": "tito", "author_url": "", "post_date": "2020-01-06T01:32:02.837000", "content": "Ilu, arutema47, thanks for comment,
\n\nJaideep,\nI replaced score -> np.random.rand(len(result_flg)) in following lines of my notebook
ap = average_precision_score(result_flg, scores)*recall\nprint_pr_curve(result_flg, scores, recall)\n", "votes": 1, "replies": [] }, { "id": 711408, "author_name": "arutema47", "author_url": "", "post_date": "2020-01-06T03:34:05.653000", "content": "
@its7171 \nThanks for the insights! Your comments and notebooks have helped a lot in this competition.
\n\nIntuitively, random confidence will mean that submitting high-confidence predictions will give higher scores (or having higher prediction thresholds) than submitting lots of predictions with low confidences.
\n\nIs this thinking right or do you have other comments?
", "votes": 1, "replies": [] }, { "id": 711586, "author_name": "tito", "author_url": "", "post_date": "2020-01-06T09:30:22.060000", "content": "Thank you arutema47,\nI have same understanding as you.
\n\nI added 2 additional images c), d) which have smaller threshold than image a), b).
\n\nLooking at a) and c), we can understand intuitively the fact that the lower threshold we use, the higher mAP we can get.
\n\nLooking at b) and d), we can see that AP of d) is decreased as you mentioned.
\n\nc) original precision-recall curve with small threshold\n
d) random confidence precision-recall curve with small threshold\n
@its7171 what's the threshold you used for your present best model?
", "votes": 0, "replies": [] }, { "id": 711603, "author_name": "tito", "author_url": "", "post_date": "2020-01-06T09:50:59.670000", "content": "It's 0.3.
", "votes": 1, "replies": [] }, { "id": 712288, "author_name": "yu4u", "author_url": "", "post_date": "2020-01-07T03:19:09.493000", "content": "I suspect that the score is calculated as \"recall * precision\" using whole results without thresholding; in other words, \"recall * precision\" at threshold = -infinity.
", "votes": 5, "replies": [] }, { "id": 712313, "author_name": "tito", "author_url": "", "post_date": "2020-01-07T04:29:38.287000", "content": "Hi yu4u,
\n\nThresholding is done before submitting.(I submitted the predictions which filtered with threshold 0.3.)\nSo I agree with that the score is calculated using whole results without thresholding in LB.
\n\nAnd I agree with you too that the score is calculated as \"recall * precision\", which is almost equal as AP with random confidence as you can see b) and d).\nI just thought random confidence AP since it is said that metrics is mAP officially.
I replaced all predictions confidence values with 1) random uniform numbers in (0,1), 2) with ones, 3) no replacement. All 3 submissions gave the same PL score (at least to the rounding accuracy we observe on PL)
", "votes": 4, "replies": [] }, { "id": 713217, "author_name": "Jaideep", "author_url": "", "post_date": "2020-01-08T03:06:02.957000", "content": "@its7171 by 0.3 threshold you mean sigmoid ? \nIf so I find that at low th points start dilating in mask image ,and more than required car start getting depicted in val set img but surprisingly in lb we might get better score, can low threshold turn out to be an over fit?
", "votes": 1, "replies": [] }, { "id": 713241, "author_name": "tito", "author_url": "", "post_date": "2020-01-08T04:15:26.353000", "content": "It's sigmoid but I think it is better to compare the number of predicted cars that are filtered by the threshold instead of the threshold it self.
\n\nIt's 16K cars for my best submission(8 cars per images).\nIt's bit smaller than training data(12 cars per Images).
", "votes": 2, "replies": [] } ] }, { "id": 716361, "author_name": "Mobassir", "author_url": "", "post_date": "2020-01-11T15:34:00.693000", "content": "Today i got Map = 0.16068738799186721 (calculated using 20% of training data)\nbut got public lb score = 0.056 :(
\n\n
If there's nothing wrong, this may be a good model. \n9% test data can't represent final score. @mobassir
", "votes": 0, "replies": [] }, { "id": 717072, "author_name": "Jaideep", "author_url": "", "post_date": "2020-01-12T17:29:20.027000", "content": "i am able to touch map of .186 but still fail to replicate max of it to lb...\nNot sure what to trust what to not..
", "votes": 0, "replies": [] } ] }, { "id": 711146, "author_name": "Jaideep", "author_url": "", "post_date": "2020-01-05T18:32:30.770000", "content": "here is my 0.142 to 0.146 + puts me to .064
", "votes": 1, "replies": [ { "id": 711346, "author_name": "Tsai29", "author_url": "", "post_date": "2020-01-06T00:38:29.090000", "content": "Thanks for the share.
", "votes": 0, "replies": [] } ] }, { "id": 711064, "author_name": "HelloAR", "author_url": "", "post_date": "2020-01-05T16:26:24.450000", "content": "I get the best LB when the logit is -0.2. But you are 0.1 right? I'm thinking is there any way to find the best logit?
", "votes": 1, "replies": [ { "id": 711344, "author_name": "Tsai29", "author_url": "", "post_date": "2020-01-06T00:37:19.233000", "content": "I think it is quite hard to answer that question, this might depends on the model you train and what penalty you design on the loss. Normally I don't play with this parameter unless in the very end stage. Since I like to keep thing simple when I still trying to improve the model architecture or loss/metric.
", "votes": 2, "replies": [] } ] }, { "id": 710392, "author_name": "HelloAR", "author_url": "", "post_date": "2020-01-04T17:20:28.193000", "content": "I met a different situation... I am wonder which code do you compute the local CV?
", "votes": 0, "replies": [ { "id": 710605, "author_name": "Tsai29", "author_url": "", "post_date": "2020-01-05T00:51:00.240000", "content": "I used this kernel to calculate the local cv. (with false negative included)\nhttps://www.kaggle.com/its7171/metrics-evaluation-script
", "votes": 0, "replies": [] }, { "id": 711047, "author_name": "HelloAR", "author_url": "", "post_date": "2020-01-05T15:43:37.983000", "content": "Thank you!
", "votes": 0, "replies": [] } ] }, { "id": 710195, "author_name": "Mobassir", "author_url": "", "post_date": "2020-01-04T12:45:25.230000", "content": "how much data you are using for calculating local cv? are you using your own gpu or kaggle kernels for this competition?
", "votes": 0, "replies": [ { "id": 710295, "author_name": "Tsai29", "author_url": "", "post_date": "2020-01-04T14:54:15.240000", "content": "My validation data is 20% of training data, which is all the data I used to calculate local cv.\nAnd since I don't have my own gpu, I use kaggle kernels and colab to do this competition.
", "votes": 3, "replies": [] }, { "id": 710311, "author_name": "Mobassir", "author_url": "", "post_date": "2020-01-04T15:22:14.307000", "content": "thats great,,,are you using tensorflow? i was trying ruslan's centernet baseline kernel but it is not working well for me,i tried 10% data for validation,,,now it is time to try 20%\nare you changing learning during training?
", "votes": 1, "replies": [] }, { "id": 710608, "author_name": "Tsai29", "author_url": "", "post_date": "2020-01-05T00:55:25.670000", "content": "Yes, I'm using keras/tensorflow for this competition which the code refer to centernet repo, so I didn't try public kernel.\nAnd yes, I had implemented a learning rate scheduler.
", "votes": 1, "replies": [] }, { "id": 710881, "author_name": "Hao Ge", "author_url": "", "post_date": "2020-01-05T11:27:09.483000", "content": "", "votes": 0, "replies": [] }, { "id": 710952, "author_name": "Tsai29", "author_url": "", "post_date": "2020-01-05T13:15:16.920000", "content": "@hekaikai Thanks a lot for your kindly invite! But I tend to finish this competition solo, so I might have no plan to team up right now. But good luck in the competition!
", "votes": 0, "replies": [] } ] }, { "id": 710700, "author_name": "", "author_url": "", "post_date": "2020-01-05T05:03:26.207000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "710175": "Hi all, since only 9% of testing data will be calculated for public score, so I guess the local cv will be much worth to be reference. Just wondering the best local CV of you guys.\nSince I kinda I hope that I can get at least a bronze medal in this competition(for competition expert), so it will be nice that you can share the local CV to me for reference. \n\nMine is 0.153 with 0.071 LB\nAnd all my submissions which can get 0.07+ LB, the local CV is swing around 0.12 ~ 0.153(...quite huge different)", "713381": "Update the new result of my submission:\n\n0.156/0.154 local CV with corresponding confidence values\n0.126/0.127 local CV with random confidence values\n0.084/0.078 LB\n\nThe conclusion is : \nThe value is much stable and much consistent with LB by using random confidence values.\nThanks again for the share @its7171.\n\n\n", "710647": "Trust CV, I tried predicting with and without the test image masks and found LB score dropped a bit while CV increased a lot. \nHere are my CV vs LB scores (not very good but):\n0.089 -> 0.051\n0.0619 -> 0.030", "716361": "Today i got Map = 0.16068738799186721 (calculated using 20% of training data)\nbut got public lb score = 0.056 :(\n\n\n", "711146": "here is my 0.142 to 0.146 + puts me to .064 \n", "711064": "I get the best LB when the logit is -0.2. But you are 0.1 right? I'm thinking is there any way to find the best logit?", "710392": "I met a different situation... I am wonder which code do you compute the local CV?", "710195": "how much data you are using for calculating local cv? are you using your own gpu or kaggle kernels for this competition?", "710700": "" } }