From the past segmentation competitions SIIM, Steel..., it seems classification on empty-mask v.s non-empty-mask always helpful to the overall performance. As far as i know, at least three options, \n- training a separate classifier and remove mask from seg model output\n- add a classification path onto the segmentation model as multi-task\n- the 'deep supervisioin'
\n\nDoes anyone had any success with classification task in any form? I'll report any progress later on.
", "messages": [ { "id": "667335", "postDate": "11/07/2019 06:04:31", "content": "From the past segmentation competitions SIIM, Steel..., it seems classification on empty-mask v.s non-empty-mask always helpful to the overall performance. As far as i know, at least three options, \n- training a separate classifier and remove mask from seg model output\n- add a classification path onto the segmentation model as multi-task\n- the 'deep supervisioin'
\n\nDoes anyone had any success with classification task in any form? I'll report any progress later on.
", "rawMarkdown": "From the past segmentation competitions SIIM, Steel..., it seems classification on empty-mask v.s non-empty-mask always helpful to the overall performance. As far as i know, at least three options, \n- training a separate classifier and remove mask from seg model output\n- add a classification path onto the segmentation model as multi-task\n- the 'deep supervisioin'\n\nDoes anyone had any success with classification task in any form? I'll report any progress later on.", "votes": null }, { "id": "667363", "postDate": "11/07/2019 06:36:06", "content": "I was using the first one but trying the second one right now.
", "rawMarkdown": "I was using the first one but trying the second one right now.", "votes": null }, { "id": "667369", "postDate": "11/07/2019 06:43:51", "content": "But I don't think my classifier was perform very great, just soso I guess. However, my segmentation models have higher recall at the selected threshold, so classifier still remove some false positive for me and gain some score.
", "rawMarkdown": "But I don't think my classifier was perform very great, just soso I guess. However, my segmentation models have higher recall at the selected threshold, so classifier still remove some false positive for me and gain some score.", "votes": null }, { "id": "667377", "postDate": "11/07/2019 06:52:30", "content": "Sounds good! Another good news for training a classifier is that 0-1 imbalance is less severe in this competition.
", "rawMarkdown": "Sounds good! Another good news for training a classifier is that 0-1 imbalance is less severe in this competition.", "votes": null }, { "id": "667412", "postDate": "11/07/2019 07:53:56", "content": "I don't think we should remove empty-mask in this competition , so how about try the second directly ? Maybe it will reduce the overfitting.
", "rawMarkdown": "I don't think we should remove empty-mask in this competition , so how about try the second directly ? Maybe it will reduce the overfitting.", "votes": null }, { "id": "667443", "postDate": "11/07/2019 08:33:26", "content": "I was using the first one method, it already boosted my score around 0.002.\nMaybe the rest methods are better for performance, but separating things helps us easier to optimize each part, I thought.
", "rawMarkdown": "I was using the first one method, it already boosted my score around 0.002.\nMaybe the rest methods are better for performance, but separating things helps us easier to optimize each part, I thought.", "votes": null }, { "id": "667575", "postDate": "11/07/2019 12:00:07", "content": "First attempt of option 1 -- seems work
\n\nTrained an efficientnet-b5 classifier.\nremoved 30 masks with highest confidence, LB improve .001
", "rawMarkdown": "First attempt of option 1 -- seems work\n\nTrained an efficientnet-b5 classifier.\nremoved 30 masks with highest confidence, LB improve .001", "votes": null }, { "id": "667586", "postDate": "11/07/2019 12:20:56", "content": "Ok , I failed at option 2 , I switch to the option 1 .
", "rawMarkdown": "Ok , I failed at option 2 , I switch to the option 1 .", "votes": null }, { "id": "667642", "postDate": "11/07/2019 13:47:19", "content": "I also had bad result on option 2. In my case, the auxiliary classifier stop to improve sooner than segmentation model, so the classifier start to overfit. The highest validation accuracy of auxiliary classifier is 0.76 only. But I think it still have much room to improve, since I only tested on 256x384 resolution with 30 epochs. But it can get about 0.645 LB score, which is pretty decent I thought.
", "rawMarkdown": "I also had bad result on option 2. In my case, the auxiliary classifier stop to improve sooner than segmentation model, so the classifier start to overfit. The highest validation accuracy of auxiliary classifier is 0.76 only. But I think it still have much room to improve, since I only tested on 256x384 resolution with 30 epochs. But it can get about 0.645 LB score, which is pretty decent I thought.", "votes": null }, { "id": "667655", "postDate": "11/07/2019 14:13:02", "content": "Yes , I'm also meet this problem like , it's a little strange here that after adding the auxiliary classifier , my models tends to overfit very quickly than before. But my GPU cost me a dollar per hour , so I will not test it again .\nBest wish to you.
", "rawMarkdown": "Yes , I'm also meet this problem like , it's a little strange here that after adding the auxiliary classifier , my models tends to overfit very quickly than before. But my GPU cost me a dollar per hour , so I will not test it again .\nBest wish to you.", "votes": null }, { "id": "667659", "postDate": "11/07/2019 14:17:46", "content": "That's quite expensive. Good luck with you.
", "rawMarkdown": "That's quite expensive. Good luck with you.", "votes": null }, { "id": "668206", "postDate": "11/08/2019 05:46:01", "content": "at first look, it seems that classifier can have only 4 class. but there are alternatives.
\n\ne.g. train predictor to output cloud area instead (area is normalized to 0 to 1 to mimic noisy label like [0, 0.1,0, 0.5]
\n\nwe consider sub class as
\n1a = fish with area > 50%\n1b = fish with area < 50%\n2a = sugar with area > 50%\n....
we can consider additional class as\n2: overlap of fish and sugar > 50% iou,\netc ...
", "rawMarkdown": "at first look, it seems that classifier can have only 4 class. but there are alternatives.\n\ne.g. train predictor to output cloud area instead (area is normalized to 0 to 1 to mimic noisy label like [0, 0.1,0, 0.5]\n\nwe consider sub class as \n1a = fish with area > 50%\n1b = fish with area < 50%\n2a = sugar with area > 50%\n....\n\n\nwe can consider additional class as\n2: overlap of fish and sugar > 50% iou,\netc ...", "votes": null }, { "id": "668211", "postDate": "11/08/2019 05:53:48", "content": "more one unconventional idea:\n- predict number of cloud type\n- another ranking predictor to sort the most likely cloud type present
", "rawMarkdown": "more one unconventional idea:\n- predict number of cloud type\n- another ranking predictor to sort the most likely cloud type present", "votes": null }, { "id": "668472", "postDate": "11/08/2019 13:11:40", "content": "A lot of ideas deserve to try, thanks!
\n\nBut I think my empty-nonempty classifier still perform bad, I compare \nmethod 1: fix pixel_threshold and min_size, use classifier to remove masks\nmethod 2: grid search pixel_threshold and min_size, use min_size to remove masks\nsame model, method 2 got higher LB
", "rawMarkdown": "A lot of ideas deserve to try, thanks!\n\nBut I think my empty-nonempty classifier still perform bad, I compare \nmethod 1: fix pixel_threshold and min_size, use classifier to remove masks\nmethod 2: grid search pixel_threshold and min_size, use min_size to remove masks\nsame model, method 2 got higher LB", "votes": null }, { "id": "671081", "postDate": "11/12/2019 08:28:01", "content": "@niuddd I am trying option 1. But CV score seems not good enough. I got only F1 0.7 (average on 4 classes). What you got?
", "rawMarkdown": "niuddd I am trying option 1. But CV score seems not good enough. I got only F1 0.7 (average on 4 classes). What you got?", "votes": null }, { "id": "671131", "postDate": "11/12/2019 09:45:12", "content": "average f1 0.75, but I still use min_size to classify empty mask...it works fine
", "rawMarkdown": "average f1 0.75, but I still use min_size to classify empty mask...it works fine", "votes": null } ], "comments": [ { "id": 667363, "author_name": "xiejialun", "author_url": "", "post_date": "11/07/2019 06:36:06", "content": "I was using the first one but trying the second one right now.
", "votes": null, "replies": [ { "id": 667369, "author_name": "xiejialun", "author_url": "", "post_date": "11/07/2019 06:43:51", "content": "But I don't think my classifier was perform very great, just soso I guess. However, my segmentation models have higher recall at the selected threshold, so classifier still remove some false positive for me and gain some score.
", "votes": null, "replies": [] }, { "id": 667377, "author_name": "niuddd", "author_url": "", "post_date": "11/07/2019 06:52:30", "content": "Sounds good! Another good news for training a classifier is that 0-1 imbalance is less severe in this competition.
", "votes": null, "replies": [] } ] }, { "id": 667412, "author_name": "xujingzhao", "author_url": "", "post_date": "11/07/2019 07:53:56", "content": "I don't think we should remove empty-mask in this competition , so how about try the second directly ? Maybe it will reduce the overfitting.
", "votes": null, "replies": [ { "id": 667586, "author_name": "xujingzhao", "author_url": "", "post_date": "11/07/2019 12:20:56", "content": "Ok , I failed at option 2 , I switch to the option 1 .
", "votes": null, "replies": [] }, { "id": 667642, "author_name": "xiejialun", "author_url": "", "post_date": "11/07/2019 13:47:19", "content": "I also had bad result on option 2. In my case, the auxiliary classifier stop to improve sooner than segmentation model, so the classifier start to overfit. The highest validation accuracy of auxiliary classifier is 0.76 only. But I think it still have much room to improve, since I only tested on 256x384 resolution with 30 epochs. But it can get about 0.645 LB score, which is pretty decent I thought.
", "votes": null, "replies": [] }, { "id": 667655, "author_name": "xujingzhao", "author_url": "", "post_date": "11/07/2019 14:13:02", "content": "Yes , I'm also meet this problem like , it's a little strange here that after adding the auxiliary classifier , my models tends to overfit very quickly than before. But my GPU cost me a dollar per hour , so I will not test it again .\nBest wish to you.
", "votes": null, "replies": [] }, { "id": 667659, "author_name": "xiejialun", "author_url": "", "post_date": "11/07/2019 14:17:46", "content": "That's quite expensive. Good luck with you.
", "votes": null, "replies": [] } ] }, { "id": 667443, "author_name": "thongpb", "author_url": "", "post_date": "11/07/2019 08:33:26", "content": "I was using the first one method, it already boosted my score around 0.002.\nMaybe the rest methods are better for performance, but separating things helps us easier to optimize each part, I thought.
", "votes": null, "replies": [] }, { "id": 667575, "author_name": "niuddd", "author_url": "", "post_date": "11/07/2019 12:00:07", "content": "First attempt of option 1 -- seems work
\n\nTrained an efficientnet-b5 classifier.\nremoved 30 masks with highest confidence, LB improve .001
", "votes": null, "replies": [] }, { "id": 668206, "author_name": "hengck23", "author_url": "", "post_date": "11/08/2019 05:46:01", "content": "at first look, it seems that classifier can have only 4 class. but there are alternatives.
\n\ne.g. train predictor to output cloud area instead (area is normalized to 0 to 1 to mimic noisy label like [0, 0.1,0, 0.5]
\n\nwe consider sub class as
\n1a = fish with area > 50%\n1b = fish with area < 50%\n2a = sugar with area > 50%\n....
we can consider additional class as\n2: overlap of fish and sugar > 50% iou,\netc ...
", "votes": null, "replies": [ { "id": 668211, "author_name": "hengck23", "author_url": "", "post_date": "11/08/2019 05:53:48", "content": "more one unconventional idea:\n- predict number of cloud type\n- another ranking predictor to sort the most likely cloud type present
", "votes": null, "replies": [] }, { "id": 668472, "author_name": "niuddd", "author_url": "", "post_date": "11/08/2019 13:11:40", "content": "A lot of ideas deserve to try, thanks!
\n\nBut I think my empty-nonempty classifier still perform bad, I compare \nmethod 1: fix pixel_threshold and min_size, use classifier to remove masks\nmethod 2: grid search pixel_threshold and min_size, use min_size to remove masks\nsame model, method 2 got higher LB
", "votes": null, "replies": [] } ] }, { "id": 671081, "author_name": "tmhung", "author_url": "", "post_date": "11/12/2019 08:28:01", "content": "@niuddd I am trying option 1. But CV score seems not good enough. I got only F1 0.7 (average on 4 classes). What you got?
", "votes": null, "replies": [ { "id": 671131, "author_name": "niuddd", "author_url": "", "post_date": "11/12/2019 09:45:12", "content": "average f1 0.75, but I still use min_size to classify empty mask...it works fine
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "667335": "From the past segmentation competitions SIIM, Steel..., it seems classification on empty-mask v.s non-empty-mask always helpful to the overall performance. As far as i know, at least three options, \n- training a separate classifier and remove mask from seg model output\n- add a classification path onto the segmentation model as multi-task\n- the 'deep supervisioin'\n\nDoes anyone had any success with classification task in any form? I'll report any progress later on.", "667363": "I was using the first one but trying the second one right now.", "667369": "But I don't think my classifier was perform very great, just soso I guess. However, my segmentation models have higher recall at the selected threshold, so classifier still remove some false positive for me and gain some score.", "667377": "Sounds good! Another good news for training a classifier is that 0-1 imbalance is less severe in this competition.", "667412": "I don't think we should remove empty-mask in this competition , so how about try the second directly ? Maybe it will reduce the overfitting.", "667443": "I was using the first one method, it already boosted my score around 0.002.\nMaybe the rest methods are better for performance, but separating things helps us easier to optimize each part, I thought.", "667575": "First attempt of option 1 -- seems work\n\nTrained an efficientnet-b5 classifier.\nremoved 30 masks with highest confidence, LB improve .001", "667586": "Ok , I failed at option 2 , I switch to the option 1 .", "667642": "I also had bad result on option 2. In my case, the auxiliary classifier stop to improve sooner than segmentation model, so the classifier start to overfit. The highest validation accuracy of auxiliary classifier is 0.76 only. But I think it still have much room to improve, since I only tested on 256x384 resolution with 30 epochs. But it can get about 0.645 LB score, which is pretty decent I thought.", "667655": "Yes , I'm also meet this problem like , it's a little strange here that after adding the auxiliary classifier , my models tends to overfit very quickly than before. But my GPU cost me a dollar per hour , so I will not test it again .\nBest wish to you.", "667659": "That's quite expensive. Good luck with you.", "668206": "at first look, it seems that classifier can have only 4 class. but there are alternatives.\n\ne.g. train predictor to output cloud area instead (area is normalized to 0 to 1 to mimic noisy label like [0, 0.1,0, 0.5]\n\nwe consider sub class as \n1a = fish with area > 50%\n1b = fish with area < 50%\n2a = sugar with area > 50%\n....\n\n\nwe can consider additional class as\n2: overlap of fish and sugar > 50% iou,\netc ...", "668211": "more one unconventional idea:\n- predict number of cloud type\n- another ranking predictor to sort the most likely cloud type present", "668472": "A lot of ideas deserve to try, thanks!\n\nBut I think my empty-nonempty classifier still perform bad, I compare \nmethod 1: fix pixel_threshold and min_size, use classifier to remove masks\nmethod 2: grid search pixel_threshold and min_size, use min_size to remove masks\nsame model, method 2 got higher LB", "671081": "niuddd I am trying option 1. But CV score seems not good enough. I got only F1 0.7 (average on 4 classes). What you got?", "671131": "average f1 0.75, but I still use min_size to classify empty mask...it works fine" }, "source": "meta" }