{ "id": 39157, "title": "some experiments and LB scores", "url": "/competitions/carvana-image-masking-challenge/discussion/39157", "author_name": "", "post_date": "2017-09-08T07:47:26.597483500Z", "votes": 35, "comment_count": 20, "views": 0, "content": "

With more decimal place for the LB scores, we can see the effects of the following experiments:

\n\n
    \n

  1. baseline results Unet-1024: LB 0.9963

    2. \n

  2. above (1) + shrink 1 pixel : LB 0.9960

    3. \n

  3. above (1) + expand 1 pixel : LB 0.9960 (better than (2) by score ranking)

    4. \n

  4. above (1) + expand 2 pixel : LB 0.9956

    5. \n

  5. above (1) + expand 4 pixel : LB 0.9944

    6. \n
\n\n
\n\n
    \n

  1. baseline results Unet-1024 (better network structure): LB 0.9966

    2. \n

  2. above (1) +pesudo-lable learning on test images : LB 0.9968

    3. \n
\n\n
\n\n
    \n

  1. baseline results Unet-1024 (trained on hq images, test on hq images): LB 0.9968

    2. \n

  2. above (1) (trained on hq images, test on old images): LB 0.9967

    3. \n
\n\n
\n\n
    \n

  1. baseline results Unet-512 : LB 0.9953

    2. \n

  2. above(1) + ensemble of test-time augmentation (shift,scale, reflect) : LB 0.9957

    3. \n
", "messages": [ { "id": "219443", "postDate": "09/08/2017 07:47:26", "content": "

With more decimal place for the LB scores, we can see the effects of the following experiments:

\n\n
    \n

  1. baseline results Unet-1024: LB 0.9963

    2. \n

  2. above (1) + shrink 1 pixel : LB 0.9960

    3. \n

  3. above (1) + expand 1 pixel : LB 0.9960 (better than (2) by score ranking)

    4. \n

  4. above (1) + expand 2 pixel : LB 0.9956

    5. \n

  5. above (1) + expand 4 pixel : LB 0.9944

    6. \n
\n\n
\n\n
    \n

  1. baseline results Unet-1024 (better network structure): LB 0.9966

    2. \n

  2. above (1) +pesudo-lable learning on test images : LB 0.9968

    3. \n
\n\n
\n\n
    \n

  1. baseline results Unet-1024 (trained on hq images, test on hq images): LB 0.9968

    2. \n

  2. above (1) (trained on hq images, test on old images): LB 0.9967

    3. \n
\n\n
\n\n
    \n

  1. baseline results Unet-512 : LB 0.9953

    2. \n

  2. above(1) + ensemble of test-time augmentation (shift,scale, reflect) : LB 0.9957

    3. \n
", "rawMarkdown": "With more decimal place for the LB scores, we can see the effects of the following experiments:\n\n 1. baseline results Unet-1024: LB 0.9963\n\n 2. above (1) + shrink 1 pixel : LB 0.9960\n\n 3. above (1) + expand 1 pixel : LB 0.9960 (better than (2) by score ranking)\n\n 4. above (1) + expand 2 pixel : LB 0.9956 \n\n 5. above (1) + expand 4 pixel : LB 0.9944 \n\n-------------------------------------------------------------\n\n 1. baseline results Unet-1024 (better network structure): LB 0.9966\n\n 2. above (1) +pesudo-lable learning on test images : LB 0.9968\n\n-------------------------------------------------------------\n\n 1. baseline results Unet-1024 (trained on hq images, test on hq images): LB 0.9968\n\n 2. above (1) (trained on hq images, test on old images): LB 0.9967\n\n-------------------------------------------------------------\n\n 1. baseline results Unet-512 : LB 0.9953\n\n 2. above(1) + ensemble of test-time augmentation (shift,scale, reflect) : LB 0.9957", "votes": null }, { "id": "219453", "postDate": "09/08/2017 08:39:31", "content": "

What is the meaning of hq images?

", "rawMarkdown": "What is the meaning of hq images?", "votes": null }, { "id": "219458", "postDate": "09/08/2017 09:00:24", "content": "

High quality images that was added recently.

", "rawMarkdown": "High quality images that was added recently.", "votes": null }, { "id": "219462", "postDate": "09/08/2017 09:28:32", "content": "

Nice insights Heng! I have two questions:

\n\n

1) Is \"better network structure\" is the structure from recent files that you have shared? So it's basically decreasing the number of layers by one (from baseline Unet 0.9963) and increasing the number of feats on the highest down layer to 24 (from 8 in baseline Unet 0.9963).

\n\n

I'm wondering because it couldn't be ran on my 1080 and max n_features on the first down layer I can use is 12 - but it doesnt increase score

\n\n

2) How did you create ensemble? I've tried concatenating the n-1 layers from different baselines unets and then simple block (conv2d_batch_relu * 3) - that gave me the same score as the best baseline UNet. Then I tried summation instead of concat - it gave lower score. I found a few papers about ensembling CNNs, but there wasn't any proposed structure. I feel like I'm missing something. Can you point out how to proper construct ensemble of CNNs?

", "rawMarkdown": "Nice insights Heng! I have two questions:\n\n1) Is \"better network structure\" is the structure from recent files that you have shared? So it's basically decreasing the number of layers by one (from baseline Unet 0.9963) and increasing the number of feats on the highest down layer to 24 (from 8 in baseline Unet 0.9963).\n\nI'm wondering because it couldn't be ran on my 1080 and max n_features on the first down layer I can use is 12 - but it doesnt increase score\n\n2) How did you create ensemble? I've tried concatenating the n-1 layers from different baselines unets and then simple block (conv2d_batch_relu * 3) - that gave me the same score as the best baseline UNet. Then I tried summation instead of concat - it gave lower score. I found a few papers about ensembling CNNs, but there wasn't any proposed structure. I feel like I'm missing something. Can you point out how to proper construct ensemble of CNNs?", "votes": null }, { "id": "219604", "postDate": "09/08/2017 18:54:10", "content": "

based on this information, it would seem that the baseline unet-1024 trained and tested on hq images (0.9968) might work well with the pseudo label learning on test images?

", "rawMarkdown": "based on this information, it would seem that the baseline unet-1024 trained and tested on hq images (0.9968) might work well with the pseudo label learning on test images?", "votes": null }, { "id": "219610", "postDate": "09/08/2017 19:34:31", "content": "

First question Heng: What's your best score on training your network can achieve? How much CAN it overfit?\nSecond question: How long does one training epoch (4000-5000 samples) take for you in minutes?

", "rawMarkdown": "First question Heng: What's your best score on training your network can achieve? How much CAN it overfit?\nSecond question: How long does one training epoch (4000-5000 samples) take for you in minutes?", "votes": null }, { "id": "219697", "postDate": "09/09/2017 07:06:56", "content": "

1) Is \"better network structure\" ...

\n\n

yes

\n\n

2) How did you create ensemble? ...

\n\n

you have to draw the prediction on the images. you will find that in some cases, you have to use 'prediction1 union prediction2'. In other cases, it is better to use intersect, average, voting,etc. It differs for different kind of augmentation like shift and scale, color/brightness changes

", "rawMarkdown": "1) Is \"better network structure\" ...\n\nyes\n\n2) How did you create ensemble? ...\n\nyou have to draw the prediction on the images. you will find that in some cases, you have to use 'prediction1 union prediction2'. In other cases, it is better to use intersect, average, voting,etc. It differs for different kind of augmentation like shift and scale, color/brightness changes", "votes": null }, { "id": "219698", "postDate": "09/09/2017 07:10:08", "content": "

1.What's your best score on training your network can achieve?

\n\n

I can train up to 0.998, by adjusting the learning rate. But if your train loss on non-augmented train images exceeds 0.99715, the LB scores decrease.

\n\n

2.How long does one training epoch (4000-5000 samples)

\n\n

for unet 1024, it takes about 16 min

", "rawMarkdown": "1.What's your best score on training your network can achieve?\n\nI can train up to 0.998, by adjusting the learning rate. But if your train loss on non-augmented train images exceeds 0.99715, the LB scores decrease.\n\n\n2.How long does one training epoch (4000-5000 samples)\n\nfor unet 1024, it takes about 16 min", "votes": null }, { "id": "219746", "postDate": "09/09/2017 12:58:32", "content": "

Interesting! Thank you very much.\nI am training on 1280 crops and cannot get good results with 1024x1024 with my architecture :/

", "rawMarkdown": "Interesting! Thank you very much.\nI am training on 1280 crops and cannot get good results with 1024x1024 with my architecture :/", "votes": null }, { "id": "219748", "postDate": "09/09/2017 13:03:11", "content": "

training 1280x1280 is too long for me. Instead of focus on single network, I suggest you train on different size (aspect 1:1, 1:2, etc ... 1024,1280, ... even upsizing) and network (uNet, etc change structure slightly, ) combinations. Then ensemble them together. This can easily get to 0.9969.

\n\n

The remaining work is some good image pre-processing methods to remove shadows i think.

", "rawMarkdown": "training 1280x1280 is too long for me. Instead of focus on single network, I suggest you train on different size (aspect 1:1, 1:2, etc ... 1024,1280, ... even upsizing) and network (uNet, etc change structure slightly, ) combinations. Then ensemble them together. This can easily get to 0.9969.\n\nThe remaining work is some good image pre-processing methods to remove shadows i think.", "votes": null }, { "id": "219750", "postDate": "09/09/2017 13:07:39", "content": "

Hi, Heng\nThanks for the info.\nWhat part of the test data do you use for pseudo-label?

\n\n

In Amazon Planet this method performed well on public LB, but on private LB the results were much worse.\nDon't you think that on this task situation will repeat?

", "rawMarkdown": "Hi, Heng\nThanks for the info.\nWhat part of the test data do you use for pseudo-label?\n\nIn Amazon Planet this method performed well on public LB, but on private LB the results were much worse.\nDon't you think that on this task situation will repeat?", "votes": null }, { "id": "219756", "postDate": "09/09/2017 13:28:05", "content": "

I can currently get 0.9967 on leaderboard without semi-supervised learning I think and training takes ~ 12 hours, while test-predictions take ~10 hours. I would love to get 0.9969 on a single network for this competition...\nDon't you have 4 Titans in your workstation if I remember correctly from the last competition? :D

", "rawMarkdown": "I can currently get 0.9967 on leaderboard without semi-supervised learning I think and training takes ~ 12 hours, while test-predictions take ~10 hours. I would love to get 0.9969 on a single network for this competition...\nDon't you have 4 Titans in your workstation if I remember correctly from the last competition? :D", "votes": null }, { "id": "220216", "postDate": "09/11/2017 14:40:45", "content": "

Hi, Heng CherKeng!\nCan you explain a bit \"pesudo-lable learning on test images\" or maybe provide a link with?

", "rawMarkdown": "Hi, Heng CherKeng!\nCan you explain a bit \"pesudo-lable learning on test images\" or maybe provide a link with?", "votes": null }, { "id": "220225", "postDate": "09/11/2017 14:58:27", "content": "

can't find this post, but basically you predict masks on test and use them to increase the train set - so your train set now = [train_images, test_images] and ground truth masks = [train_masks, predicted_test_masks]

", "rawMarkdown": "can't find this post, but basically you predict masks on test and use them to increase the train set - so your train set now = [train_images, test_images] and ground truth masks = [train_masks, predicted_test_masks]", "votes": null }, { "id": "220246", "postDate": "09/11/2017 15:42:54", "content": "

https://shaoanlu.wordpress.com/2017/04/10/a-simple-pseudo-labeling-function-implementation-in-keras/

", "rawMarkdown": "https://shaoanlu.wordpress.com/2017/04/10/a-simple-pseudo-labeling-function-implementation-in-keras/", "votes": null }, { "id": "220247", "postDate": "09/11/2017 15:44:34", "content": "

I remember the post, but why doesn't this just overfit the data and not generalize well?

", "rawMarkdown": "I remember the post, but why doesn't this just overfit the data and not generalize well?", "votes": null }, { "id": "220507", "postDate": "09/12/2017 11:08:07", "content": "

I think pseudo-labeling acts somehow like adding (informative but not so helpful) noise to each step of gradient descent. And this is perhaps why we usually blend [train_images test_images] in certain ratio, say, 3:1.

\n\n

As a result, the trained model will be more robust and have better generalization.

", "rawMarkdown": "I think pseudo-labeling acts somehow like adding (informative but not so helpful) noise to each step of gradient descent. And this is perhaps why we usually blend [train_images test_images] in certain ratio, say, 3:1. \n\nAs a result, the trained model will be more robust and have better generalization.", "votes": null }, { "id": "220785", "postDate": "09/13/2017 03:46:54", "content": "

How many test images did you use when dealing with the pseudo label learning on test images?

", "rawMarkdown": "How many test images did you use when dealing with the pseudo label learning on test images?", "votes": null }, { "id": "222944", "postDate": "09/20/2017 16:48:59", "content": "

Hi Heng\nDo you mind sharing how much GPU memory your model takes when training? I have a similar model Unet 1024 to the one you posted. I am using the following code to determine how much memory my model needs: https://stackoverflow.com/a/46216013\nThat code returns 9gb but my script still reports out of memory error when allocating some tensors. I have tesla k80 gpu with 11gb gpu memory.\nI was just wondering how you trained with a model and how you determine how much memory its going to take?

\n\n

Thanks

", "rawMarkdown": "Hi Heng\nDo you mind sharing how much GPU memory your model takes when training? I have a similar model Unet 1024 to the one you posted. I am using the following code to determine how much memory my model needs: https://stackoverflow.com/a/46216013\nThat code returns 9gb but my script still reports out of memory error when allocating some tensors. I have tesla k80 gpu with 11gb gpu memory.\nI was just wondering how you trained with a model and how you determine how much memory its going to take?\n\nThanks", "votes": null }, { "id": "223228", "postDate": "09/21/2017 14:20:07", "content": "

Just wondering how to change to better network structure, is there any guideline?Or just keep trying?

", "rawMarkdown": "Just wondering how to change to better network structure, is there any guideline?Or just keep trying?", "votes": null }, { "id": "223256", "postDate": "09/21/2017 16:01:27", "content": "
    \n

  1. trial and error

    2. \n

  2. read paper and understand why and how they use certain structure

    3. \n

  3. look at the error and think of why and how to solve

    4. \n

  4. measure gradient and signal flow (in forward and backward propagation) and feature map activation values. Improve flow in both direction.

    5. \n

  5. look at feature map results , convolution filters.

    6. \n
\n\n

General hints:

\n\n", "rawMarkdown": "1. trial and error\n\n 2. read paper and understand why and how they use certain structure\n\n 3. look at the error and think of why and how to solve\n\n 4. measure gradient and signal flow (in forward and backward propagation) and feature map activation values. Improve flow in both direction. \n\n 5. look at feature map results , convolution filters.\n\n\nGeneral hints:\n\n - make it deep if you have lots of data. \n\n - increase channels for each conv until you see no improvement\n\n - handle multiscale (and rotation , but rotation is not for this competiton). most CNN are weak in scale and rotation\n\n - ensemble effect in your network, e.g. dropout, residual add connection\n\n - for high resolution segmentation (and small object detection), increase context (receptive field).e.g. devolution, dilated filter, large fitler", "votes": null } ], "comments": [ { "id": 219453, "author_name": "zhangsongwei", "author_url": "", "post_date": "09/08/2017 08:39:31", "content": "

What is the meaning of hq images?

", "votes": null, "replies": [ { "id": 219458, "author_name": "sheriytm", "author_url": "", "post_date": "09/08/2017 09:00:24", "content": "

High quality images that was added recently.

", "votes": null, "replies": [] } ] }, { "id": 219462, "author_name": "heyt0ny", "author_url": "", "post_date": "09/08/2017 09:28:32", "content": "

Nice insights Heng! I have two questions:

\n\n

1) Is \"better network structure\" is the structure from recent files that you have shared? So it's basically decreasing the number of layers by one (from baseline Unet 0.9963) and increasing the number of feats on the highest down layer to 24 (from 8 in baseline Unet 0.9963).

\n\n

I'm wondering because it couldn't be ran on my 1080 and max n_features on the first down layer I can use is 12 - but it doesnt increase score

\n\n

2) How did you create ensemble? I've tried concatenating the n-1 layers from different baselines unets and then simple block (conv2d_batch_relu * 3) - that gave me the same score as the best baseline UNet. Then I tried summation instead of concat - it gave lower score. I found a few papers about ensembling CNNs, but there wasn't any proposed structure. I feel like I'm missing something. Can you point out how to proper construct ensemble of CNNs?

", "votes": null, "replies": [ { "id": 219697, "author_name": "hengck23", "author_url": "", "post_date": "09/09/2017 07:06:56", "content": "

1) Is \"better network structure\" ...

\n\n

yes

\n\n

2) How did you create ensemble? ...

\n\n

you have to draw the prediction on the images. you will find that in some cases, you have to use 'prediction1 union prediction2'. In other cases, it is better to use intersect, average, voting,etc. It differs for different kind of augmentation like shift and scale, color/brightness changes

", "votes": null, "replies": [] }, { "id": 222944, "author_name": "amalhotra", "author_url": "", "post_date": "09/20/2017 16:48:59", "content": "

Hi Heng\nDo you mind sharing how much GPU memory your model takes when training? I have a similar model Unet 1024 to the one you posted. I am using the following code to determine how much memory my model needs: https://stackoverflow.com/a/46216013\nThat code returns 9gb but my script still reports out of memory error when allocating some tensors. I have tesla k80 gpu with 11gb gpu memory.\nI was just wondering how you trained with a model and how you determine how much memory its going to take?

\n\n

Thanks

", "votes": null, "replies": [] } ] }, { "id": 219604, "author_name": "gw00207", "author_url": "", "post_date": "09/08/2017 18:54:10", "content": "

based on this information, it would seem that the baseline unet-1024 trained and tested on hq images (0.9968) might work well with the pseudo label learning on test images?

", "votes": null, "replies": [] }, { "id": 219610, "author_name": "timjoseph", "author_url": "", "post_date": "09/08/2017 19:34:31", "content": "

First question Heng: What's your best score on training your network can achieve? How much CAN it overfit?\nSecond question: How long does one training epoch (4000-5000 samples) take for you in minutes?

", "votes": null, "replies": [ { "id": 219698, "author_name": "hengck23", "author_url": "", "post_date": "09/09/2017 07:10:08", "content": "

1.What's your best score on training your network can achieve?

\n\n

I can train up to 0.998, by adjusting the learning rate. But if your train loss on non-augmented train images exceeds 0.99715, the LB scores decrease.

\n\n

2.How long does one training epoch (4000-5000 samples)

\n\n

for unet 1024, it takes about 16 min

", "votes": null, "replies": [] }, { "id": 219746, "author_name": "timjoseph", "author_url": "", "post_date": "09/09/2017 12:58:32", "content": "

Interesting! Thank you very much.\nI am training on 1280 crops and cannot get good results with 1024x1024 with my architecture :/

", "votes": null, "replies": [] }, { "id": 219748, "author_name": "hengck23", "author_url": "", "post_date": "09/09/2017 13:03:11", "content": "

training 1280x1280 is too long for me. Instead of focus on single network, I suggest you train on different size (aspect 1:1, 1:2, etc ... 1024,1280, ... even upsizing) and network (uNet, etc change structure slightly, ) combinations. Then ensemble them together. This can easily get to 0.9969.

\n\n

The remaining work is some good image pre-processing methods to remove shadows i think.

", "votes": null, "replies": [] }, { "id": 219756, "author_name": "timjoseph", "author_url": "", "post_date": "09/09/2017 13:28:05", "content": "

I can currently get 0.9967 on leaderboard without semi-supervised learning I think and training takes ~ 12 hours, while test-predictions take ~10 hours. I would love to get 0.9969 on a single network for this competition...\nDon't you have 4 Titans in your workstation if I remember correctly from the last competition? :D

", "votes": null, "replies": [] } ] }, { "id": 219750, "author_name": "markpopov", "author_url": "", "post_date": "09/09/2017 13:07:39", "content": "

Hi, Heng\nThanks for the info.\nWhat part of the test data do you use for pseudo-label?

\n\n

In Amazon Planet this method performed well on public LB, but on private LB the results were much worse.\nDon't you think that on this task situation will repeat?

", "votes": null, "replies": [] }, { "id": 220216, "author_name": "wanizz", "author_url": "", "post_date": "09/11/2017 14:40:45", "content": "

Hi, Heng CherKeng!\nCan you explain a bit \"pesudo-lable learning on test images\" or maybe provide a link with?

", "votes": null, "replies": [ { "id": 220225, "author_name": "heyt0ny", "author_url": "", "post_date": "09/11/2017 14:58:27", "content": "

can't find this post, but basically you predict masks on test and use them to increase the train set - so your train set now = [train_images, test_images] and ground truth masks = [train_masks, predicted_test_masks]

", "votes": null, "replies": [] }, { "id": 220246, "author_name": "atom1231", "author_url": "", "post_date": "09/11/2017 15:42:54", "content": "

https://shaoanlu.wordpress.com/2017/04/10/a-simple-pseudo-labeling-function-implementation-in-keras/

", "votes": null, "replies": [] }, { "id": 220247, "author_name": "stevenknguyen", "author_url": "", "post_date": "09/11/2017 15:44:34", "content": "

I remember the post, but why doesn't this just overfit the data and not generalize well?

", "votes": null, "replies": [] }, { "id": 220507, "author_name": "shaoanlu", "author_url": "", "post_date": "09/12/2017 11:08:07", "content": "

I think pseudo-labeling acts somehow like adding (informative but not so helpful) noise to each step of gradient descent. And this is perhaps why we usually blend [train_images test_images] in certain ratio, say, 3:1.

\n\n

As a result, the trained model will be more robust and have better generalization.

", "votes": null, "replies": [] } ] }, { "id": 220785, "author_name": "zhangsongwei", "author_url": "", "post_date": "09/13/2017 03:46:54", "content": "

How many test images did you use when dealing with the pseudo label learning on test images?

", "votes": null, "replies": [] }, { "id": 223228, "author_name": "strideradu", "author_url": "", "post_date": "09/21/2017 14:20:07", "content": "

Just wondering how to change to better network structure, is there any guideline?Or just keep trying?

", "votes": null, "replies": [ { "id": 223256, "author_name": "hengck23", "author_url": "", "post_date": "09/21/2017 16:01:27", "content": "
    \n

  1. trial and error

    2. \n

  2. read paper and understand why and how they use certain structure

    3. \n

  3. look at the error and think of why and how to solve

    4. \n

  4. measure gradient and signal flow (in forward and backward propagation) and feature map activation values. Improve flow in both direction.

    5. \n

  5. look at feature map results , convolution filters.

    6. \n
\n\n

General hints:

\n\n", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "219443": "With more decimal place for the LB scores, we can see the effects of the following experiments:\n\n 1. baseline results Unet-1024: LB 0.9963\n\n 2. above (1) + shrink 1 pixel : LB 0.9960\n\n 3. above (1) + expand 1 pixel : LB 0.9960 (better than (2) by score ranking)\n\n 4. above (1) + expand 2 pixel : LB 0.9956 \n\n 5. above (1) + expand 4 pixel : LB 0.9944 \n\n-------------------------------------------------------------\n\n 1. baseline results Unet-1024 (better network structure): LB 0.9966\n\n 2. above (1) +pesudo-lable learning on test images : LB 0.9968\n\n-------------------------------------------------------------\n\n 1. baseline results Unet-1024 (trained on hq images, test on hq images): LB 0.9968\n\n 2. above (1) (trained on hq images, test on old images): LB 0.9967\n\n-------------------------------------------------------------\n\n 1. baseline results Unet-512 : LB 0.9953\n\n 2. above(1) + ensemble of test-time augmentation (shift,scale, reflect) : LB 0.9957", "219453": "What is the meaning of hq images?", "219458": "High quality images that was added recently.", "219462": "Nice insights Heng! I have two questions:\n\n1) Is \"better network structure\" is the structure from recent files that you have shared? So it's basically decreasing the number of layers by one (from baseline Unet 0.9963) and increasing the number of feats on the highest down layer to 24 (from 8 in baseline Unet 0.9963).\n\nI'm wondering because it couldn't be ran on my 1080 and max n_features on the first down layer I can use is 12 - but it doesnt increase score\n\n2) How did you create ensemble? I've tried concatenating the n-1 layers from different baselines unets and then simple block (conv2d_batch_relu * 3) - that gave me the same score as the best baseline UNet. Then I tried summation instead of concat - it gave lower score. I found a few papers about ensembling CNNs, but there wasn't any proposed structure. I feel like I'm missing something. Can you point out how to proper construct ensemble of CNNs?", "219604": "based on this information, it would seem that the baseline unet-1024 trained and tested on hq images (0.9968) might work well with the pseudo label learning on test images?", "219610": "First question Heng: What's your best score on training your network can achieve? How much CAN it overfit?\nSecond question: How long does one training epoch (4000-5000 samples) take for you in minutes?", "219697": "1) Is \"better network structure\" ...\n\nyes\n\n2) How did you create ensemble? ...\n\nyou have to draw the prediction on the images. you will find that in some cases, you have to use 'prediction1 union prediction2'. In other cases, it is better to use intersect, average, voting,etc. It differs for different kind of augmentation like shift and scale, color/brightness changes", "219698": "1.What's your best score on training your network can achieve?\n\nI can train up to 0.998, by adjusting the learning rate. But if your train loss on non-augmented train images exceeds 0.99715, the LB scores decrease.\n\n\n2.How long does one training epoch (4000-5000 samples)\n\nfor unet 1024, it takes about 16 min", "219746": "Interesting! Thank you very much.\nI am training on 1280 crops and cannot get good results with 1024x1024 with my architecture :/", "219748": "training 1280x1280 is too long for me. Instead of focus on single network, I suggest you train on different size (aspect 1:1, 1:2, etc ... 1024,1280, ... even upsizing) and network (uNet, etc change structure slightly, ) combinations. Then ensemble them together. This can easily get to 0.9969.\n\nThe remaining work is some good image pre-processing methods to remove shadows i think.", "219750": "Hi, Heng\nThanks for the info.\nWhat part of the test data do you use for pseudo-label?\n\nIn Amazon Planet this method performed well on public LB, but on private LB the results were much worse.\nDon't you think that on this task situation will repeat?", "219756": "I can currently get 0.9967 on leaderboard without semi-supervised learning I think and training takes ~ 12 hours, while test-predictions take ~10 hours. I would love to get 0.9969 on a single network for this competition...\nDon't you have 4 Titans in your workstation if I remember correctly from the last competition? :D", "220216": "Hi, Heng CherKeng!\nCan you explain a bit \"pesudo-lable learning on test images\" or maybe provide a link with?", "220225": "can't find this post, but basically you predict masks on test and use them to increase the train set - so your train set now = [train_images, test_images] and ground truth masks = [train_masks, predicted_test_masks]", "220246": "https://shaoanlu.wordpress.com/2017/04/10/a-simple-pseudo-labeling-function-implementation-in-keras/", "220247": "I remember the post, but why doesn't this just overfit the data and not generalize well?", "220507": "I think pseudo-labeling acts somehow like adding (informative but not so helpful) noise to each step of gradient descent. And this is perhaps why we usually blend [train_images test_images] in certain ratio, say, 3:1. \n\nAs a result, the trained model will be more robust and have better generalization.", "220785": "How many test images did you use when dealing with the pseudo label learning on test images?", "222944": "Hi Heng\nDo you mind sharing how much GPU memory your model takes when training? I have a similar model Unet 1024 to the one you posted. I am using the following code to determine how much memory my model needs: https://stackoverflow.com/a/46216013\nThat code returns 9gb but my script still reports out of memory error when allocating some tensors. I have tesla k80 gpu with 11gb gpu memory.\nI was just wondering how you trained with a model and how you determine how much memory its going to take?\n\nThanks", "223228": "Just wondering how to change to better network structure, is there any guideline?Or just keep trying?", "223256": "1. trial and error\n\n 2. read paper and understand why and how they use certain structure\n\n 3. look at the error and think of why and how to solve\n\n 4. measure gradient and signal flow (in forward and backward propagation) and feature map activation values. Improve flow in both direction. \n\n 5. look at feature map results , convolution filters.\n\n\nGeneral hints:\n\n - make it deep if you have lots of data. \n\n - increase channels for each conv until you see no improvement\n\n - handle multiscale (and rotation , but rotation is not for this competiton). most CNN are weak in scale and rotation\n\n - ensemble effect in your network, e.g. dropout, residual add connection\n\n - for high resolution segmentation (and small object detection), increase context (receptive field).e.g. devolution, dilated filter, large fitler" }, "source": "meta" }