\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3492127%2F33dd4f76a9b3e7f7a1d29a9f859e80fb%2F1.png?generation=1606395471230692&alt=media\")
\n[
Left: When 57,000 steps - 19.194 loss, Right: 60,000 steps - 19.089 loss]I joined this competition late. It was difficult to try many things.
\nSo, I tried a few things.
\n(Maybe my solution is too simple. :) )
My first goal was to be in the top 100.
\nI used Resnet18 and train_full.zarr that have 224 raster_size.
I got 20.56 lb score after about 80,000 steps.
I got 19.23 after 25,000 steps. The performance has been greatly improved.
\nSo, I trained more steps and finally got 15.080.
Ensemble using public kernel's way didn't work for me.
\nAnd ensemble using a simple average way with same model helped public and private lb.
\n(But I didn't have many models. So, I used basic one and weights averaged model i.e. polyak.)
2 days ago I switched my baseline model to resnet50.
\nWhen I trained resnet50, I found the interesting part between valid score and public lb.
After 50,000 steps, public lb was 19.132 that it is bad compared to resnet34. But valid score is a pretty good.
I checked resnet34 and got a little bad valid score.
\nAnd also I observed mean loss when I trained resnet50.
\n
\n[Left: When 57,000 steps - 19.194 loss, Right: 60,000 steps - 19.089 loss]
In that points, lb scores are:
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3492127%2Fe307542c1bcee60540d08583a092e717%2F2.png?generation=1606396024722504&alt=media\")
I just trained resnet50 after 100,000 steps and got public lb score 16.076.
\n(I have no time for training. haha)
I selected resnet34 and resnet50.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3492127%2Fcf29ed6aadd3b67951817754a9691fbe%2F3.png?generation=1606396517653027&alt=media\")
As you can see, resnet50 got better private lb despite public lb score.
\n(I wasn't sure, so I didn't choose the ensemble model.)
I did some other experiments such as raster size - 300 and finetuning model using high raster size, but It is not good for me. Single model is best. haha
\nI tried @pestipeti advices for improving training speed but my hardware is not good for this. (no spaces and no time) Anyway, Thanks all. :)
I joined this competition late. It was difficult to try many things.
\nSo, I tried a few things.
\n(Maybe my solution is too simple. :) )
My first goal was to be in the top 100.
\nI used Resnet18 and train_full.zarr that have 224 raster_size.
I got 20.56 lb score after about 80,000 steps.
I got 19.23 after 25,000 steps. The performance has been greatly improved.
\nSo, I trained more steps and finally got 15.080.
Ensemble using public kernel's way didn't work for me.
\nAnd ensemble using a simple average way with same model helped public and private lb.
\n(But I didn't have many models. So, I used basic one and weights averaged model i.e. polyak.)
2 days ago I switched my baseline model to resnet50.
\nWhen I trained resnet50, I found the interesting part between valid score and public lb.
After 50,000 steps, public lb was 19.132 that it is bad compared to resnet34. But valid score is a pretty good.
I checked resnet34 and got a little bad valid score.
\nAnd also I observed mean loss when I trained resnet50.
\n
\n[Left: When 57,000 steps - 19.194 loss, Right: 60,000 steps - 19.089 loss]
In that points, lb scores are:
\n
I just trained resnet50 after 100,000 steps and got public lb score 16.076.
\n(I have no time for training. haha)
I selected resnet34 and resnet50.
\n
As you can see, resnet50 got better private lb despite public lb score.
\n(I wasn't sure, so I didn't choose the ensemble model.)
I did some other experiments such as raster size - 300 and finetuning model using high raster size, but It is not good for me. Single model is best. haha
\nI tried @pestipeti advices for improving training speed but my hardware is not good for this. (no spaces and no time) Anyway, Thanks all. :)
Great job! What was the batch size that you used?
", "rawMarkdown": "Great job! What was the batch size that you used?", "votes": 1, "replies": [ { "id": 1093355, "postDate": "2020-11-27T16:30:21.413Z", "content": "I used 256 batch size for train. :)
\n@louis925
I see! Looks like larger batch size is really important.
", "rawMarkdown": "I see! Looks like larger batch size is really important.", "votes": 1 }, { "id": 1093519, "postDate": "2020-11-27T19:41:22.740Z", "content": "I just trained 100,000 steps. Maybe I trained model more. lb score is much higher I think
", "rawMarkdown": "I just trained 100,000 steps. Maybe I trained model more. lb score is much higher I think" } ] }, { "id": 1092234, "postDate": "2020-11-26T16:25:40.650Z", "rawMarkdown": "", "votes": 1, "isDeleted": true, "replies": [ { "id": 1092254, "postDate": "2020-11-26T16:40:41.023Z", "content": "Yes, it is a single model.
Opt - Adam
\nlr - 1e5 that is initial and I used linear schedule with warmup. I also tried other lr scheduler i.e. CosineAnnealingWarmRestarts but it is not good for me.
please, see 'linear schedule with warmup' haha @kramadhari
", "rawMarkdown": "please, see 'linear schedule with warmup' haha @kramadhari " } ] } ], "comments": [ { "id": 1092478, "author_name": "Louis Yang", "author_url": "", "post_date": "2020-11-26T21:16:34.050000", "content": "Great job! What was the batch size that you used?
", "votes": 1, "replies": [ { "id": 1093355, "author_name": "Heroseo", "author_url": "", "post_date": "2020-11-27T16:30:21.413000", "content": "I used 256 batch size for train. :)
\n@louis925
I see! Looks like larger batch size is really important.
", "votes": 1, "replies": [] }, { "id": 1093519, "author_name": "Heroseo", "author_url": "", "post_date": "2020-11-27T19:41:22.740000", "content": "I just trained 100,000 steps. Maybe I trained model more. lb score is much higher I think
", "votes": 0, "replies": [] } ] }, { "id": 1092234, "author_name": "", "author_url": "", "post_date": "2020-11-26T16:25:40.650000", "content": "", "votes": 1, "replies": [ { "id": 1092254, "author_name": "Heroseo", "author_url": "", "post_date": "2020-11-26T16:40:41.023000", "content": "Yes, it is a single model.
Opt - Adam
\nlr - 1e5 that is initial and I used linear schedule with warmup. I also tried other lr scheduler i.e. CosineAnnealingWarmRestarts but it is not good for me.
please, see 'linear schedule with warmup' haha @kramadhari
", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "1092009": "# Intro\nI joined this competition late. It was difficult to try many things.\nSo, I tried a few things.\n(Maybe my solution is too simple. :) )\n\n\n# First Step\nMy first goal was to be in the top 100.\nI used Resnet18 and train_full.zarr that have 224 raster_size.\n\nI got **20.56** lb score after about `80,000` steps.\n\n\n# Second Step - Switch to Resnet34\nI got **19.23** after `25,000` steps. The performance has been greatly improved.\nSo, I trained more steps and finally got 15.080.\n\nEnsemble using public kernel's way didn't work for me.\nAnd ensemble using a simple average way with same model helped public and private lb.\n(But I didn't have many models. So, I used basic one and weights averaged model i.e. polyak.)\n\n- I got public lb score of **14.91** \n\n\n\n# Third Step - Switch to Resnet50\n2 days ago I switched my baseline model to resnet50.\nWhen I trained resnet50, I found the interesting part between valid score and public lb.\n\nAfter `50,000` steps, public lb was **19.132** that it is bad compared to resnet34. But valid score is a pretty good.\n- **12.504**\n\nI checked resnet34 and got a little bad valid score.\n- **16.66**\n\nAnd also I observed mean loss when I trained resnet50.\n\n[`Left`: When `57,000` steps - 19.194 loss, `Right`: `60,000` steps - 19.089 loss]\n\nIn that points, lb scores are:\n\n\nI just trained resnet50 after `100,000` steps and got public lb score **16.076**.\n(I have no time for training. haha)\n\n# Final\nI selected resnet34 and resnet50.\n\n\nAs you can see, resnet50 got better private lb despite public lb score.\n(I wasn't sure, so I didn't choose the ensemble model.)\n\n# Etc.\nI did some other experiments such as raster size - 300 and finetuning model using high raster size, but It is not good for me. Single model is best. haha\nI tried @pestipeti advices for improving training speed but my hardware is not good for this. (no spaces and no time) Anyway, Thanks all. :)", "1092478": "Great job! What was the batch size that you used?", "1092234": "" } }