{ "id": 243761, "title": "9th place: SWIN/VIT/CAIT + Heng decoder with code: tugstugi's part (updated)", "url": "/competitions/bms-molecular-translation/discussion/243761", "author_name": "", "post_date": "2021-06-03T23:57:21.002960400Z", "votes": 45, "comment_count": 17, "views": 0, "content": "

Team summary: https://www.kaggle.com/c/bms-molecular-translation/discussion/244031
\nCode: https://github.com/tugstugi/pytorch-bms

\n

First, I want thank my team mates @zfturbo @youhanlee @drhabib and the competition organizers. Special thanks to @yasufuminakama providing a starter notebook and @hengck23 for his transformer idea.

\n

I have first started with the Y.Nakama's tokenization and tried resnest with LSTM/GRU with different image sizes. Result so far was showing that using bigger images brings boost in LB. After Heng has released his starter code, I went immediately to 384x384 vit_deit_base_patch16_384 with 3 decoder layers and was able to get CV 1.0 and LB 1.3 without any data augmentation and jumped to the 4th place. Even I had access to a 8x GPU machine, I didn't feel confident enough to continue alone, so merged with @zfturbo @youhanlee @drhabib which was a good decision. They were already training with pseudo, so I have finetuned my model on their pseudo and scored LB0.86. At same time, @yohanlee has trained swin_base_patch4_window12_384. There was an interesting observation, VIT showed sometimes NAN during mixed precision training, but SWIN didn't have this issue! So my main models were now the swin transformer encoders (VIT was better but slower). With the 3x decoder layer, our highest LB was around 0.8. So I have started to experiment, with deeper decoders and different head configurations. Best single model so far was swin_base_patch4_window12_384 with 12x decoder layers and 16 heads which reached LB 0.74. For experementing with different decoders, I first freeze the encoder and train only decoder and next stage unfreeze everything which enables faster experimenting. @yohanlee also trained cait_m36_384 which scored only LB 0.8 but helped in the final ensembling.

\n

What worked so far:

\n\n

Not sure (no impact on LB, but also didn't hurt it):

\n\n

What didn't work:

\n\n

Even the beam search was working good for LSTM, I haven't implemented a beam search for transformer, because the inferencing was slow, so I thought it was not worth.

\n

After investigating my model outputs, I have discovered many errors in m0/m1. So i have trained some resnet18 and resnet50 models on 256x256 and 512x512 images, which helped to bring also 0.02 gain on LB.

", "messages": [ { "id": "1334966", "postDate": "06/03/2021 23:57:21", "content": "

Team summary: https://www.kaggle.com/c/bms-molecular-translation/discussion/244031
\nCode: https://github.com/tugstugi/pytorch-bms

\n

First, I want thank my team mates @zfturbo @youhanlee @drhabib and the competition organizers. Special thanks to @yasufuminakama providing a starter notebook and @hengck23 for his transformer idea.

\n

I have first started with the Y.Nakama's tokenization and tried resnest with LSTM/GRU with different image sizes. Result so far was showing that using bigger images brings boost in LB. After Heng has released his starter code, I went immediately to 384x384 vit_deit_base_patch16_384 with 3 decoder layers and was able to get CV 1.0 and LB 1.3 without any data augmentation and jumped to the 4th place. Even I had access to a 8x GPU machine, I didn't feel confident enough to continue alone, so merged with @zfturbo @youhanlee @drhabib which was a good decision. They were already training with pseudo, so I have finetuned my model on their pseudo and scored LB0.86. At same time, @yohanlee has trained swin_base_patch4_window12_384. There was an interesting observation, VIT showed sometimes NAN during mixed precision training, but SWIN didn't have this issue! So my main models were now the swin transformer encoders (VIT was better but slower). With the 3x decoder layer, our highest LB was around 0.8. So I have started to experiment, with deeper decoders and different head configurations. Best single model so far was swin_base_patch4_window12_384 with 12x decoder layers and 16 heads which reached LB 0.74. For experementing with different decoders, I first freeze the encoder and train only decoder and next stage unfreeze everything which enables faster experimenting. @yohanlee also trained cait_m36_384 which scored only LB 0.8 but helped in the final ensembling.

\n

What worked so far:

\n\n

Not sure (no impact on LB, but also didn't hurt it):

\n\n

What didn't work:

\n\n

Even the beam search was working good for LSTM, I haven't implemented a beam search for transformer, because the inferencing was slow, so I thought it was not worth.

\n

After investigating my model outputs, I have discovered many errors in m0/m1. So i have trained some resnet18 and resnet50 models on 256x256 and 512x512 images, which helped to bring also 0.02 gain on LB.

", "rawMarkdown": "Team summary: https://www.kaggle.com/c/bms-molecular-translation/discussion/244031\nCode: https://github.com/tugstugi/pytorch-bms\n\nFirst, I want thank my team mates @zfturbo @youhanlee @drhabib and the competition organizers. Special thanks to @yasufuminakama providing a starter notebook and @hengck23 for his transformer idea.\n\nI have first started with the Y.Nakama's tokenization and tried resnest with LSTM/GRU with different image sizes. Result so far was showing that using [bigger images brings boost in LB](https://www.kaggle.com/c/bms-molecular-translation/discussion/231172). After Heng has released his starter code, I went immediately to 384x384 `vit_deit_base_patch16_384` with 3 decoder layers and was able to get CV 1.0 and LB 1.3 without any data augmentation and jumped to the 4th place. Even I had access to a 8x GPU machine, I didn't feel confident enough to continue alone, so merged with @zfturbo @youhanlee @drhabib which was a good decision. They were already training with pseudo, so I have finetuned my model on their pseudo and scored LB0.86. At same time, @yohanlee has trained `swin_base_patch4_window12_384`. There was an interesting observation, VIT showed sometimes NAN during mixed precision training, but SWIN didn't have this issue! So my main models were now the swin transformer encoders (VIT was better but slower). With the 3x decoder layer, our highest LB was around 0.8. So I have started to experiment, with deeper decoders and different head configurations. Best single model so far was `swin_base_patch4_window12_384` with 12x decoder layers and 16 heads which reached LB 0.74. For experementing with different decoders, I first freeze the encoder and train only decoder and next stage unfreeze everything which enables faster experimenting. @yohanlee also trained `cait_m36_384` which scored only LB 0.8 but helped in the final ensembling.\n\nWhat worked so far:\n* SWA which brings usually 0.02 improvement\n* cropping and rotation as augmentation\n* rotation and crop TTA also 0.02 improvement (downside inferencing much slower)\n* swin as encoder with 12 layers -> No NaNs!\n\nNot sure (no impact on LB, but also didn't hurt it):\n* more heads\n* label smoothing\n* external images (i have used 1m external images created by rdkit)\n* reverse inchi models\n\nWhat didn't work:\n* decoder step ensembling. Was really surprised that it worked out for the 7th place team.\n\nEven the beam search was working [good for LSTM](https://www.kaggle.com/c/bms-molecular-translation/discussion/231065), I haven't implemented a beam search for transformer, because the inferencing was slow, so I thought it was not worth.\n\nAfter investigating my model outputs, I have discovered many errors in m0/m1. So i have trained some resnet18 and resnet50 models on 256x256 and 512x512 images, which helped to bring also 0.02 gain on LB.", "votes": null }, { "id": "1334974", "postDate": "06/04/2021 00:05:51", "content": "

Congrats GM!!
\nYou deserved it!

", "rawMarkdown": "Congrats GM!!\nYou deserved it!", "votes": null }, { "id": "1334977", "postDate": "06/04/2021 00:06:10", "content": "

Congrats GM!

", "rawMarkdown": "Congrats GM!", "votes": null }, { "id": "1334980", "postDate": "06/04/2021 00:08:11", "content": "

Congratulations on gold finish @tugstugi and becoming grandmaster. 🎉

", "rawMarkdown": "Congratulations on gold finish @tugstugi and becoming grandmaster. 🎉", "votes": null }, { "id": "1334988", "postDate": "06/04/2021 00:20:01", "content": "

Congrats GM!

", "rawMarkdown": "Congrats GM!", "votes": null }, { "id": "1334990", "postDate": "06/04/2021 00:21:44", "content": "

Thanks. Your notebook helped me a lot to start in this competition.

", "rawMarkdown": "Thanks. Your notebook helped me a lot to start in this competition.", "votes": null }, { "id": "1334991", "postDate": "06/04/2021 00:22:04", "content": "

Congrats on Grandmaster!

", "rawMarkdown": "Congrats on Grandmaster!", "votes": null }, { "id": "1334994", "postDate": "06/04/2021 00:25:31", "content": "

Congrats GM

", "rawMarkdown": "Congrats GM", "votes": null }, { "id": "1335016", "postDate": "06/04/2021 00:50:58", "content": "

Congrats GM!

", "rawMarkdown": "Congrats GM!", "votes": null }, { "id": "1335028", "postDate": "06/04/2021 01:07:58", "content": "

Congratulations on gold finish @tugstugi and becoming a grandmaster. You guys didn't ensemble models

", "rawMarkdown": "Congratulations on gold finish @tugstugi and becoming a grandmaster. You guys didn't ensemble models", "votes": null }, { "id": "1335033", "postDate": "06/04/2021 01:16:11", "content": "

we ensemble… @zfturbo wrote custom ensemble code which operates on InChI (very cool stuff) .. he will probably create separate topic =)

", "rawMarkdown": "we ensemble... @zfturbo wrote custom ensemble code which operates on InChI (very cool stuff) .. he will probably create separate topic =)", "votes": null }, { "id": "1335067", "postDate": "06/04/2021 02:04:27", "content": "

Congrats on you and your team's 9th place and becoming a competition grandmaster! @tugstugi

", "rawMarkdown": "Congrats on you and your team's 9th place and becoming a competition grandmaster! @tugstugi", "votes": null }, { "id": "1335092", "postDate": "06/04/2021 02:47:09", "content": "

Congrats on becoming GM @tugstugi

", "rawMarkdown": "Congrats on becoming GM @tugstugi", "votes": null }, { "id": "1335243", "postDate": "06/04/2021 05:33:54", "content": "

thx for sharing. How long It takes to train for a LB around 1.0?

", "rawMarkdown": "thx for sharing. How long It takes to train for a LB around 1.0?", "votes": null }, { "id": "1335428", "postDate": "06/04/2021 07:52:24", "content": "

Congrats @tugstugi

", "rawMarkdown": "Congrats @tugstugi", "votes": null }, { "id": "1335790", "postDate": "06/04/2021 12:52:34", "content": "

Congrats to be a GM!! But I thought you already are;)

", "rawMarkdown": "Congrats to be a GM!! But I thought you already are;)", "votes": null }, { "id": "1336512", "postDate": "06/05/2021 02:02:02", "content": "

At least 1 day on a 8x GPU machine. This competition is really computation heavy.

", "rawMarkdown": "At least 1 day on a 8x GPU machine. This competition is really computation heavy.", "votes": null }, { "id": "1339545", "postDate": "06/07/2021 10:05:37", "content": "

Great! Congratulations @tugstugi

", "rawMarkdown": "Great! Congratulations @tugstugi", "votes": null } ], "comments": [ { "id": 1334974, "author_name": "youhanlee", "author_url": "", "post_date": "06/04/2021 00:05:51", "content": "

Congrats GM!!
\nYou deserved it!

", "votes": null, "replies": [] }, { "id": 1334977, "author_name": "underwearfitting", "author_url": "", "post_date": "06/04/2021 00:06:10", "content": "

Congrats GM!

", "votes": null, "replies": [] }, { "id": 1334980, "author_name": "nischaydnk", "author_url": "", "post_date": "06/04/2021 00:08:11", "content": "

Congratulations on gold finish @tugstugi and becoming grandmaster. 🎉

", "votes": null, "replies": [] }, { "id": 1334988, "author_name": "yasufuminakama", "author_url": "", "post_date": "06/04/2021 00:20:01", "content": "

Congrats GM!

", "votes": null, "replies": [ { "id": 1334990, "author_name": "tugstugi", "author_url": "", "post_date": "06/04/2021 00:21:44", "content": "

Thanks. Your notebook helped me a lot to start in this competition.

", "votes": null, "replies": [] } ] }, { "id": 1334991, "author_name": "andrewshao05", "author_url": "", "post_date": "06/04/2021 00:22:04", "content": "

Congrats on Grandmaster!

", "votes": null, "replies": [] }, { "id": 1334994, "author_name": "koukinn", "author_url": "", "post_date": "06/04/2021 00:25:31", "content": "

Congrats GM

", "votes": null, "replies": [] }, { "id": 1335016, "author_name": "alexlwh", "author_url": "", "post_date": "06/04/2021 00:50:58", "content": "

Congrats GM!

", "votes": null, "replies": [] }, { "id": 1335028, "author_name": "morizin", "author_url": "", "post_date": "06/04/2021 01:07:58", "content": "

Congratulations on gold finish @tugstugi and becoming a grandmaster. You guys didn't ensemble models

", "votes": null, "replies": [ { "id": 1335033, "author_name": "drhabib", "author_url": "", "post_date": "06/04/2021 01:16:11", "content": "

we ensemble… @zfturbo wrote custom ensemble code which operates on InChI (very cool stuff) .. he will probably create separate topic =)

", "votes": null, "replies": [] } ] }, { "id": 1335067, "author_name": "piantic", "author_url": "", "post_date": "06/04/2021 02:04:27", "content": "

Congrats on you and your team's 9th place and becoming a competition grandmaster! @tugstugi

", "votes": null, "replies": [] }, { "id": 1335092, "author_name": "duykhanh99", "author_url": "", "post_date": "06/04/2021 02:47:09", "content": "

Congrats on becoming GM @tugstugi

", "votes": null, "replies": [] }, { "id": 1335243, "author_name": "dragonzhang", "author_url": "", "post_date": "06/04/2021 05:33:54", "content": "

thx for sharing. How long It takes to train for a LB around 1.0?

", "votes": null, "replies": [ { "id": 1336512, "author_name": "tugstugi", "author_url": "", "post_date": "06/05/2021 02:02:02", "content": "

At least 1 day on a 8x GPU machine. This competition is really computation heavy.

", "votes": null, "replies": [] } ] }, { "id": 1335428, "author_name": "adamgrygielski", "author_url": "", "post_date": "06/04/2021 07:52:24", "content": "

Congrats @tugstugi

", "votes": null, "replies": [] }, { "id": 1335790, "author_name": "bamps53", "author_url": "", "post_date": "06/04/2021 12:52:34", "content": "

Congrats to be a GM!! But I thought you already are;)

", "votes": null, "replies": [] }, { "id": 1339545, "author_name": "grzegorzmierzwa", "author_url": "", "post_date": "06/07/2021 10:05:37", "content": "

Great! Congratulations @tugstugi

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1334966": "Team summary: https://www.kaggle.com/c/bms-molecular-translation/discussion/244031\nCode: https://github.com/tugstugi/pytorch-bms\n\nFirst, I want thank my team mates @zfturbo @youhanlee @drhabib and the competition organizers. Special thanks to @yasufuminakama providing a starter notebook and @hengck23 for his transformer idea.\n\nI have first started with the Y.Nakama's tokenization and tried resnest with LSTM/GRU with different image sizes. Result so far was showing that using [bigger images brings boost in LB](https://www.kaggle.com/c/bms-molecular-translation/discussion/231172). After Heng has released his starter code, I went immediately to 384x384 `vit_deit_base_patch16_384` with 3 decoder layers and was able to get CV 1.0 and LB 1.3 without any data augmentation and jumped to the 4th place. Even I had access to a 8x GPU machine, I didn't feel confident enough to continue alone, so merged with @zfturbo @youhanlee @drhabib which was a good decision. They were already training with pseudo, so I have finetuned my model on their pseudo and scored LB0.86. At same time, @yohanlee has trained `swin_base_patch4_window12_384`. There was an interesting observation, VIT showed sometimes NAN during mixed precision training, but SWIN didn't have this issue! So my main models were now the swin transformer encoders (VIT was better but slower). With the 3x decoder layer, our highest LB was around 0.8. So I have started to experiment, with deeper decoders and different head configurations. Best single model so far was `swin_base_patch4_window12_384` with 12x decoder layers and 16 heads which reached LB 0.74. For experementing with different decoders, I first freeze the encoder and train only decoder and next stage unfreeze everything which enables faster experimenting. @yohanlee also trained `cait_m36_384` which scored only LB 0.8 but helped in the final ensembling.\n\nWhat worked so far:\n* SWA which brings usually 0.02 improvement\n* cropping and rotation as augmentation\n* rotation and crop TTA also 0.02 improvement (downside inferencing much slower)\n* swin as encoder with 12 layers -> No NaNs!\n\nNot sure (no impact on LB, but also didn't hurt it):\n* more heads\n* label smoothing\n* external images (i have used 1m external images created by rdkit)\n* reverse inchi models\n\nWhat didn't work:\n* decoder step ensembling. Was really surprised that it worked out for the 7th place team.\n\nEven the beam search was working [good for LSTM](https://www.kaggle.com/c/bms-molecular-translation/discussion/231065), I haven't implemented a beam search for transformer, because the inferencing was slow, so I thought it was not worth.\n\nAfter investigating my model outputs, I have discovered many errors in m0/m1. So i have trained some resnet18 and resnet50 models on 256x256 and 512x512 images, which helped to bring also 0.02 gain on LB.", "1334974": "Congrats GM!!\nYou deserved it!", "1334977": "Congrats GM!", "1334980": "Congratulations on gold finish @tugstugi and becoming grandmaster. 🎉", "1334988": "Congrats GM!", "1334990": "Thanks. Your notebook helped me a lot to start in this competition.", "1334991": "Congrats on Grandmaster!", "1334994": "Congrats GM", "1335016": "Congrats GM!", "1335028": "Congratulations on gold finish @tugstugi and becoming a grandmaster. You guys didn't ensemble models", "1335033": "we ensemble... @zfturbo wrote custom ensemble code which operates on InChI (very cool stuff) .. he will probably create separate topic =)", "1335067": "Congrats on you and your team's 9th place and becoming a competition grandmaster! @tugstugi", "1335092": "Congrats on becoming GM @tugstugi", "1335243": "thx for sharing. How long It takes to train for a LB around 1.0?", "1335428": "Congrats @tugstugi", "1335790": "Congrats to be a GM!! But I thought you already are;)", "1336512": "At least 1 day on a 8x GPU machine. This competition is really computation heavy.", "1339545": "Great! Congratulations @tugstugi" }, "source": "meta" }