{ "id": 244166, "title": "13th Place Team (a story from a Kaggle newbie's view)", "url": "/competitions/bms-molecular-translation/writeups/mfl-eindhoven-13th-place-team-a-story-from-a-kaggl", "author_name": "", "post_date": "2021-06-07T23:39:04.550Z", "votes": 30, "comment_count": 10, "views": 0, "content": "

Hello all,

\n

It was an amazing competition. I would like to thank my team members ( @eakdag, @ubique, @aerdem4, @proletheus ) for their great effort. We missed the gold medal at the last moment, but I learned a lot and had nice friends.

\n

As a Kaggle newbie, I will explain the story of our team from my view. It could be a bit long so I divided it into sections and I will fill them in time.

\n

[Introduction]

\n

I and Erkut were friends from high school (MFL) at Konya/Turkey. After 12 years we met in Eindhoven/Netherlands by chance. We are both pursuing a PhD at TU/e. I am working on cheminformatics applications and Erkut is working on computer vision. We had discussed trying a Kaggle competition together 2 weeks before the BMS competition started. When this competition popped up, it was an amazing opportunity :D. So we set up \"MFL Eindhoven\" team.

\n

[Preparetion]

\n

I had no prior experience with computer vision except playing with some toy datasets at the beginning of the competition. I took a quick computer vision course from Kaggle. Then I developed my first kernel which detects the stereochemical layer in the image.

\n

At the same time, I analyzed InChI and listed possible solution ideas. (see the picture of my early research in the comments)

\n

[Training First Models]

\n

First, we started image captioning with @yasufuminakama's famous ResNet+LSTM notebook. However, we had limited resources and training was super slow. We wanted to make many different experiments, so we needed a faster way. (LB:7.06)

\n

Mark's TPU notebook : @markwijkhuizen shared an amazing end-to-end TPU notebook. We used eff_B1+LSTM for training our models. Each epoch was taking around (30 min to 1 hour) which allowed us to make the following experiments:

\n\n

Now, we had several models (LB: btw 4.0 to 7.0) and it was time to ensemble them which will boost our score.
\n\"\"
\nPS: Special thanks to @markwijkhuizen ! These experiments would not have been possible without your sharing.

\n

[Boost of Merge Algorithms]

\n

Gold Rush
\nOur first merge algorithm was \"Gold Rush\". It simply merges two submissions by selecting valid molecules detected by Rdkit. When we apply Gold Rush to our submissions from different models, our score improved to LB: ~3.50.
\n\"\"

\n

The Collector of Lost Souls
\nAfter we merge all submissions, there are still around 100k invalid molecules. In order to find them, we developed an algorithm that takes inference from each epoch for only invalid molecules. Using this method, we collected 20k more valid molecules and our score improved to LB: ~3.30.

\n

During that time @hengck23 shared his transformer and the inference script. We inferred two submissions (LB:3.10 and 3.11). Then we also included these submissions into Gold Rush and we got a high improvement to LB:2.30.

\n

Multi Merge
\nAfter this point, I started to analyze my validation data. I found that avg error of valid molecules was 0.8. So even if we found all valid molecules, 0.8 was the top score we could reach. Then I started to compare different submissions and found different valid InChIs for the same molecules. It meant that valid molecules also need to be validated :D. So I developed a new algorithm based on voting and minimum distance. That really worked and our score improved to LB: ~2.00.
\n\"\"
\nNow, we had a great merge algorithm and we were in a good LB position, it was a good time to develop a team-up strategy…

\n

[Team Up Strategy]

\n

Our algorithm boosted 3.xx models to 2.00 level, so if we find new team members with high-scored models we could have played for top positions. Our algorithm gets more benefit from diverse models, so we always took this into account. We decided to go until 5 members. After each member joined our score would increase and we would have a better chance to merge with higher LBs. So we decided to go round by round and started to invite other top LBs.

\n

PS: There were some suspicious accounts at the top positions, we decided not to invite them.

\n

[Boost of New Members]

\n

In the first round, @ubique accepted our invitation. He had SMILES model trained with ResNest (LB:2.03). When we added his models in our algorithm we reached (LB:1.16).

\n

Before the second round, @aerdem4 contacted us for a possible merge. He had just finished another competition and recently joined this one. He had an InChI Transformer model (LB:3.71) which predicts each layer separately. We thought that his experience would be very useful for the team and probably will improve his model quickly (it exactly came true). Since his InChI layers ensembled from separate models, valid ones were very reliable. His model boosted us to (LB:0.99).

\n

We reached that level, using relatively weak models. We needed a locomotive single model (preferably InChI). In the last round, @proletheus joined our team, and we found our locomotive (LB:1.42 Transformer InChI). By including his model into multi-merge, we reached (LB:0.78).

\n

[Developed Models]

\n

After that point, every day we worked hard and improved our models. In the end, our merged score reached (LB:0.62). The main models we developed are listed with scores (before pseudo-labeling) below.

\n\n

[Applied Methods]

\n", "messages": [ { "id": "1337156", "postDate": "06/05/2021 12:57:11", "content": "

Hello all,

\n

It was an amazing competition. I would like to thank my team members ( @eakdag, @ubique, @aerdem4, @proletheus ) for their great effort. We missed the gold medal at the last moment, but I learned a lot and had nice friends.

\n

As a Kaggle newbie, I will explain the story of our team from my view. It could be a bit long so I divided it into sections and I will fill them in time.

\n

[Introduction]

\n

I and Erkut were friends from high school (MFL) at Konya/Turkey. After 12 years we met in Eindhoven/Netherlands by chance. We are both pursuing a PhD at TU/e. I am working on cheminformatics applications and Erkut is working on computer vision. We had discussed trying a Kaggle competition together 2 weeks before the BMS competition started. When this competition popped up, it was an amazing opportunity :D. So we set up \"MFL Eindhoven\" team.

\n

[Preparetion]

\n

I had no prior experience with computer vision except playing with some toy datasets at the beginning of the competition. I took a quick computer vision course from Kaggle. Then I developed my first kernel which detects the stereochemical layer in the image.

\n

At the same time, I analyzed InChI and listed possible solution ideas. (see the picture of my early research in the comments)

\n

[Training First Models]

\n

First, we started image captioning with @yasufuminakama's famous ResNet+LSTM notebook. However, we had limited resources and training was super slow. We wanted to make many different experiments, so we needed a faster way. (LB:7.06)

\n

Mark's TPU notebook : @markwijkhuizen shared an amazing end-to-end TPU notebook. We used eff_B1+LSTM for training our models. Each epoch was taking around (30 min to 1 hour) which allowed us to make the following experiments:

\n\n

Now, we had several models (LB: btw 4.0 to 7.0) and it was time to ensemble them which will boost our score.
\n\"\"
\nPS: Special thanks to @markwijkhuizen ! These experiments would not have been possible without your sharing.

\n

[Boost of Merge Algorithms]

\n

Gold Rush
\nOur first merge algorithm was \"Gold Rush\". It simply merges two submissions by selecting valid molecules detected by Rdkit. When we apply Gold Rush to our submissions from different models, our score improved to LB: ~3.50.
\n\"\"

\n

The Collector of Lost Souls
\nAfter we merge all submissions, there are still around 100k invalid molecules. In order to find them, we developed an algorithm that takes inference from each epoch for only invalid molecules. Using this method, we collected 20k more valid molecules and our score improved to LB: ~3.30.

\n

During that time @hengck23 shared his transformer and the inference script. We inferred two submissions (LB:3.10 and 3.11). Then we also included these submissions into Gold Rush and we got a high improvement to LB:2.30.

\n

Multi Merge
\nAfter this point, I started to analyze my validation data. I found that avg error of valid molecules was 0.8. So even if we found all valid molecules, 0.8 was the top score we could reach. Then I started to compare different submissions and found different valid InChIs for the same molecules. It meant that valid molecules also need to be validated :D. So I developed a new algorithm based on voting and minimum distance. That really worked and our score improved to LB: ~2.00.
\n\"\"
\nNow, we had a great merge algorithm and we were in a good LB position, it was a good time to develop a team-up strategy…

\n

[Team Up Strategy]

\n

Our algorithm boosted 3.xx models to 2.00 level, so if we find new team members with high-scored models we could have played for top positions. Our algorithm gets more benefit from diverse models, so we always took this into account. We decided to go until 5 members. After each member joined our score would increase and we would have a better chance to merge with higher LBs. So we decided to go round by round and started to invite other top LBs.

\n

PS: There were some suspicious accounts at the top positions, we decided not to invite them.

\n

[Boost of New Members]

\n

In the first round, @ubique accepted our invitation. He had SMILES model trained with ResNest (LB:2.03). When we added his models in our algorithm we reached (LB:1.16).

\n

Before the second round, @aerdem4 contacted us for a possible merge. He had just finished another competition and recently joined this one. He had an InChI Transformer model (LB:3.71) which predicts each layer separately. We thought that his experience would be very useful for the team and probably will improve his model quickly (it exactly came true). Since his InChI layers ensembled from separate models, valid ones were very reliable. His model boosted us to (LB:0.99).

\n

We reached that level, using relatively weak models. We needed a locomotive single model (preferably InChI). In the last round, @proletheus joined our team, and we found our locomotive (LB:1.42 Transformer InChI). By including his model into multi-merge, we reached (LB:0.78).

\n

[Developed Models]

\n

After that point, every day we worked hard and improved our models. In the end, our merged score reached (LB:0.62). The main models we developed are listed with scores (before pseudo-labeling) below.

\n\n

[Applied Methods]

\n", "rawMarkdown": "Hello all,\n\nIt was an amazing competition. I would like to thank my team members ( @eakdag, @ubique, @aerdem4, @proletheus ) for their great effort. We missed the gold medal at the last moment, but I learned a lot and had nice friends.\n\nAs a Kaggle newbie, I will explain the story of our team from my view. It could be a bit long so I divided it into sections and I will fill them in time.\n\n### **[Introduction]**\n\nI and Erkut were friends from high school (MFL) at Konya/Turkey. After 12 years we met in Eindhoven/Netherlands by chance. We are both pursuing a PhD at TU/e. I am working on cheminformatics applications and Erkut is working on computer vision. We had discussed trying a Kaggle competition together 2 weeks before the BMS competition started. When this competition popped up, it was an amazing opportunity :D. So we set up \"MFL Eindhoven\" team.\n\n### **[Preparetion]**\n\nI had no prior experience with computer vision except playing with some toy datasets at the beginning of the competition. I took a quick [computer vision course](https://www.kaggle.com/learn/computer-vision) from Kaggle. Then I developed [my first kernel which detects the stereochemical layer](https://www.kaggle.com/c/bms-molecular-translation/discussion/229168#1255349) in the image. \n\nAt the same time, I analyzed InChI and listed possible solution ideas. (see the picture of my early research in the comments)\n\n### **[Training First Models]**\n\nFirst, we started image captioning with @yasufuminakama's famous [ResNet+LSTM notebook](https://www.kaggle.com/yasufuminakama/inchi-resnet-lstm-with-attention-starter). However, we had limited resources and training was super slow. We wanted to make many different experiments, so we needed a faster way. (**LB:7.06**)\n\n[Mark's TPU notebook ](https://www.kaggle.com/markwijkhuizen/tensorflow-tpu-training-baseline-lb-16-92): @markwijkhuizen shared an amazing end-to-end TPU notebook. We used eff_B1+LSTM for training our models. Each epoch was taking around (30 min to 1 hour) which allowed us to make the following experiments:\n\n- Larger Image size\n- Different tokenizer \n- Train with only longer sequences\n(see comments for the list of other possible improvements)\n\nNow, we had several models (**LB: btw 4.0 to 7.0**) and it was time to ensemble them which will boost our score. \n![](https://i.imgur.com/Hovv3C0.png)\nPS: Special thanks to @markwijkhuizen ! These experiments would not have been possible without your sharing.\n\n\n### **[Boost of Merge Algorithms]**\n\n**Gold Rush**\nOur first merge algorithm was \"Gold Rush\". It simply merges two submissions by selecting valid molecules detected by Rdkit. When we apply Gold Rush to our submissions from different models, our score improved to **LB: ~3.50**.\n![](https://i.imgur.com/UX4r21n.png)\n\n\n**The Collector of Lost Souls**\nAfter we merge all submissions, there are still around 100k invalid molecules. In order to find them, we developed an algorithm that takes inference from each epoch for only invalid molecules. Using this method, we collected 20k more valid molecules and our score improved to **LB: ~3.30**.\n\nDuring that time @hengck23 shared his [transformer and the inference script](https://www.kaggle.com/c/bms-molecular-translation/discussion/231190). We inferred two submissions (**LB:3.10 and 3.11**). Then we also included these submissions into Gold Rush and we got a high improvement to **LB:2.30**. \n\n\n**Multi Merge**\nAfter this point, I started to analyze my validation data. I found that avg error of valid molecules was 0.8. So even if we found all valid molecules, 0.8 was the top score we could reach. Then I started to compare different submissions and found different valid InChIs for the same molecules. It meant that valid molecules also need to be validated :D. So I developed a new algorithm based on voting and minimum distance. That really worked and our score improved to **LB: ~2.00**.\n![](https://i.imgur.com/5nHXrdf.png)\nNow, we had a great merge algorithm and we were in a good LB position, it was a good time to develop a team-up strategy...\n\n### **[Team Up Strategy]**\nOur algorithm boosted 3.xx models to 2.00 level, so if we find new team members with high-scored models we could have played for top positions. Our algorithm gets more benefit from diverse models, so we always took this into account. We decided to go until 5 members. After each member joined our score would increase and we would have a better chance to merge with higher LBs. So we decided to go round by round and started to invite other top LBs. \n\n**PS:** There were some suspicious accounts at the top positions, we decided not to invite them.\n\n### **[Boost of New Members]**\nIn the first round, @ubique accepted our invitation. He had SMILES model trained with ResNest (**LB:2.03**). When we added his models in our algorithm we reached (**LB:1.16**).\n\nBefore the second round, @aerdem4 contacted us for a possible merge. He had just finished another competition and recently joined this one. He had an InChI Transformer model (**LB:3.71**) which predicts each layer separately. We thought that his experience would be very useful for the team and probably will improve his model quickly (it exactly came true). Since his InChI layers ensembled from separate models, valid ones were very reliable. His model boosted us to (**LB:0.99**).\n\nWe reached that level, using relatively weak models. We needed a locomotive single model (preferably InChI). In the last round, @proletheus joined our team, and we found our locomotive (**LB:1.42** Transformer InChI). By including his model into multi-merge, we reached (**LB:0.78**). \n\n### **[Developed Models]**\nAfter that point, every day we worked hard and improved our models. In the end, our merged score reached (**LB:0.62**). The main models we developed are listed with scores (before pseudo-labeling) below.\n\n- InChI / Transformer / @proletheus / **LB: 1.37**\n- InChI (layered) / Transformer / @aerdem4 / **LB: 2.01**\n- InChI / eff_B1+LSTM / @sorkun & @eakdag / **LB: 3.89**\n- SMILES / eff_B1+LSTM / @sorkun & @eakdag/ **LB: 1.89**\n- SMILES / ResNest+GRU / @ubique / **LB: 2.03**\n- SELFIES / Transformer / @proletheus / **LB: 2.52**\n- SELFIES / eff_B1+LSTM / @sorkun & @eakdag / **LB: 3.74**\n\n### **[Applied Methods]**\n\n- Molecular Reps (InChI, SMILES, SELFIES)\n- OHEM\n- Pseudo-labeling (improved single models but not improved the merged score)", "votes": null }, { "id": "1337768", "postDate": "06/05/2021 21:53:00", "content": "

Preparation section is updated.

\n

Preliminary research on InChI and some Ideas…
\n\"\"

", "rawMarkdown": "**Preparation** section is updated. \n\nPreliminary research on InChI and some Ideas...\n![](https://i.imgur.com/MrDswM7.jpg)", "votes": null }, { "id": "1338374", "postDate": "06/06/2021 11:47:34", "content": "

Training First Models section is updated.

\n

More ideas about possible improvements…
\n\"\"

", "rawMarkdown": "**Training First Models** section is updated.\n\nMore ideas about possible improvements...\n![](https://i.imgur.com/4naXMsM.png)", "votes": null }, { "id": "1338461", "postDate": "06/06/2021 13:32:53", "content": "

Congrats for the silver! Look forward to your full write-up (and your story)
\nDid u mainly use Kaggle TPU in the beginning stage?

", "rawMarkdown": "Congrats for the silver! Look forward to your full write-up (and your story)\nDid u mainly use Kaggle TPU in the beginning stage?", "votes": null }, { "id": "1339013", "postDate": "06/07/2021 00:26:38", "content": "

Thanks! Yes, we used Kaggle and Colab for our TPU experiments.

", "rawMarkdown": "Thanks! Yes, we used Kaggle and Colab for our TPU experiments.", "votes": null }, { "id": "1339017", "postDate": "06/07/2021 00:32:19", "content": "

Nice!
\nI didnt take part in the competition but I know the dataset is gigantic. Running an experiment could take so long. Did u do anything special in order to run your experiments more efficiently in Kaggle/ Colab?

", "rawMarkdown": "Nice! \nI didnt take part in the competition but I know the dataset is gigantic. Running an experiment could take so long. Did u do anything special in order to run your experiments more efficiently in Kaggle/ Colab?", "votes": null }, { "id": "1339028", "postDate": "06/07/2021 01:04:40", "content": "

What I see is TPU memory allows you to go very large batch sizes like 128*8. I used @markwijkhuizen's reference notebook for TPU experiments. It took around 30 mins for each epoch, so I could try different ideas within a day.

", "rawMarkdown": "What I see is TPU memory allows you to go very large batch sizes like 128*8. I used @markwijkhuizen's reference notebook for TPU experiments. It took around 30 mins for each epoch, so I could try different ideas within a day.", "votes": null }, { "id": "1339035", "postDate": "06/07/2021 01:23:08", "content": "

Boost of Merge Algorithms section is updated.

\n

Multi Merge algorithm first draft…
\n\"\"

", "rawMarkdown": "**Boost of Merge Algorithms** section is updated.\n\nMulti Merge algorithm first draft…\n![](https://i.imgur.com/o4Bq1XP.jpg)", "votes": null }, { "id": "1340435", "postDate": "06/07/2021 23:32:09", "content": "

The story has been completed. It was an enjoyable but tiring experience for me.
\n I hope you also enjoyed reading!

\n

Here is the final image, a small part of our merge experiments!
\n\"\"

", "rawMarkdown": "The story has been completed. It was an enjoyable but tiring experience for me.\n I hope you also enjoyed reading!\n\nHere is the final image, a small part of our merge experiments! \n![](https://i.imgur.com/fMlfEWn.png)", "votes": null }, { "id": "1348887", "postDate": "06/14/2021 11:06:21", "content": "

Congrats :) Also, amazing storytelling!

", "rawMarkdown": "Congrats :) Also, amazing storytelling!", "votes": null }, { "id": "1351028", "postDate": "06/16/2021 03:00:12", "content": "

Wonderful explanation! 👍
\nI will also helpful for newbies who are still afraid to enter in competitions! 👊

", "rawMarkdown": "Wonderful explanation! 👍 \nI will also helpful for newbies who are still afraid to enter in competitions! 👊", "votes": null } ], "comments": [ { "id": 1337768, "author_name": "sorkun", "author_url": "", "post_date": "06/05/2021 21:53:00", "content": "

Preparation section is updated.

\n

Preliminary research on InChI and some Ideas…
\n\"\"

", "votes": null, "replies": [] }, { "id": 1338374, "author_name": "sorkun", "author_url": "", "post_date": "06/06/2021 11:47:34", "content": "

Training First Models section is updated.

\n

More ideas about possible improvements…
\n\"\"

", "votes": null, "replies": [] }, { "id": 1338461, "author_name": "alexlwh", "author_url": "", "post_date": "06/06/2021 13:32:53", "content": "

Congrats for the silver! Look forward to your full write-up (and your story)
\nDid u mainly use Kaggle TPU in the beginning stage?

", "votes": null, "replies": [ { "id": 1339013, "author_name": "sorkun", "author_url": "", "post_date": "06/07/2021 00:26:38", "content": "

Thanks! Yes, we used Kaggle and Colab for our TPU experiments.

", "votes": null, "replies": [] }, { "id": 1339017, "author_name": "alexlwh", "author_url": "", "post_date": "06/07/2021 00:32:19", "content": "

Nice!
\nI didnt take part in the competition but I know the dataset is gigantic. Running an experiment could take so long. Did u do anything special in order to run your experiments more efficiently in Kaggle/ Colab?

", "votes": null, "replies": [] }, { "id": 1339028, "author_name": "sorkun", "author_url": "", "post_date": "06/07/2021 01:04:40", "content": "

What I see is TPU memory allows you to go very large batch sizes like 128*8. I used @markwijkhuizen's reference notebook for TPU experiments. It took around 30 mins for each epoch, so I could try different ideas within a day.

", "votes": null, "replies": [] } ] }, { "id": 1339035, "author_name": "sorkun", "author_url": "", "post_date": "06/07/2021 01:23:08", "content": "

Boost of Merge Algorithms section is updated.

\n

Multi Merge algorithm first draft…
\n\"\"

", "votes": null, "replies": [] }, { "id": 1340435, "author_name": "sorkun", "author_url": "", "post_date": "06/07/2021 23:32:09", "content": "

The story has been completed. It was an enjoyable but tiring experience for me.
\n I hope you also enjoyed reading!

\n

Here is the final image, a small part of our merge experiments!
\n\"\"

", "votes": null, "replies": [] }, { "id": 1348887, "author_name": "cihanyatbaz", "author_url": "", "post_date": "06/14/2021 11:06:21", "content": "

Congrats :) Also, amazing storytelling!

", "votes": null, "replies": [] }, { "id": 1351028, "author_name": "mudassar66", "author_url": "", "post_date": "06/16/2021 03:00:12", "content": "

Wonderful explanation! 👍
\nI will also helpful for newbies who are still afraid to enter in competitions! 👊

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1337156": "Hello all,\n\nIt was an amazing competition. I would like to thank my team members ( @eakdag, @ubique, @aerdem4, @proletheus ) for their great effort. We missed the gold medal at the last moment, but I learned a lot and had nice friends.\n\nAs a Kaggle newbie, I will explain the story of our team from my view. It could be a bit long so I divided it into sections and I will fill them in time.\n\n### **[Introduction]**\n\nI and Erkut were friends from high school (MFL) at Konya/Turkey. After 12 years we met in Eindhoven/Netherlands by chance. We are both pursuing a PhD at TU/e. I am working on cheminformatics applications and Erkut is working on computer vision. We had discussed trying a Kaggle competition together 2 weeks before the BMS competition started. When this competition popped up, it was an amazing opportunity :D. So we set up \"MFL Eindhoven\" team.\n\n### **[Preparetion]**\n\nI had no prior experience with computer vision except playing with some toy datasets at the beginning of the competition. I took a quick [computer vision course](https://www.kaggle.com/learn/computer-vision) from Kaggle. Then I developed [my first kernel which detects the stereochemical layer](https://www.kaggle.com/c/bms-molecular-translation/discussion/229168#1255349) in the image. \n\nAt the same time, I analyzed InChI and listed possible solution ideas. (see the picture of my early research in the comments)\n\n### **[Training First Models]**\n\nFirst, we started image captioning with @yasufuminakama's famous [ResNet+LSTM notebook](https://www.kaggle.com/yasufuminakama/inchi-resnet-lstm-with-attention-starter). However, we had limited resources and training was super slow. We wanted to make many different experiments, so we needed a faster way. (**LB:7.06**)\n\n[Mark's TPU notebook ](https://www.kaggle.com/markwijkhuizen/tensorflow-tpu-training-baseline-lb-16-92): @markwijkhuizen shared an amazing end-to-end TPU notebook. We used eff_B1+LSTM for training our models. Each epoch was taking around (30 min to 1 hour) which allowed us to make the following experiments:\n\n- Larger Image size\n- Different tokenizer \n- Train with only longer sequences\n(see comments for the list of other possible improvements)\n\nNow, we had several models (**LB: btw 4.0 to 7.0**) and it was time to ensemble them which will boost our score. \n![](https://i.imgur.com/Hovv3C0.png)\nPS: Special thanks to @markwijkhuizen ! These experiments would not have been possible without your sharing.\n\n\n### **[Boost of Merge Algorithms]**\n\n**Gold Rush**\nOur first merge algorithm was \"Gold Rush\". It simply merges two submissions by selecting valid molecules detected by Rdkit. When we apply Gold Rush to our submissions from different models, our score improved to **LB: ~3.50**.\n![](https://i.imgur.com/UX4r21n.png)\n\n\n**The Collector of Lost Souls**\nAfter we merge all submissions, there are still around 100k invalid molecules. In order to find them, we developed an algorithm that takes inference from each epoch for only invalid molecules. Using this method, we collected 20k more valid molecules and our score improved to **LB: ~3.30**.\n\nDuring that time @hengck23 shared his [transformer and the inference script](https://www.kaggle.com/c/bms-molecular-translation/discussion/231190). We inferred two submissions (**LB:3.10 and 3.11**). Then we also included these submissions into Gold Rush and we got a high improvement to **LB:2.30**. \n\n\n**Multi Merge**\nAfter this point, I started to analyze my validation data. I found that avg error of valid molecules was 0.8. So even if we found all valid molecules, 0.8 was the top score we could reach. Then I started to compare different submissions and found different valid InChIs for the same molecules. It meant that valid molecules also need to be validated :D. So I developed a new algorithm based on voting and minimum distance. That really worked and our score improved to **LB: ~2.00**.\n![](https://i.imgur.com/5nHXrdf.png)\nNow, we had a great merge algorithm and we were in a good LB position, it was a good time to develop a team-up strategy...\n\n### **[Team Up Strategy]**\nOur algorithm boosted 3.xx models to 2.00 level, so if we find new team members with high-scored models we could have played for top positions. Our algorithm gets more benefit from diverse models, so we always took this into account. We decided to go until 5 members. After each member joined our score would increase and we would have a better chance to merge with higher LBs. So we decided to go round by round and started to invite other top LBs. \n\n**PS:** There were some suspicious accounts at the top positions, we decided not to invite them.\n\n### **[Boost of New Members]**\nIn the first round, @ubique accepted our invitation. He had SMILES model trained with ResNest (**LB:2.03**). When we added his models in our algorithm we reached (**LB:1.16**).\n\nBefore the second round, @aerdem4 contacted us for a possible merge. He had just finished another competition and recently joined this one. He had an InChI Transformer model (**LB:3.71**) which predicts each layer separately. We thought that his experience would be very useful for the team and probably will improve his model quickly (it exactly came true). Since his InChI layers ensembled from separate models, valid ones were very reliable. His model boosted us to (**LB:0.99**).\n\nWe reached that level, using relatively weak models. We needed a locomotive single model (preferably InChI). In the last round, @proletheus joined our team, and we found our locomotive (**LB:1.42** Transformer InChI). By including his model into multi-merge, we reached (**LB:0.78**). \n\n### **[Developed Models]**\nAfter that point, every day we worked hard and improved our models. In the end, our merged score reached (**LB:0.62**). The main models we developed are listed with scores (before pseudo-labeling) below.\n\n- InChI / Transformer / @proletheus / **LB: 1.37**\n- InChI (layered) / Transformer / @aerdem4 / **LB: 2.01**\n- InChI / eff_B1+LSTM / @sorkun & @eakdag / **LB: 3.89**\n- SMILES / eff_B1+LSTM / @sorkun & @eakdag/ **LB: 1.89**\n- SMILES / ResNest+GRU / @ubique / **LB: 2.03**\n- SELFIES / Transformer / @proletheus / **LB: 2.52**\n- SELFIES / eff_B1+LSTM / @sorkun & @eakdag / **LB: 3.74**\n\n### **[Applied Methods]**\n\n- Molecular Reps (InChI, SMILES, SELFIES)\n- OHEM\n- Pseudo-labeling (improved single models but not improved the merged score)", "1337768": "**Preparation** section is updated. \n\nPreliminary research on InChI and some Ideas...\n![](https://i.imgur.com/MrDswM7.jpg)", "1338374": "**Training First Models** section is updated.\n\nMore ideas about possible improvements...\n![](https://i.imgur.com/4naXMsM.png)", "1338461": "Congrats for the silver! Look forward to your full write-up (and your story)\nDid u mainly use Kaggle TPU in the beginning stage?", "1339013": "Thanks! Yes, we used Kaggle and Colab for our TPU experiments.", "1339017": "Nice! \nI didnt take part in the competition but I know the dataset is gigantic. Running an experiment could take so long. Did u do anything special in order to run your experiments more efficiently in Kaggle/ Colab?", "1339028": "What I see is TPU memory allows you to go very large batch sizes like 128*8. I used @markwijkhuizen's reference notebook for TPU experiments. It took around 30 mins for each epoch, so I could try different ideas within a day.", "1339035": "**Boost of Merge Algorithms** section is updated.\n\nMulti Merge algorithm first draft…\n![](https://i.imgur.com/o4Bq1XP.jpg)", "1340435": "The story has been completed. It was an enjoyable but tiring experience for me.\n I hope you also enjoyed reading!\n\nHere is the final image, a small part of our merge experiments! \n![](https://i.imgur.com/fMlfEWn.png)", "1348887": "Congrats :) Also, amazing storytelling!", "1351028": "Wonderful explanation! 👍 \nI will also helpful for newbies who are still afraid to enter in competitions! 👊" }, "source": "meta" }