Hi all!
\nThanks to the competition host and for higly useful advices in the discussion! It’s my first competition and I am glad to get a bronze :)
\nMy solution is very simple and based on smile targets. It was hard to make enough experiments due to I have had only a single 2080ti. To some speed up the training, I used torch amp.
\nSummary of the solution
\nThe main pipeline is similar to @yasufuminakama great notebook, but as encoder I use EfficienNet_B3. The main difference - instead of using InChI-target I use Smile notation. On inference I convert Smile back to InChI via rdkit and as tokenizer I use XinhaoLi74/SmilesPE. Smiles is much more simpler for prediction - shorter notation and less token-classes - and this gave me great improvement in accuracy (from 6.31 to 2.22).
\nData
\nI use syntheic images and extra_approved_InChIs.csv of course - about 1 million images (thanks for the great notebook by @tuckerarrants and notebook by @stainsby)
\nI have make a lot of experiments - in hyperparameters, batchsampler, backbones, etc., but not as many as I wanted because training was soooo long. I ended up with 288x288 images size, as a compromise between accuracy and training speed.
\nI made an adaptive batchsampler that was quite helpful - it increase probability for a sample to be added in batch by sample's loss during training. So the model was trained on \"harder\" examples, and samples with almost zero loss wasn’t put in batch often.
\nSome other notes
\nThere was some little accuracy improvement (about 1.8 -> 1.6) due to freeze encoder and train LSTM-decored only. It significantly speed up training and allowed to increase batch size in final stage of the training.
\nAlso as optimizer I use AdamW with weight_decay 0.01 (as article suggest) but don't sure that it get any influence at all :) As image augmentations were used only gaussian blur and various rotations (by randomly transpose and vertical/horizontal flipping).
\nOther experiments I wanted to try
\nThe source code (pytorch) is available in the github repository.
\nCongrat to all who took a part in the competition:)
", "messages": [ { "id": "1335609", "postDate": "06/04/2021 10:09:01", "content": "Hi all!
\nThanks to the competition host and for higly useful advices in the discussion! It’s my first competition and I am glad to get a bronze :)
\nMy solution is very simple and based on smile targets. It was hard to make enough experiments due to I have had only a single 2080ti. To some speed up the training, I used torch amp.
\nSummary of the solution
\nThe main pipeline is similar to @yasufuminakama great notebook, but as encoder I use EfficienNet_B3. The main difference - instead of using InChI-target I use Smile notation. On inference I convert Smile back to InChI via rdkit and as tokenizer I use XinhaoLi74/SmilesPE. Smiles is much more simpler for prediction - shorter notation and less token-classes - and this gave me great improvement in accuracy (from 6.31 to 2.22).
\nData
\nI use syntheic images and extra_approved_InChIs.csv of course - about 1 million images (thanks for the great notebook by @tuckerarrants and notebook by @stainsby)
\nI have make a lot of experiments - in hyperparameters, batchsampler, backbones, etc., but not as many as I wanted because training was soooo long. I ended up with 288x288 images size, as a compromise between accuracy and training speed.
\nI made an adaptive batchsampler that was quite helpful - it increase probability for a sample to be added in batch by sample's loss during training. So the model was trained on \"harder\" examples, and samples with almost zero loss wasn’t put in batch often.
\nSome other notes
\nThere was some little accuracy improvement (about 1.8 -> 1.6) due to freeze encoder and train LSTM-decored only. It significantly speed up training and allowed to increase batch size in final stage of the training.
\nAlso as optimizer I use AdamW with weight_decay 0.01 (as article suggest) but don't sure that it get any influence at all :) As image augmentations were used only gaussian blur and various rotations (by randomly transpose and vertical/horizontal flipping).
\nOther experiments I wanted to try
\nThe source code (pytorch) is available in the github repository.
\nCongrat to all who took a part in the competition:)
", "rawMarkdown": "Hi all!\n\nThanks to the competition host and for higly useful advices in the discussion! It’s my first competition and I am glad to get a bronze :)\n\nMy solution is very simple and based on smile targets. It was hard to make enough experiments due to I have had only a single 2080ti. To some speed up the training, I used torch amp.\n\n**Summary of the solution**\n\nThe main pipeline is similar to @yasufuminakama great [notebook](https://www.kaggle.com/yasufuminakama/inchi-resnet-lstm-with-attention-starter), but as encoder I use EfficienNet_B3. The main difference - instead of using InChI-target I use [Smile](https://en.wikipedia.org/wiki/Simplified_molecular-input_line-entry_system) notation. On inference I convert Smile back to InChI via [rdkit](https://github.com/kuelumbus/rdkit_platform_wheels) and as tokenizer I use [XinhaoLi74/SmilesPE](https://github.com/XinhaoLi74/SmilesPE). Smiles is much more simpler for prediction - shorter notation and less token-classes - and this gave me great improvement in accuracy (from **6.31** to **2.22**).\n\n**Data**\n\nI use syntheic images and extra_approved_InChIs.csv of course - about 1 million images (thanks for the great [notebook](https://www.kaggle.com/tuckerarrants/inchi-allowed-external-data) by @tuckerarrants and [notebook](https://www.kaggle.com/stainsby/improved-synthetic-data-for-bms-competition-v3) by @stainsby)\n\nI have make a lot of experiments - in hyperparameters, batchsampler, backbones, etc., but not as many as I wanted because training was soooo long. I ended up with 288x288 images size, as a compromise between accuracy and training speed.\n\nI made an adaptive batchsampler that was quite helpful - it increase probability for a sample to be added in batch by sample's loss during training. So the model was trained on \"harder\" examples, and samples with almost zero loss wasn’t put in batch often.\n\n**Some other notes**\n\nThere was some little accuracy improvement (about 1.8 -> 1.6) due to freeze encoder and train LSTM-decored only. It significantly speed up training and allowed to increase batch size in final stage of the training.\n\nAlso as optimizer I use AdamW with weight_decay 0.01 (as [article](https://www.fast.ai/2018/07/02/adam-weight-decay/) suggest) but don't sure that it get any influence at all :) As image augmentations were used only gaussian blur and various rotations (by randomly transpose and vertical/horizontal flipping).\n\n**Other experiments I wanted to try**\n\n- To split InChI up to 8 indepenpent string layers (separated by the \"/\" notation: \"/b\", \"/t\", \"/m\", \"/s\", etc.) and than train individual models for each layer. Each layer in the InChI describe different information about a molecule, and several models, trained on separate inchi-layers, presumably can get good results.\n- To add rotation transoform to any angles (it might help the model to catch some useful feature on images)\n- To add beam search and some models ensembling.\n- Trying Transformers instead of LSTM\n\n\n\nThe source code (pytorch) is available in the github [repository](https://github.com/skalinin/BMS-competition). \n\nCongrat to all who took a part in the competition:)", "votes": null }, { "id": "1338791", "postDate": "06/06/2021 18:20:06", "content": "Thanks for the great write-up and release of your code! 👍
", "rawMarkdown": "Thanks for the great write-up and release of your code! 👍", "votes": null }, { "id": "1340381", "postDate": "06/07/2021 20:37:46", "content": "Awesome solution and github repo, very well organized. Thanks for sharing it!
", "rawMarkdown": "Awesome solution and github repo, very well organized. Thanks for sharing it!", "votes": null } ], "comments": [ { "id": 1338791, "author_name": "alexlwh", "author_url": "", "post_date": "06/06/2021 18:20:06", "content": "Thanks for the great write-up and release of your code! 👍
", "votes": null, "replies": [] }, { "id": 1340381, "author_name": "hinepo", "author_url": "", "post_date": "06/07/2021 20:37:46", "content": "Awesome solution and github repo, very well organized. Thanks for sharing it!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1335609": "Hi all!\n\nThanks to the competition host and for higly useful advices in the discussion! It’s my first competition and I am glad to get a bronze :)\n\nMy solution is very simple and based on smile targets. It was hard to make enough experiments due to I have had only a single 2080ti. To some speed up the training, I used torch amp.\n\n**Summary of the solution**\n\nThe main pipeline is similar to @yasufuminakama great [notebook](https://www.kaggle.com/yasufuminakama/inchi-resnet-lstm-with-attention-starter), but as encoder I use EfficienNet_B3. The main difference - instead of using InChI-target I use [Smile](https://en.wikipedia.org/wiki/Simplified_molecular-input_line-entry_system) notation. On inference I convert Smile back to InChI via [rdkit](https://github.com/kuelumbus/rdkit_platform_wheels) and as tokenizer I use [XinhaoLi74/SmilesPE](https://github.com/XinhaoLi74/SmilesPE). Smiles is much more simpler for prediction - shorter notation and less token-classes - and this gave me great improvement in accuracy (from **6.31** to **2.22**).\n\n**Data**\n\nI use syntheic images and extra_approved_InChIs.csv of course - about 1 million images (thanks for the great [notebook](https://www.kaggle.com/tuckerarrants/inchi-allowed-external-data) by @tuckerarrants and [notebook](https://www.kaggle.com/stainsby/improved-synthetic-data-for-bms-competition-v3) by @stainsby)\n\nI have make a lot of experiments - in hyperparameters, batchsampler, backbones, etc., but not as many as I wanted because training was soooo long. I ended up with 288x288 images size, as a compromise between accuracy and training speed.\n\nI made an adaptive batchsampler that was quite helpful - it increase probability for a sample to be added in batch by sample's loss during training. So the model was trained on \"harder\" examples, and samples with almost zero loss wasn’t put in batch often.\n\n**Some other notes**\n\nThere was some little accuracy improvement (about 1.8 -> 1.6) due to freeze encoder and train LSTM-decored only. It significantly speed up training and allowed to increase batch size in final stage of the training.\n\nAlso as optimizer I use AdamW with weight_decay 0.01 (as [article](https://www.fast.ai/2018/07/02/adam-weight-decay/) suggest) but don't sure that it get any influence at all :) As image augmentations were used only gaussian blur and various rotations (by randomly transpose and vertical/horizontal flipping).\n\n**Other experiments I wanted to try**\n\n- To split InChI up to 8 indepenpent string layers (separated by the \"/\" notation: \"/b\", \"/t\", \"/m\", \"/s\", etc.) and than train individual models for each layer. Each layer in the InChI describe different information about a molecule, and several models, trained on separate inchi-layers, presumably can get good results.\n- To add rotation transoform to any angles (it might help the model to catch some useful feature on images)\n- To add beam search and some models ensembling.\n- Trying Transformers instead of LSTM\n\n\n\nThe source code (pytorch) is available in the github [repository](https://github.com/skalinin/BMS-competition). \n\nCongrat to all who took a part in the competition:)", "1338791": "Thanks for the great write-up and release of your code! 👍", "1340381": "Awesome solution and github repo, very well organized. Thanks for sharing it!" }, "source": "meta" }