First of all, I would like to thanks the organizers for organizing such an exciting competition. Second I would like to thank my teammates. We had a great working environment with everybody working hard, and last one week, we had to stay up until 3 am local times. Below I will try to summarize the main idea of our best-performing model.
\nOne of the challenging things about this competition was that the chemical had very different structures, some were super long, and others were very short. This also meant that images were highly diverse in size and shapes (e.g., super long in 1 axis or super small). Therefore it was essential to preprocess these images correctly. After some initial discussion we decided to try to use two methods:
\n1) simple rescaling
\nplus - we will get all the information (kind off)
\nminus - feature of the images were become distorted (extreme shrinking across one axis)
\n2) rescaling with preserving the aspect ratio
\nplus - This method will preserve the feature
\nminus - bigger images, when downscaled, will have extremely small features
\nBoth of these methods have some advantages and disadvantages, but we reasoned that it would be best that each of our team members pick one of these methods and stick to it. We also found this an excellent article showing that most modern computer vision libraries employ very buggy resizing methods (https://arxiv.org/pdf/2104.11222.pdf). Since we are dealing with a lot of lines when downscaling, it was essential to preserve most of the information (tl dr: use PIL)
We decided to go with natural image augmentations that were found in the dataset. But to make things a little bit diverse, we decided that everyone should train on a slightly different set of augmentations.
\nSet 1:
\nRandomRotate90
\nSet 2
RandomBorderCropPixel (cropping 5-20 pixel at random from the border of the image)\nRandomRotate90\nRandomNoiseAugment\nSet 3
\nRandomRotate90\nRandomBorderCropPixel (cropping 5-20 pixel at random from the border of the image)\nRandomNoiseAugment\nRandomLineRemoval\nRandomPiexlThreshold (some of the chemical lines had different sheds of black pixel values so we perform)\nthr = random.randint(200, 250)\nimg[img < thr] = 0\nimg[img >= thr] = 255\n@yasufuminakama tokenizers was great, but we were wondering if there is a better way to tokenize. We found a lot of research on different tokenization strategies using SMILES but almost no information about InChI. After some thinking, we decided to try huggingface BPE tokenization.
vocab = 128 # can be any\ntokenizer = Tokenizer(BPE(unk_token=\"[<unk>]\", fuse_unk=False))\npre_tokenizer = pre_tokenizers.Sequence([Whitespace(), Digits()])\ntokenizer.pre_tokenizer = pre_tokenizer\ntrainer = BpeTrainer(vocab = max_len, special_tokens=[\"[<pad>]\", \"[<start>]\", \"[<end>]\", \"[<unk>]\"])\nWe run a simple small-scale experiment using resnet34 and several BPE tokenizers using different vocab sizes. Below are results
42 character tokenzation - 9.7\n64 BPE tokenization - 8.9\n128 BPE tokenization - 8.01\n246 BPE tokenization - 6.9 \nIt clearly shows that more vocab size leads to better-learned embeddings. As per usual, in the end, we decided to use three different tokenizers to bring more diversity.
\n1) Y.nakama tokenizer
\n2) BPE - 512
\n3) BPE - 1024
We decided to train several different models with everything mentioned above, with progressive pseudo labeling on test data (take only valid inchis). In total, we had the following modules
\n1)effnetv2 -> encoder(6) -> decoder(8, 10, 12)
\n2)effnet3 -> encoder(3) -> decoder(3)
\n3)vit/cait/swin
\nFor vision transformers, we used hengs decoders.
\nFor hybrid CNN transformers to increase the diversity of our models, we used both Encoder/Decoder with attention layer with following modifications:
\n1) gating with GLUE (https://arxiv.org/abs/2002.05202). In feed-forward layer, use GLUE in place of the first linear transformation and the activation function.
\n2) like 4th place solutions, we use relative positional embedding as in T5
We trained our model, everything looks great, but we needed to come with an ensembling strategy because everyone was using different tokenizers, so combining probabilities was not an option. Luckily @zfturbo wrote a custom ensemble script that operates on string levels. It deserves its own post, which he will share later. Using @zfturbo script, we were able to get from weakly scoring models (0.74 - 0.90) ~0.67
We noticed that most of our valid predictions were different only by m0/m1. To solve this problem, we build a simple resnet50 model classifying images into 3 classes (no m0/m1, m0,m1). Using this classifier to post-process our InChI, we gain a boost from 0.67 to 0.65.
We observed that our models tend to make a more wrong prediction for long InChIs. As a result last 4 days we decided to finetune our model only on long InChIs (bigger then 180). After finetuning, we took the longest 100k InChI in our test and made new predictions. We also applied TTA to images and got those predictions as well. Once we have those predictions, we used a modified version of the zfturbo script to make an ensemble only on the longest InChI. This eventually brought us to our final LB standing 0.65 -> 0.61.
I would like to thank my teammates for being a great teamates and working really hard @zfturbo, @youhanlee and new GM @tugstugi =)
\nP.S also a big congratulations to all new GMs =)
First of all, I would like to thanks the organizers for organizing such an exciting competition. Second I would like to thank my teammates. We had a great working environment with everybody working hard, and last one week, we had to stay up until 3 am local times. Below I will try to summarize the main idea of our best-performing model.
\nOne of the challenging things about this competition was that the chemical had very different structures, some were super long, and others were very short. This also meant that images were highly diverse in size and shapes (e.g., super long in 1 axis or super small). Therefore it was essential to preprocess these images correctly. After some initial discussion we decided to try to use two methods:
\n1) simple rescaling
\nplus - we will get all the information (kind off)
\nminus - feature of the images were become distorted (extreme shrinking across one axis)
\n2) rescaling with preserving the aspect ratio
\nplus - This method will preserve the feature
\nminus - bigger images, when downscaled, will have extremely small features
\nBoth of these methods have some advantages and disadvantages, but we reasoned that it would be best that each of our team members pick one of these methods and stick to it. We also found this an excellent article showing that most modern computer vision libraries employ very buggy resizing methods (https://arxiv.org/pdf/2104.11222.pdf). Since we are dealing with a lot of lines when downscaling, it was essential to preserve most of the information (tl dr: use PIL)
We decided to go with natural image augmentations that were found in the dataset. But to make things a little bit diverse, we decided that everyone should train on a slightly different set of augmentations.
\nSet 1:
\nRandomRotate90
\nSet 2
RandomBorderCropPixel (cropping 5-20 pixel at random from the border of the image)\nRandomRotate90\nRandomNoiseAugment\nSet 3
\nRandomRotate90\nRandomBorderCropPixel (cropping 5-20 pixel at random from the border of the image)\nRandomNoiseAugment\nRandomLineRemoval\nRandomPiexlThreshold (some of the chemical lines had different sheds of black pixel values so we perform)\nthr = random.randint(200, 250)\nimg[img < thr] = 0\nimg[img >= thr] = 255\n@yasufuminakama tokenizers was great, but we were wondering if there is a better way to tokenize. We found a lot of research on different tokenization strategies using SMILES but almost no information about InChI. After some thinking, we decided to try huggingface BPE tokenization.
vocab = 128 # can be any\ntokenizer = Tokenizer(BPE(unk_token=\"[<unk>]\", fuse_unk=False))\npre_tokenizer = pre_tokenizers.Sequence([Whitespace(), Digits()])\ntokenizer.pre_tokenizer = pre_tokenizer\ntrainer = BpeTrainer(vocab = max_len, special_tokens=[\"[<pad>]\", \"[<start>]\", \"[<end>]\", \"[<unk>]\"])\nWe run a simple small-scale experiment using resnet34 and several BPE tokenizers using different vocab sizes. Below are results
42 character tokenzation - 9.7\n64 BPE tokenization - 8.9\n128 BPE tokenization - 8.01\n246 BPE tokenization - 6.9 \nIt clearly shows that more vocab size leads to better-learned embeddings. As per usual, in the end, we decided to use three different tokenizers to bring more diversity.
\n1) Y.nakama tokenizer
\n2) BPE - 512
\n3) BPE - 1024
We decided to train several different models with everything mentioned above, with progressive pseudo labeling on test data (take only valid inchis). In total, we had the following modules
\n1)effnetv2 -> encoder(6) -> decoder(8, 10, 12)
\n2)effnet3 -> encoder(3) -> decoder(3)
\n3)vit/cait/swin
\nFor vision transformers, we used hengs decoders.
\nFor hybrid CNN transformers to increase the diversity of our models, we used both Encoder/Decoder with attention layer with following modifications:
\n1) gating with GLUE (https://arxiv.org/abs/2002.05202). In feed-forward layer, use GLUE in place of the first linear transformation and the activation function.
\n2) like 4th place solutions, we use relative positional embedding as in T5
We trained our model, everything looks great, but we needed to come with an ensembling strategy because everyone was using different tokenizers, so combining probabilities was not an option. Luckily @zfturbo wrote a custom ensemble script that operates on string levels. It deserves its own post, which he will share later. Using @zfturbo script, we were able to get from weakly scoring models (0.74 - 0.90) ~0.67
We noticed that most of our valid predictions were different only by m0/m1. To solve this problem, we build a simple resnet50 model classifying images into 3 classes (no m0/m1, m0,m1). Using this classifier to post-process our InChI, we gain a boost from 0.67 to 0.65.
We observed that our models tend to make a more wrong prediction for long InChIs. As a result last 4 days we decided to finetune our model only on long InChIs (bigger then 180). After finetuning, we took the longest 100k InChI in our test and made new predictions. We also applied TTA to images and got those predictions as well. Once we have those predictions, we used a modified version of the zfturbo script to make an ensemble only on the longest InChI. This eventually brought us to our final LB standing 0.65 -> 0.61.
I would like to thank my teammates for being a great teamates and working really hard @zfturbo, @youhanlee and new GM @tugstugi =)
\nP.S also a big congratulations to all new GMs =)
Thx for sharing. BPE and other tricks learned.
", "rawMarkdown": "Thx for sharing. BPE and other tricks learned.", "votes": null }, { "id": "1336482", "postDate": "06/05/2021 00:51:12", "content": "Great work, thanks for sharing your insights on the tokenisers. Looking forward to seeing the ensemble script from @zfturbo.
", "rawMarkdown": "Great work, thanks for sharing your insights on the tokenisers. Looking forward to seeing the ensemble script from @zfturbo.", "votes": null }, { "id": "1336487", "postDate": "06/05/2021 01:04:12", "content": "\"We found a lot of research on different tokenization strategies using SMILES but almost no information about InChI. \"
\nyou can check this:
\nhttps://www.aclweb.org/anthology/2020.aacl-main.19.pdf
\nTransformer-based Approach for Predicting Chemical Compound Structures
\"For tokenizing InChI strings, the model was learned on SentencePiece (Kudo and Richardson, 2018), a unigram-based unsupervised training method for word segmentation. \"
", "rawMarkdown": "\"We found a lot of research on different tokenization strategies using SMILES but almost no information about InChI. \"\n\nyou can check this:\nhttps://www.aclweb.org/anthology/2020.aacl-main.19.pdf\nTransformer-based Approach for Predicting Chemical Compound Structures\n\n\"For tokenizing InChI strings, the model was learned on SentencePiece (Kudo and Richardson, 2018), a unigram-based unsupervised training method for word segmentation. \"", "votes": null }, { "id": "1336489", "postDate": "06/05/2021 01:08:14", "content": "Thank you, I missed this article!
", "rawMarkdown": "Thank you, I missed this article!", "votes": null }, { "id": "1336549", "postDate": "06/05/2021 03:14:28", "content": "Wonderful! Thanks for sharing. I am wondering how the \"RandomLineRemoval\" and \"RandomPixelThreshold\" augmentations are implemented.
", "rawMarkdown": "Wonderful! Thanks for sharing. I am wondering how the \"RandomLineRemoval\" and \"RandomPixelThreshold\" augmentations are implemented.", "votes": null } ], "comments": [ { "id": 1336481, "author_name": "dragonzhang", "author_url": "", "post_date": "06/05/2021 00:49:48", "content": "Thx for sharing. BPE and other tricks learned.
", "votes": null, "replies": [] }, { "id": 1336482, "author_name": "talktocharles", "author_url": "", "post_date": "06/05/2021 00:51:12", "content": "Great work, thanks for sharing your insights on the tokenisers. Looking forward to seeing the ensemble script from @zfturbo.
", "votes": null, "replies": [] }, { "id": 1336487, "author_name": "hengck23", "author_url": "", "post_date": "06/05/2021 01:04:12", "content": "\"We found a lot of research on different tokenization strategies using SMILES but almost no information about InChI. \"
\nyou can check this:
\nhttps://www.aclweb.org/anthology/2020.aacl-main.19.pdf
\nTransformer-based Approach for Predicting Chemical Compound Structures
\"For tokenizing InChI strings, the model was learned on SentencePiece (Kudo and Richardson, 2018), a unigram-based unsupervised training method for word segmentation. \"
", "votes": null, "replies": [ { "id": 1336489, "author_name": "drhabib", "author_url": "", "post_date": "06/05/2021 01:08:14", "content": "Thank you, I missed this article!
", "votes": null, "replies": [] } ] }, { "id": 1336549, "author_name": "houndcl", "author_url": "", "post_date": "06/05/2021 03:14:28", "content": "Wonderful! Thanks for sharing. I am wondering how the \"RandomLineRemoval\" and \"RandomPixelThreshold\" augmentations are implemented.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1336470": "First of all, I would like to thanks the organizers for organizing such an exciting competition. Second I would like to thank my teammates. We had a great working environment with everybody working hard, and last one week, we had to stay up until 3 am local times. Below I will try to summarize the main idea of our best-performing model.\n\n## Image Preprocessing\nOne of the challenging things about this competition was that the chemical had very different structures, some were super long, and others were very short. This also meant that images were highly diverse in size and shapes (e.g., super long in 1 axis or super small). Therefore it was essential to preprocess these images correctly. After some initial discussion we decided to try to use two methods: