![\"\"](\"https://i.ibb.co/TYt5h7m/Selection-193.png\")
to be updated ….
\nthere are two part:
\nTNT/VIT/CAIT ensemble model (public/private LB 0.71/0.71)
\nwe train 5 models : 3x patched based VIT (patch size 16, image scale = 0.8,0.9,0.1), 1x TNT with image size 384 and 1x CAIT with image size 240. For details, please refer to https://www.kaggle.com/c/bms-molecular-translation/discussion/231190
\n(local CV 0.78)
we do 5 iterations of image perturbation. if an image has gives an invalid inchi (verification by rdkit), we perturb the image and try inference again.
\n( local CV 0.76)
finally, we apply rdkit normalisation.
\n(public/private LB 0.73/0.74, local CV 0.72)
The ensemble is our top score submission. We also have submissions for each of the individual models. We then sort all submission cvs by LB score. To create the final submission:
\n
we use random shift and scale to perturb image
\nhere is a tip on ensemble strategy:
\n![\"\"](\"https://i.ibb.co/F3Yns33/Selection-202.png\")
we note that lb score is correlated to the number of invalid inchi (verified from rdkit). If you have fewer invalid inchi, you are likely to better LB score.
\ninstead of making many submissions, you can use the number of invalid inchi to estimate your lb score. To know the limit of LB score of your current method, you can extrapolate your graph to the lowest number of possible inchi. If that doesn't work well, you can change your ensemble strategy so that you have a steeper graph. (i.e. better lb score per invalid inchi)
\nI am grateful to Z by HP & NVIDIA for sponsoring me a Z8 Workstation with dual RTX8000 GPU for this competition. In the final days, I need to run TNT/VIT/CAIT ensemble (5 transformers) on the 1.6 million test images. Some of the test images are large and have long sequences. Besides high computative power, we also need high GPU memory.
\nOne RTX8000 GPU has 48GB onboard. It takes about 7 hours to process all images using the workstation. This is very fast and definitely help our team to secure a gold medal of 10th placing in the final ranking!
", "messages": [ { "id": "1334981", "postDate": "06/04/2021 00:11:46", "content": "to be updated ….
\nthere are two part:
\nTNT/VIT/CAIT ensemble model (public/private LB 0.71/0.71)
\nwe train 5 models : 3x patched based VIT (patch size 16, image scale = 0.8,0.9,0.1), 1x TNT with image size 384 and 1x CAIT with image size 240. For details, please refer to https://www.kaggle.com/c/bms-molecular-translation/discussion/231190
\n(local CV 0.78)
we do 5 iterations of image perturbation. if an image has gives an invalid inchi (verification by rdkit), we perturb the image and try inference again.
\n( local CV 0.76)
finally, we apply rdkit normalisation.
\n(public/private LB 0.73/0.74, local CV 0.72)
The ensemble is our top score submission. We also have submissions for each of the individual models. We then sort all submission cvs by LB score. To create the final submission:
\n
we use random shift and scale to perturb image
\nhere is a tip on ensemble strategy:
\n
we note that lb score is correlated to the number of invalid inchi (verified from rdkit). If you have fewer invalid inchi, you are likely to better LB score.
\ninstead of making many submissions, you can use the number of invalid inchi to estimate your lb score. To know the limit of LB score of your current method, you can extrapolate your graph to the lowest number of possible inchi. If that doesn't work well, you can change your ensemble strategy so that you have a steeper graph. (i.e. better lb score per invalid inchi)
\nI am grateful to Z by HP & NVIDIA for sponsoring me a Z8 Workstation with dual RTX8000 GPU for this competition. In the final days, I need to run TNT/VIT/CAIT ensemble (5 transformers) on the 1.6 million test images. Some of the test images are large and have long sequences. Besides high computative power, we also need high GPU memory.
\nOne RTX8000 GPU has 48GB onboard. It takes about 7 hours to process all images using the workstation. This is very fast and definitely help our team to secure a gold medal of 10th placing in the final ranking!
", "rawMarkdown": "to be updated ....\n\nthere are two part:\n- part one: TNT/VIT/CAIT ensemble model (public/private LB 0.71/0.71)\nhttps://www.kaggle.com/c/bms-molecular-translation/discussion/243766 (this post)\n- part two: OCR + MolBuilder + object detection CNN (public/private LB 0.62/0.61)\nhttps://www.kaggle.com/c/bms-molecular-translation/discussion/243809\n\n\n---\n\nTNT/VIT/CAIT ensemble model (public/private LB 0.71/0.71)\n\n- we train 5 models : 3x patched based VIT (patch size 16, image scale = 0.8,0.9,0.1), 1x TNT with image size 384 and 1x CAIT with image size 240. For details, please refer to https://www.kaggle.com/c/bms-molecular-translation/discussion/231190\n(local CV 0.78) \n\n- we do 5 iterations of image perturbation. if an image has gives an invalid inchi (verification by rdkit), we perturb the image and try inference again.\n( local CV 0.76) \n\n- finally, we apply rdkit normalisation.\n(public/private LB 0.73/0.74, local CV 0.72) \n\nThe ensemble is our top score submission. We also have submissions for each of the individual models. We then sort all submission cvs by LB score. To create the final submission:\n- 1. We start off will predictions from the top score submission csv. \n- 2. For a current prediction has invalid inchi, it will be replaced from the corresponding one from the next higher score submission csv if ( and only if ) that corresponding one is valid\n- 3. The process is repeated for all csvs. \nFinal LB score is (public/private LB 0.71/0.71, local CV 0.70)\n\n\n\nwe use random shift and scale to perturb image\n\n---\n\nhere is a tip on ensemble strategy:\n\n\nwe note that lb score is correlated to the number of invalid inchi (verified from rdkit). If you have fewer invalid inchi, you are likely to better LB score.\n\ninstead of making many submissions, you can use the number of invalid inchi to estimate your lb score. To know the limit of LB score of your current method, you can extrapolate your graph to the lowest number of possible inchi. If that doesn't work well, you can change your ensemble strategy so that you have a steeper graph. (i.e. better lb score per invalid inchi)\n\n---\n\nAcknowledgment\n---\n\nI am grateful to Z by HP & NVIDIA for sponsoring me a Z8 Workstation with dual RTX8000 GPU for this competition. In the final days, I need to run TNT/VIT/CAIT ensemble (5 transformers) on the 1.6 million test images. Some of the test images are large and have long sequences. Besides high computative power, we also need high GPU memory.\n\nOne RTX8000 GPU has 48GB onboard. It takes about 7 hours to process all images using the workstation. This is very fast and definitely help our team to secure a gold medal of 10th placing in the final ranking!", "votes": null }, { "id": "1335009", "postDate": "06/04/2021 00:38:00", "content": "the kaggle dataset is rendered by indigo toolkit (use old version 1.4.0-beta)
\nhttps://lifescience.opensource.epam.com/indigo/
![\"\"](\"https://i.ibb.co/ZzSv1mV/Selection-194.png\")
Wow really missed that. We were training with 1m extra images created by rdkit. It didn't hurt but did't help much.
", "rawMarkdown": "Wow really missed that. We were training with 1m extra images created by rdkit. It didn't hurt but did't help much.", "votes": null }, { "id": "1335018", "postDate": "06/04/2021 00:55:21", "content": "![\"\"](\"https://i.ibb.co/4Jqh27S/Selection-195.png\")
This means your model outperforms ours. A lot to learn from different teams! Big congrats to GM!
", "rawMarkdown": "This means your model outperforms ours. A lot to learn from different teams! Big congrats to GM!", "votes": null }, { "id": "1335021", "postDate": "06/04/2021 00:56:41", "content": "i trained with extra 2 million indigo images. i didn't exactly analyse/compare the results (due to lack of time). But it seems that it didn't help much in local cross validation.
\nI may think that learning super resolution is a good solution, but i didn't have time to implement.
\nIf we validate only using clean indigo images, local CV score is about 0.65 (instead of 1.20 with kaggle noisy images)
\nwith the original indigo image, we are able to analyze/model the domain difference of test and train data. In fact, our local CV and public are almost identical. This makes it easier to design our experiments.
", "rawMarkdown": "i trained with extra 2 million indigo images. i didn't exactly analyse/compare the results (due to lack of time). But it seems that it didn't help much in local cross validation.\n\nI may think that learning super resolution is a good solution, but i didn't have time to implement. \n\nIf we validate only using clean indigo images, local CV score is about 0.65 (instead of 1.20 with kaggle noisy images)\n\n---\n\nwith the original indigo image, we are able to analyze/model the domain difference of test and train data. In fact, our local CV and public are almost identical. This makes it easier to design our experiments.", "votes": null }, { "id": "1335036", "postDate": "06/04/2021 01:20:39", "content": "example of results:
\n![\"\"](\"https://i.ibb.co/ZzL5HLk/Selection-196.png\")
Big congrats and thank you for sharing your code!
", "rawMarkdown": "Big congrats and thank you for sharing your code!", "votes": null }, { "id": "1335342", "postDate": "06/04/2021 06:54:25", "content": "Big Congratulations @hengck23 , I am sure a lot of people have got their medals owing to your pipeline . Using object detection is another good idea , great work
", "rawMarkdown": "Big Congratulations @hengck23 , I am sure a lot of people have got their medals owing to your pipeline . Using object detection is another good idea , great work", "votes": null }, { "id": "1335432", "postDate": "06/04/2021 07:55:12", "content": "Dear frog bro
\nwhy I cannot get a score as same as yours
\nI also training a VIT from scratch on TPU , I think the score should be more near ,
\nbut after 10 epoch I only get 88% Acc,
# transformer decoder params\nnum_layer=4\nd_model=768\ndff= 512\nnum_heads=8\ntarget_vocab_size=197\ndropout_rate=0.1 \n\n# vision encoder params\nimage_size = (IMG_SHAPE[0], IMG_SHAPE[1]) # We'll resize input images to this size\npatch_size = 32 # Size of the patches to be extract from the input images\nnum_patches = (image_size[0] // patch_size) * (image_size[1] // patch_size) \nprojection_dim = d_model\nvit_heads = 12\ntransformer_units = [\n projection_dim * 2,\n projection_dim,\n] # Size of the transformer layers\ntransformer_layers = 12\nI use gradient clip to keep stable
\ngradients, _ = tf.clip_by_global_norm(gradients, 10.0)\nHere is my notebook: maybe you can take a look at the model part
\nhttps://www.kaggle.com/drzhuzhe/bms-vision-transformer?scriptVersionId=64760387
training is using spike learning rate
\nIt's the first time i implement Transformer via bare hand
", "rawMarkdown": "Dear frog bro\n\n\nwhy I cannot get a score as same as yours \nI also training a VIT from scratch on TPU , I think the score should be more near , \nbut after 10 epoch I only get 88% Acc,\n\n```\n# transformer decoder params\nnum_layer=4\nd_model=768\ndff= 512\nnum_heads=8\ntarget_vocab_size=197\ndropout_rate=0.1 \n\n# vision encoder params\nimage_size = (IMG_SHAPE[0], IMG_SHAPE[1]) # We'll resize input images to this size\npatch_size = 32 # Size of the patches to be extract from the input images\nnum_patches = (image_size[0] // patch_size) * (image_size[1] // patch_size) \nprojection_dim = d_model\nvit_heads = 12\ntransformer_units = [\n projection_dim * 2,\n projection_dim,\n] # Size of the transformer layers\ntransformer_layers = 12\n```\n\n I use gradient clip to keep stable\n\n```\ngradients, _ = tf.clip_by_global_norm(gradients, 10.0)\n```\n\nHere is my notebook: maybe you can take a look at the model part\nhttps://www.kaggle.com/drzhuzhe/bms-vision-transformer?scriptVersionId=64760387\n\ntraining is using spike learning rate\n\nIt's the first time i implement Transformer via bare hand", "votes": null }, { "id": "1335742", "postDate": "06/04/2021 12:16:04", "content": "I've been looking for this for a long time and tried different renderers, but I couldn't find the one used in the competition, so I gave up. :D
", "rawMarkdown": "I've been looking for this for a long time and tried different renderers, but I couldn't find the one used in the competition, so I gave up. :D", "votes": null }, { "id": "1336028", "postDate": "06/04/2021 15:35:41", "content": "Congrats for the gold medal ended, and thanks for sharing, always! ;)
", "rawMarkdown": "Congrats for the gold medal ended, and thanks for sharing, always! ;)", "votes": null }, { "id": "1336111", "postDate": "06/04/2021 16:20:03", "content": "Ha-ha, I wonder how many other teams did discover this as well :)
", "rawMarkdown": "Ha-ha, I wonder how many other teams did discover this as well :)", "votes": null }, { "id": "1336227", "postDate": "06/04/2021 18:21:45", "content": "Always a pleasure reading your posts. Congratulations on your place!
", "rawMarkdown": "Always a pleasure reading your posts. Congratulations on your place!", "votes": null }, { "id": "1343154", "postDate": "06/10/2021 04:43:21", "content": "we also make some modifications to our patch-based transformer.
\nwe note that resize artifacts are only present in the resized images.
\nwe want the model to handle resized and original images differently.
![\"\"](\"https://i.ibb.co/nC5tC3N/Selection-201.png\")
the kaggle dataset is rendered by indigo toolkit (use old version 1.4.0-beta)
\nhttps://lifescience.opensource.epam.com/indigo/
Wow really missed that. We were training with 1m extra images created by rdkit. It didn't hurt but did't help much.
", "votes": null, "replies": [] }, { "id": 1335019, "author_name": "houndcl", "author_url": "", "post_date": "06/04/2021 00:55:59", "content": "This means your model outperforms ours. A lot to learn from different teams! Big congrats to GM!
", "votes": null, "replies": [] }, { "id": 1335021, "author_name": "hengck23", "author_url": "", "post_date": "06/04/2021 00:56:41", "content": "i trained with extra 2 million indigo images. i didn't exactly analyse/compare the results (due to lack of time). But it seems that it didn't help much in local cross validation.
\nI may think that learning super resolution is a good solution, but i didn't have time to implement.
\nIf we validate only using clean indigo images, local CV score is about 0.65 (instead of 1.20 with kaggle noisy images)
\nwith the original indigo image, we are able to analyze/model the domain difference of test and train data. In fact, our local CV and public are almost identical. This makes it easier to design our experiments.
", "votes": null, "replies": [] }, { "id": 1335742, "author_name": "sorkun", "author_url": "", "post_date": "06/04/2021 12:16:04", "content": "I've been looking for this for a long time and tried different renderers, but I couldn't find the one used in the competition, so I gave up. :D
", "votes": null, "replies": [] }, { "id": 1336111, "author_name": "stassl", "author_url": "", "post_date": "06/04/2021 16:20:03", "content": "Ha-ha, I wonder how many other teams did discover this as well :)
", "votes": null, "replies": [] } ] }, { "id": 1335018, "author_name": "hengck23", "author_url": "", "post_date": "06/04/2021 00:55:21", "content": "
example of results:
\n
Big congrats and thank you for sharing your code!
", "votes": null, "replies": [] }, { "id": 1335342, "author_name": "tanulsingh077", "author_url": "", "post_date": "06/04/2021 06:54:25", "content": "Big Congratulations @hengck23 , I am sure a lot of people have got their medals owing to your pipeline . Using object detection is another good idea , great work
", "votes": null, "replies": [] }, { "id": 1335432, "author_name": "drzhuzhe", "author_url": "", "post_date": "06/04/2021 07:55:12", "content": "Dear frog bro
\nwhy I cannot get a score as same as yours
\nI also training a VIT from scratch on TPU , I think the score should be more near ,
\nbut after 10 epoch I only get 88% Acc,
# transformer decoder params\nnum_layer=4\nd_model=768\ndff= 512\nnum_heads=8\ntarget_vocab_size=197\ndropout_rate=0.1 \n\n# vision encoder params\nimage_size = (IMG_SHAPE[0], IMG_SHAPE[1]) # We'll resize input images to this size\npatch_size = 32 # Size of the patches to be extract from the input images\nnum_patches = (image_size[0] // patch_size) * (image_size[1] // patch_size) \nprojection_dim = d_model\nvit_heads = 12\ntransformer_units = [\n projection_dim * 2,\n projection_dim,\n] # Size of the transformer layers\ntransformer_layers = 12\nI use gradient clip to keep stable
\ngradients, _ = tf.clip_by_global_norm(gradients, 10.0)\nHere is my notebook: maybe you can take a look at the model part
\nhttps://www.kaggle.com/drzhuzhe/bms-vision-transformer?scriptVersionId=64760387
training is using spike learning rate
\nIt's the first time i implement Transformer via bare hand
", "votes": null, "replies": [] }, { "id": 1336028, "author_name": "haqishen", "author_url": "", "post_date": "06/04/2021 15:35:41", "content": "Congrats for the gold medal ended, and thanks for sharing, always! ;)
", "votes": null, "replies": [] }, { "id": 1336227, "author_name": "yassinealouini", "author_url": "", "post_date": "06/04/2021 18:21:45", "content": "Always a pleasure reading your posts. Congratulations on your place!
", "votes": null, "replies": [] }, { "id": 1343154, "author_name": "hengck23", "author_url": "", "post_date": "06/10/2021 04:43:21", "content": "we also make some modifications to our patch-based transformer.
\nwe note that resize artifacts are only present in the resized images.
\nwe want the model to handle resized and original images differently.