\n
![\"\"](\"https://i.ibb.co/BK6DWwX/builder.png\")
Part 1 by the great leader @hengck23:
\nhttps://www.kaggle.com/c/bms-molecular-translation/discussion/243766 the TNT/VIT/CAIT ensemble model has provided a good baseline for my last-minute shot.
From the competition overview:
\n\n\nExisting tools produce 90% accuracy but only under optimal conditions. Historical sources often have some level of image corruption, which reduces performance to near zero.
\n
I come from the so said existing tools world with zero deep learning experience. I joined the competition to prove this overview is not true (but also true). My colleagues are the developers of MolVec: https://molvec.ncats.io. \"Traditional OCR method is able to achieve >90% accuracy on near-perfect images\", but the first submission slapped my face hard: LD ~ 80, just slightly better than the H2O naïve baseline. Some heuristic fine tuning (link disconnected components, atomic/bond fixers) has made molvec perform much better on noised images, but we found that pure OCR method stuck on the LD = 5 line. Improvement is possible but it is limited by \"what is presented by the image\" rather than \"what is encoded in the image\". So I wrote about ghost atom problem in image resizing. In addition, OCR methods which heavily rely on heuristics (e.g., the appearance of atom/bond/connection) has a hard time dealing with complex cross-bond structure (example). I will discuss about OCR method in the thoughts section.
\nSo, while top teams are approaching LD ~ 1, what's the use of traditional OCR method which is stuck on LD = 5? We abandoned our arrogance and shifted overall strategy: patch deep learning predictions for what it doesn't work well.
\n\"Traditional OCR method is able to achieve >90% accuracy on near-perfect images\".
\n| \n | \n |
|---|---|
| Training | \n248k randomly selected kaggle image - Indigo rendered image pairs | \n
| Model | \nResNet34 + U-Net with self-attention fastai tutorial | \n
| Image size | \n832 x 832 | \n
| Inference | \n- small image (any dimension < 750): upscale to 832. big image (any dimension 750-1200): upscale to 1248. xbig image (any dimension > 1200): upscale to 1504 | \n
| InChI generator | \nsuper resolution images -> molvec -> .mol -> InChI | \n
| Performance | \nLD ~ 3 | \n
| Fail | \ncomplex molecule | \n
From the discussion board, there are lots of complaints like \"my model doesn't work well on large images\". But OCR works decently regardless of image size (10,000 pixels is fine)
\nNo training. Just feed super resolution images to molvec -> .mol -> InChI
\nI didn't benchmark the performance for big images, but with 90% accuracy, it should outperforms deep learning model that downscaled to 384 or 224 in which most molecular features are lost.
Idea identical to the DACON winning solution, but to build a valid molecule with acceptable layout from the detected objects on noised image requires lots of \"fixers\" as well as atom/bond imputation. We introduced the atom group idea to help the valency identification: e.g., CH2 and NH are different than C and N. We separate implicit carbon (shown as vertex) and explicit carbon (shown as Cxx). Meanwhile we enlarge the bbox a bit during the training to include some local context, e.g., a CH2 group usually has 2 bond tips in the bbox. There are totally 54 atom groups and bond types in the 1.6M training set, and we selected 49 for training.
\n| \n | \n |
|---|---|
| Training | \n691k training set images. Bounding boxes of atom/atom group/bond were obtained from Indigo rendered svg and custom parser. These 691k includes hard images with a. cross-bond structure or b. atom-atom close contact or c. rare atom/atom groups (e.g, isotope, explicit H, explicit CH2, etc) | \n
| Model | \nEfficientDet-D3 | \n
| Image size | \n896 x 896 (bigger the better) | \n
| Augmentation | \nrotation / flip / salt & pepper | \n
| InChI generator | \nbbox -> custom molecular builder script -> .mol -> InChI | \n
| Performance | \noverall mAP=0.87. LD~0.65 for simple molecule. LD~3.5 for complex molecule | \n
| Fail | \nbig images dim > 896 | \n
Manually check several hundreds hard molecules in the validation set where VIT failed to predict valid InChI, and found that on average MolBuild could reduce LD by ~25 even if MolBuild predicted the wrong molecule. Performance could be better if we have more time on training. It is almost a last-minute model.
\n
LB 0.71 -> 0.61 by updating only 7,679 invalid InChIs (totally 9,800) from VIT in the final submission. Criteria of substitution:
\nReplacing 0.48% test set gains 0.1 overall LB score, indicating that on average each substitution reduces Levenshtein distance by 21.
\nFirst I'd like to thank leader @hengck23 and members @mathurinache @ruchi798 who have worked very hard in the past few months. Special thank to @DrHB's team who crashed my day dream that OCR is undefeatable early in this competition. Also thank my colleagues who developed SOTA OCR method and fine tune the parameters. Last but not least, it's been a great & tough competition. Thank you organizers & staff behind the scene.
\nWith more training data augmented with different renderer parameters, I am convinced that deep learning will win with a big margin, at least for \"plain\" & small molecule. On the other hand, traditional OCR method which is more generalized can be complementary to SOTA DL model when training data is unreachable (e.g., complex layout). There is always a sweet spot between domain knowledge which offers heuristics and data science which offers precision. Domain knowledge provides indispensable shortcut to the core of problem, and is cost-efficient in pre/post-processing and when big & clean data / annotation are lacking.
\nFirst we need to understand what is being learned:
\nFrom my understanding, most image-driven, encoder-decoder model is good at learning layout (#2) & noise (#3). InChI validation introduces the chemistry rules (#1) elegantly. There is a great discussion here by @nofreewill. On the other hand, OCR method which mainly looks at chemistry will be defeated when noise / layout is out of its domain. The message is that when we understand the problem better, we will solve it more efficiently.
\nHappy think out of the box.
", "messages": [ { "id": "1335176", "postDate": "06/04/2021 03:54:01", "content": "Part 1 by the great leader @hengck23:
\nhttps://www.kaggle.com/c/bms-molecular-translation/discussion/243766 the TNT/VIT/CAIT ensemble model has provided a good baseline for my last-minute shot.
From the competition overview:
\n\n\nExisting tools produce 90% accuracy but only under optimal conditions. Historical sources often have some level of image corruption, which reduces performance to near zero.
\n
I come from the so said existing tools world with zero deep learning experience. I joined the competition to prove this overview is not true (but also true). My colleagues are the developers of MolVec: https://molvec.ncats.io. \"Traditional OCR method is able to achieve >90% accuracy on near-perfect images\", but the first submission slapped my face hard: LD ~ 80, just slightly better than the H2O naïve baseline. Some heuristic fine tuning (link disconnected components, atomic/bond fixers) has made molvec perform much better on noised images, but we found that pure OCR method stuck on the LD = 5 line. Improvement is possible but it is limited by \"what is presented by the image\" rather than \"what is encoded in the image\". So I wrote about ghost atom problem in image resizing. In addition, OCR methods which heavily rely on heuristics (e.g., the appearance of atom/bond/connection) has a hard time dealing with complex cross-bond structure (example). I will discuss about OCR method in the thoughts section.
\nSo, while top teams are approaching LD ~ 1, what's the use of traditional OCR method which is stuck on LD = 5? We abandoned our arrogance and shifted overall strategy: patch deep learning predictions for what it doesn't work well.
\n\"Traditional OCR method is able to achieve >90% accuracy on near-perfect images\".
\n| \n | \n |
|---|---|
| Training | \n248k randomly selected kaggle image - Indigo rendered image pairs | \n
| Model | \nResNet34 + U-Net with self-attention fastai tutorial | \n
| Image size | \n832 x 832 | \n
| Inference | \n- small image (any dimension < 750): upscale to 832. big image (any dimension 750-1200): upscale to 1248. xbig image (any dimension > 1200): upscale to 1504 | \n
| InChI generator | \nsuper resolution images -> molvec -> .mol -> InChI | \n
| Performance | \nLD ~ 3 | \n
| Fail | \ncomplex molecule | \n
From the discussion board, there are lots of complaints like \"my model doesn't work well on large images\". But OCR works decently regardless of image size (10,000 pixels is fine)
\nNo training. Just feed super resolution images to molvec -> .mol -> InChI
\nI didn't benchmark the performance for big images, but with 90% accuracy, it should outperforms deep learning model that downscaled to 384 or 224 in which most molecular features are lost.
Idea identical to the DACON winning solution, but to build a valid molecule with acceptable layout from the detected objects on noised image requires lots of \"fixers\" as well as atom/bond imputation. We introduced the atom group idea to help the valency identification: e.g., CH2 and NH are different than C and N. We separate implicit carbon (shown as vertex) and explicit carbon (shown as Cxx). Meanwhile we enlarge the bbox a bit during the training to include some local context, e.g., a CH2 group usually has 2 bond tips in the bbox. There are totally 54 atom groups and bond types in the 1.6M training set, and we selected 49 for training.
\n| \n | \n |
|---|---|
| Training | \n691k training set images. Bounding boxes of atom/atom group/bond were obtained from Indigo rendered svg and custom parser. These 691k includes hard images with a. cross-bond structure or b. atom-atom close contact or c. rare atom/atom groups (e.g, isotope, explicit H, explicit CH2, etc) | \n
| Model | \nEfficientDet-D3 | \n
| Image size | \n896 x 896 (bigger the better) | \n
| Augmentation | \nrotation / flip / salt & pepper | \n
| InChI generator | \nbbox -> custom molecular builder script -> .mol -> InChI | \n
| Performance | \noverall mAP=0.87. LD~0.65 for simple molecule. LD~3.5 for complex molecule | \n
| Fail | \nbig images dim > 896 | \n
Manually check several hundreds hard molecules in the validation set where VIT failed to predict valid InChI, and found that on average MolBuild could reduce LD by ~25 even if MolBuild predicted the wrong molecule. Performance could be better if we have more time on training. It is almost a last-minute model.
\n
LB 0.71 -> 0.61 by updating only 7,679 invalid InChIs (totally 9,800) from VIT in the final submission. Criteria of substitution:
\nReplacing 0.48% test set gains 0.1 overall LB score, indicating that on average each substitution reduces Levenshtein distance by 21.
\nFirst I'd like to thank leader @hengck23 and members @mathurinache @ruchi798 who have worked very hard in the past few months. Special thank to @DrHB's team who crashed my day dream that OCR is undefeatable early in this competition. Also thank my colleagues who developed SOTA OCR method and fine tune the parameters. Last but not least, it's been a great & tough competition. Thank you organizers & staff behind the scene.
\nWith more training data augmented with different renderer parameters, I am convinced that deep learning will win with a big margin, at least for \"plain\" & small molecule. On the other hand, traditional OCR method which is more generalized can be complementary to SOTA DL model when training data is unreachable (e.g., complex layout). There is always a sweet spot between domain knowledge which offers heuristics and data science which offers precision. Domain knowledge provides indispensable shortcut to the core of problem, and is cost-efficient in pre/post-processing and when big & clean data / annotation are lacking.
\nFirst we need to understand what is being learned:
\nFrom my understanding, most image-driven, encoder-decoder model is good at learning layout (#2) & noise (#3). InChI validation introduces the chemistry rules (#1) elegantly. There is a great discussion here by @nofreewill. On the other hand, OCR method which mainly looks at chemistry will be defeated when noise / layout is out of its domain. The message is that when we understand the problem better, we will solve it more efficiently.
\nHappy think out of the box.
", "rawMarkdown": "Part 1 by the great leader @hengck23: \nhttps://www.kaggle.com/c/bms-molecular-translation/discussion/243766 the TNT/VIT/CAIT ensemble model has provided a good baseline for my last-minute shot.\n\n#### Before technical details\nFrom the competition overview:\n> Existing tools produce 90% accuracy but only under optimal conditions. Historical sources often have some level of image corruption, which reduces performance to near zero. \n\nI come from the so said *existing tools* world with zero deep learning experience. I joined the competition to prove this overview is not true (but also true). My colleagues are the developers of MolVec: https://molvec.ncats.io. \"Traditional OCR method is able to achieve >90% accuracy on near-perfect images\", but the first submission slapped my face hard: LD ~ 80, just slightly better than the H2O naïve baseline. Some heuristic fine tuning (link disconnected components, atomic/bond fixers) has made molvec perform much better on noised images, but we found that pure OCR method stuck on the LD = 5 line. Improvement is possible but it is limited by *\"what is presented by the image\"* rather than *\"what is encoded in the image\"*. So I wrote about [ghost atom](https://www.kaggle.com/c/bms-molecular-translation/discussion/233927) problem in image resizing. In addition, OCR methods which heavily rely on heuristics (e.g., the appearance of atom/bond/connection) has a hard time dealing with complex [cross-bond structure (example)](https://molvec.ncats.io/c3bb46b5a4). I will discuss about OCR method in the thoughts section. \n\nSo, while top teams are approaching LD ~ 1, what's the use of traditional OCR method which is stuck on LD = 5? We abandoned our arrogance and shifted overall strategy: patch deep learning predictions for what it doesn't work well.\n1. by feeding what OCR is good at (near-perfect image) \n2. on the areas where other method doesn't work (big image)\n3. borrow the idea (aka, build molecule atom by atom using chemistry rules)\n\n#### 1. Feed OCR with super-resolution image\n\"Traditional OCR method is able to achieve >90% accuracy on near-perfect images\". \n| | |\n| --- | --- |\n| Training | 248k randomly selected kaggle image - Indigo rendered image pairs |\n| Model | ResNet34 + U-Net with self-attention [fastai tutorial](https://github.com/fastai/fastai/blob/master/dev_nbs/course/lesson7-superres.ipynb) |\n| Image size | 832 x 832 |\n| Inference | - small image (any dimension < 750): upscale to 832. big image (any dimension 750-1200): upscale to 1248. xbig image (any dimension > 1200): upscale to 1504\n|\n|InChI generator | super resolution images -> molvec -> .mol -> InChI |\n|Performance | LD ~ 3 |\n|Fail | complex molecule |\n\n#### 2. Use OCR on extra large image (e.g., width > 1000 pixels)\nFrom the discussion board, there are lots of complaints like \"my model doesn't work well on large images\". But OCR works decently regardless of image size (10,000 pixels is fine)\n\n*No training*. Just feed super resolution images to molvec -> .mol -> InChI\nI didn't benchmark the performance for big images, but with 90% accuracy, it should outperforms deep learning model that downscaled to 384 or 224 in which most molecular features are lost.\n\n#### 3. Object detection route to build complex molecules from scratch\nIdea identical to the DACON winning solution, but to build a valid molecule with acceptable layout from the detected objects on *noised* image requires lots of \"fixers\" as well as atom/bond imputation. We introduced the **atom group** idea to help the valency identification: e.g., CH2 and NH are different than C and N. We separate implicit carbon (shown as vertex) and explicit carbon (shown as Cxx). Meanwhile we enlarge the bbox a bit during the training to include some local context, e.g., a CH2 group usually has 2 bond tips in the bbox. There are totally 54 atom groups and bond types in the 1.6M training set, and we selected 49 for training.\n\n| | |\n| --- | --- |\n| Training | 691k training set images. Bounding boxes of atom/atom group/bond were obtained from Indigo rendered svg and custom parser. These 691k includes hard images with a. cross-bond structure or b. atom-atom close contact or c. rare atom/atom groups (e.g, isotope, explicit H, explicit CH2, etc) |\n| Model | EfficientDet-D3|\n| Image size| 896 x 896 (bigger the better) |\n| Augmentation | rotation / flip / salt & pepper|\n| InChI generator | bbox -> custom molecular builder script -> .mol -> InChI |\n| Performance | overall mAP=0.87. LD~0.65 for simple molecule. LD~3.5 for complex molecule |\n| Fail | big images dim > 896 |\n\nManually check several hundreds hard molecules in the validation set where VIT failed to predict valid InChI, and found that on average MolBuild could reduce LD by ~25 even if MolBuild predicted the wrong molecule. Performance could be better if we have more time on training. It is almost a last-minute model.\n\n\n#### 4. Merge with TNT/VIT/CAIT ensemble model\n**LB 0.71 -> 0.61** by updating only **7,679** invalid InChIs (totally 9,800) from VIT in the final submission. Criteria of substitution:\n\n1. Top tier: the formula component of InChI (the C3H4O2 part) matches the VIT prediction. \n2. Second tier: OCR/MolBuilder-to-VIT edit distance is lower than 10. \n3. Third tier: OCR matches MolBuilder prediction\n4. Finally, push in all OCR result for big and xbig images, regardless of distance\n\nReplacing 0.48% test set gains 0.1 overall LB score, indicating that **on average each substitution reduces Levenshtein distance by 21**. \n\n#### Thanks\nFirst I'd like to thank leader @hengck23 and members @mathurinache @ruchi798 who have worked very hard in the past few months. Special thank to @DrHB's team who crashed my day dream that OCR is undefeatable early in this competition. Also thank my colleagues who developed SOTA OCR method and fine tune the parameters. Last but not least, it's been a great & tough competition. Thank you organizers & staff behind the scene.\n\n#### Thoughts\n###### End of molecular OCR?\nWith more training data augmented with different renderer parameters, I am convinced that deep learning will win with a big margin, at least for \"plain\" & small molecule. On the other hand, traditional OCR method which is more generalized can be complementary to SOTA DL model when training data is unreachable (e.g., complex layout). **There is always a sweet spot between domain knowledge which offers heuristics and data science which offers precision.** Domain knowledge provides indispensable shortcut to the core of problem, and is cost-efficient in pre/post-processing and when big & clean data / annotation are lacking. \n\n###### Why InChI validation by RDKit works?\nFirst we need to understand what is being learned: \n1. chemistry rules (e.g., a carbon cannot have 10 bonds. InChI string has strict rules)\n2. layout (for 99.9% cases, a benzene ring will be rendered as a hexagon)\n3. noise (vertical/horizontal lines are more likely to disappear in resizing)\n4. synthetic feasibility (wild structure could hardly be synthesized)\n\nFrom my understanding, most image-driven, encoder-decoder model is good at learning layout (#2) & noise (#3). InChI validation introduces the chemistry rules (#1) elegantly. There is a great discussion [here](https://www.kaggle.com/c/bms-molecular-translation/discussion/243102) by @nofreewill. On the other hand, OCR method which mainly looks at chemistry will be defeated when noise / layout is out of its domain. The message is that when we understand the problem better, we will solve it more efficiently.\n\nHappy think out of the box.", "votes": null }, { "id": "1335262", "postDate": "06/04/2021 05:44:43", "content": "Wow, this is a really cool trick) Congratulations on your gold medal!
", "rawMarkdown": "Wow, this is a really cool trick) Congratulations on your gold medal!", "votes": null }, { "id": "1335713", "postDate": "06/04/2021 11:50:03", "content": "It's truly amazing to see solutions like this ;)
", "rawMarkdown": "It's truly amazing to see solutions like this ;)", "votes": null }, { "id": "1597039", "postDate": "11/27/2021 06:24:12", "content": "Dear @houndcl, I really like your solution! Do you by any chance have ideas why the competition host used Indigo to visualize the structures? Would any other tools have worked similarly or does Indigo have some advantages compared to the other tools? Thanks!
", "rawMarkdown": "Dear @houndcl, I really like your solution! Do you by any chance have ideas why the competition host used Indigo to visualize the structures? Would any other tools have worked similarly or does Indigo have some advantages compared to the other tools? Thanks!", "votes": null } ], "comments": [ { "id": 1335262, "author_name": "timofeev25", "author_url": "", "post_date": "06/04/2021 05:44:43", "content": "Wow, this is a really cool trick) Congratulations on your gold medal!
", "votes": null, "replies": [] }, { "id": 1335713, "author_name": "artnotintelligence", "author_url": "", "post_date": "06/04/2021 11:50:03", "content": "It's truly amazing to see solutions like this ;)
", "votes": null, "replies": [] }, { "id": 1597039, "author_name": "nurlybek17", "author_url": "", "post_date": "11/27/2021 06:24:12", "content": "Dear @houndcl, I really like your solution! Do you by any chance have ideas why the competition host used Indigo to visualize the structures? Would any other tools have worked similarly or does Indigo have some advantages compared to the other tools? Thanks!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1335176": "Part 1 by the great leader @hengck23: \nhttps://www.kaggle.com/c/bms-molecular-translation/discussion/243766 the TNT/VIT/CAIT ensemble model has provided a good baseline for my last-minute shot.\n\n#### Before technical details\nFrom the competition overview:\n> Existing tools produce 90% accuracy but only under optimal conditions. Historical sources often have some level of image corruption, which reduces performance to near zero. \n\nI come from the so said *existing tools* world with zero deep learning experience. I joined the competition to prove this overview is not true (but also true). My colleagues are the developers of MolVec: https://molvec.ncats.io. \"Traditional OCR method is able to achieve >90% accuracy on near-perfect images\", but the first submission slapped my face hard: LD ~ 80, just slightly better than the H2O naïve baseline. Some heuristic fine tuning (link disconnected components, atomic/bond fixers) has made molvec perform much better on noised images, but we found that pure OCR method stuck on the LD = 5 line. Improvement is possible but it is limited by *\"what is presented by the image\"* rather than *\"what is encoded in the image\"*. So I wrote about [ghost atom](https://www.kaggle.com/c/bms-molecular-translation/discussion/233927) problem in image resizing. In addition, OCR methods which heavily rely on heuristics (e.g., the appearance of atom/bond/connection) has a hard time dealing with complex [cross-bond structure (example)](https://molvec.ncats.io/c3bb46b5a4). I will discuss about OCR method in the thoughts section. \n\nSo, while top teams are approaching LD ~ 1, what's the use of traditional OCR method which is stuck on LD = 5? We abandoned our arrogance and shifted overall strategy: patch deep learning predictions for what it doesn't work well.\n1. by feeding what OCR is good at (near-perfect image) \n2. on the areas where other method doesn't work (big image)\n3. borrow the idea (aka, build molecule atom by atom using chemistry rules)\n\n#### 1. Feed OCR with super-resolution image\n\"Traditional OCR method is able to achieve >90% accuracy on near-perfect images\". \n| | |\n| --- | --- |\n| Training | 248k randomly selected kaggle image - Indigo rendered image pairs |\n| Model | ResNet34 + U-Net with self-attention [fastai tutorial](https://github.com/fastai/fastai/blob/master/dev_nbs/course/lesson7-superres.ipynb) |\n| Image size | 832 x 832 |\n| Inference | - small image (any dimension < 750): upscale to 832. big image (any dimension 750-1200): upscale to 1248. xbig image (any dimension > 1200): upscale to 1504\n|\n|InChI generator | super resolution images -> molvec -> .mol -> InChI |\n|Performance | LD ~ 3 |\n|Fail | complex molecule |\n\n#### 2. Use OCR on extra large image (e.g., width > 1000 pixels)\nFrom the discussion board, there are lots of complaints like \"my model doesn't work well on large images\". But OCR works decently regardless of image size (10,000 pixels is fine)\n\n*No training*. Just feed super resolution images to molvec -> .mol -> InChI\nI didn't benchmark the performance for big images, but with 90% accuracy, it should outperforms deep learning model that downscaled to 384 or 224 in which most molecular features are lost.\n\n#### 3. Object detection route to build complex molecules from scratch\nIdea identical to the DACON winning solution, but to build a valid molecule with acceptable layout from the detected objects on *noised* image requires lots of \"fixers\" as well as atom/bond imputation. We introduced the **atom group** idea to help the valency identification: e.g., CH2 and NH are different than C and N. We separate implicit carbon (shown as vertex) and explicit carbon (shown as Cxx). Meanwhile we enlarge the bbox a bit during the training to include some local context, e.g., a CH2 group usually has 2 bond tips in the bbox. There are totally 54 atom groups and bond types in the 1.6M training set, and we selected 49 for training.\n\n| | |\n| --- | --- |\n| Training | 691k training set images. Bounding boxes of atom/atom group/bond were obtained from Indigo rendered svg and custom parser. These 691k includes hard images with a. cross-bond structure or b. atom-atom close contact or c. rare atom/atom groups (e.g, isotope, explicit H, explicit CH2, etc) |\n| Model | EfficientDet-D3|\n| Image size| 896 x 896 (bigger the better) |\n| Augmentation | rotation / flip / salt & pepper|\n| InChI generator | bbox -> custom molecular builder script -> .mol -> InChI |\n| Performance | overall mAP=0.87. LD~0.65 for simple molecule. LD~3.5 for complex molecule |\n| Fail | big images dim > 896 |\n\nManually check several hundreds hard molecules in the validation set where VIT failed to predict valid InChI, and found that on average MolBuild could reduce LD by ~25 even if MolBuild predicted the wrong molecule. Performance could be better if we have more time on training. It is almost a last-minute model.\n\n\n#### 4. Merge with TNT/VIT/CAIT ensemble model\n**LB 0.71 -> 0.61** by updating only **7,679** invalid InChIs (totally 9,800) from VIT in the final submission. Criteria of substitution:\n\n1. Top tier: the formula component of InChI (the C3H4O2 part) matches the VIT prediction. \n2. Second tier: OCR/MolBuilder-to-VIT edit distance is lower than 10. \n3. Third tier: OCR matches MolBuilder prediction\n4. Finally, push in all OCR result for big and xbig images, regardless of distance\n\nReplacing 0.48% test set gains 0.1 overall LB score, indicating that **on average each substitution reduces Levenshtein distance by 21**. \n\n#### Thanks\nFirst I'd like to thank leader @hengck23 and members @mathurinache @ruchi798 who have worked very hard in the past few months. Special thank to @DrHB's team who crashed my day dream that OCR is undefeatable early in this competition. Also thank my colleagues who developed SOTA OCR method and fine tune the parameters. Last but not least, it's been a great & tough competition. Thank you organizers & staff behind the scene.\n\n#### Thoughts\n###### End of molecular OCR?\nWith more training data augmented with different renderer parameters, I am convinced that deep learning will win with a big margin, at least for \"plain\" & small molecule. On the other hand, traditional OCR method which is more generalized can be complementary to SOTA DL model when training data is unreachable (e.g., complex layout). **There is always a sweet spot between domain knowledge which offers heuristics and data science which offers precision.** Domain knowledge provides indispensable shortcut to the core of problem, and is cost-efficient in pre/post-processing and when big & clean data / annotation are lacking. \n\n###### Why InChI validation by RDKit works?\nFirst we need to understand what is being learned: \n1. chemistry rules (e.g., a carbon cannot have 10 bonds. InChI string has strict rules)\n2. layout (for 99.9% cases, a benzene ring will be rendered as a hexagon)\n3. noise (vertical/horizontal lines are more likely to disappear in resizing)\n4. synthetic feasibility (wild structure could hardly be synthesized)\n\nFrom my understanding, most image-driven, encoder-decoder model is good at learning layout (#2) & noise (#3). InChI validation introduces the chemistry rules (#1) elegantly. There is a great discussion [here](https://www.kaggle.com/c/bms-molecular-translation/discussion/243102) by @nofreewill. On the other hand, OCR method which mainly looks at chemistry will be defeated when noise / layout is out of its domain. The message is that when we understand the problem better, we will solve it more efficiently.\n\nHappy think out of the box.", "1335262": "Wow, this is a really cool trick) Congratulations on your gold medal!", "1335713": "It's truly amazing to see solutions like this ;)", "1597039": "Dear @houndcl, I really like your solution! Do you by any chance have ideas why the competition host used Indigo to visualize the structures? Would any other tools have worked similarly or does Indigo have some advantages compared to the other tools? Thanks!" }, "source": "meta" }