I believe many (like us) started with YNakama's great notebook and used the tokenizer in it. So I guess it may be worth share that there exists a bug in the word segmentation part.
\nFor example, the training sample 002f57184f1f has InChI InChI=1S/C7H2Cl3F3N4/c8-6(9,10)5-16-3(7(11,12)13)2(1-14)4(15)17-5/h(H2,15,16,17) and its word segmentation result should be
C 7 H 2 Cl 3 F 3 N 4 /c 8 - 6 ( 9 , 10 ) 5 - 16 - 3 ( 7 ( 11 , 12 ) 13 ) 2 ( 1 - 14 ) 4 ( 15 ) 17 - 5 /h ( H 2 , 15 , 16 , 17 )\nwhile it actually is
\nC 7 H 2 Cl 3 F 3 N 4 /c 8 - 6 ( 9 , 10 ) 5 - 16 - 3 ( 7 ( 11 , 12 ) 13 ) 2 ( 1 - 14 ) 4 ( 15 ) 17 - 5 /h 2 , 15 , 16 , 17 )\n.
\nBut I doubt this bug would have any real influences beacause it only affects 414 training samples and results in 790 Levenshtein distance in total. :P
", "messages": [ { "id": "1314411", "postDate": "05/19/2021 06:46:21", "content": "I believe many (like us) started with YNakama's great notebook and used the tokenizer in it. So I guess it may be worth share that there exists a bug in the word segmentation part.
\nFor example, the training sample 002f57184f1f has InChI InChI=1S/C7H2Cl3F3N4/c8-6(9,10)5-16-3(7(11,12)13)2(1-14)4(15)17-5/h(H2,15,16,17) and its word segmentation result should be
C 7 H 2 Cl 3 F 3 N 4 /c 8 - 6 ( 9 , 10 ) 5 - 16 - 3 ( 7 ( 11 , 12 ) 13 ) 2 ( 1 - 14 ) 4 ( 15 ) 17 - 5 /h ( H 2 , 15 , 16 , 17 )\nwhile it actually is
\nC 7 H 2 Cl 3 F 3 N 4 /c 8 - 6 ( 9 , 10 ) 5 - 16 - 3 ( 7 ( 11 , 12 ) 13 ) 2 ( 1 - 14 ) 4 ( 15 ) 17 - 5 /h 2 , 15 , 16 , 17 )\n.
\nBut I doubt this bug would have any real influences beacause it only affects 414 training samples and results in 790 Levenshtein distance in total. :P
", "rawMarkdown": "I believe many (like us) started with YNakama's great notebook and used the tokenizer in it. So I guess it may be worth share that there exists a bug in the word segmentation part. \n\nFor example, the training sample `002f57184f1f` has InChI `InChI=1S/C7H2Cl3F3N4/c8-6(9,10)5-16-3(7(11,12)13)2(1-14)4(15)17-5/h(H2,15,16,17)` and its word segmentation result should be\n```\nC 7 H 2 Cl 3 F 3 N 4 /c 8 - 6 ( 9 , 10 ) 5 - 16 - 3 ( 7 ( 11 , 12 ) 13 ) 2 ( 1 - 14 ) 4 ( 15 ) 17 - 5 /h ( H 2 , 15 , 16 , 17 )\n```\nwhile it actually is\n```\nC 7 H 2 Cl 3 F 3 N 4 /c 8 - 6 ( 9 , 10 ) 5 - 16 - 3 ( 7 ( 11 , 12 ) 13 ) 2 ( 1 - 14 ) 4 ( 15 ) 17 - 5 /h 2 , 15 , 16 , 17 )\n```\n.\n\nBut I doubt this bug would have any real influences beacause it only affects 414 training samples and results in 790 Levenshtein distance in total. :P", "votes": null }, { "id": "1314413", "postDate": "05/19/2021 06:48:52", "content": "My code:
\nimport re\nPATTEN = re.compile('\\d+|[A-Z][a-z]?|[^A-Za-z\\d/]|/[a-z]')\ndef l_split(s):\n return ' '.join(re.findall(PATTEN,s))\n\ninchi = 'InChI=1S/C7H2Cl3F3N4/c8-6(9,10)5-16-3(7(11,12)13)2(1-14)4(15)17-5/h(H2,15,16,17)'\nsplited_inchi = l_split(inchi[9:]) \n\nsplited_inchi \n# 'C 7 H 2 Cl 3 F 3 N 4 /c 8 - 6 ( 9 , 10 ) 5 - 16 - 3 ( 7 ( 11 , 12 ) 13 ) 2 ( 1 - 14 ) 4 ( 15 ) 17 - 5 /h ( H 2 , 15 , 16 , 17 )'\nI didn't noticed it⦠thanks for sharing!
", "rawMarkdown": "I didn't noticed it... thanks for sharing!", "votes": null }, { "id": "1315522", "postDate": "05/19/2021 21:32:33", "content": "A good unit-test for (reversible) tokenization is to de-tokenize the tokenized text again and check if it matches the original:
\nInChI -> tokenize -> save -> de-tokenize -> InchI == De-tokenized-InchI?
Comparable tests can be done for other common pre-processing steps.
", "rawMarkdown": "A good unit-test for (reversible) tokenization is to de-tokenize the tokenized text again and check if it matches the original:\nInChI -> tokenize -> save -> de-tokenize -> InchI == De-tokenized-InchI?\n\nComparable tests can be done for other common pre-processing steps.", "votes": null }, { "id": "1315573", "postDate": "05/20/2021 00:02:15", "content": "is there a fix for the bug?
", "rawMarkdown": "is there a fix for the bug?", "votes": null }, { "id": "1315642", "postDate": "05/20/2021 03:24:34", "content": "Maybe try the l_split function in the comment to split inchis?
\nI have not run the unit test as @cepheidq suggests yet, so I'm not one hundred percent sure.
Thank you for the great starter notebook. I learned a lot!π
", "rawMarkdown": "Thank you for the great starter notebook. I learned a lot!π", "votes": null }, { "id": "1316169", "postDate": "05/20/2021 10:58:11", "content": "@ee1yii thanks a lot for sharing this! I did a quick check of your code and it looks like it provides the correct segmentation for those 414 molecules. I also doubt that fixing the tokenizer has any tangible effect on the score, but it is always good to make sure all the labels are correct :)
", "rawMarkdown": "ee1yii thanks a lot for sharing this! I did a quick check of your code and it looks like it provides the correct segmentation for those 414 molecules. I also doubt that fixing the tokenizer has any tangible effect on the score, but it is always good to make sure all the labels are correct :)", "votes": null }, { "id": "1316257", "postDate": "05/20/2021 12:06:59", "content": "I am using the YNakama's tokenizer, but I will not fix the bug and regard this as noise :) Seems to hurt not much because I am able to get 0.74 with a single model.
", "rawMarkdown": "I am using the YNakama's tokenizer, but I will not fix the bug and regard this as noise :) Seems to hurt not much because I am able to get 0.74 with a single model.", "votes": null } ], "comments": [ { "id": 1314413, "author_name": "ee1yii", "author_url": "", "post_date": "05/19/2021 06:48:52", "content": "My code:
\nimport re\nPATTEN = re.compile('\\d+|[A-Z][a-z]?|[^A-Za-z\\d/]|/[a-z]')\ndef l_split(s):\n return ' '.join(re.findall(PATTEN,s))\n\ninchi = 'InChI=1S/C7H2Cl3F3N4/c8-6(9,10)5-16-3(7(11,12)13)2(1-14)4(15)17-5/h(H2,15,16,17)'\nsplited_inchi = l_split(inchi[9:]) \n\nsplited_inchi \n# 'C 7 H 2 Cl 3 F 3 N 4 /c 8 - 6 ( 9 , 10 ) 5 - 16 - 3 ( 7 ( 11 , 12 ) 13 ) 2 ( 1 - 14 ) 4 ( 15 ) 17 - 5 /h ( H 2 , 15 , 16 , 17 )'\n@ee1yii thanks a lot for sharing this! I did a quick check of your code and it looks like it provides the correct segmentation for those 414 molecules. I also doubt that fixing the tokenizer has any tangible effect on the score, but it is always good to make sure all the labels are correct :)
", "votes": null, "replies": [] } ] }, { "id": 1314927, "author_name": "yasufuminakama", "author_url": "", "post_date": "05/19/2021 13:08:21", "content": "I didn't noticed it⦠thanks for sharing!
", "votes": null, "replies": [ { "id": 1315644, "author_name": "ee1yii", "author_url": "", "post_date": "05/20/2021 03:27:35", "content": "Thank you for the great starter notebook. I learned a lot!π
", "votes": null, "replies": [] } ] }, { "id": 1315522, "author_name": "cepheidq", "author_url": "", "post_date": "05/19/2021 21:32:33", "content": "A good unit-test for (reversible) tokenization is to de-tokenize the tokenized text again and check if it matches the original:
\nInChI -> tokenize -> save -> de-tokenize -> InchI == De-tokenized-InchI?
Comparable tests can be done for other common pre-processing steps.
", "votes": null, "replies": [] }, { "id": 1315573, "author_name": "hengck23", "author_url": "", "post_date": "05/20/2021 00:02:15", "content": "is there a fix for the bug?
", "votes": null, "replies": [ { "id": 1315642, "author_name": "ee1yii", "author_url": "", "post_date": "05/20/2021 03:24:34", "content": "Maybe try the l_split function in the comment to split inchis?
\nI have not run the unit test as @cepheidq suggests yet, so I'm not one hundred percent sure.
I am using the YNakama's tokenizer, but I will not fix the bug and regard this as noise :) Seems to hurt not much because I am able to get 0.74 with a single model.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1314411": "I believe many (like us) started with YNakama's great notebook and used the tokenizer in it. So I guess it may be worth share that there exists a bug in the word segmentation part. \n\nFor example, the training sample `002f57184f1f` has InChI `InChI=1S/C7H2Cl3F3N4/c8-6(9,10)5-16-3(7(11,12)13)2(1-14)4(15)17-5/h(H2,15,16,17)` and its word segmentation result should be\n```\nC 7 H 2 Cl 3 F 3 N 4 /c 8 - 6 ( 9 , 10 ) 5 - 16 - 3 ( 7 ( 11 , 12 ) 13 ) 2 ( 1 - 14 ) 4 ( 15 ) 17 - 5 /h ( H 2 , 15 , 16 , 17 )\n```\nwhile it actually is\n```\nC 7 H 2 Cl 3 F 3 N 4 /c 8 - 6 ( 9 , 10 ) 5 - 16 - 3 ( 7 ( 11 , 12 ) 13 ) 2 ( 1 - 14 ) 4 ( 15 ) 17 - 5 /h 2 , 15 , 16 , 17 )\n```\n.\n\nBut I doubt this bug would have any real influences beacause it only affects 414 training samples and results in 790 Levenshtein distance in total. :P", "1314413": "My code:\n```ptyhon\nimport re\nPATTEN = re.compile('\\d+|[A-Z][a-z]?|[^A-Za-z\\d/]|/[a-z]')\ndef l_split(s):\n return ' '.join(re.findall(PATTEN,s))\n\ninchi = 'InChI=1S/C7H2Cl3F3N4/c8-6(9,10)5-16-3(7(11,12)13)2(1-14)4(15)17-5/h(H2,15,16,17)'\nsplited_inchi = l_split(inchi[9:]) \n\nsplited_inchi \n# 'C 7 H 2 Cl 3 F 3 N 4 /c 8 - 6 ( 9 , 10 ) 5 - 16 - 3 ( 7 ( 11 , 12 ) 13 ) 2 ( 1 - 14 ) 4 ( 15 ) 17 - 5 /h ( H 2 , 15 , 16 , 17 )'\n```", "1314927": "I didn't noticed it... thanks for sharing!", "1315522": "A good unit-test for (reversible) tokenization is to de-tokenize the tokenized text again and check if it matches the original:\nInChI -> tokenize -> save -> de-tokenize -> InchI == De-tokenized-InchI?\n\nComparable tests can be done for other common pre-processing steps.", "1315573": "is there a fix for the bug?", "1315642": "Maybe try the `l_split` function in the comment to split inchis? \nI have not run the unit test as @cepheidq suggests yet, so I'm not one hundred percent sure.", "1315644": "Thank you for the great starter notebook. I learned a lot!π", "1316169": "ee1yii thanks a lot for sharing this! I did a quick check of your code and it looks like it provides the correct segmentation for those 414 molecules. I also doubt that fixing the tokenizer has any tangible effect on the score, but it is always good to make sure all the labels are correct :)", "1316257": "I am using the YNakama's tokenizer, but I will not fix the bug and regard this as noise :) Seems to hurt not much because I am able to get 0.74 with a single model." }, "source": "meta" }