\n
![\"\"](\"https://pbs.twimg.com/media/Ev8bem6UcAQ2e45?format=png&name=medium\")
Scientifically, the Levenshtein Distance between targeted InChI string and predicted InChI string doesn't reflect the real chemical distance. One can find many \"one-pixel attack\" examples where small tweak on structure changes the entire InChI string. Another critical issue is the validity of output: not any InChI is syntactically meaningful. What's the point of predicting an invalid InChI even if it's highly similar to the real string?
\nThe proper metric is: 1) % exact InChI matches; or 2) mean chemical similarity or 3) Levenshtein distance for valid InChI (or whatever identifier)
\nBelow shows some interesting examples (aka, hacks):
\nExample 1
\n
Example 2
\n
\n", "messages": [ { "id": "1230601", "postDate": "03/08/2021 08:57:00", "content": "UPDATE: I'm less worried about the bias from direct InChI comparison, as the leaderboard is approaching LD ~ 10 with ~65% perfect predictions (aka, LD = 0).
\n
Scientifically, the Levenshtein Distance between targeted InChI string and predicted InChI string doesn't reflect the real chemical distance. One can find many \"one-pixel attack\" examples where small tweak on structure changes the entire InChI string. Another critical issue is the validity of output: not any InChI is syntactically meaningful. What's the point of predicting an invalid InChI even if it's highly similar to the real string?
\nThe proper metric is: 1) % exact InChI matches; or 2) mean chemical similarity or 3) Levenshtein distance for valid InChI (or whatever identifier)
\nBelow shows some interesting examples (aka, hacks):
\nExample 1
\n
Example 2
\n
\n", "rawMarkdown": "Scientifically, the *Levenshtein Distance* between targeted InChI string and predicted InChI string doesn't reflect the real chemical distance. One can find many \"one-pixel attack\" examples where small tweak on structure changes the entire InChI string. Another critical issue is the validity of output: not any InChI is syntactically meaningful. What's the point of predicting an *invalid* InChI even if it's highly similar to the real string?\n\nThe proper metric is: 1) % exact InChI matches; or 2) mean chemical similarity or 3) Levenshtein distance for valid InChI (or whatever identifier)\n\nBelow shows some interesting examples (aka, hacks): \n\nExample 1\n\n\nExample 2\n\n\n> UPDATE: I'm less worried about the bias from direct InChI comparison, as the leaderboard is approaching LD ~ 10 with ~65% perfect predictions (aka, LD = 0).", "votes": null }, { "id": "1230953", "postDate": "03/08/2021 15:05:42", "content": "UPDATE: I'm less worried about the bias from direct InChI comparison, as the leaderboard is approaching LD ~ 10 with ~65% perfect predictions (aka, LD = 0).
\n
Great analysis! I’ll have to take a deeper look at this soon
", "rawMarkdown": "Great analysis! I’ll have to take a deeper look at this soon", "votes": null }, { "id": "1231069", "postDate": "03/08/2021 16:39:00", "content": "@jakealbrecht1337 kindly address the concern raised by @houndcl regarding the evaluation metric
", "rawMarkdown": "jakealbrecht1337 kindly address the concern raised by @houndcl regarding the evaluation metric", "votes": null }, { "id": "1231178", "postDate": "03/08/2021 18:19:40", "content": "I don't think that the outputs are going to be used for purposes where predictions must be valid InChIs.
\nI can think of use cases like searching the most similar substances to a query string amongst millions of predictions to images. Then you can look at the papers where the most similar ones came from.
Thank you for your question! Yes, the choice of a metric was a challenge. The metric of fraction exactly matched was initially proposed, but we wanted to give more partial credit as in your examples. Other metrics like Tanimoto (Jaccard) similarity or checks for valid InChIs were considered but would have been too demanding for automatic scoring. Because the internal structure of InChIs makes them a poor target for direct prediction, ideally the winning submission would convert them to another format (e.g. deepsmiles) or molecular graphs prior to training, with the predictions converted back to InChI. For these reasons, the risk of hacking the leaderboard was deemed low and a simple distance metric was used.
Thank you for the comprehensive response! :)
", "rawMarkdown": "Thank you for the comprehensive response! :)", "votes": null }, { "id": "1232604", "postDate": "03/09/2021 21:16:33", "content": "Thanks for the thorough response! With the best model pushing LD closer to 10, most are probably perfect predictions. @drhabib has confirmed this in his thread (65% LD = 0). This is amazing. I originally thought that the performance wouldn't be this good (e.g., LD ~ 40), which are more likely to suffer the mentioned bias from direct InChI comparison.
", "rawMarkdown": "Thanks for the thorough response! With the best model pushing LD closer to 10, most are probably perfect predictions. @drhabib has confirmed this in his thread (65% LD = 0). This is amazing. I originally thought that the performance wouldn't be this good (e.g., LD ~ 40), which are more likely to suffer the mentioned bias from direct InChI comparison.", "votes": null }, { "id": "1235130", "postDate": "03/11/2021 21:19:35", "content": "I've thought about this and there are some implications for predicting the InChI directly vs using a format like SMILES and converting. With the conversion approach, you can have your model predict a shorter, simpler sequence and then convert over, which feels easier. But this relies on having the ability to convert. If you use SMILES for example, you can have a single character syntax error that makes the entire string invalid, you lose the whole sample because you can't convert back to an InChI.
\nPredicting InChI strings directly is more complicated (longer sequence to predict) but you're much more likely to get partially correct answers, which might perform better with the current metric.
", "rawMarkdown": "I've thought about this and there are some implications for predicting the InChI directly vs using a format like SMILES and converting. With the conversion approach, you can have your model predict a shorter, simpler sequence and then convert over, which feels easier. But this relies on having the ability to convert. If you use SMILES for example, you can have a single character syntax error that makes the entire string invalid, you lose the whole sample because you can't convert back to an InChI.\n\nPredicting InChI strings directly is more complicated (longer sequence to predict) but you're much more likely to get partially correct answers, which might perform better with the current metric.", "votes": null }, { "id": "1235346", "postDate": "03/12/2021 04:32:43", "content": "The layered structure of the InChI itself makes it amenable to some levels of fuzzy matching. For example a comparison could be done solely between the main layer (https://www.inchi-trust.org/technical-faq-2/#4.3) by string splitting.
\nTechnically you could get even fuzzier and consider just the molecular formula (i.e. have the labels in the depiction been succesfully identified), albeit I expect you'd need to ignore hydrogen count, as if the connection/hydrogen layers are wrong this count is likely also wrong.
From a cheminformatics viewpoint, ideally the dataset should have both SMILES and InChI, even if the InChI is the evaluation target. InChI, by design, considers difference resonance forms or tautomers to have the same InChI, but this distinction is made in chemical depictions (and also SMILES).
\ni.e. conversion from InChI to SMILES is a lossy conversion as there are multiple canonical SMILES for a single InChI.
In practice the depictions appear to have been generated from the deterministically chosen structure you get if you convert an InChI to structure, so this is somewhat of a moot issue. One visible quirk of this is that the tautomer chosen for amides is not the typical one, hence you have C(O)=N instead of C(=O)N
", "rawMarkdown": "The layered structure of the InChI itself makes it amenable to some levels of fuzzy matching. For example a comparison could be done solely between the main layer (https://www.inchi-trust.org/technical-faq-2/#4.3) by string splitting.\nTechnically you could get even fuzzier and consider just the molecular formula (i.e. have the labels in the depiction been succesfully identified), albeit I expect you'd need to ignore hydrogen count, as if the connection/hydrogen layers are wrong this count is likely also wrong.\n\nFrom a cheminformatics viewpoint, ideally the dataset should have both SMILES and InChI, even if the InChI is the evaluation target. InChI, by design, considers difference resonance forms or tautomers to have the same InChI, but this distinction is made in chemical depictions (and also SMILES). \ni.e. conversion from InChI to SMILES is a lossy conversion as there are multiple **canonical** SMILES for a single InChI.\n\nIn practice the depictions appear to have been generated from the deterministically chosen structure you get if you convert an InChI to structure, so this is somewhat of a moot issue. One visible quirk of this is that the tautomer chosen for amides is not the typical one, hence you have C(O)=N instead of C(=O)N", "votes": null }, { "id": "1235606", "postDate": "03/12/2021 09:45:11", "content": "indeed InChi to canonical SMILES is a lossy compression, but in the train dataset you get >99.99% valid InChi's when converting to SMILES and back. This discrepancies might become relevant when squeezing the last bits of performance towards the end of the competition
", "rawMarkdown": "indeed InChi to canonical SMILES is a lossy compression, but in the train dataset you get >99.99% valid InChi's when converting to SMILES and back. This discrepancies might become relevant when squeezing the last bits of performance towards the end of the competition", "votes": null }, { "id": "1236214", "postDate": "03/12/2021 22:15:14", "content": "Good discussion on the challenges of predicting invalid chemical strings. I'm curious: have teams explored representations of the molecules based on generative models e.g. MolGAN ?
", "rawMarkdown": "Good discussion on the challenges of predicting invalid chemical strings. I'm curious: have teams explored representations of the molecules based on generative models e.g. [MolGAN](https://deepchem.readthedocs.io/en/latest/api_reference/featurizers.html#molganfeaturizer) ?", "votes": null }, { "id": "1238230", "postDate": "03/14/2021 19:04:22", "content": "I published a baseline notebook to show that we can get LB 71.1 by always predicting the same InChI :
\nhttps://www.kaggle.com/datanewb/always-predict-same-inchi/
\nI found this \"central\" InChI with a method similar to https://rawgit.com/ztane/python-Levenshtein/master/docs/Levenshtein.html#Levenshtein-setmedian
\nThis set-median InChI is from the training InChIs but we could do even better than 71.1 with a median invalid InChI.
\nBut finding a median string among a set of strings is an NP-complete problem :
\n\"The generalized median is the more general concept and therefore provides a better representation of the set than the set median, but de la Higuera and Casacuberta [3] proved that the computation of the former is NP-complete, in other words very hard to compute.\" (c.f. http://www.ok.sc.e.titech.ac.jp/res/PCS/courses/comp644/)
\nThis is just to provide a constant prediction baseline and with top LB scores already below 3 it is impossible to \"hack\" the leaderboard with a median InChI.
I do, i'm very instered on using latent spaces from graph autoencoders. I'm not sure if it would be the right approach but i'm going to learn a lot doing it.
", "rawMarkdown": "I do, i'm very instered on using **latent spaces** from **graph autoencoders**. I'm not sure if it would be the right approach but i'm going to learn a lot doing it.", "votes": null }, { "id": "1244270", "postDate": "03/18/2021 21:13:16", "content": "@jakealbrecht1337 \"the winning submission would convert them to another format (e.g. deepsmiles) or molecular graphs prior to training, with the predictions converted back to InChI\"
\nI would like to do as you say i.e. predict some intermediate representation for the molecular graph but the big issue is that we then miss a clear algorithm to produce a unique InChI from any molecular graph.
@datanewb Do you mean something like https://www.rdkit.org/docs/source/rdkit.Chem.inchi.html#rdkit.Chem.inchi.MolToInchi ?
\nInChIs are produced using a specific algorithm, that for a given molecular graph (and variants of the graph that are chemically equivalent) produces the same InChI. Partially due to the complexity of the algorithm, and partially to guarantee that the same algorithm is used, there is only one implementation. This library is then bundled with most of the major chemistry toolkits e.g. RDKit, Epam's Indigo, Chemistry Development Kit (CDK), and hence for a given molecular graph any of these will give the same InChI
\nSELFIES (https://github.com/aspuru-guzik-group/selfies) are also worth considering as the intermediate representation.
", "rawMarkdown": "datanewb Do you mean something like https://www.rdkit.org/docs/source/rdkit.Chem.inchi.html#rdkit.Chem.inchi.MolToInchi ?\n\nInChIs are produced using a specific algorithm, that for a given molecular graph (and variants of the graph that are chemically equivalent) produces the same InChI. Partially due to the complexity of the algorithm, and partially to guarantee that the same algorithm is used, there is only one implementation. This library is then bundled with most of the major chemistry toolkits e.g. RDKit, Epam's Indigo, Chemistry Development Kit (CDK), and hence for a given molecular graph any of these will give the same InChI\n\nSELFIES (https://github.com/aspuru-guzik-group/selfies) are also worth considering as the intermediate representation.", "votes": null } ], "comments": [ { "id": 1230953, "author_name": "matthewmasters", "author_url": "", "post_date": "03/08/2021 15:05:42", "content": "Great analysis! I’ll have to take a deeper look at this soon
", "votes": null, "replies": [] }, { "id": 1231069, "author_name": "usharengaraju", "author_url": "", "post_date": "03/08/2021 16:39:00", "content": "@jakealbrecht1337 kindly address the concern raised by @houndcl regarding the evaluation metric
", "votes": null, "replies": [] }, { "id": 1231178, "author_name": "nofreewill", "author_url": "", "post_date": "03/08/2021 18:19:40", "content": "I don't think that the outputs are going to be used for purposes where predictions must be valid InChIs.
\nI can think of use cases like searching the most similar substances to a query string amongst millions of predictions to images. Then you can look at the papers where the most similar ones came from.
Thank you for your question! Yes, the choice of a metric was a challenge. The metric of fraction exactly matched was initially proposed, but we wanted to give more partial credit as in your examples. Other metrics like Tanimoto (Jaccard) similarity or checks for valid InChIs were considered but would have been too demanding for automatic scoring. Because the internal structure of InChIs makes them a poor target for direct prediction, ideally the winning submission would convert them to another format (e.g. deepsmiles) or molecular graphs prior to training, with the predictions converted back to InChI. For these reasons, the risk of hacking the leaderboard was deemed low and a simple distance metric was used.
Thank you for the comprehensive response! :)
", "votes": null, "replies": [] }, { "id": 1232604, "author_name": "houndcl", "author_url": "", "post_date": "03/09/2021 21:16:33", "content": "Thanks for the thorough response! With the best model pushing LD closer to 10, most are probably perfect predictions. @drhabib has confirmed this in his thread (65% LD = 0). This is amazing. I originally thought that the performance wouldn't be this good (e.g., LD ~ 40), which are more likely to suffer the mentioned bias from direct InChI comparison.
", "votes": null, "replies": [] }, { "id": 1235130, "author_name": "towardsentropy", "author_url": "", "post_date": "03/11/2021 21:19:35", "content": "I've thought about this and there are some implications for predicting the InChI directly vs using a format like SMILES and converting. With the conversion approach, you can have your model predict a shorter, simpler sequence and then convert over, which feels easier. But this relies on having the ability to convert. If you use SMILES for example, you can have a single character syntax error that makes the entire string invalid, you lose the whole sample because you can't convert back to an InChI.
\nPredicting InChI strings directly is more complicated (longer sequence to predict) but you're much more likely to get partially correct answers, which might perform better with the current metric.
", "votes": null, "replies": [] }, { "id": 1235346, "author_name": "infy2097", "author_url": "", "post_date": "03/12/2021 04:32:43", "content": "The layered structure of the InChI itself makes it amenable to some levels of fuzzy matching. For example a comparison could be done solely between the main layer (https://www.inchi-trust.org/technical-faq-2/#4.3) by string splitting.
\nTechnically you could get even fuzzier and consider just the molecular formula (i.e. have the labels in the depiction been succesfully identified), albeit I expect you'd need to ignore hydrogen count, as if the connection/hydrogen layers are wrong this count is likely also wrong.
From a cheminformatics viewpoint, ideally the dataset should have both SMILES and InChI, even if the InChI is the evaluation target. InChI, by design, considers difference resonance forms or tautomers to have the same InChI, but this distinction is made in chemical depictions (and also SMILES).
\ni.e. conversion from InChI to SMILES is a lossy conversion as there are multiple canonical SMILES for a single InChI.
In practice the depictions appear to have been generated from the deterministically chosen structure you get if you convert an InChI to structure, so this is somewhat of a moot issue. One visible quirk of this is that the tautomer chosen for amides is not the typical one, hence you have C(O)=N instead of C(=O)N
", "votes": null, "replies": [] }, { "id": 1235606, "author_name": "bulatz", "author_url": "", "post_date": "03/12/2021 09:45:11", "content": "indeed InChi to canonical SMILES is a lossy compression, but in the train dataset you get >99.99% valid InChi's when converting to SMILES and back. This discrepancies might become relevant when squeezing the last bits of performance towards the end of the competition
", "votes": null, "replies": [] }, { "id": 1236214, "author_name": "jakealbrecht1337", "author_url": "", "post_date": "03/12/2021 22:15:14", "content": "Good discussion on the challenges of predicting invalid chemical strings. I'm curious: have teams explored representations of the molecules based on generative models e.g. MolGAN ?
", "votes": null, "replies": [] }, { "id": 1238483, "author_name": "hiramcho", "author_url": "", "post_date": "03/15/2021 03:03:15", "content": "I do, i'm very instered on using latent spaces from graph autoencoders. I'm not sure if it would be the right approach but i'm going to learn a lot doing it.
", "votes": null, "replies": [] }, { "id": 1244270, "author_name": "datanewb", "author_url": "", "post_date": "03/18/2021 21:13:16", "content": "@jakealbrecht1337 \"the winning submission would convert them to another format (e.g. deepsmiles) or molecular graphs prior to training, with the predictions converted back to InChI\"
\nI would like to do as you say i.e. predict some intermediate representation for the molecular graph but the big issue is that we then miss a clear algorithm to produce a unique InChI from any molecular graph.
@datanewb Do you mean something like https://www.rdkit.org/docs/source/rdkit.Chem.inchi.html#rdkit.Chem.inchi.MolToInchi ?
\nInChIs are produced using a specific algorithm, that for a given molecular graph (and variants of the graph that are chemically equivalent) produces the same InChI. Partially due to the complexity of the algorithm, and partially to guarantee that the same algorithm is used, there is only one implementation. This library is then bundled with most of the major chemistry toolkits e.g. RDKit, Epam's Indigo, Chemistry Development Kit (CDK), and hence for a given molecular graph any of these will give the same InChI
\nSELFIES (https://github.com/aspuru-guzik-group/selfies) are also worth considering as the intermediate representation.
", "votes": null, "replies": [] } ] }, { "id": 1238230, "author_name": "datanewb", "author_url": "", "post_date": "03/14/2021 19:04:22", "content": "I published a baseline notebook to show that we can get LB 71.1 by always predicting the same InChI :
\nhttps://www.kaggle.com/datanewb/always-predict-same-inchi/
\nI found this \"central\" InChI with a method similar to https://rawgit.com/ztane/python-Levenshtein/master/docs/Levenshtein.html#Levenshtein-setmedian
\nThis set-median InChI is from the training InChIs but we could do even better than 71.1 with a median invalid InChI.
\nBut finding a median string among a set of strings is an NP-complete problem :
\n\"The generalized median is the more general concept and therefore provides a better representation of the set than the set median, but de la Higuera and Casacuberta [3] proved that the computation of the former is NP-complete, in other words very hard to compute.\" (c.f. http://www.ok.sc.e.titech.ac.jp/res/PCS/courses/comp644/)
\nThis is just to provide a constant prediction baseline and with top LB scores already below 3 it is impossible to \"hack\" the leaderboard with a median InChI.
![\"\"](\"https://pbs.twimg.com/media/Ev2XsfMVEAMDdfs?format=jpg&name=medium\")