If you have multiple submissions, just modify the normalization script so that you note what happened with each InChI (valid, modified, mol_is_none, exception, segfault) and choose valid/modified ones based on some strategy.
\nYou can also mine the molecules for which you have no valid and/or modified predictions and predict only those with more resources, i.e. with more beams.
\nI don't know how I missed this so far… it's all there throughout the discussions
\n@hengck23 thank you for opening my eyes!
If you have multiple submissions, just modify the normalization script so that you note what happened with each InChI (valid, modified, mol_is_none, exception, segfault) and choose valid/modified ones based on some strategy.
\nYou can also mine the molecules for which you have no valid and/or modified predictions and predict only those with more resources, i.e. with more beams.
\nI don't know how I missed this so far… it's all there throughout the discussions
\n@hengck23 thank you for opening my eyes!
to be exact \"validation of inchi is more crucial\"
\nrdkit validation doesn't means correct (although it is mostly correct). If you assume valid = correct, then :
\nyour score is upper bounded by error of prediction that is valid.
\nyou may need to train a score predictor at the end of your decoder, or sum up the prob of each element of the seq.
\ngiven a set of trained models, you are using heuristics to determine if the prediction is correct., e.g. valid inchi of rdkit. train a model to predict the LD score (given multiple predictions from different models and rdkit) may be better
", "rawMarkdown": "to be exact \"validation of inchi ~~rdkit~~ is more crucial\"\n\nrdkit validation doesn't means correct (although it is mostly correct). If you assume valid = correct, then :\n\nyour score is upper bounded by error of prediction that is valid. \n\n---\n\nyou may need to train a score predictor at the end of your decoder, or sum up the prob of each element of the seq.\n\ngiven a set of trained models, you are using heuristics to determine if the prediction is correct., e.g. valid inchi of rdkit. train a model to predict the LD score (given multiple predictions from different models and rdkit) may be better", "votes": 5, "replies": [ { "id": 3140688, "postDate": "2025-03-04T20:26:45.493Z", "content": "\"rdkit validation doesn't means correct\" - I haven't realised back then that this was also a low hanging fruit I haven't touched.
\nThere were so many possibilities for small improvements here-and-there.
Also, to this day I have flashbacks that I basically robbed someone from their solo gold and with that to get the kaggle comp gm title.
", "rawMarkdown": "\"rdkit validation doesn't means correct\" - I haven't realised back then that this was also a low hanging fruit I haven't touched.\nThere were so many possibilities for small improvements here-and-there.\n\nAlso, to this day I have flashbacks that I basically robbed someone from their solo gold and with that to get the kaggle comp gm title." } ] }, { "id": 1332519, "postDate": "2021-06-02T06:53:15.360Z", "content": "Thanks. It did help me to decrease LB from 1.08 to 1.03.
", "rawMarkdown": "Thanks. It did help me to decrease LB from 1.08 to 1.03. ", "votes": 1, "replies": [ { "id": 1332566, "postDate": "2021-06-02T07:30:11.483Z", "content": "Glad to hear that, thank you for letting me know! :)
", "rawMarkdown": "Glad to hear that, thank you for letting me know! :)", "votes": -1 }, { "id": 1332574, "postDate": "2021-06-02T07:37:50.063Z", "content": "We had an improvement (maybe a big) with this too before. not sure sure about it now. But you are idea is good and amazing. and also crucial
", "rawMarkdown": "We had an improvement (maybe a big) with this too before. not sure sure about it now. But you are idea is good and amazing. and also crucial", "votes": 1 }, { "id": 1332876, "postDate": "2021-06-02T11:07:49.210Z", "content": "It's not my idea, there are a couple mentions of it in the discussions. I just made it more available.
", "rawMarkdown": "It's not my idea, there are a couple mentions of it in the discussions. I just made it more available.", "votes": -1 } ] }, { "id": 1332477, "postDate": "2021-06-02T06:29:00.053Z", "content": "Thanks for sharing @nofreewill
", "rawMarkdown": "Thanks for sharing @nofreewill", "votes": 1, "replies": [ { "id": 1332490, "postDate": "2021-06-02T06:37:17.200Z", "content": "Did it help you? You're welcome! :))
", "rawMarkdown": "Did it help you? You're welcome! :))", "votes": -1 }, { "id": 1332506, "postDate": "2021-06-02T06:44:34.610Z", "content": "Not yet, still trying our best to figure out how to use it , trying to decipher your words haha
", "rawMarkdown": "Not yet, still trying our best to figure out how to use it , trying to decipher your words haha" }, { "id": 1332564, "postDate": "2021-06-02T07:27:33.660Z", "content": "In my validation:
\nThen
\nThen
\nBut now I'm stuck here, I think that my models are just not good enough (as all of them have CV higher than 1 LD)
\nI may be able to work out some sophisticated strategies on how to choose between all the predictions if there are multiple ones being the same, but sadly I just have no time for that despite the extension.
@nofreewill its humble of you to share but you should not have done it before the deadline , this might be unfair to people who had figured this out on their own . This could prove vital for a win and would have required good amount of digging
", "rawMarkdown": "@nofreewill its humble of you to share but you should not have done it before the deadline , this might be unfair to people who had figured this out on their own . This could prove vital for a win and would have required good amount of digging ", "votes": 6 }, { "id": 1332761, "postDate": "2021-06-02T09:27:03.553Z", "content": "That's a good amount of downvotes.
\nWhat I said is all out in the discussions, so I don't agree with that hate.
There are also a lot of nuances that needs a lot of work.
\nI.e. how does beam size effect LD of shorter and longer inchis? How do you select your prediction if there are more inchis that are valid, and how do you take into account the beam size effect? How do you take into account if your models are trained on different aspect ratios? And many other things that needs a lot of work to optimize.
So I guess that the downvotes come from teams that enjoyed that this idea is less available and didn't want to work harder on these issues.
", "rawMarkdown": "So I guess that the downvotes come from teams that enjoyed that this idea is less available and didn't want to work harder on these issues.", "votes": 2 }, { "id": 1333385, "postDate": "2021-06-02T17:14:21.663Z", "content": "don't worry that can be normalised with upvotes ;)
", "rawMarkdown": "don't worry that can be normalised with upvotes ;)", "votes": 2 }, { "id": 1333397, "postDate": "2021-06-02T17:29:39.847Z", "content": "haha, thank you! :D I'm keeping an eye on the number of votes, it's a roller coaster.. :D
\nI get why one might be angry, but really.. this is such a simple idea that it would be a shame if 3 month of one's hard work would go into garbage because he missed it but others did not.
I appreciate your humble sharing in the competition and It only worked in my favour that you shared exactly what you did and saved us time .
\nI just said on a general note that it might hurt people , there is a reason why all the gms and top teams don't share their secret sauce right untill the end of competition no matter how easy it is 😉
Just to put things into perpective , the first competition I participated in people found a very easy post-processing like this that could be utilized but no one shared untill the end of the competition
", "rawMarkdown": "I appreciate your humble sharing in the competition and It only worked in my favour that you shared exactly what you did and saved us time .\nI just said on a general note that it might hurt people , there is a reason why all the gms and top teams don't share their secret sauce right untill the end of competition no matter how easy it is 😉\n\nJust to put things into perpective , the first competition I participated in people found a very easy post-processing like this that could be utilized but no one shared untill the end of the competition", "votes": 5 }, { "id": 1333481, "postDate": "2021-06-02T18:54:46.103Z", "content": "You are right @tanulsingh077, the teams that are at the top which has already done it and found it useful to improve their score might think as a spoiler alert and downvotes it.
", "rawMarkdown": "You are right @tanulsingh077, the teams that are at the top which has already done it and found it useful to improve their score might think as a spoiler alert and downvotes it. " }, { "id": 1333499, "postDate": "2021-06-02T19:02:23.203Z", "content": "Don't worry. It is just a sensitive moment. Eventually, people only remember your method and generosity.
", "rawMarkdown": "Don't worry. It is just a sensitive moment. Eventually, people only remember your method and generosity.", "votes": 1 }, { "id": 1333501, "postDate": "2021-06-02T19:12:49.517Z", "content": "@ybwu01 wow, thanks for your kind words! :)
\nIt's a little funny how by downvoting a thing you know that works you actually confirm that it does work and encourage others to go onto that path.
This happens all the time and I believe someone summed it up best in another competition as \"people already using the trick will be unhappy while people new to it will be happy\".
\nI personally think it's Kaggle etiquette not to share anything that might have a material impact on the LB late in the competition - the caution on high-scoring notebooks is another facet of it.
", "rawMarkdown": "This happens all the time and I believe someone summed it up best in another competition as \"people already using the trick will be unhappy while people new to it will be happy\".\n\nI personally think it's Kaggle etiquette not to share anything that might have a material impact on the LB late in the competition - the caution on high-scoring notebooks is another facet of it.", "votes": 4 }, { "id": 1334141, "postDate": "2021-06-03T09:41:52.520Z", "content": "With the recent changes on the leaderboard we at least know who already used the trick and who didn't.
\nQuite surprising for me to see so interrupting changes. But we will see if carries to the private board.
I myself have decided to not upload a submission at all. I have done a lot of experiments to reproduce some papers and trying out stuff on myself. It was fun and I learned a lot. Meanwhile I never went for the highest-most score, just an efficient model (more on the encoder side). In the end I run out of time to train for long enough to be competitive (enough for my taste).
\nI think I will make a post after the submission end, detailing my most interesting findings.
\nEdit: Cheers to @charmq for keeping up with the teams in the gold-region so far.
", "rawMarkdown": "With the recent changes on the leaderboard we at least know who already used the trick and who didn't.\nQuite surprising for me to see so interrupting changes. But we will see if carries to the private board.\n\nI myself have decided to not upload a submission at all. I have done a lot of experiments to reproduce some papers and trying out stuff on myself. It was fun and I learned a lot. Meanwhile I never went for the highest-most score, just an efficient model (more on the encoder side). In the end I run out of time to train for long enough to be competitive (enough for my taste).\n\nI think I will make a post after the submission end, detailing my most interesting findings.\n\nEdit: Cheers to @charmq for keeping up with the teams in the gold-region so far.", "votes": 2 }, { "id": 1334958, "postDate": "2021-06-03T23:37:02.507Z", "content": "I have to wonder how much of the shake-up in the gold zone was driven by this reveal (or maybe not at all).
\nI would not be happy if I was in one of the teams being punished…
", "rawMarkdown": "I have to wonder how much of the shake-up in the gold zone was driven by this reveal (or maybe not at all).\n\nI would not be happy if I was in one of the teams being punished... ", "votes": 1 } ] }, { "id": 1331111, "postDate": "2021-06-01T09:26:30.843Z", "content": "Wow! This idea is very very important! Thanks a lot for sharing!
", "rawMarkdown": "Wow! This idea is very very important! Thanks a lot for sharing! ", "votes": 1, "replies": [ { "id": 1331120, "postDate": "2021-06-01T09:37:33.890Z", "content": "It was worth sharing, then! :) You're welcome!
", "rawMarkdown": "It was worth sharing, then! :) You're welcome!" } ] }, { "id": 1330928, "postDate": "2021-06-01T07:09:02.070Z", "content": "Well, then I hope that is enough for you to achieve the solo gold you wanted :)
", "rawMarkdown": "Well, then I hope that is enough for you to achieve the solo gold you wanted :)", "votes": 1, "replies": [ { "id": 1330936, "postDate": "2021-06-01T07:14:44.117Z", "content": "Thank you! :)
\nI still have doubts, but now I have a little hope, at least.
But I guess that a lot of teams left their submissions to the end, so… :/ :D
", "rawMarkdown": "Thank you! :)\nI still have doubts, but now I have a little hope, at least.\n\nBut I guess that a lot of teams left their submissions to the end, so... :/ :D", "votes": 1 }, { "id": 1331031, "postDate": "2021-06-01T08:27:02.570Z", "content": "i estimate gold level should be better 0.65~0.62
", "rawMarkdown": "i estimate gold level should be better 0.65~0.62", "votes": 1 }, { "id": 1331136, "postDate": "2021-06-01T09:47:54.717Z", "content": "@nofreewill i think you were hiding your score for a month. I think you have a better model and will win gold I think.
\nI estimate gold will be <0.62
", "rawMarkdown": "@nofreewill i think you were hiding your score for a month. I think you have a better model and will win gold I think.\n\nI estimate gold will be <0.62", "votes": 1 }, { "id": 1331142, "postDate": "2021-06-01T09:53:10.190Z", "content": "I also think gold will be less than 0.62
\nBut sadly, no, my best score I can achieve is 0.70 right now..
\nI don't think I'll have gold, but I'm trying my best :)
OT: Has here anybody enough experience to confidently say, if the public leader board compared to the private + public (complete) leaderboard may either be:
\nIn other words: Is the public test-set usually representative of the private one or not? Or does this strictly vary with the competition?
", "rawMarkdown": "OT: Has here anybody enough experience to confidently say, if the public leader board compared to the private + public (complete) leaderboard may either be:\n\n1. A random sub-sample\n2. A stratified sub-sample\n3. A purposefully easier sub-sample.\n\nIn other words: Is the public test-set usually representative of the private one or not? Or does this strictly vary with the competition?" }, { "id": 1331607, "postDate": "2021-06-01T15:17:26.760Z", "content": "Good Question.. Ideally, it should be a stratified sub sample but based on my bast experience most of the times it's random. That's why shakeup happens
", "rawMarkdown": "Good Question.. Ideally, it should be a stratified sub sample but based on my bast experience most of the times it's random. That's why shakeup happens" }, { "id": 1331677, "postDate": "2021-06-01T16:15:17.263Z", "content": "@cepheidq sampling of the private set from whole test set can any of your mention and it varies with each competition.
\nIn this competition it might be random sampled or length wise sampling as @nofreewill did for his CV.
\npurposeful maybe but lower probability.
to be exact \"validation of inchi is more crucial\"
\nrdkit validation doesn't means correct (although it is mostly correct). If you assume valid = correct, then :
\nyour score is upper bounded by error of prediction that is valid.
\nyou may need to train a score predictor at the end of your decoder, or sum up the prob of each element of the seq.
\ngiven a set of trained models, you are using heuristics to determine if the prediction is correct., e.g. valid inchi of rdkit. train a model to predict the LD score (given multiple predictions from different models and rdkit) may be better
", "votes": 5, "replies": [ { "id": 3140688, "author_name": "nofreewill42", "author_url": "", "post_date": "2025-03-04T20:26:45.493000", "content": "\"rdkit validation doesn't means correct\" - I haven't realised back then that this was also a low hanging fruit I haven't touched.
\nThere were so many possibilities for small improvements here-and-there.
Also, to this day I have flashbacks that I basically robbed someone from their solo gold and with that to get the kaggle comp gm title.
", "votes": 0, "replies": [] } ] }, { "id": 1332519, "author_name": "Yibing Wu", "author_url": "", "post_date": "2021-06-02T06:53:15.360000", "content": "Thanks. It did help me to decrease LB from 1.08 to 1.03.
", "votes": 1, "replies": [ { "id": 1332566, "author_name": "nofreewill42", "author_url": "", "post_date": "2021-06-02T07:30:11.483000", "content": "Glad to hear that, thank you for letting me know! :)
", "votes": -1, "replies": [] }, { "id": 1332574, "author_name": "Mohammed Rizin V K", "author_url": "", "post_date": "2021-06-02T07:37:50.063000", "content": "We had an improvement (maybe a big) with this too before. not sure sure about it now. But you are idea is good and amazing. and also crucial
", "votes": 1, "replies": [] }, { "id": 1332876, "author_name": "nofreewill42", "author_url": "", "post_date": "2021-06-02T11:07:49.210000", "content": "It's not my idea, there are a couple mentions of it in the discussions. I just made it more available.
", "votes": -1, "replies": [] } ] }, { "id": 1332477, "author_name": "Mr_KnowNothing", "author_url": "", "post_date": "2021-06-02T06:29:00.053000", "content": "Thanks for sharing @nofreewill
", "votes": 1, "replies": [ { "id": 1332490, "author_name": "nofreewill42", "author_url": "", "post_date": "2021-06-02T06:37:17.200000", "content": "Did it help you? You're welcome! :))
", "votes": -1, "replies": [] }, { "id": 1332506, "author_name": "Mr_KnowNothing", "author_url": "", "post_date": "2021-06-02T06:44:34.610000", "content": "Not yet, still trying our best to figure out how to use it , trying to decipher your words haha
", "votes": 0, "replies": [] }, { "id": 1332564, "author_name": "nofreewill42", "author_url": "", "post_date": "2021-06-02T07:27:33.660000", "content": "In my validation:
\nThen
\nThen
\nBut now I'm stuck here, I think that my models are just not good enough (as all of them have CV higher than 1 LD)
\nI may be able to work out some sophisticated strategies on how to choose between all the predictions if there are multiple ones being the same, but sadly I just have no time for that despite the extension.
@nofreewill its humble of you to share but you should not have done it before the deadline , this might be unfair to people who had figured this out on their own . This could prove vital for a win and would have required good amount of digging
", "votes": 6, "replies": [] }, { "id": 1332761, "author_name": "nofreewill42", "author_url": "", "post_date": "2021-06-02T09:27:03.553000", "content": "That's a good amount of downvotes.
\nWhat I said is all out in the discussions, so I don't agree with that hate.
There are also a lot of nuances that needs a lot of work.
\nI.e. how does beam size effect LD of shorter and longer inchis? How do you select your prediction if there are more inchis that are valid, and how do you take into account the beam size effect? How do you take into account if your models are trained on different aspect ratios? And many other things that needs a lot of work to optimize.
So I guess that the downvotes come from teams that enjoyed that this idea is less available and didn't want to work harder on these issues.
", "votes": 2, "replies": [] }, { "id": 1333385, "author_name": "Vikrant", "author_url": "", "post_date": "2021-06-02T17:14:21.663000", "content": "don't worry that can be normalised with upvotes ;)
", "votes": 2, "replies": [] }, { "id": 1333397, "author_name": "nofreewill42", "author_url": "", "post_date": "2021-06-02T17:29:39.847000", "content": "haha, thank you! :D I'm keeping an eye on the number of votes, it's a roller coaster.. :D
\nI get why one might be angry, but really.. this is such a simple idea that it would be a shame if 3 month of one's hard work would go into garbage because he missed it but others did not.
I appreciate your humble sharing in the competition and It only worked in my favour that you shared exactly what you did and saved us time .
\nI just said on a general note that it might hurt people , there is a reason why all the gms and top teams don't share their secret sauce right untill the end of competition no matter how easy it is 😉
Just to put things into perpective , the first competition I participated in people found a very easy post-processing like this that could be utilized but no one shared untill the end of the competition
", "votes": 5, "replies": [] }, { "id": 1333481, "author_name": "Mohammed Rizin V K", "author_url": "", "post_date": "2021-06-02T18:54:46.103000", "content": "You are right @tanulsingh077, the teams that are at the top which has already done it and found it useful to improve their score might think as a spoiler alert and downvotes it.
", "votes": 0, "replies": [] }, { "id": 1333499, "author_name": "Yibing Wu", "author_url": "", "post_date": "2021-06-02T19:02:23.203000", "content": "Don't worry. It is just a sensitive moment. Eventually, people only remember your method and generosity.
", "votes": 1, "replies": [] }, { "id": 1333501, "author_name": "nofreewill42", "author_url": "", "post_date": "2021-06-02T19:12:49.517000", "content": "@ybwu01 wow, thanks for your kind words! :)
\nIt's a little funny how by downvoting a thing you know that works you actually confirm that it does work and encourage others to go onto that path.
This happens all the time and I believe someone summed it up best in another competition as \"people already using the trick will be unhappy while people new to it will be happy\".
\nI personally think it's Kaggle etiquette not to share anything that might have a material impact on the LB late in the competition - the caution on high-scoring notebooks is another facet of it.
", "votes": 4, "replies": [] }, { "id": 1334141, "author_name": "Gabriel Lindenmaier", "author_url": "", "post_date": "2021-06-03T09:41:52.520000", "content": "With the recent changes on the leaderboard we at least know who already used the trick and who didn't.
\nQuite surprising for me to see so interrupting changes. But we will see if carries to the private board.
I myself have decided to not upload a submission at all. I have done a lot of experiments to reproduce some papers and trying out stuff on myself. It was fun and I learned a lot. Meanwhile I never went for the highest-most score, just an efficient model (more on the encoder side). In the end I run out of time to train for long enough to be competitive (enough for my taste).
\nI think I will make a post after the submission end, detailing my most interesting findings.
\nEdit: Cheers to @charmq for keeping up with the teams in the gold-region so far.
", "votes": 2, "replies": [] }, { "id": 1334958, "author_name": "Chun Ming Lee", "author_url": "", "post_date": "2021-06-03T23:37:02.507000", "content": "I have to wonder how much of the shake-up in the gold zone was driven by this reveal (or maybe not at all).
\nI would not be happy if I was in one of the teams being punished…
", "votes": 1, "replies": [] } ] }, { "id": 1331111, "author_name": "shinewine", "author_url": "", "post_date": "2021-06-01T09:26:30.843000", "content": "Wow! This idea is very very important! Thanks a lot for sharing!
", "votes": 1, "replies": [ { "id": 1331120, "author_name": "nofreewill42", "author_url": "", "post_date": "2021-06-01T09:37:33.890000", "content": "It was worth sharing, then! :) You're welcome!
", "votes": 0, "replies": [] } ] }, { "id": 1330928, "author_name": "Gabriel Lindenmaier", "author_url": "", "post_date": "2021-06-01T07:09:02.070000", "content": "Well, then I hope that is enough for you to achieve the solo gold you wanted :)
", "votes": 1, "replies": [ { "id": 1330936, "author_name": "nofreewill42", "author_url": "", "post_date": "2021-06-01T07:14:44.117000", "content": "Thank you! :)
\nI still have doubts, but now I have a little hope, at least.
But I guess that a lot of teams left their submissions to the end, so… :/ :D
", "votes": 1, "replies": [] }, { "id": 1331031, "author_name": "hengck23", "author_url": "", "post_date": "2021-06-01T08:27:02.570000", "content": "i estimate gold level should be better 0.65~0.62
", "votes": 1, "replies": [] }, { "id": 1331136, "author_name": "Mohammed Rizin V K", "author_url": "", "post_date": "2021-06-01T09:47:54.717000", "content": "@nofreewill i think you were hiding your score for a month. I think you have a better model and will win gold I think.
\nI estimate gold will be <0.62
", "votes": 1, "replies": [] }, { "id": 1331142, "author_name": "nofreewill42", "author_url": "", "post_date": "2021-06-01T09:53:10.190000", "content": "I also think gold will be less than 0.62
\nBut sadly, no, my best score I can achieve is 0.70 right now..
\nI don't think I'll have gold, but I'm trying my best :)
OT: Has here anybody enough experience to confidently say, if the public leader board compared to the private + public (complete) leaderboard may either be:
\nIn other words: Is the public test-set usually representative of the private one or not? Or does this strictly vary with the competition?
", "votes": 0, "replies": [] }, { "id": 1331607, "author_name": "Vikrant", "author_url": "", "post_date": "2021-06-01T15:17:26.760000", "content": "Good Question.. Ideally, it should be a stratified sub sample but based on my bast experience most of the times it's random. That's why shakeup happens
", "votes": 0, "replies": [] }, { "id": 1331677, "author_name": "Mohammed Rizin V K", "author_url": "", "post_date": "2021-06-01T16:15:17.263000", "content": "@cepheidq sampling of the private set from whole test set can any of your mention and it varies with each competition.
\nIn this competition it might be random sampled or length wise sampling as @nofreewill did for his CV.
\npurposeful maybe but lower probability.