Only one week left and nearly 2000 entries!! There are some very impressive LB scores and the whole project has been very interesting. What do people think will happen one the Private LB scores are revealed? Obviously it will all come down to two things:
\n1) how well the Public LB set represents the Private LB set
\n2) how well entries' CV represented the Private LB set
What do you think will happen?
", "messages": [ { "id": 2899812, "postDate": "2024-07-01T21:22:00.417Z", "content": "Private will not represent Public well. IIUC, 33% (nefarious new library) is not part of public - we are totally blind on it and can't even slightly probe (unless someone actually does docking). The correlation between CV and public LB for shared and non-shared parts is quite ok for me, given that you do local simulation of shared/non-shared split correctly. BUT, problem is - I've mostly flattened out on the shared part improvement and got lucky couple of times on non-shared. It maybe is just luck as well and shake-up will come from non-shared and nefarious segment. This will mean a LOT of random stuff will happen. In theory folks with many models of different sorts will have an upper hand. I have 1 model currently, lol
", "rawMarkdown": "Private will not represent Public well. IIUC, 33% (nefarious new library) is not part of public - we are totally blind on it and can't even slightly probe (unless someone actually does docking). The correlation between CV and public LB for shared and non-shared parts is quite ok for me, given that you do local simulation of shared/non-shared split correctly. BUT, problem is - I've mostly flattened out on the shared part improvement and got lucky couple of times on non-shared. It maybe is just luck as well and shake-up will come from non-shared and nefarious segment. This will mean a LOT of random stuff will happen. In theory folks with many models of different sorts will have an upper hand. I have 1 model currently, lol", "votes": 9, "replies": [ { "id": 2899818, "postDate": "2024-07-01T21:28:40.680Z", "content": "Agreed, there's a LOT of uncertainties added by the nefarious new library… Really impressive that you achieved the current LB with one model though! lol
", "rawMarkdown": "Agreed, there's a LOT of uncertainties added by the nefarious new library... Really impressive that you achieved the current LB with one model though! lol", "votes": 7, "replies": [ { "id": 2899888, "postDate": "2024-07-01T22:44:19.550Z", "content": "Congrats to both of you on your positions on the LB!
\nI'm WAY down the list hoping for a massive shake up. 😅
\nBest of luck to both of you in the final LB!! I look forward to reading the write ups of what models make it to the top.
", "rawMarkdown": "Congrats to both of you on your positions on the LB! \n\nI'm WAY down the list hoping for a massive shake up. 😅\n\nBest of luck to both of you in the final LB!! I look forward to reading the write ups of what models make it to the top.", "votes": 1 }, { "id": 2901490, "postDate": "2024-07-02T19:44:47.063Z", "content": "wait wait. I went all in and got a second GPU )
", "rawMarkdown": "wait wait. I went all in and got a second GPU )", "votes": 3 }, { "id": 2903459, "postDate": "2024-07-03T19:47:11.683Z", "content": "I'm also looking forward to the debriefs, breakdowns and analyses… there's so much surface area here (data sample creation, data preprocessing, model design, etc…)
", "rawMarkdown": "I'm also looking forward to the debriefs, breakdowns and analyses... there's so much surface area here (data sample creation, data preprocessing, model design, etc...)", "votes": 1 } ] }, { "id": 2904970, "postDate": "2024-07-04T16:13:30.370Z", "content": "The new library introduction you mentioned and the correlation of shared and non-shared parts does add to the uncertainty of the final ranking. In particular, there is still some randomness in the performance of the non-shared parts, which is very noteworthy. Being able to perform well on shared parts shows the power of your model and the sophistication of your tuning
", "rawMarkdown": "The new library introduction you mentioned and the correlation of shared and non-shared parts does add to the uncertainty of the final ranking. In particular, there is still some randomness in the performance of the non-shared parts, which is very noteworthy. Being able to perform well on shared parts shows the power of your model and the sophistication of your tuning", "isDeleted": true } ] }, { "id": 2899805, "postDate": "2024-07-01T21:16:16.397Z", "content": "Only one week left and nearly 2000 entries!! There are some very impressive LB scores and the whole project has been very interesting. What do people think will happen one the Private LB scores are revealed? Obviously it will all come down to two things:
\n1) how well the Public LB set represents the Private LB set
\n2) how well entries' CV represented the Private LB set
What do you think will happen?
", "rawMarkdown": "Only one week left and nearly 2000 entries!! There are some very impressive LB scores and the whole project has been very interesting. What do people think will happen one the Private LB scores are revealed? Obviously it will all come down to two things:\n\n1) how well the Public LB set represents the Private LB set\n2) how well entries' CV represented the Private LB set\n\nWhat do you think will happen?", "votes": 7 }, { "id": 2900388, "postDate": "2024-07-02T08:43:10.990Z", "content": "Welcome to the notriazine lottery
", "rawMarkdown": "Welcome to the notriazine lottery", "votes": 8, "replies": [ { "id": 2902270, "postDate": "2024-07-03T06:54:38.657Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 2910523, "postDate": "2024-07-07T18:06:24.953Z", "content": "In a real sense you are only predicting a which of a small handful of building blocks are active. So in addition to what's already been said, there's also all the problems of small datasets in my view. High chance of randomness and shake-up.
\nThe alternative is if all scores are just sooo low on the non shared and nefarious, then if a small number of competitors are way ahead of the pack on shared BB predictions, then the magnitude of the random shake-up could be only among people near each other and actually small. But public LB won't always indicate who is near to each other in shared BB preds, as there could be some that are way overfit on public non shared BBs. Guess we'll see
", "rawMarkdown": "In a real sense you are only predicting a which of a small handful of building blocks are active. So in addition to what's already been said, there's also all the problems of small datasets in my view. High chance of randomness and shake-up. \n\nThe alternative is if all scores are just sooo low on the non shared and nefarious, then if a small number of competitors are way ahead of the pack on shared BB predictions, then the magnitude of the random shake-up could be only among people near each other and actually small. But public LB won't always indicate who is near to each other in shared BB preds, as there could be some that are way overfit on public non shared BBs. Guess we'll see", "votes": 3 }, { "id": 2900614, "postDate": "2024-07-02T11:54:25.787Z", "content": "I think a good portion of the private LB set is out-of-domain for my models, so I am not at all confident regarding robustness to shake-up. I had no measurable success with docking, so am relying on some combination of descriptor-based cheminformatics and tokenised SMILES models. The shake-up will not be small.
", "rawMarkdown": "I think a good portion of the private LB set is out-of-domain for my models, so I am not at all confident regarding robustness to shake-up. I had no measurable success with docking, so am relying on some combination of descriptor-based cheminformatics and tokenised SMILES models. The shake-up will not be small.", "votes": 3 }, { "id": 2912480, "postDate": "2024-07-08T23:38:36.207Z", "content": "tl;dr: I don't know about others, but I will probably fall far. 😆
\nIt's really hard to say how much shake-up there will be, because I have the suspicion that proper holdout (CV) will be pretty trustworthy, and that the final unknown library might be still close enough to \"non-shared\" that what works best on non-shared holdout will work on it.
\nBut the \"only small to moderate shakeup\" prediction would only be for people who are submitting predictions tuned against the train data and NOT the public LB with the tiny non-shared section. So the only person that I can fully predict is myself… I climbed a bunch of spots in the last hour today and definitely about to go back down, lol. I took a long break so I came back and just threw all the public ensemble kitchen sink at the non-share, tuning it to the public LB. Definitely not taking my own advice, lol.
", "rawMarkdown": "tl;dr: I don't know about others, but I will probably fall far. 😆\n\nIt's really hard to say how much shake-up there will be, because I have the suspicion that proper holdout (CV) will be pretty trustworthy, and that the final unknown library might be still close enough to \"non-shared\" that what works best on non-shared holdout will work on it.\n\nBut the \"only small to moderate shakeup\" prediction would only be for people who are submitting predictions tuned against the train data and NOT the public LB with the tiny non-shared section. So the only person that I can fully predict is myself... I climbed a bunch of spots in the last hour today and definitely about to go back down, lol. I took a long break so I came back and just threw all the public ensemble kitchen sink at the non-share, tuning it to the public LB. Definitely not taking my own advice, lol.", "replies": [ { "id": 2912492, "postDate": "2024-07-09T00:05:48.367Z", "content": "+800, +1000, +1000, +1000… @hengck23
\nI rest my case, lol. Tons of shake-up, but at least one person survived by relying on CV. With so many competitors, and a certain degree of pure luck, there's probably counter-examples, but big congrats to @hengck23 for top 5 and navigating the shakeup!
", "rawMarkdown": "+800, +1000, +1000, +1000... @hengck23 \n\nI rest my case, lol. Tons of shake-up, but at least one person survived by relying on CV. With so many competitors, and a certain degree of pure luck, there's probably counter-examples, but big congrats to @hengck23 for top 5 and navigating the shakeup!", "votes": 2 } ] }, { "id": 2905944, "postDate": "2024-07-05T09:02:22.580Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 2899812, "author_name": "yamu_duck", "author_url": "", "post_date": "2024-07-01T21:22:00.417000", "content": "Private will not represent Public well. IIUC, 33% (nefarious new library) is not part of public - we are totally blind on it and can't even slightly probe (unless someone actually does docking). The correlation between CV and public LB for shared and non-shared parts is quite ok for me, given that you do local simulation of shared/non-shared split correctly. BUT, problem is - I've mostly flattened out on the shared part improvement and got lucky couple of times on non-shared. It maybe is just luck as well and shake-up will come from non-shared and nefarious segment. This will mean a LOT of random stuff will happen. In theory folks with many models of different sorts will have an upper hand. I have 1 model currently, lol
", "votes": 9, "replies": [ { "id": 2899818, "author_name": "Steven_Y", "author_url": "", "post_date": "2024-07-01T21:28:40.680000", "content": "Agreed, there's a LOT of uncertainties added by the nefarious new library… Really impressive that you achieved the current LB with one model though! lol
", "votes": 7, "replies": [ { "id": 2899888, "author_name": "KirkDCO", "author_url": "", "post_date": "2024-07-01T22:44:19.550000", "content": "Congrats to both of you on your positions on the LB!
\nI'm WAY down the list hoping for a massive shake up. 😅
\nBest of luck to both of you in the final LB!! I look forward to reading the write ups of what models make it to the top.
", "votes": 1, "replies": [] }, { "id": 2901490, "author_name": "yamu_duck", "author_url": "", "post_date": "2024-07-02T19:44:47.063000", "content": "wait wait. I went all in and got a second GPU )
", "votes": 3, "replies": [] }, { "id": 2903459, "author_name": "Matt McDonagh", "author_url": "", "post_date": "2024-07-03T19:47:11.683000", "content": "I'm also looking forward to the debriefs, breakdowns and analyses… there's so much surface area here (data sample creation, data preprocessing, model design, etc…)
", "votes": 1, "replies": [] } ] }, { "id": 2904970, "author_name": "", "author_url": "", "post_date": "2024-07-04T16:13:30.370000", "content": "The new library introduction you mentioned and the correlation of shared and non-shared parts does add to the uncertainty of the final ranking. In particular, there is still some randomness in the performance of the non-shared parts, which is very noteworthy. Being able to perform well on shared parts shows the power of your model and the sophistication of your tuning
", "votes": 0, "replies": [] } ] }, { "id": 2900388, "author_name": "Ricardo Colomer", "author_url": "", "post_date": "2024-07-02T08:43:10.990000", "content": "Welcome to the notriazine lottery
", "votes": 8, "replies": [ { "id": 2902270, "author_name": "", "author_url": "", "post_date": "2024-07-03T06:54:38.657000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 2910523, "author_name": "Robert Hatch", "author_url": "", "post_date": "2024-07-07T18:06:24.953000", "content": "In a real sense you are only predicting a which of a small handful of building blocks are active. So in addition to what's already been said, there's also all the problems of small datasets in my view. High chance of randomness and shake-up.
\nThe alternative is if all scores are just sooo low on the non shared and nefarious, then if a small number of competitors are way ahead of the pack on shared BB predictions, then the magnitude of the random shake-up could be only among people near each other and actually small. But public LB won't always indicate who is near to each other in shared BB preds, as there could be some that are way overfit on public non shared BBs. Guess we'll see
", "votes": 3, "replies": [] }, { "id": 2900614, "author_name": "John Mitchell", "author_url": "", "post_date": "2024-07-02T11:54:25.787000", "content": "I think a good portion of the private LB set is out-of-domain for my models, so I am not at all confident regarding robustness to shake-up. I had no measurable success with docking, so am relying on some combination of descriptor-based cheminformatics and tokenised SMILES models. The shake-up will not be small.
", "votes": 3, "replies": [] }, { "id": 2912480, "author_name": "Robert Hatch", "author_url": "", "post_date": "2024-07-08T23:38:36.207000", "content": "tl;dr: I don't know about others, but I will probably fall far. 😆
\nIt's really hard to say how much shake-up there will be, because I have the suspicion that proper holdout (CV) will be pretty trustworthy, and that the final unknown library might be still close enough to \"non-shared\" that what works best on non-shared holdout will work on it.
\nBut the \"only small to moderate shakeup\" prediction would only be for people who are submitting predictions tuned against the train data and NOT the public LB with the tiny non-shared section. So the only person that I can fully predict is myself… I climbed a bunch of spots in the last hour today and definitely about to go back down, lol. I took a long break so I came back and just threw all the public ensemble kitchen sink at the non-share, tuning it to the public LB. Definitely not taking my own advice, lol.
", "votes": 0, "replies": [ { "id": 2912492, "author_name": "Robert Hatch", "author_url": "", "post_date": "2024-07-09T00:05:48.367000", "content": "+800, +1000, +1000, +1000… @hengck23
\nI rest my case, lol. Tons of shake-up, but at least one person survived by relying on CV. With so many competitors, and a certain degree of pure luck, there's probably counter-examples, but big congrats to @hengck23 for top 5 and navigating the shakeup!
", "votes": 2, "replies": [] } ] }, { "id": 2905944, "author_name": "", "author_url": "", "post_date": "2024-07-05T09:02:22.580000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2899812": "Private will not represent Public well. IIUC, 33% (nefarious new library) is not part of public - we are totally blind on it and can't even slightly probe (unless someone actually does docking). The correlation between CV and public LB for shared and non-shared parts is quite ok for me, given that you do local simulation of shared/non-shared split correctly. BUT, problem is - I've mostly flattened out on the shared part improvement and got lucky couple of times on non-shared. It maybe is just luck as well and shake-up will come from non-shared and nefarious segment. This will mean a LOT of random stuff will happen. In theory folks with many models of different sorts will have an upper hand. I have 1 model currently, lol", "2899805": "Only one week left and nearly 2000 entries!! There are some very impressive LB scores and the whole project has been very interesting. What do people think will happen one the Private LB scores are revealed? Obviously it will all come down to two things:\n\n1) how well the Public LB set represents the Private LB set\n2) how well entries' CV represented the Private LB set\n\nWhat do you think will happen?", "2900388": "Welcome to the notriazine lottery", "2910523": "In a real sense you are only predicting a which of a small handful of building blocks are active. So in addition to what's already been said, there's also all the problems of small datasets in my view. High chance of randomness and shake-up. \n\nThe alternative is if all scores are just sooo low on the non shared and nefarious, then if a small number of competitors are way ahead of the pack on shared BB predictions, then the magnitude of the random shake-up could be only among people near each other and actually small. But public LB won't always indicate who is near to each other in shared BB preds, as there could be some that are way overfit on public non shared BBs. Guess we'll see", "2900614": "I think a good portion of the private LB set is out-of-domain for my models, so I am not at all confident regarding robustness to shake-up. I had no measurable success with docking, so am relying on some combination of descriptor-based cheminformatics and tokenised SMILES models. The shake-up will not be small.", "2912480": "tl;dr: I don't know about others, but I will probably fall far. 😆\n\nIt's really hard to say how much shake-up there will be, because I have the suspicion that proper holdout (CV) will be pretty trustworthy, and that the final unknown library might be still close enough to \"non-shared\" that what works best on non-shared holdout will work on it.\n\nBut the \"only small to moderate shakeup\" prediction would only be for people who are submitting predictions tuned against the train data and NOT the public LB with the tiny non-shared section. So the only person that I can fully predict is myself... I climbed a bunch of spots in the last hour today and definitely about to go back down, lol. I took a long break so I came back and just threw all the public ensemble kitchen sink at the non-share, tuning it to the public LB. Definitely not taking my own advice, lol.", "2905944": "" } }