![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F779570%2Fac637e0191a60f55be4a9997fb405f62%2FBRD_allFPPrecision_CI.jpg?generation=1732482321497325&alt=media\")
After a long hiatus, I've finally been able to start looking at this again. Hopefully some of the folks out there are still around - I may tag a few of you at the bottom of this message.
\nDuring the competition, I had used Pharmacophore Fingerprints as I thought that would provide a more generalizable representation of the structures. Also, I used XGBoost as my modeling method. The combination didn't do terrible but was certainly not a high performer - only middle of the pack. I wanted to look at other fingerprints to see if any provided improved generalization, and luckily, ScikitFingerprints was pointed out early in this competition. If you've not tried it, you really should. This is an excellent, easy to use package. Huge thanks to @michaszafarczyk and his colleagues for this package!
\nHere's what I did:
\nResults:
\nI have all the results for all the subsets of test compounds (Public, Private, Share, Non-share, kin0), but I'm only discussing kin0, the non-triazine set of compounds from the Private test set, as those were the most interesting.
The results were similar to what was seen before - performance on the kin0 set isn't very good, but then I was wondering, what should I expect from random, and are any of these results actually better than random? (One of the conclusions of the competition is that results for the kin0 set were no better than random.) The expected Precision for any given dataset is the proportion of binders to total structures in that set. For the three targets in this set - BRD4 -> .00111; HSA -> .00126; sEH->.00124. To get a confidence interval on that expectation, I performed 10,000 bootstrap samplings of the data and calculated the upper 95th and 99th percentiles of of the observed precisions from bootstrap sampling.
\nThe figures below show the distribution of precisions seen over the 10,000 bootstrap samples with vertical lines indicating the mean, 95th percentile, and 99th percentile. The legends list the precisions of every fingerprint I tried along with how much better than random that precision is. A star indicates that fingerprint's performance was above the 99th percentile, and and asterisk indicates is it above the 95th percentile. Something of an empirical, bootstrap significance test.
\nNotice here that for BRD4 and sEH, the best performing fingerprint was 7X and 10.2X, resp., better than random. HSA was very slightly better than random, but not much as 1.4X. These results outperform most of the winning submissions form the competition.. One reason for this may be because ECFP fingerprints were very commonly used during the competition, but they only perform well for kin0 on BRD4, and are actually quite poor on HSA and sEH.
\nThere's a huge caveat here though - there is no way to know which fingerprint will perform best given the training set and the public test set. Looking at the correlations between public test set performances and the kin0 set performance (not shown), there is very little guidance on which fingerprint to choose. Also, no one fingerprint consistently performed best across all three targets. RDKit and Klekota-Roth fingerprints were significantly better than random on all three targets, however. (I'm not familiar with Klekota-Roth FPs.) It would be interesting to see if any of the better performing methods improve with RDKit rather than ECFP fingerprints.
\nAre we back to square one and the conclusion that there is no improvement over random selection? I originally commented that I was \"disturbed\" by that result. Given these results and especially that the vast majority of winning submissions did indeed perform better than random of the kin0 set, I would say that I am now \"disappointed\" rather than disturbed. There is certainly an improvement over random, but that improvement on the kin0 set is very small. There is great potential there - 10X would be fantastic! - but there is very little if any guidance on how to get there without knowing the result a priori.
\nI'm still going to work on this set further to investigate the question of representations and bridging between different chemotypes. I truly appreciate any feedback and discussion.
\nTagging a few folks here that might be interested in this. Let me know if you would rather not be tagged in future posts. @andrewdblevins @antoninadolgorukova @ahsuna123 @lililycai @hengck23 @chemdatafarmer @roberthatch
\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F779570%2F9d438dbbc3cd06b6145f4c90dce0ea56%2FHSA_allFPPrecision_CI.jpg?generation=1732482333127056&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F779570%2F18b439319497d4a683a0800156e91d88%2FsEH_allFPPrecision_CI.jpg?generation=1732482345825732&alt=media\")
After a long hiatus, I've finally been able to start looking at this again. Hopefully some of the folks out there are still around - I may tag a few of you at the bottom of this message.
\nDuring the competition, I had used Pharmacophore Fingerprints as I thought that would provide a more generalizable representation of the structures. Also, I used XGBoost as my modeling method. The combination didn't do terrible but was certainly not a high performer - only middle of the pack. I wanted to look at other fingerprints to see if any provided improved generalization, and luckily, ScikitFingerprints was pointed out early in this competition. If you've not tried it, you really should. This is an excellent, easy to use package. Huge thanks to @michaszafarczyk and his colleagues for this package!
\nHere's what I did:
\nResults:
\nI have all the results for all the subsets of test compounds (Public, Private, Share, Non-share, kin0), but I'm only discussing kin0, the non-triazine set of compounds from the Private test set, as those were the most interesting.
The results were similar to what was seen before - performance on the kin0 set isn't very good, but then I was wondering, what should I expect from random, and are any of these results actually better than random? (One of the conclusions of the competition is that results for the kin0 set were no better than random.) The expected Precision for any given dataset is the proportion of binders to total structures in that set. For the three targets in this set - BRD4 -> .00111; HSA -> .00126; sEH->.00124. To get a confidence interval on that expectation, I performed 10,000 bootstrap samplings of the data and calculated the upper 95th and 99th percentiles of of the observed precisions from bootstrap sampling.
\nThe figures below show the distribution of precisions seen over the 10,000 bootstrap samples with vertical lines indicating the mean, 95th percentile, and 99th percentile. The legends list the precisions of every fingerprint I tried along with how much better than random that precision is. A star indicates that fingerprint's performance was above the 99th percentile, and and asterisk indicates is it above the 95th percentile. Something of an empirical, bootstrap significance test.
\nNotice here that for BRD4 and sEH, the best performing fingerprint was 7X and 10.2X, resp., better than random. HSA was very slightly better than random, but not much as 1.4X. These results outperform most of the winning submissions form the competition.. One reason for this may be because ECFP fingerprints were very commonly used during the competition, but they only perform well for kin0 on BRD4, and are actually quite poor on HSA and sEH.
\nThere's a huge caveat here though - there is no way to know which fingerprint will perform best given the training set and the public test set. Looking at the correlations between public test set performances and the kin0 set performance (not shown), there is very little guidance on which fingerprint to choose. Also, no one fingerprint consistently performed best across all three targets. RDKit and Klekota-Roth fingerprints were significantly better than random on all three targets, however. (I'm not familiar with Klekota-Roth FPs.) It would be interesting to see if any of the better performing methods improve with RDKit rather than ECFP fingerprints.
\nAre we back to square one and the conclusion that there is no improvement over random selection? I originally commented that I was \"disturbed\" by that result. Given these results and especially that the vast majority of winning submissions did indeed perform better than random of the kin0 set, I would say that I am now \"disappointed\" rather than disturbed. There is certainly an improvement over random, but that improvement on the kin0 set is very small. There is great potential there - 10X would be fantastic! - but there is very little if any guidance on how to get there without knowing the result a priori.
\nI'm still going to work on this set further to investigate the question of representations and bridging between different chemotypes. I truly appreciate any feedback and discussion.
\nTagging a few folks here that might be interested in this. Let me know if you would rather not be tagged in future posts. @andrewdblevins @antoninadolgorukova @ahsuna123 @lililycai @hengck23 @chemdatafarmer @roberthatch
\n
Amazing analysis and results. It makes me think docking methods may have higher score on kin0 set compared with fingerprint training
", "rawMarkdown": "Amazing analysis and results. It makes me think docking methods may have higher score on kin0 set compared with fingerprint training", "votes": null }, { "id": "3115989", "postDate": "02/05/2025 13:51:44", "content": "Thanks! 🙏🏻
\nSpeaking of docking, i just finished docking for all three proteins yesterday. I'm going to try to analyze those results and write up the results this weekend. 🤞🏻
", "rawMarkdown": "Thanks! 🙏🏻 \n\nSpeaking of docking, i just finished docking for all three proteins yesterday. I'm going to try to analyze those results and write up the results this weekend. 🤞🏻", "votes": null }, { "id": "3116046", "postDate": "02/05/2025 15:18:12", "content": "Wow. Looking forward for the results
", "rawMarkdown": "Wow. Looking forward for the results", "votes": null }, { "id": "3119915", "postDate": "02/09/2025 21:54:29", "content": "Docking Performance result are up!
", "rawMarkdown": "[Docking Performance](https://www.kaggle.com/competitions/leash-BELKA/discussion/562096) result are up!", "votes": null } ], "comments": [ { "id": 3115966, "author_name": "lililycai", "author_url": "", "post_date": "02/05/2025 13:30:46", "content": "Amazing analysis and results. It makes me think docking methods may have higher score on kin0 set compared with fingerprint training
", "votes": null, "replies": [ { "id": 3115989, "author_name": "kirkdco", "author_url": "", "post_date": "02/05/2025 13:51:44", "content": "Thanks! 🙏🏻
\nSpeaking of docking, i just finished docking for all three proteins yesterday. I'm going to try to analyze those results and write up the results this weekend. 🤞🏻
", "votes": null, "replies": [ { "id": 3116046, "author_name": "lililycai", "author_url": "", "post_date": "02/05/2025 15:18:12", "content": "Wow. Looking forward for the results
", "votes": null, "replies": [ { "id": 3119915, "author_name": "kirkdco", "author_url": "", "post_date": "02/09/2025 21:54:29", "content": "Docking Performance result are up!
", "votes": null, "replies": [] } ] } ] } ] } ], "raw_markdown_by_id": { "3054582": "After a long hiatus, I've finally been able to start looking at this again. Hopefully some of the folks out there are still around - I may tag a few of you at the bottom of this message.\n\nDuring the competition, I had used Pharmacophore Fingerprints as I thought that would provide a more generalizable representation of the structures. Also, I used XGBoost as my modeling method. The combination didn't do terrible but was certainly not a high performer - only middle of the pack. I wanted to look at other fingerprints to see if any provided improved generalization, and luckily, [ScikitFingerprints](https://github.com/scikit-fingerprints/scikit-fingerprints) [was pointed out early in this competition](https://www.kaggle.com/competitions/leash-BELKA/discussion/499455). If you've not tried it, you really should. This is an excellent, easy to use package. Huge thanks to @michaszafarczyk and his colleagues for this package!\n\nHere's what I did:\n* Stuck with XGBoost and the tuned hyperparameters I used during the competition, for simplicity.\n* Calculated all 2D fingerprints and RDKit Descriptors for all Building Blocks.\n* For each molecule in the training set, I concatenated the fingerprints for each Building Block and used that as an input. This is similar to a [Free-Wilson analysis](https://practicalcheminformatics.blogspot.com/2018/05/free-wilson-analysis.html), but using a fingerprint vector as a representation for the R-groups rather than simply R-group identity. [Here's a presentation ](https://bigchem.eu/sites/default/files/Online4_Chen.pdf)describing a similar approach. The [paper](https://pubs.acs.org/doi/abs/10.1021/ci4001376) is behind a paywall, unfortunately.\n* Training sets contained all the binders and a randomly selected 1,000,000 nonbinders.\n\n_Results:_\nI have all the results for all the subsets of test compounds (Public, Private, Share, Non-share, kin0), but I'm only discussing kin0, the non-triazine set of compounds from the Private test set, as those were the most interesting. \n\nThe results were similar to what was seen before - performance on the kin0 set isn't very good, but then I was wondering, what should I expect from random, and are any of these results actually better than random? (One of the conclusions of the competition is that results for the kin0 set were no better than random.) The expected Precision for any given dataset is the proportion of binders to total structures in that set. For the three targets in this set - BRD4 -> .00111; HSA -> .00126; sEH->.00124. To get a confidence interval on that expectation, I performed 10,000 bootstrap samplings of the data and calculated the upper 95th and 99th percentiles of of the observed precisions from bootstrap sampling. \n\nThe figures below show the distribution of precisions seen over the 10,000 bootstrap samples with vertical lines indicating the mean, 95th percentile, and 99th percentile. The legends list the precisions of every fingerprint I tried along with how much better than random that precision is. A star indicates that fingerprint's performance was above the 99th percentile, and and asterisk indicates is it above the 95th percentile. Something of an empirical, bootstrap significance test.\n\nNotice here that for BRD4 and sEH, the best performing fingerprint was 7X and 10.2X, resp., better than random. HSA was very slightly better than random, but not much as 1.4X. [These results outperform most of the winning submissions form the competition.](https://www.kaggle.com/competitions/leash-BELKA/discussion/523779). One reason for this may be because ECFP fingerprints were very commonly used during the competition, but they only perform well for kin0 on BRD4, and are actually quite poor on HSA and sEH.\n\nThere's a huge caveat here though - there is no way to know which fingerprint will perform best given the training set and the public test set. Looking at the correlations between public test set performances and the kin0 set performance (not shown), there is very little guidance on which fingerprint to choose. Also, no one fingerprint consistently performed best across all three targets. RDKit and Klekota-Roth fingerprints were significantly better than random on all three targets, however. (I'm not familiar with Klekota-Roth FPs.) It would be interesting to see if any of the better performing methods improve with RDKit rather than ECFP fingerprints.\n\nAre we back to square one and the conclusion that there is no improvement over random selection? I originally commented that I was \"disturbed\" by that result. Given these results and especially that the vast majority of winning submissions did indeed perform better than random of the kin0 set, I would say that I am now \"disappointed\" rather than disturbed. There is certainly an improvement over random, but that improvement on the kin0 set is very small. There is great potential there - 10X would be fantastic! - but there is very little if any guidance on how to get there without knowing the result _a priori_.\n\nI'm still going to work on this set further to investigate the question of representations and bridging between different chemotypes. I truly appreciate any feedback and discussion.\n\nTagging a few folks here that might be interested in this. Let me know if you would rather not be tagged in future posts. @andrewdblevins @antoninadolgorukova @ahsuna123 @lililycai @hengck23 @chemdatafarmer @roberthatch \n\n\n\n\n\n\n", "3115966": "Amazing analysis and results. It makes me think docking methods may have higher score on kin0 set compared with fingerprint training", "3115989": "Thanks! 🙏🏻 \n\nSpeaking of docking, i just finished docking for all three proteins yesterday. I'm going to try to analyze those results and write up the results this weekend. 🤞🏻", "3116046": "Wow. Looking forward for the results", "3119915": "[Docking Performance](https://www.kaggle.com/competitions/leash-BELKA/discussion/562096) result are up!" }, "source": "meta" }