![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F779570%2F65285bf03f53048398a89bc2e8647e6d%2FPerformanceTable.jpeg?generation=1722723111614211&alt=media\")
Thanks to @andrewdblevins for posting the answer key.
\nIt would be very interesting to see exactly how different models performed across proteins and subsets of the data. I've created this notebook to calculate the Average Precision across those 9 different groups.
\nI think it would be incredibly informative to see how the different models performed especially in the non-triazine sets of test compounds. The conclusion was that models did not generalize well to that group, but I'm curious how any of the models generalize if at all, especially the top performing ones. Please run the notebook and post your results here.
\nFor me, I tried to be clever and use pharmacophore similarities to represent the R-groups which didn't work as hoped. Here are my results. I notice that my Mean Average Precision does not match that posted for the Private LB. It isn't that different but I'm curious if there are any ideas why they don't match. I'm using the same SciKit-Learn function as shown in the evaluation notebook. EDIT - I realized that this is due to the fact that I did not separate the Public and Private sets, but used the entire dataset.
\nThanks again to @andrewdblevins, the Kaggle organizers and all the participants. This has been a great experience.
\nBTW - in the table produced by the notebook, kin0 refers to the non-triazine set, non_share refers to the triazine core but with non-shared BBs, and share refers to the triazine core with shared BBs.
Update 24.08.05 A dataset was created with the 21 submission files gathered from the Code section, and a .csv with performances (shown below in table form).
\n Average Precision: .\n non_share share\n . . .\n . . .\n . . .\nThanks to @andrewdblevins for posting the answer key.
\nIt would be very interesting to see exactly how different models performed across proteins and subsets of the data. I've created this notebook to calculate the Average Precision across those 9 different groups.
\nI think it would be incredibly informative to see how the different models performed especially in the non-triazine sets of test compounds. The conclusion was that models did not generalize well to that group, but I'm curious how any of the models generalize if at all, especially the top performing ones. Please run the notebook and post your results here.
\nFor me, I tried to be clever and use pharmacophore similarities to represent the R-groups which didn't work as hoped. Here are my results. I notice that my Mean Average Precision does not match that posted for the Private LB. It isn't that different but I'm curious if there are any ideas why they don't match. I'm using the same SciKit-Learn function as shown in the evaluation notebook. EDIT - I realized that this is due to the fact that I did not separate the Public and Private sets, but used the entire dataset.
\nThanks again to @andrewdblevins, the Kaggle organizers and all the participants. This has been a great experience.
\nBTW - in the table produced by the notebook, kin0 refers to the non-triazine set, non_share refers to the triazine core but with non-shared BBs, and share refers to the triazine core with shared BBs.
Update 24.08.05 A dataset was created with the 21 submission files gathered from the Code section, and a .csv with performances (shown below in table form).
\n Average Precision: .\n non_share share\n . . .\n . . .\n . . .\nThanks for the notebook! Just to confirm, \"kin0\" is non-triazine, \"share\" is triazine with shared BBs, and \"non-share\" is triazine core with non-shared BBs?
", "rawMarkdown": "Thanks for the notebook! Just to confirm, \"kin0\" is non-triazine, \"share\" is triazine with shared BBs, and \"non-share\" is triazine core with non-shared BBs?", "votes": null }, { "id": "2945896", "postDate": "08/03/2024 20:20:29", "content": "Yes, that is correct. That is how they are labelled in the solution file, but I should make a note of that on the notebook itself.
", "rawMarkdown": "Yes, that is correct. That is how they are labelled in the solution file, but I should make a note of that on the notebook itself.", "votes": null }, { "id": "2946010", "postDate": "08/03/2024 22:11:57", "content": "From the Code section I gathered 20 submissions with Private LB scores noted from 0.210 to 0.310 and ran my evaluation notebook to get a comparison of how they all did across the various proteins and groups. I also added one of my own submissions which had the highest Private LB score, but which I hadn't selected as a possible submission. 🫠Below is the table of results.
\nNote that one of these submissions, from Ah, had a higher Private LB score than the 1st place solution. It appears to have been a late submission - perhaps one that hadn't been selected for judging.
\nThe general conclusion is the same and described by the organizers, that Private LB scores were largely driven by the triazines with shared BBs, and some submissions had comparable scores for the triazine non-shared BB group. They all did poorly on the non-triazine group, however, even the high scoring submission by Ah I mentioned above. I had hoped than some of the better scoring algorithms showed some degree of generalization to the non-triazine compounds, alas, it was not to be.
\nThis result is troubling to me as this is exactly what computational chemistry and cheminformatics is supposed to do - translate to previously unseen structures. While there is certainly a benefit and enrichment for the triazine non-shared BBs, the non-triazine set shows no enrichment for hits based on predictions. It is an interesting result and certainly an interesting problem that remains to be solved.
\nI will create a dataset shortly containing all the submissions I hand scraped from the Code section along with a file with tabulated results. I'll update this thread with a link to the dataset.
\n
Use your methods to check @hengck23 's submissions ![\"image\"](\"https://i.ibb.co/bdH20zd/1.png\")
, check this notebook
Same story as the others.
\nI had thought that the use of any models trained using the SMILES strings would inherently have difficulty in generalization. To me, the SMILES representation will lead to over-training by its very nature. Even a GNN is biased due to the specific representations in SMILES strings. It almost seems that the best performing models were those that were the most over-trained. ðŸ«
\nThat's why I went with the 2D pharmacophore, but unfortunately that also didn't generalize. I'm curious as to what kind of representation would allow generalization. 🤔
\nI've said for a long time (decades?) that we don't need better modeling methods, we need better representations. Assuming the data is accurate given the complexities of the DEL assay, this is a beautiful set to search for new representations, IMO.
", "rawMarkdown": "Same story as the others.\n\nI had thought that the use of any models trained using the SMILES strings would inherently have difficulty in generalization. To me, the SMILES representation will lead to over-training by its very nature. Even a GNN is biased due to the specific representations in SMILES strings. It almost seems that the best performing models were those that were the most over-trained. 🫠\n\nThat's why I went with the 2D pharmacophore, but unfortunately that also didn't generalize. I'm curious as to what kind of representation would allow generalization. 🤔\n\nI've said for a long time (decades?) that we don't need better modeling methods, we need better representations. Assuming the data is accurate given the complexities of the DEL assay, this is a beautiful set to search for new representations, IMO.", "votes": null }, { "id": "2946127", "postDate": "08/04/2024 01:54:34", "content": "Agree. This is a good dataset to explore new representations for generalizations. Some physics based models should work, but needs lots of computation resources.
", "rawMarkdown": "Agree. This is a good dataset to explore new representations for generalizations. Some physics based models should work, but needs lots of computation resources.", "votes": null }, { "id": "2955534", "postDate": "08/11/2024 08:47:58", "content": "@kirkdco A little correction : It wasn't a late submisison 😬, it was during the submission timeline just didn't end up selecting as one of the final two submissions! :)
\nKudos to you for compiling the results! Very helpful! :)
Thanks for the clarification, @ahsuna123, and for the kind words. Kudos for the great work and contributions during the competition!!
", "rawMarkdown": "Thanks for the clarification, @ahsuna123, and for the kind words. Kudos for the great work and contributions during the competition!!", "votes": null }, { "id": "2975566", "postDate": "09/01/2024 03:08:41", "content": "Hi! Sorry for the late reply, was busy with another competition :)
\nI scored >90 submissions, which I saved, and the top models are here: https://www.kaggle.com/code/antoninadolgorukova/belka-score-submissions.
I have come to the following conclusions:
\n2.Limitations of Private mAP Scores: When considering private mAP scores averaged across both share and non-share groups, the overall reliability of these scores as indicators of generalization ability may be misleading. It might overestimate generalization ability if dominated by the more consistent performance in share groups or underestimate it if heavily influenced by variability in non-share groups.
", "rawMarkdown": "Hi! Sorry for the late reply, was busy with another competition :)\nI scored >90 submissions, which I saved, and the top models are here: https://www.kaggle.com/code/antoninadolgorukova/belka-score-submissions.\n\nI have come to the following conclusions:\n\n1. No Single Best Model Across All Contexts: The analysis highlights that there is no single best model that performs optimally across all proteins and test groups:\n- CNNs are the best models for all proteins but only for molecules with shared building blocks \n- ChemBerta models and XGBoost models showed the strongest performance on molecules with unseen building blocks, though their success depends on specific parameters and training settings \n- Chemprop models perform relatively better than other models in the highly challenging non-triazines group, although all models struggle in this category\n\n2.Limitations of Private mAP Scores: When considering private mAP scores averaged across both share and non-share groups, the overall reliability of these scores as indicators of generalization ability may be misleading. It might overestimate generalization ability if dominated by the more consistent performance in share groups or underestimate it if heavily influenced by variability in non-share groups.", "votes": null }, { "id": "2976219", "postDate": "09/01/2024 16:11:55", "content": "This is an amazing amount of excellent work. Thank you!
", "rawMarkdown": "This is an amazing amount of excellent work. Thank you!", "votes": null } ], "comments": [ { "id": 2945893, "author_name": "lililycai", "author_url": "", "post_date": "08/03/2024 20:18:21", "content": "Thanks for the notebook! Just to confirm, \"kin0\" is non-triazine, \"share\" is triazine with shared BBs, and \"non-share\" is triazine core with non-shared BBs?
", "votes": null, "replies": [ { "id": 2945896, "author_name": "kirkdco", "author_url": "", "post_date": "08/03/2024 20:20:29", "content": "Yes, that is correct. That is how they are labelled in the solution file, but I should make a note of that on the notebook itself.
", "votes": null, "replies": [] } ] }, { "id": 2946010, "author_name": "kirkdco", "author_url": "", "post_date": "08/03/2024 22:11:57", "content": "From the Code section I gathered 20 submissions with Private LB scores noted from 0.210 to 0.310 and ran my evaluation notebook to get a comparison of how they all did across the various proteins and groups. I also added one of my own submissions which had the highest Private LB score, but which I hadn't selected as a possible submission. 🫠Below is the table of results.
\nNote that one of these submissions, from Ah, had a higher Private LB score than the 1st place solution. It appears to have been a late submission - perhaps one that hadn't been selected for judging.
\nThe general conclusion is the same and described by the organizers, that Private LB scores were largely driven by the triazines with shared BBs, and some submissions had comparable scores for the triazine non-shared BB group. They all did poorly on the non-triazine group, however, even the high scoring submission by Ah I mentioned above. I had hoped than some of the better scoring algorithms showed some degree of generalization to the non-triazine compounds, alas, it was not to be.
\nThis result is troubling to me as this is exactly what computational chemistry and cheminformatics is supposed to do - translate to previously unseen structures. While there is certainly a benefit and enrichment for the triazine non-shared BBs, the non-triazine set shows no enrichment for hits based on predictions. It is an interesting result and certainly an interesting problem that remains to be solved.
\nI will create a dataset shortly containing all the submissions I hand scraped from the Code section along with a file with tabulated results. I'll update this thread with a link to the dataset.
\n
@kirkdco A little correction : It wasn't a late submisison 😬, it was during the submission timeline just didn't end up selecting as one of the final two submissions! :)
\nKudos to you for compiling the results! Very helpful! :)
Thanks for the clarification, @ahsuna123, and for the kind words. Kudos for the great work and contributions during the competition!!
", "votes": null, "replies": [] } ] } ] }, { "id": 2946056, "author_name": "lililycai", "author_url": "", "post_date": "08/03/2024 23:29:19", "content": "Use your methods to check @hengck23 's submissions , check this notebook
Same story as the others.
\nI had thought that the use of any models trained using the SMILES strings would inherently have difficulty in generalization. To me, the SMILES representation will lead to over-training by its very nature. Even a GNN is biased due to the specific representations in SMILES strings. It almost seems that the best performing models were those that were the most over-trained. ðŸ«
\nThat's why I went with the 2D pharmacophore, but unfortunately that also didn't generalize. I'm curious as to what kind of representation would allow generalization. 🤔
\nI've said for a long time (decades?) that we don't need better modeling methods, we need better representations. Assuming the data is accurate given the complexities of the DEL assay, this is a beautiful set to search for new representations, IMO.
", "votes": null, "replies": [ { "id": 2946127, "author_name": "lililycai", "author_url": "", "post_date": "08/04/2024 01:54:34", "content": "Agree. This is a good dataset to explore new representations for generalizations. Some physics based models should work, but needs lots of computation resources.
", "votes": null, "replies": [] } ] } ] }, { "id": 2975566, "author_name": "antoninadolgorukova", "author_url": "", "post_date": "09/01/2024 03:08:41", "content": "Hi! Sorry for the late reply, was busy with another competition :)
\nI scored >90 submissions, which I saved, and the top models are here: https://www.kaggle.com/code/antoninadolgorukova/belka-score-submissions.
I have come to the following conclusions:
\n2.Limitations of Private mAP Scores: When considering private mAP scores averaged across both share and non-share groups, the overall reliability of these scores as indicators of generalization ability may be misleading. It might overestimate generalization ability if dominated by the more consistent performance in share groups or underestimate it if heavily influenced by variability in non-share groups.
", "votes": null, "replies": [ { "id": 2976219, "author_name": "kirkdco", "author_url": "", "post_date": "09/01/2024 16:11:55", "content": "This is an amazing amount of excellent work. Thank you!
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2944629": "Thanks to @andrewdblevins for posting the [answer key](https://www.kaggle.com/competitions/leash-BELKA/discussion/523644). \n \nIt would be very interesting to see exactly how different models performed across proteins and subsets of the data. I've created [this notebook](https://www.kaggle.com/code/kirkdco/precision-by-protein-and-group?scriptVersionId=190931649) to calculate the Average Precision across those 9 different groups. \n\nI think it would be incredibly informative to see how the different models performed especially in the non-triazine sets of test compounds. The conclusion was that models did not generalize well to that group, but I'm curious how any of the models generalize if at all, especially the top performing ones. Please run the notebook and post your results here.\n\nFor me, I tried to be clever and use [pharmacophore similarities to represent the R-groups](https://www.kaggle.com/competitions/leash-BELKA/discussion/519152) which didn't work as hoped. Here are my results. I notice that my Mean Average Precision does not match that posted for the Private LB. It isn't that different but I'm curious if there are any ideas why they don't match. I'm using the same SciKit-Learn function as shown in the [evaluation notebook](https://www.kaggle.com/code/metric/leash-average-map). **EDIT - I realized that this is due to the fact that I did not separate the Public and Private sets, but used the entire dataset.**\n\nThanks again to @andrewdblevins, the Kaggle organizers and all the participants. This has been a great experience.\n\nBTW - in the table produced by the notebook, `kin0` refers to the non-triazine set, non_share refers to the triazine core but with non-shared BBs, and share refers to the triazine core with shared BBs. \n\n**Update 24.08.05** [A dataset](https://www.kaggle.com/datasets/kirkdco/leashbio-competition-subs/data) was created with the 21 submission files gathered from the Code section, and a .csv with performances (shown below in table form).\n\n```\nMean Average Precision: 0.21924157357132887\n kin0\tnon_share\tshare\nBRD4\t0.001740\t0.175586\t0.517075\nHSA\t0.001165\t0.088662\t0.270757\nsEH\t0.000782\t0.059996\t0.857411\n```", "2945893": "Thanks for the notebook! Just to confirm, \"kin0\" is non-triazine, \"share\" is triazine with shared BBs, and \"non-share\" is triazine core with non-shared BBs?", "2945896": "Yes, that is correct. That is how they are labelled in the solution file, but I should make a note of that on the notebook itself.", "2946010": "From the Code section I gathered 20 submissions with Private LB scores noted from 0.210 to 0.310 and ran my evaluation notebook to get a comparison of how they all did across the various proteins and groups. I also added one of my own submissions which had the highest Private LB score, but which I hadn't selected as a possible submission. 🫠Below is the table of results.\n\nNote that [one of these submissions, from Ah,](https://www.kaggle.com/code/ahsuna123/belka1dcnn-0-310-private-based-on-ah-s-notebook) had a higher Private LB score than the 1st place solution. It appears to have been a late submission - perhaps one that hadn't been selected for judging.\n\nThe general conclusion is the same and described by the organizers, that Private LB scores were largely driven by the triazines with shared BBs, and some submissions had comparable scores for the triazine non-shared BB group. They all did poorly on the non-triazine group, however, even the high scoring submission by Ah I mentioned above. I had hoped than some of the better scoring algorithms showed some degree of generalization to the non-triazine compounds, alas, it was not to be.\n\nThis result is troubling to me as this is exactly what computational chemistry and cheminformatics is supposed to do - translate to previously unseen structures. While there is certainly a benefit and enrichment for the triazine non-shared BBs, the non-triazine set shows no enrichment for hits based on predictions. It is an interesting result and certainly an interesting problem that remains to be solved.\n\nI will create a dataset shortly containing all the submissions I hand scraped from the Code section along with a file with tabulated results. I'll update this thread with a link to the dataset.\n\n", "2946056": "Use your methods to check @hengck23 's submissions , check this [notebook] (https://www.kaggle.com/code/lililycai/ap-score-by-protein-and-group)", "2946089": "Same story as the others.\n\nI had thought that the use of any models trained using the SMILES strings would inherently have difficulty in generalization. To me, the SMILES representation will lead to over-training by its very nature. Even a GNN is biased due to the specific representations in SMILES strings. It almost seems that the best performing models were those that were the most over-trained. 🫠\n\nThat's why I went with the 2D pharmacophore, but unfortunately that also didn't generalize. I'm curious as to what kind of representation would allow generalization. 🤔\n\nI've said for a long time (decades?) that we don't need better modeling methods, we need better representations. Assuming the data is accurate given the complexities of the DEL assay, this is a beautiful set to search for new representations, IMO.", "2946127": "Agree. This is a good dataset to explore new representations for generalizations. Some physics based models should work, but needs lots of computation resources.", "2955534": "kirkdco A little correction : It wasn't a late submisison 😬, it was during the submission timeline just didn't end up selecting as one of the final two submissions! :)\nKudos to you for compiling the results! Very helpful! :)", "2955681": "Thanks for the clarification, @ahsuna123, and for the kind words. Kudos for the great work and contributions during the competition!!", "2975566": "Hi! Sorry for the late reply, was busy with another competition :)\nI scored >90 submissions, which I saved, and the top models are here: https://www.kaggle.com/code/antoninadolgorukova/belka-score-submissions.\n\nI have come to the following conclusions:\n\n1. No Single Best Model Across All Contexts: The analysis highlights that there is no single best model that performs optimally across all proteins and test groups:\n- CNNs are the best models for all proteins but only for molecules with shared building blocks \n- ChemBerta models and XGBoost models showed the strongest performance on molecules with unseen building blocks, though their success depends on specific parameters and training settings \n- Chemprop models perform relatively better than other models in the highly challenging non-triazines group, although all models struggle in this category\n\n2.Limitations of Private mAP Scores: When considering private mAP scores averaged across both share and non-share groups, the overall reliability of these scores as indicators of generalization ability may be misleading. It might overestimate generalization ability if dominated by the more consistent performance in share groups or underestimate it if heavily influenced by variability in non-share groups.", "2976219": "This is an amazing amount of excellent work. Thank you!" }, "source": "meta" }