It seems that this competition uses a common method for early drug screening.
\nI believe that there must be technical limitations to the affinity values used in this competition.
Therefore, I have the following questions.
\nI don't think there is a perfect zero value, at least not in the data set, given the presence of nonspecific binding of the drug to the protein, but it is included in the data set.
\nIf the measurements are compiled, what would be the exact number of significant digits?
It seems that this competition uses a common method for early drug screening.
\nI believe that there must be technical limitations to the affinity values used in this competition.
Therefore, I have the following questions.
\nI don't think there is a perfect zero value, at least not in the data set, given the presence of nonspecific binding of the drug to the protein, but it is included in the data set.
\nIf the measurements are compiled, what would be the exact number of significant digits?
They did not measure affinity. They just tested if a molecule binds or not.
", "rawMarkdown": "They did not measure affinity. They just tested if a molecule binds or not.", "votes": null }, { "id": "2779119", "postDate": "04/27/2024 14:22:44", "content": "Thank you for answering my question.
\nSo it is a Binary class that may or may not be combined? I had overlooked the most elementary part.
I was mistaken because it was accepted even if it was not a Binary class in the Submit format.
\nPlease, thank you.
", "rawMarkdown": "Thank you for answering my question.\nSo it is a Binary class that may or may not be combined? I had overlooked the most elementary part.\n\nI was mistaken because it was accepted even if it was not a Binary class in the Submit format.\n\nPlease, thank you.", "votes": null }, { "id": "2779437", "postDate": "04/27/2024 16:51:56", "content": "@daikikurosu - I believe the submission form expects predictions to be probabilities between 0 and 1. Giving a simple 0 or 1 will likely skew the evaluation metric, which operates similar to AUROC.
\nOf course, please correct me if I'm wrong. I don't want to unintentionally (nor intentionally) spread misinformation.
", "rawMarkdown": "daikikurosu - I believe the submission form expects predictions to be probabilities between 0 and 1. Giving a simple 0 or 1 will likely skew the evaluation metric, which operates similar to AUROC.\n\nOf course, please correct me if I'm wrong. I don't want to unintentionally (nor intentionally) spread misinformation.", "votes": null }, { "id": "2779696", "postDate": "04/27/2024 19:13:32", "content": "The organizers wrote:
\n\n\nThis metric for this competition is the Mean Average Precision (micro) between the predicted probability and the observed target.
\n
So it seems safe to assume they are using exactly that function. From playing around with that function, it cares about (and only about) the ranks (relative order) of every prediction. So yes, giving nuanced probabilities will give better scores than ranking everything as only 0 or 1.
", "rawMarkdown": "The organizers wrote:\n\n> This metric for this competition is the [Mean Average Precision](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.average_precision_score.html) (micro) between the predicted probability and the observed target.\n\nSo it seems safe to assume they are using exactly that function. From playing around with that function, it cares about (and only about) the ranks (relative order) of every prediction. So yes, giving nuanced probabilities will give better scores than ranking everything as only 0 or 1.", "votes": null }, { "id": "2781446", "postDate": "04/28/2024 19:33:28", "content": "In DEL screens, the proxy binding affinity is expressed as the enrichment factor which is a function of the DNA read count on the \"with-protein\" condition relative to the control \"no-protein\" condition. The exact threshold used to transform the enrichment factor to boolean may vary depending on the specific experimental protocol.
", "rawMarkdown": "In DEL screens, the proxy binding affinity is expressed as the enrichment factor which is a function of the DNA read count on the \"with-protein\" condition relative to the control \"no-protein\" condition. The exact threshold used to transform the enrichment factor to boolean may vary depending on the specific experimental protocol.", "votes": null } ], "comments": [ { "id": 2777557, "author_name": "gyulamaloveczky4", "author_url": "", "post_date": "04/26/2024 17:50:44", "content": "They did not measure affinity. They just tested if a molecule binds or not.
", "votes": null, "replies": [ { "id": 2779119, "author_name": "daikikurosu", "author_url": "", "post_date": "04/27/2024 14:22:44", "content": "Thank you for answering my question.
\nSo it is a Binary class that may or may not be combined? I had overlooked the most elementary part.
I was mistaken because it was accepted even if it was not a Binary class in the Submit format.
\nPlease, thank you.
", "votes": null, "replies": [ { "id": 2779437, "author_name": "kirkdco", "author_url": "", "post_date": "04/27/2024 16:51:56", "content": "@daikikurosu - I believe the submission form expects predictions to be probabilities between 0 and 1. Giving a simple 0 or 1 will likely skew the evaluation metric, which operates similar to AUROC.
\nOf course, please correct me if I'm wrong. I don't want to unintentionally (nor intentionally) spread misinformation.
", "votes": null, "replies": [ { "id": 2779696, "author_name": "roberthatch", "author_url": "", "post_date": "04/27/2024 19:13:32", "content": "The organizers wrote:
\n\n\nThis metric for this competition is the Mean Average Precision (micro) between the predicted probability and the observed target.
\n
So it seems safe to assume they are using exactly that function. From playing around with that function, it cares about (and only about) the ranks (relative order) of every prediction. So yes, giving nuanced probabilities will give better scores than ranking everything as only 0 or 1.
", "votes": null, "replies": [] } ] } ] } ] }, { "id": 2781446, "author_name": "ruelcedeno", "author_url": "", "post_date": "04/28/2024 19:33:28", "content": "In DEL screens, the proxy binding affinity is expressed as the enrichment factor which is a function of the DNA read count on the \"with-protein\" condition relative to the control \"no-protein\" condition. The exact threshold used to transform the enrichment factor to boolean may vary depending on the specific experimental protocol.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2776249": "It seems that this competition uses a common method for early drug screening.\nI believe that there must be technical limitations to the affinity values used in this competition.\n\nTherefore, I have the following questions.\nI don't think there is a perfect zero value, at least not in the data set, given the presence of nonspecific binding of the drug to the protein, but it is included in the data set.\nIf the measurements are compiled, what would be the exact number of significant digits?", "2777557": "They did not measure affinity. They just tested if a molecule binds or not.", "2779119": "Thank you for answering my question.\nSo it is a Binary class that may or may not be combined? I had overlooked the most elementary part.\n\nI was mistaken because it was accepted even if it was not a Binary class in the Submit format.\n\nPlease, thank you.", "2779437": "daikikurosu - I believe the submission form expects predictions to be probabilities between 0 and 1. Giving a simple 0 or 1 will likely skew the evaluation metric, which operates similar to AUROC.\n\nOf course, please correct me if I'm wrong. I don't want to unintentionally (nor intentionally) spread misinformation.", "2779696": "The organizers wrote:\n\n> This metric for this competition is the [Mean Average Precision](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.average_precision_score.html) (micro) between the predicted probability and the observed target.\n\nSo it seems safe to assume they are using exactly that function. From playing around with that function, it cares about (and only about) the ranks (relative order) of every prediction. So yes, giving nuanced probabilities will give better scores than ranking everything as only 0 or 1.", "2781446": "In DEL screens, the proxy binding affinity is expressed as the enrichment factor which is a function of the DNA read count on the \"with-protein\" condition relative to the control \"no-protein\" condition. The exact threshold used to transform the enrichment factor to boolean may vary depending on the specific experimental protocol." }, "source": "meta" }