One of the key goals of 3D RNA structure prediction is to identify all the RNA segments across natural RNA sequences that might have well-defined 3D structure. We'd like your help to get us there!
\nWe'd like to get Kaggle codes run across all natural RNA sequences. For the highest confidence predictions that are novel, we could prioritize these segments for high resolution experimental 3D structure determination.
\nWe're one step away. The thing is: we don't want a lot of false positives. So we'd love to have each notebook give a confidence score that predicts the TM-score that the target would get when compared to an experimentally solved structure.
\nWe'd especially like to identify cases where TM is predicted to be >0.45, the typical cutoff for a reasonable fold prediction.
\nIf you'd like to help, take your favorite notebook and attach it to this data set
\nhttps://www.kaggle.com/datasets/rhijudas/stanford-rna-3d-folding-pdb-summer2025
\nIt holds 13 targets that have come out in the PDB since the close of the training phase.
\nAnd then see if you can come up with a confidence score that achieves a high precision in predicting which of the 13 predictions actually has Tm>0.45.
\nHere's an example based on a notebook that runs template search with MMseqs2 and then estimates confidence based on the fraction of the target covered by the template:
\nhttps://www.kaggle.com/code/rhijudas/confidence-score-mmseqs2-3d-rna-template-id
\nThe notebook gets mean 3D accuracy of Tm = 0.368 and Precision, Recall, and F1 of confidence score accuracy of 1.0
\nYour notebooks should do better in terms of 3D accuracy and, hopefully maintain precision of 1.0. Please post statistics and notebook links here if you make progress!
\n(Precision, recall, and F1 values updated after fix to notebook's comparison of ref and pred.)
", "messages": [ { "id": "3298264", "postDate": "10/05/2025 02:54:04", "content": "One of the key goals of 3D RNA structure prediction is to identify all the RNA segments across natural RNA sequences that might have well-defined 3D structure. We'd like your help to get us there!
\nWe'd like to get Kaggle codes run across all natural RNA sequences. For the highest confidence predictions that are novel, we could prioritize these segments for high resolution experimental 3D structure determination.
\nWe're one step away. The thing is: we don't want a lot of false positives. So we'd love to have each notebook give a confidence score that predicts the TM-score that the target would get when compared to an experimentally solved structure.
\nWe'd especially like to identify cases where TM is predicted to be >0.45, the typical cutoff for a reasonable fold prediction.
\nIf you'd like to help, take your favorite notebook and attach it to this data set
\nhttps://www.kaggle.com/datasets/rhijudas/stanford-rna-3d-folding-pdb-summer2025
\nIt holds 13 targets that have come out in the PDB since the close of the training phase.
\nAnd then see if you can come up with a confidence score that achieves a high precision in predicting which of the 13 predictions actually has Tm>0.45.
\nHere's an example based on a notebook that runs template search with MMseqs2 and then estimates confidence based on the fraction of the target covered by the template:
\nhttps://www.kaggle.com/code/rhijudas/confidence-score-mmseqs2-3d-rna-template-id
\nThe notebook gets mean 3D accuracy of Tm = 0.368 and Precision, Recall, and F1 of confidence score accuracy of 1.0
\nYour notebooks should do better in terms of 3D accuracy and, hopefully maintain precision of 1.0. Please post statistics and notebook links here if you make progress!
\n(Precision, recall, and F1 values updated after fix to notebook's comparison of ref and pred.)
", "rawMarkdown": "One of the key goals of 3D RNA structure prediction is to identify all the RNA segments across natural RNA sequences that might have well-defined 3D structure. We'd like your help to get us there!\n\nWe'd like to get Kaggle codes run across all natural RNA sequences. For the highest confidence predictions that are novel, we could prioritize these segments for high resolution experimental 3D structure determination.\n\nWe're one step away. The thing is: *we don't want a lot of false positives*. So we'd love to have each notebook give a confidence score that predicts the TM-score that the target would get when compared to an experimentally solved structure. \n\nWe'd especially like to identify cases where TM is predicted to be >0.45, the typical cutoff for a reasonable fold prediction.\n\nIf you'd like to help, take your favorite notebook and attach it to this data set\n\nhttps://www.kaggle.com/datasets/rhijudas/stanford-rna-3d-folding-pdb-summer2025\n\nIt holds 13 targets that have come out in the PDB since the close of the training phase.\n\nAnd then see if you can come up with a confidence score that achieves a high precision in predicting which of the 13 predictions actually has Tm>0.45. \n\nHere's an example based on a notebook that runs template search with MMseqs2 and then estimates confidence based on the fraction of the target covered by the template: \n\nhttps://www.kaggle.com/code/rhijudas/confidence-score-mmseqs2-3d-rna-template-id\n \nThe notebook gets mean 3D accuracy of Tm = 0.368 and Precision, Recall, and F1 of confidence score accuracy of 1.0\n\nYour notebooks should do better in terms of 3D accuracy and, hopefully maintain precision of 1.0. Please post statistics and notebook links here if you make progress!\n\n(*Precision, recall, and F1 values updated after fix to notebook's comparison of ref and pred.*)", "votes": null }, { "id": "3298525", "postDate": "10/05/2025 18:35:44", "content": "Hi, I'm not entirely sure what output format you would need. I ran a simulation on these 13 targets and got results like the following (see attachment)(the TM-scores were calculated only for informational purposes, using the tm_align library)
\n| MAX_SCORE | \nCONFIDENCE(SUM) | \n
|---|---|
| 0.488270057555273 | \n0.938955225050449 | \n
| 0.331262390937843 | \n0.514765247702599 | \n
| 0.370054298339892 | \n0.642883621156216 | \n
| 0.283441785709035 | \n0.719378303736448 | \n
| 0.249054987745784 | \n0.642032139003277 | \n
| 0.798223644046592 | \n0.456463485956192 | \n
| 0.636943080236158 | \n0.660994231700897 | \n
| 0.548786849733515 | \n0.721843557432294 | \n
| 0.344998093726893 | \n0.57373333349824 | \n
| 0.741880618462633 | \n0.804388243705034 | \n
| 0.625168415230373 | \n0.724774695932865 | \n
| 0.317178276413226 | \n0.724590353667736 | \n
| 0.763040445662887 | \n0.99974361560453 | \n
| mean | \n\n |
| 0.499869457215393 | \n0.701888158011291 | \n
Thanks!
\nDo you see a way to define a confidence score such that when it is high, and only when it is high, it accurately predicts that the prediction has TM-score > 0.45?
\nFor example in my notebook, I calculated the fraction of the target covered by the template (target_frac_template), and was able to achieve this:
TMscore target_frac_template\n G4P . .\n G4R . .\n G4Q . .\n IWF . .\n J09 . .\n MMG . .\n MME . .\n VQV . .\n E9Q . .\n J4N . .\n J4O . .\n JGM . .\n LKU . .\nIn particular, precision is 100% – there were 6 targets with target_frac_template>0.45, and all 6 indeed had TM-score>0.45 when compared to the experimental structure.
TM-score: .\n: . (/)\n : . (/)\n Score : .\nThat's nice except that my notebook only found a template for 6 of the 13 cases, and its mean TM-score is only 0.368.
\nMost Kaggle notebooks (including yours) will have higher mean TM-score -- can they also define a confidence score that would achieve 100% precision in predicting which targets have TM-score > 0.45?
\nP.S. your question prompted me to find a bug in my notebook's comparison of predicted TM-score to actual TM-score. Thanks!
", "rawMarkdown": "Thanks! \n\nDo you see a way to define a confidence score such that when it is high, and only when it is high, it accurately predicts that the prediction has TM-score > 0.45?\n\nFor example in my [notebook](https://www.kaggle.com/code/rhijudas/confidence-score-mmseqs2-3d-rna-template-id), I calculated the fraction of the target covered by the template (`target_frac_template`), and was able to achieve this:\n\n```\n target_id TMscore target_frac_template\n0 9G4P 0.02752 0.000000\n1 9G4R 0.02251 0.000000\n2 9G4Q 0.02234 0.000000\n3 9IWF 0.02473 0.000000\n4 9J09 0.02990 0.000000\n5 9MMG 0.91128 0.894828\n6 9MME 0.02618 0.000000\n7 8VQV 0.92762 1.000000\n8 9E9Q 0.57674 1.000000\n9 9J4N 0.70860 1.000000\n10 9J4O 0.60490 0.876404\n11 9JGM 0.02941 0.000000\n12 9LKU 0.87714 0.953846\n```\n\nIn particular, precision is 100% – there were 6 targets with `target_frac_template`>0.45, and all 6 indeed had `TM-score`>0.45 when compared to the experimental structure.\n\n```\nMean TM-score: 0.3684\nPrecision: 1.0000 (6/6)\nRecall : 1.0000 (6/6)\nF1 Score : 1.0000\n```\n\nThat's nice except that my notebook only found a template for 6 of the 13 cases, and its mean TM-score is only 0.368. \n\nMost Kaggle notebooks (including yours) will have higher mean TM-score -- can they also define a confidence score that would achieve 100% precision in predicting which targets have TM-score > 0.45?\n\nP.S. your question prompted me to find a bug in my notebook's comparison of predicted TM-score to actual TM-score. Thanks!", "votes": null }, { "id": "3298549", "postDate": "10/05/2025 19:48:02", "content": "Below is a comparison of TM-scores between the Template-Based Model (TBM) and Protenix across several targets.
\nThe best_model column highlights which model achieved the higher TM-score for each case.
| target_id | \nTM_TBM | \nTM_Protenix | \nbest_model | \nbest_TM | \ntarget_frac_template | \n
|---|---|---|---|---|---|
| 8VQV | \n0.91708 | \n0.74352 | \nTBM | \n0.91708 | \n1 | \n
| 9E9Q | \n0.56803 | \n0.60023 | \nProtenix | \n0.60023 | \n1 | \n
| 9G4P | \n0.46631 | \n0.31505 | \nTBM | \n0.46631 | \n1 | \n
| 9G4Q | \n0.33465 | \n0.43069 | \nProtenix | \n0.43069 | \n1 | \n
| 9G4R | \n0.23963 | \n0.43340 | \nProtenix | \n0.43340 | \n1 | \n
| 9IWF | \n0.66022 | \n0.59655 | \nTBM | \n0.66022 | \n1 | \n
| 9J09 | \n0.26842 | \n0.38660 | \nProtenix | \n0.38660 | \n1 | \n
| 9J4N | \n0.73014 | \n0.74350 | \nProtenix | \n0.74350 | \n1 | \n
| 9J4O | \n0.69472 | \n0.64824 | \nTBM | \n0.69472 | \n1 | \n
| 9JGM | \n0.66172 | \n0.76514 | \nProtenix | \n0.76514 | \n1 | \n
| 9LKU | \n0.83085 | \n0.86004 | \nProtenix | \n0.86004 | \n1 | \n
| 9MME | \n0.74523 | \n0.20920 | \nTBM | \n0.74523 | \n1 | \n
| 9MMG | \n0.91277 | \n0.24754 | \nTBM | \n0.91277 | \n1 | \n
Precision: 0.7692
\nRecall: 1.0000
\nF1 Score: 0.8696
A hybrid ensemble approach—using TBM for high-template coverage targets and Protenix for difficult/no-template ones—achieves more balanced and robust overall performance.
\nTM score Private: 0.61793 and public 0.65366
", "rawMarkdown": "## 🧬 Template-Based vs. Protenix Model Comparison\n\nBelow is a comparison of **TM-scores** between the **Template-Based Model (TBM)** and **Protenix** across several targets. \nThe **`best_model`** column highlights which model achieved the higher TM-score for each case.\n\n| target_id | TM_TBM | TM_Protenix | best_model | best_TM | target_frac_template |\n|:-----------|--------:|-------------:|:------------|---------:|----------------------:|\n| 8VQV | 0.91708 | 0.74352 | **TBM** | 0.91708 | 1 |\n| 9E9Q | 0.56803 | 0.60023 | **Protenix** | 0.60023 | 1 |\n| 9G4P | 0.46631 | 0.31505 | **TBM** | 0.46631 | 1 |\n| 9G4Q | 0.33465 | 0.43069 | **Protenix** | 0.43069 | 1 |\n| 9G4R | 0.23963 | 0.43340 | **Protenix** | 0.43340 | 1 |\n| 9IWF | 0.66022 | 0.59655 | **TBM** | 0.66022 | 1 |\n| 9J09 | 0.26842 | 0.38660 | **Protenix** | 0.38660 | 1 |\n| 9J4N | 0.73014 | 0.74350 | **Protenix** | 0.74350 | 1 |\n| 9J4O | 0.69472 | 0.64824 | **TBM** | 0.69472 | 1 |\n| 9JGM | 0.66172 | 0.76514 | **Protenix** | 0.76514 | 1 |\n| 9LKU | 0.83085 | 0.86004 | **Protenix** | 0.86004 | 1 |\n| 9MME | 0.74523 | 0.20920 | **TBM** | 0.74523 | 1 |\n| 9MMG | 0.91277 | 0.24754 | **TBM** | 0.91277 | 1 |\n\n---\nPrecision: 0.7692\nRecall: 1.0000\nF1 Score: 0.8696\n\n### ✅ Observations\n- **Protenix** outperforms TBM on **6 out of 13** targets, particularly on **complex or non-template-driven** folds. \n- **TBM** remains stronger on **template-rich and well-structured** targets (e.g., 8VQV, 9MMG). \n\n### 💡 Takeaway\nA **hybrid ensemble** approach—using **TBM for high-template coverage** targets and **Protenix for difficult/no-template** ones—achieves more balanced and robust overall performance. \n\n### Hybrid approach (TBM and Proteinx)\n*TM score* **Private**: 0.61793 and **public** 0.65366", "votes": null }, { "id": "3298946", "postDate": "10/06/2025 19:23:22", "content": "Arun, thanks! Is there some score you could compute in a blind setting (i.e. without knowing the experimental structure) that would allow for selection of all 10/13 targets with actual TM>0.45 with 100% precision?
", "rawMarkdown": "Arun, thanks! Is there some score you could compute in a blind setting (i.e. without knowing the experimental structure) that would allow for selection of all 10/13 targets with actual TM>0.45 with 100% precision?", "votes": null }, { "id": "3299043", "postDate": "10/07/2025 04:07:02", "content": "Tanks guys for the
", "rawMarkdown": "Tanks guys for the", "votes": null }, { "id": "3299315", "postDate": "10/07/2025 18:49:18", "content": "@rhijudas I'm working on it
", "rawMarkdown": "rhijudas I'm working on it", "votes": null }, { "id": "3301495", "postDate": "10/13/2025 13:59:24", "content": "Hi @rhijudas!
\nHere's my attempt at the confidence scoring challenge.
\nI developed a hybrid system that combines template-based search (MMseqs2) with a reference-based fallback, using multi-factor confidence scoring designed with a precision-first philosophy.
\nResults on the 13 PDB summer 2025 targets:
\nMean TM-score: 0.5377 (+46% improvement over baseline 0.3684)
\nPrecision: 1.0000 (zero false positives!)
\nRecall: 0.7143 (5/7 targets with TM > 0.45 correctly identified)
\nF1 Score: 0.8333
The system correctly identifies 5 high-confidence targets (9MMG, 8VQV, 9J4N, 9LKU, 9E9Q) and appropriately flags 2 borderline cases (9MME and 9J4O) as low confidence due to genuine prediction uncertainty.
\nhttps://www.kaggle.com/code/fernandosr85/hybrid-confidence-scoring-system
", "rawMarkdown": "Hi @rhijudas! \n\nHere's my attempt at the confidence scoring challenge.\n\nI developed a hybrid system that combines template-based search (MMseqs2) with a reference-based fallback, using multi-factor confidence scoring designed with a precision-first philosophy.\n\nResults on the 13 PDB summer 2025 targets:\n\nMean TM-score: 0.5377 (+46% improvement over baseline 0.3684)\nPrecision: 1.0000 (zero false positives!)\nRecall: 0.7143 (5/7 targets with TM > 0.45 correctly identified)\nF1 Score: 0.8333\n\nThe system correctly identifies 5 high-confidence targets (9MMG, 8VQV, 9J4N, 9LKU, 9E9Q) and appropriately flags 2 borderline cases (9MME and 9J4O) as low confidence due to genuine prediction uncertainty.\n\nhttps://www.kaggle.com/code/fernandosr85/hybrid-confidence-scoring-system", "votes": null } ], "comments": [ { "id": 3298525, "author_name": "raulenrique", "author_url": "", "post_date": "10/05/2025 18:35:44", "content": "Hi, I'm not entirely sure what output format you would need. I ran a simulation on these 13 targets and got results like the following (see attachment)(the TM-scores were calculated only for informational purposes, using the tm_align library)
\n| MAX_SCORE | \nCONFIDENCE(SUM) | \n
|---|---|
| 0.488270057555273 | \n0.938955225050449 | \n
| 0.331262390937843 | \n0.514765247702599 | \n
| 0.370054298339892 | \n0.642883621156216 | \n
| 0.283441785709035 | \n0.719378303736448 | \n
| 0.249054987745784 | \n0.642032139003277 | \n
| 0.798223644046592 | \n0.456463485956192 | \n
| 0.636943080236158 | \n0.660994231700897 | \n
| 0.548786849733515 | \n0.721843557432294 | \n
| 0.344998093726893 | \n0.57373333349824 | \n
| 0.741880618462633 | \n0.804388243705034 | \n
| 0.625168415230373 | \n0.724774695932865 | \n
| 0.317178276413226 | \n0.724590353667736 | \n
| 0.763040445662887 | \n0.99974361560453 | \n
| mean | \n\n |
| 0.499869457215393 | \n0.701888158011291 | \n
Thanks!
\nDo you see a way to define a confidence score such that when it is high, and only when it is high, it accurately predicts that the prediction has TM-score > 0.45?
\nFor example in my notebook, I calculated the fraction of the target covered by the template (target_frac_template), and was able to achieve this:
TMscore target_frac_template\n G4P . .\n G4R . .\n G4Q . .\n IWF . .\n J09 . .\n MMG . .\n MME . .\n VQV . .\n E9Q . .\n J4N . .\n J4O . .\n JGM . .\n LKU . .\nIn particular, precision is 100% – there were 6 targets with target_frac_template>0.45, and all 6 indeed had TM-score>0.45 when compared to the experimental structure.
TM-score: .\n: . (/)\n : . (/)\n Score : .\nThat's nice except that my notebook only found a template for 6 of the 13 cases, and its mean TM-score is only 0.368.
\nMost Kaggle notebooks (including yours) will have higher mean TM-score -- can they also define a confidence score that would achieve 100% precision in predicting which targets have TM-score > 0.45?
\nP.S. your question prompted me to find a bug in my notebook's comparison of predicted TM-score to actual TM-score. Thanks!
", "votes": null, "replies": [] } ] }, { "id": 3298549, "author_name": "arunodhayan", "author_url": "", "post_date": "10/05/2025 19:48:02", "content": "Below is a comparison of TM-scores between the Template-Based Model (TBM) and Protenix across several targets.
\nThe best_model column highlights which model achieved the higher TM-score for each case.
| target_id | \nTM_TBM | \nTM_Protenix | \nbest_model | \nbest_TM | \ntarget_frac_template | \n
|---|---|---|---|---|---|
| 8VQV | \n0.91708 | \n0.74352 | \nTBM | \n0.91708 | \n1 | \n
| 9E9Q | \n0.56803 | \n0.60023 | \nProtenix | \n0.60023 | \n1 | \n
| 9G4P | \n0.46631 | \n0.31505 | \nTBM | \n0.46631 | \n1 | \n
| 9G4Q | \n0.33465 | \n0.43069 | \nProtenix | \n0.43069 | \n1 | \n
| 9G4R | \n0.23963 | \n0.43340 | \nProtenix | \n0.43340 | \n1 | \n
| 9IWF | \n0.66022 | \n0.59655 | \nTBM | \n0.66022 | \n1 | \n
| 9J09 | \n0.26842 | \n0.38660 | \nProtenix | \n0.38660 | \n1 | \n
| 9J4N | \n0.73014 | \n0.74350 | \nProtenix | \n0.74350 | \n1 | \n
| 9J4O | \n0.69472 | \n0.64824 | \nTBM | \n0.69472 | \n1 | \n
| 9JGM | \n0.66172 | \n0.76514 | \nProtenix | \n0.76514 | \n1 | \n
| 9LKU | \n0.83085 | \n0.86004 | \nProtenix | \n0.86004 | \n1 | \n
| 9MME | \n0.74523 | \n0.20920 | \nTBM | \n0.74523 | \n1 | \n
| 9MMG | \n0.91277 | \n0.24754 | \nTBM | \n0.91277 | \n1 | \n
Precision: 0.7692
\nRecall: 1.0000
\nF1 Score: 0.8696
A hybrid ensemble approach—using TBM for high-template coverage targets and Protenix for difficult/no-template ones—achieves more balanced and robust overall performance.
\nTM score Private: 0.61793 and public 0.65366
", "votes": null, "replies": [ { "id": 3298946, "author_name": "rhijudas", "author_url": "", "post_date": "10/06/2025 19:23:22", "content": "Arun, thanks! Is there some score you could compute in a blind setting (i.e. without knowing the experimental structure) that would allow for selection of all 10/13 targets with actual TM>0.45 with 100% precision?
", "votes": null, "replies": [ { "id": 3299315, "author_name": "arunodhayan", "author_url": "", "post_date": "10/07/2025 18:49:18", "content": "@rhijudas I'm working on it
", "votes": null, "replies": [] } ] } ] }, { "id": 3299043, "author_name": "phumlani93tshabalala", "author_url": "", "post_date": "10/07/2025 04:07:02", "content": "Tanks guys for the
", "votes": null, "replies": [] }, { "id": 3301495, "author_name": "fernandosr85", "author_url": "", "post_date": "10/13/2025 13:59:24", "content": "Hi @rhijudas!
\nHere's my attempt at the confidence scoring challenge.
\nI developed a hybrid system that combines template-based search (MMseqs2) with a reference-based fallback, using multi-factor confidence scoring designed with a precision-first philosophy.
\nResults on the 13 PDB summer 2025 targets:
\nMean TM-score: 0.5377 (+46% improvement over baseline 0.3684)
\nPrecision: 1.0000 (zero false positives!)
\nRecall: 0.7143 (5/7 targets with TM > 0.45 correctly identified)
\nF1 Score: 0.8333
The system correctly identifies 5 high-confidence targets (9MMG, 8VQV, 9J4N, 9LKU, 9E9Q) and appropriately flags 2 borderline cases (9MME and 9J4O) as low confidence due to genuine prediction uncertainty.
\nhttps://www.kaggle.com/code/fernandosr85/hybrid-confidence-scoring-system
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "3298264": "One of the key goals of 3D RNA structure prediction is to identify all the RNA segments across natural RNA sequences that might have well-defined 3D structure. We'd like your help to get us there!\n\nWe'd like to get Kaggle codes run across all natural RNA sequences. For the highest confidence predictions that are novel, we could prioritize these segments for high resolution experimental 3D structure determination.\n\nWe're one step away. The thing is: *we don't want a lot of false positives*. So we'd love to have each notebook give a confidence score that predicts the TM-score that the target would get when compared to an experimentally solved structure. \n\nWe'd especially like to identify cases where TM is predicted to be >0.45, the typical cutoff for a reasonable fold prediction.\n\nIf you'd like to help, take your favorite notebook and attach it to this data set\n\nhttps://www.kaggle.com/datasets/rhijudas/stanford-rna-3d-folding-pdb-summer2025\n\nIt holds 13 targets that have come out in the PDB since the close of the training phase.\n\nAnd then see if you can come up with a confidence score that achieves a high precision in predicting which of the 13 predictions actually has Tm>0.45. \n\nHere's an example based on a notebook that runs template search with MMseqs2 and then estimates confidence based on the fraction of the target covered by the template: \n\nhttps://www.kaggle.com/code/rhijudas/confidence-score-mmseqs2-3d-rna-template-id\n \nThe notebook gets mean 3D accuracy of Tm = 0.368 and Precision, Recall, and F1 of confidence score accuracy of 1.0\n\nYour notebooks should do better in terms of 3D accuracy and, hopefully maintain precision of 1.0. Please post statistics and notebook links here if you make progress!\n\n(*Precision, recall, and F1 values updated after fix to notebook's comparison of ref and pred.*)", "3298525": "Hi, I'm not entirely sure what output format you would need. I ran a simulation on these 13 targets and got results like the following (see attachment)(the TM-scores were calculated only for informational purposes, using the tm_align library)\n\n| MAX_SCORE | CONFIDENCE(SUM) |\n|-----------------|-----------------|\n| 0.488270057555273 | 0.938955225050449 |\n| 0.331262390937843 | 0.514765247702599 |\n| 0.370054298339892 | 0.642883621156216 |\n| 0.283441785709035 | 0.719378303736448 |\n| 0.249054987745784 | 0.642032139003277 |\n| 0.798223644046592 | 0.456463485956192 |\n| 0.636943080236158 | 0.660994231700897 |\n| 0.548786849733515 | 0.721843557432294 |\n| 0.344998093726893 | 0.57373333349824 |\n| 0.741880618462633 | 0.804388243705034 |\n| 0.625168415230373 | 0.724774695932865 |\n| 0.317178276413226 | 0.724590353667736 |\n| 0.763040445662887 | 0.99974361560453 |\n| **mean** | |\n| 0.499869457215393 | 0.701888158011291 |", "3298545": "Thanks! \n\nDo you see a way to define a confidence score such that when it is high, and only when it is high, it accurately predicts that the prediction has TM-score > 0.45?\n\nFor example in my [notebook](https://www.kaggle.com/code/rhijudas/confidence-score-mmseqs2-3d-rna-template-id), I calculated the fraction of the target covered by the template (`target_frac_template`), and was able to achieve this:\n\n```\n target_id TMscore target_frac_template\n0 9G4P 0.02752 0.000000\n1 9G4R 0.02251 0.000000\n2 9G4Q 0.02234 0.000000\n3 9IWF 0.02473 0.000000\n4 9J09 0.02990 0.000000\n5 9MMG 0.91128 0.894828\n6 9MME 0.02618 0.000000\n7 8VQV 0.92762 1.000000\n8 9E9Q 0.57674 1.000000\n9 9J4N 0.70860 1.000000\n10 9J4O 0.60490 0.876404\n11 9JGM 0.02941 0.000000\n12 9LKU 0.87714 0.953846\n```\n\nIn particular, precision is 100% – there were 6 targets with `target_frac_template`>0.45, and all 6 indeed had `TM-score`>0.45 when compared to the experimental structure.\n\n```\nMean TM-score: 0.3684\nPrecision: 1.0000 (6/6)\nRecall : 1.0000 (6/6)\nF1 Score : 1.0000\n```\n\nThat's nice except that my notebook only found a template for 6 of the 13 cases, and its mean TM-score is only 0.368. \n\nMost Kaggle notebooks (including yours) will have higher mean TM-score -- can they also define a confidence score that would achieve 100% precision in predicting which targets have TM-score > 0.45?\n\nP.S. your question prompted me to find a bug in my notebook's comparison of predicted TM-score to actual TM-score. Thanks!", "3298549": "## 🧬 Template-Based vs. Protenix Model Comparison\n\nBelow is a comparison of **TM-scores** between the **Template-Based Model (TBM)** and **Protenix** across several targets. \nThe **`best_model`** column highlights which model achieved the higher TM-score for each case.\n\n| target_id | TM_TBM | TM_Protenix | best_model | best_TM | target_frac_template |\n|:-----------|--------:|-------------:|:------------|---------:|----------------------:|\n| 8VQV | 0.91708 | 0.74352 | **TBM** | 0.91708 | 1 |\n| 9E9Q | 0.56803 | 0.60023 | **Protenix** | 0.60023 | 1 |\n| 9G4P | 0.46631 | 0.31505 | **TBM** | 0.46631 | 1 |\n| 9G4Q | 0.33465 | 0.43069 | **Protenix** | 0.43069 | 1 |\n| 9G4R | 0.23963 | 0.43340 | **Protenix** | 0.43340 | 1 |\n| 9IWF | 0.66022 | 0.59655 | **TBM** | 0.66022 | 1 |\n| 9J09 | 0.26842 | 0.38660 | **Protenix** | 0.38660 | 1 |\n| 9J4N | 0.73014 | 0.74350 | **Protenix** | 0.74350 | 1 |\n| 9J4O | 0.69472 | 0.64824 | **TBM** | 0.69472 | 1 |\n| 9JGM | 0.66172 | 0.76514 | **Protenix** | 0.76514 | 1 |\n| 9LKU | 0.83085 | 0.86004 | **Protenix** | 0.86004 | 1 |\n| 9MME | 0.74523 | 0.20920 | **TBM** | 0.74523 | 1 |\n| 9MMG | 0.91277 | 0.24754 | **TBM** | 0.91277 | 1 |\n\n---\nPrecision: 0.7692\nRecall: 1.0000\nF1 Score: 0.8696\n\n### ✅ Observations\n- **Protenix** outperforms TBM on **6 out of 13** targets, particularly on **complex or non-template-driven** folds. \n- **TBM** remains stronger on **template-rich and well-structured** targets (e.g., 8VQV, 9MMG). \n\n### 💡 Takeaway\nA **hybrid ensemble** approach—using **TBM for high-template coverage** targets and **Protenix for difficult/no-template** ones—achieves more balanced and robust overall performance. \n\n### Hybrid approach (TBM and Proteinx)\n*TM score* **Private**: 0.61793 and **public** 0.65366", "3298946": "Arun, thanks! Is there some score you could compute in a blind setting (i.e. without knowing the experimental structure) that would allow for selection of all 10/13 targets with actual TM>0.45 with 100% precision?", "3299043": "Tanks guys for the", "3299315": "rhijudas I'm working on it", "3301495": "Hi @rhijudas! \n\nHere's my attempt at the confidence scoring challenge.\n\nI developed a hybrid system that combines template-based search (MMseqs2) with a reference-based fallback, using multi-factor confidence scoring designed with a precision-first philosophy.\n\nResults on the 13 PDB summer 2025 targets:\n\nMean TM-score: 0.5377 (+46% improvement over baseline 0.3684)\nPrecision: 1.0000 (zero false positives!)\nRecall: 0.7143 (5/7 targets with TM > 0.45 correctly identified)\nF1 Score: 0.8333\n\nThe system correctly identifies 5 high-confidence targets (9MMG, 8VQV, 9J4N, 9LKU, 9E9Q) and appropriately flags 2 borderline cases (9MME and 9J4O) as low confidence due to genuine prediction uncertainty.\n\nhttps://www.kaggle.com/code/fernandosr85/hybrid-confidence-scoring-system" }, "source": "meta" }