\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F10147343%2Fffb52fe84fe77af3ef6ea2f2ccaaaec7%2FStanfordRSNA_SubmissionSel.jpg?generation=1758772761681643&alt=media\")
The competition has closed, and the Private Leaderboard has posted!
\nThe final leaderboard
\nWe are pleased to announce that the three leading teams from the training phase of the competition are also the top three teams in the final rankings. John @jaejohn, odat @odat1248, and team EIGEN @yekim102 @ryankim99 @ouiqdmw are our leaderboard prize winners!
\nImportantly, the final Private Leaderboard involves 20 targets with no overlap with the Public Leaderboard used in the training phase.
\nIn fact many of the Private Leaderboard targets are not yet described publicly but have been made available specially to this Kaggle competition by experimental collaborators to permit rigorous blind evaluation.
\nThe lack of major shake-up at the top of the leaderboard supports the good generalization of the top models.
\nIn addition, the score gap between these top Kaggle notebooks and leading prior algorithms, AlphaFold 3 and the trRosettaRNA server (top automated method in CASP16), both posted to the leaderboard as baselines, support the generality of their solutions.
\nRemarkably, we have evidence that the top teams used different strategies to achieve high modeling accuracy, which you can read about in their forum posts. Indeed different teams did well on different targets. Stay tuned for more as hosts collaboratively synthesize these insights into single models that we expect to go into wide use throughout science and medicine.
\nWe look forward to seeing the write-ups of all Kaggle teams, and we are grateful already for interactions with many teams over the summer.
\nComparison to human experts
\nA major goal of this competition was to uncover automated strategies that might be competitive with human experts at RNA structure prediction.
\nWe have been fortunate to collaborate with the RNA-Puzzles organizers to present 10 of the 20 Private Leaderboard targets as confidential RNA-Puzzles to the prediction community over the recent months.
\nAnd we are grateful to Shi-Jie Chen’s Vfold group at U. Missouri –the top team by a clear margin in last year’s CASP16 RNA category– for sharing their RNA-Puzzle blind predictions as a baseline.
\nOver the 10 special targets shared between the Kaggle Private Leaderboard and RNA-Puzzles, the mean best-of-5 TM-score (C1’) of the Vfold human expert team is essentially tied with the top Kaggle notebooks:
\nvfold_human_experts (corrected from 0.4592)john EIGEN odat AF3_rMSA trRosettaRNA server In particular, john's notebook and vfold_human_experts turn out to be statistically indistinguishable in their scores. So while the top Kaggle notebook is not ‘super-human’, it certainly appears human-competitive. This is a milestone in the RNA structure prediction problem!
What’s next?\nThe RNA structure prediction problem isn’t solved yet. For the typical distribution of targets presented in CASP, RNA-Puzzles, and this Kaggle challenge, the maximum achievable TM-score should be about 0.8. We’re still not quite at 0.6. But we’ve certainly made progress in automated RNA structure prediction, and the codes from this Kaggle challenge will likely form the springboard for achieving the solution.
\nThe hosts are focusing on consolidating the Kaggle results into a scientific paper, synthesizing the Kaggle models into a single, easily useable model, and making these models easy to use. We’re shooting to get this paper out by early 2026 so that the broader computer science and biology community can build on this research by the time of the CASP17 challenge in April 2026.
\nUpdate: the preprint has been posted on bioRxiv!
\nFor participants who have become excited about the RNA structure prediction problem and want to develop new models, please consider entering the ongoing RNA-puzzles challenges as well as next year’s CASP17. If you have interest in joining the Das lab to work on the problem full-time, positions are available both through Stanford University and through the new AI@HHMI initiative. Hope to see you in the RNA world!
\nCongratulations to the leaderboard winners and to all who participated!
\nFor the hosts\nRhiju Das @rhijudas
\nWith much thanks to:
\nP.S. No one claimed the Early Sharing Prize!
\nP.P.S. The Public Leaderboard scores shifted slightly compared to the training phase. That's due to some stochasticity in the re-run and also because scores are now shown only for the two notebooks the team chose as their submissions rather than best over all submitted notebooks (which was the case in the training phase). In any case, focus on the Private Leaderboard!
\n[Edited: Jan 5, 2026 to provide link to preprint, and update Vfold score after final evaluation.]
", "messages": [ { "id": 3293776, "postDate": "2025-09-24T16:11:34.503Z", "content": "The competition has closed, and the Private Leaderboard has posted!
\nThe final leaderboard
\nWe are pleased to announce that the three leading teams from the training phase of the competition are also the top three teams in the final rankings. John @jaejohn, odat @odat1248, and team EIGEN @yekim102 @ryankim99 @ouiqdmw are our leaderboard prize winners!
\nImportantly, the final Private Leaderboard involves 20 targets with no overlap with the Public Leaderboard used in the training phase.
\nIn fact many of the Private Leaderboard targets are not yet described publicly but have been made available specially to this Kaggle competition by experimental collaborators to permit rigorous blind evaluation.
\nThe lack of major shake-up at the top of the leaderboard supports the good generalization of the top models.
\nIn addition, the score gap between these top Kaggle notebooks and leading prior algorithms, AlphaFold 3 and the trRosettaRNA server (top automated method in CASP16), both posted to the leaderboard as baselines, support the generality of their solutions.
\nRemarkably, we have evidence that the top teams used different strategies to achieve high modeling accuracy, which you can read about in their forum posts. Indeed different teams did well on different targets. Stay tuned for more as hosts collaboratively synthesize these insights into single models that we expect to go into wide use throughout science and medicine.
\nWe look forward to seeing the write-ups of all Kaggle teams, and we are grateful already for interactions with many teams over the summer.
\nComparison to human experts
\nA major goal of this competition was to uncover automated strategies that might be competitive with human experts at RNA structure prediction.
\nWe have been fortunate to collaborate with the RNA-Puzzles organizers to present 10 of the 20 Private Leaderboard targets as confidential RNA-Puzzles to the prediction community over the recent months.
\nAnd we are grateful to Shi-Jie Chen’s Vfold group at U. Missouri –the top team by a clear margin in last year’s CASP16 RNA category– for sharing their RNA-Puzzle blind predictions as a baseline.
\nOver the 10 special targets shared between the Kaggle Private Leaderboard and RNA-Puzzles, the mean best-of-5 TM-score (C1’) of the Vfold human expert team is essentially tied with the top Kaggle notebooks:
\nvfold_human_experts (corrected from 0.4592)john EIGEN odat AF3_rMSA trRosettaRNA server In particular, john's notebook and vfold_human_experts turn out to be statistically indistinguishable in their scores. So while the top Kaggle notebook is not ‘super-human’, it certainly appears human-competitive. This is a milestone in the RNA structure prediction problem!
What’s next?\nThe RNA structure prediction problem isn’t solved yet. For the typical distribution of targets presented in CASP, RNA-Puzzles, and this Kaggle challenge, the maximum achievable TM-score should be about 0.8. We’re still not quite at 0.6. But we’ve certainly made progress in automated RNA structure prediction, and the codes from this Kaggle challenge will likely form the springboard for achieving the solution.
\nThe hosts are focusing on consolidating the Kaggle results into a scientific paper, synthesizing the Kaggle models into a single, easily useable model, and making these models easy to use. We’re shooting to get this paper out by early 2026 so that the broader computer science and biology community can build on this research by the time of the CASP17 challenge in April 2026.
\nUpdate: the preprint has been posted on bioRxiv!
\nFor participants who have become excited about the RNA structure prediction problem and want to develop new models, please consider entering the ongoing RNA-puzzles challenges as well as next year’s CASP17. If you have interest in joining the Das lab to work on the problem full-time, positions are available both through Stanford University and through the new AI@HHMI initiative. Hope to see you in the RNA world!
\nCongratulations to the leaderboard winners and to all who participated!
\nFor the hosts\nRhiju Das @rhijudas
\nWith much thanks to:
\nP.S. No one claimed the Early Sharing Prize!
\nP.P.S. The Public Leaderboard scores shifted slightly compared to the training phase. That's due to some stochasticity in the re-run and also because scores are now shown only for the two notebooks the team chose as their submissions rather than best over all submitted notebooks (which was the case in the training phase). In any case, focus on the Private Leaderboard!
\n[Edited: Jan 5, 2026 to provide link to preprint, and update Vfold score after final evaluation.]
", "rawMarkdown": "The competition has closed, and the [Private Leaderboard](https://www.kaggle.com/competitions/stanford-rna-3d-folding/leaderboard?tab=private) has posted! \n\n**The final leaderboard**\n\nWe are pleased to announce that the three leading teams from the training phase of the competition are also the top three teams in the final rankings. John @jaejohn, odat @odat1248, and team EIGEN @yekim102 @ryankim99 @ouiqdmw are our leaderboard prize winners!\n\nImportantly, the final Private Leaderboard involves 20 targets with no overlap with the Public Leaderboard used in the training phase.\n\nIn fact many of the Private Leaderboard targets are not yet described publicly but have been made available specially to this Kaggle competition by experimental collaborators to permit rigorous blind evaluation. \n\nThe lack of major shake-up at the top of the leaderboard supports the good generalization of the top models.\n\nIn addition, the score gap between these top Kaggle notebooks and leading prior algorithms, AlphaFold 3 and the trRosettaRNA server (top automated method in CASP16), both posted to the leaderboard as baselines, support the generality of their solutions.\n\nRemarkably, we have evidence that the top teams used *different* strategies to achieve high modeling accuracy, which you can read about in their forum posts. Indeed different teams did well on different targets. Stay tuned for more as hosts collaboratively synthesize these insights into single models that we expect to go into wide use throughout science and medicine.\n\nWe look forward to seeing the write-ups of all Kaggle teams, and we are grateful already for interactions with many teams over the summer.\n\n**Comparison to human experts**\n\nA major goal of this competition was to uncover automated strategies that might be competitive with human experts at RNA structure prediction. \n\nWe have been fortunate to collaborate with the RNA-Puzzles organizers to present 10 of the 20 Private Leaderboard targets as confidential RNA-Puzzles to the prediction community over the recent months. \n\nAnd we are grateful to Shi-Jie Chen’s [Vfold](https://doi.org/10.1002/prot.26856) group at U. Missouri –the top team by a clear margin in last year’s [CASP16 RNA](https://www.biorxiv.org/content/10.1101/2025.05.06.652459v1) category– for sharing their RNA-Puzzle blind predictions as a baseline.\n\nOver the 10 special targets shared between the Kaggle Private Leaderboard and RNA-Puzzles, the mean best-of-5 TM-score (C1’) of the Vfold human expert team is essentially tied with the top Kaggle notebooks:\n\n- 0.5071 `vfold_human_experts` *(corrected from 0.4592)*\n- 0.4907 `john` \n- 0.4250 `EIGEN` \n- 0.4179 `odat` \n- 0.3951 `AF3_rMSA` \n- 0.3400 `trRosettaRNA server` \n\nIn particular, `john`'s notebook and `vfold_human_experts` turn out to be statistically indistinguishable in their scores. So while the top Kaggle notebook is not ‘super-human’, it certainly appears human-competitive. This is a milestone in the RNA structure prediction problem!\n\n**What’s next?**\nThe RNA structure prediction problem isn’t solved yet. For the typical distribution of targets presented in CASP, RNA-Puzzles, and this Kaggle challenge, the maximum achievable TM-score should be about 0.8. We’re still not quite at 0.6. But we’ve certainly made progress in automated RNA structure prediction, and the codes from this Kaggle challenge will likely form the springboard for achieving the solution.\n\nThe hosts are focusing on consolidating the Kaggle results into a scientific paper, synthesizing the Kaggle models into a single, easily useable model, and making these models easy to use. We’re shooting to get this paper out by early 2026 so that the broader computer science and biology community can build on this research by the time of the CASP17 challenge in April 2026. \n\n**Update: the preprint has been posted on [bioRxiv](https://www.biorxiv.org/content/10.64898/2025.12.30.696949v1)!**\n\nFor participants who have become excited about the RNA structure prediction problem and want to develop new models, please consider entering the ongoing [RNA-puzzles](https://www.rnapuzzles.org/) challenges as well as next year’s [CASP17](https://predictioncenter.org/). If you have interest in joining the [Das lab](https://daslab.stanford.edu/) to work on the problem full-time, positions are available both through Stanford University and through the new [AI@HHMI](https://ai.hhmi.org/) initiative. Hope to see you in the RNA world!\n\n*Congratulations to the leaderboard winners and to all who participated!*\n\nFor the hosts\nRhiju Das @rhijudas\n\nWith much thanks to:\n* Kaggle co-hosts @shujun717 and, from Nvidia, @chrismunley@theoviel @youhanlee\n* Members of the Das lab providing new structures and analysis, @rkretsch and @alissahummer\n* CASP16 organizers, @andriyca and John Moult\n* RNA-Puzzles organizers, Chichau Miao and Eric Westhof\n* The global RNA structural biology community providing blind prediction targets, esp. the groups of Koirala (UMBC), Chiu (Stanford), Marcia (Upsalla), Zhang (USTC), Su (Sichuan), Huang/Taylor (Case Western), and Piccirilli (U. Chicago)\n* The RNA RFdiffusion/MPNN team at the Institute of Protein Design, Andrew Favor @andrewfavor, Andrew Kubaney, and David Baker\n* Shi-Jie Chen and the Vfold team for providing Vfold_human_expert baseline predictions.\n* Public leaderboard leaders who contributed code and model descriptions to hosts over summer 2025: @alanchen1115 @johnhsu7 @aypyaypy @yekim102 @ryankim99 @ouiqdmw @alejopaullier @zoushuxian @biancochiu @hiranorm @jaejohn @lihaoweicvch @nguyenhoa @hengck23 @arunodhayan @raulenrique @odat1248 @koooeo @shunkuraishi @sweetyheehee\n\nP.S. No one claimed the Early Sharing Prize! \n\nP.P.S. The Public Leaderboard scores shifted slightly compared to the training phase. That's due to some stochasticity in the re-run and also because scores are now shown only for the two notebooks the team chose as their submissions rather than best over all submitted notebooks (which was the case in the training phase). In any case, focus on the Private Leaderboard!\n\n*[Edited: Jan 5, 2026 to provide link to preprint, and update Vfold score after final evaluation.]*\n\n", "votes": 23 }, { "id": 3293951, "postDate": "2025-09-25T04:00:05.167Z", "content": "The notebooks selected are not scored however, last two notebooks submitted were scored. Can you please clarify.
\n
We have the same issue. Selected submits: v.8 and v.12 of inference notebook. Scored on LB: v.11 and v.12
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F682731%2Fa3773b2f895737f3c7b3ce9e47d2f45a%2Frna.JPG?generation=1758779104690786&alt=media\")
Weird, we'll have a look at it.
", "rawMarkdown": "Weird, we'll have a look at it." }, { "id": 3294091, "postDate": "2025-09-25T10:36:40.927Z", "content": "The scored notebooks are the duplicates of the selected notebooks rather than the latest ones. However, I don't know why there is discrepancy in the public scores.
", "rawMarkdown": "The scored notebooks are the duplicates of the selected notebooks rather than the latest ones. However, I don't know why there is discrepancy in the public scores.", "replies": [ { "id": 3294108, "postDate": "2025-09-25T11:14:56.377Z", "content": "And the difference is too big.
", "rawMarkdown": "And the difference is too big." }, { "id": 3294378, "postDate": "2025-09-26T03:06:50.150Z", "content": "For most notebooks, the rerun scores are about 0.01-0.02 different in Public LB. For some notebooks that are fully deterministic, the Public LB scores don't change at all in the rerun.
\nCan you check though? Try to rerun your selected notebooks and let us know how much the score varies.
", "rawMarkdown": "For most notebooks, the rerun scores are about 0.01-0.02 different in Public LB. For some notebooks that are fully deterministic, the Public LB scores don't change at all in the rerun. \n\nCan you check though? Try to rerun your selected notebooks and let us know how much the score varies." }, { "id": 3294381, "postDate": "2025-09-26T03:18:53.417Z", "rawMarkdown": "", "isDeleted": true }, { "id": 3294382, "postDate": "2025-09-26T03:20:14.517Z", "rawMarkdown": "", "isDeleted": true }, { "id": 3294384, "postDate": "2025-09-26T03:24:07.440Z", "content": "@rhijudas For me, when the ranking was closed in May, my score was 0.41107. To prevent OOM, I also chose a truncated version with a score of 0.40729. The scores after the public ranking was refreshed were:
\n0.41107 - 0.37069 The difference is 0.04038
\n0.40729 - 0.37145 The difference is 0.03584
\nThere are two very strange things here. First, the score dropped drastically. Second, the score of the truncated version is higher than the normal version. So I doubt it's a problem with the notebook, because I have already set the random seed. I just resubmitted my notebook and I'm not sure what happened.
@rhijudas How can I retry the process? The submission has been blocked.
", "rawMarkdown": "@rhijudas How can I retry the process? The submission has been blocked." }, { "id": 3294389, "postDate": "2025-09-26T03:35:39.620Z", "content": "Looking into it. Thanks to you and @alexxanderlarko for reporting!
", "rawMarkdown": "Looking into it. Thanks to you and @alexxanderlarko for reporting!", "votes": 2 }, { "id": 3294829, "postDate": "2025-09-26T21:30:42.027Z", "content": "Please try now!
", "rawMarkdown": "Please try now!" } ] }, { "id": 3296726, "postDate": "2025-10-01T14:15:48.500Z", "content": "As the competition has already closed, we investigated these reports with the immediate goal of determining if there was any meaningful change to the competition leaderboard, and found none. We're continuing to look into the underlying cause, but are happy to report the leaderboard for this competition won't require an update. Thanks for surfacing!
", "rawMarkdown": "As the competition has already closed, we investigated these reports with the immediate goal of determining if there was any meaningful change to the competition leaderboard, and found none. We're continuing to look into the underlying cause, but are happy to report the leaderboard for this competition won't require an update. Thanks for surfacing!" } ] }, { "id": 3293811, "postDate": "2025-09-24T17:28:23.983Z", "content": "@rhijudas, one of my notebooks listed a ![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2101769%2Faa2b6b372a74a5bfa3b712464420a4f0%2FScreenshot%20from%202025-09-24%2019-25-55.png?generation=1758734868038829&alt=media\")
score of 0.80, i don't understand what this score
I'm also very curious about it.
", "rawMarkdown": "I'm also very curious about it." }, { "id": 3293818, "postDate": "2025-09-24T17:46:28.723Z", "content": "In the middle of the training phase, there was a 'data refresh', where we removed targets from both Public and Private Leaderboards whose structures had become available publicly. The apparent Private Leaderboard scores of >0.8 you're seeing are from before this data refresh.
", "rawMarkdown": "In the middle of the training phase, there was a 'data refresh', where we removed targets from both Public and Private Leaderboards whose structures had become available publicly. The apparent Private Leaderboard scores of >0.8 you're seeing are from before this data refresh.", "votes": 1 } ] }, { "id": 3293802, "postDate": "2025-09-24T16:56:02.723Z", "content": "To all the organizers,
\nThis competition has been truly educational and has sparked my deep interest in RNA structure prediction.
\nI plan to continue developing models for structure prediction over time, and I’m very grateful for this opportunity.
\nThank you very much!
One more thing: we have re-opened this Kaggle competition to allow download of the competition data and accept submissions.
\nWe will not be able to publicly release the hidden test sequences or solutions until the experimental labs put out their papers, likely in 2026. However, you can use the scoring feedback (which will be private to you and not posted on the leaderboard) for your own hill climbing campaigns.
\nIf you do that, be careful that the strongest apparent boost you’ll get will be from ‘leakage’!
\nThat is, if you use training data that has become public after this competition’s training phase, you will see your scores go up, but the performance is unlikely to generalize to future targets.
\nYou may want to set May 29, 2025 as a temporal cutoff during your model development to avoid unpleasant surprises in future blind tests in RNA-Puzzles or CASP17.
", "rawMarkdown": "One more thing: we have re-opened this Kaggle competition to allow download of the competition data and accept submissions. \n\nWe will not be able to publicly release the hidden test sequences or solutions until the experimental labs put out their papers, likely in 2026. However, you can use the scoring feedback (which will be private to you and not posted on the leaderboard) for your own hill climbing campaigns.\n\nIf you do that, be careful that the strongest apparent boost you’ll get will be from ‘leakage’!\n\nThat is, if you use training data that has become public after this competition’s training phase, you will see your scores go up, but the performance is unlikely to generalize to future targets. \n\nYou may want to set **May 29, 2025** as a temporal cutoff during your model development to avoid unpleasant surprises in future blind tests in RNA-Puzzles or CASP17. ", "votes": 1, "replies": [ { "id": 3293782, "postDate": "2025-09-24T16:19:39.943Z", "content": "I'm not quite clear. Is the public ranking the same as before? Is it just due to randomness? Why has my ranking dropped by more than three hundred places on the public ranking
", "rawMarkdown": "I'm not quite clear. Is the public ranking the same as before? Is it just due to randomness? Why has my ranking dropped by more than three hundred places on the public ranking" }, { "id": 3293790, "postDate": "2025-09-24T16:32:25.117Z", "content": "My previous ranking on the public ranking list was 32nd, and now it's 344th on the public ranking list and 718th on the private ranking list. Even more astonishingly, the score of the notebook that I truncated the sequence length to prevent OOM is even better than that of the one that didn't
", "rawMarkdown": "My previous ranking on the public ranking list was 32nd, and now it's 344th on the public ranking list and 718th on the private ranking list. Even more astonishingly, the score of the notebook that I truncated the sequence length to prevent OOM is even better than that of the one that didn't", "replies": [ { "id": 3293820, "postDate": "2025-09-24T17:55:02.460Z", "content": "Thanks for pointing this out! There were a large number of submissions that were very similar in score in the middle of the leaderboard, and their relative rankings were very sensitive to noise. Only at the top of the leaderboard were rankings largely preserved. I've edited the post to clarify.
", "rawMarkdown": "Thanks for pointing this out! There were a large number of submissions that were very similar in score in the middle of the leaderboard, and their relative rankings were very sensitive to noise. Only at the top of the leaderboard were rankings largely preserved. I've edited the post to clarify." }, { "id": 3293897, "postDate": "2025-09-25T00:33:02.823Z", "content": "Thank you for your answer. This is the biggest shock I have ever experienced. Haha. Since I only used training data and no external data, I thought this would not happen.😂
", "rawMarkdown": "Thank you for your answer. This is the biggest shock I have ever experienced. Haha. Since I only used training data and no external data, I thought this would not happen.😂" } ] } ] }, { "id": 3294379, "postDate": "2025-09-26T03:11:45.323Z", "content": "Hello!
\nThe submit button isn't working.
\nCan this be fixed?
Thanks for reporting. Please try now!
", "rawMarkdown": "Thanks for reporting. Please try now!", "votes": 1 } ] }, { "id": 3294027, "postDate": "2025-09-25T08:14:40.740Z", "content": "I find the large variability between the public and private leaderboards in some cases very striking, with some cases showing a considerable increase or decrease in score between the two. This leads me to a question: as far as I know, some models perform considerably better on short sequences than on long ones, or vice versa. Do you know if the proportion of sequence lengths in the private dataset is similar to that in the public dataset? Thanks
", "rawMarkdown": "I find the large variability between the public and private leaderboards in some cases very striking, with some cases showing a considerable increase or decrease in score between the two. This leads me to a question: as far as I know, some models perform considerably better on short sequences than on long ones, or vice versa. Do you know if the proportion of sequence lengths in the private dataset is similar to that in the public dataset? Thanks", "replies": [ { "id": 3294386, "postDate": "2025-09-26T03:33:12.130Z", "content": "The length distributions are quite similar. For example, the fraction of sequences with lengths > 200 matches exactly between the Public and Private datasets.
\nBut the molecules' functions and folds are quite diverse within and between the datasets. There appears to be a lot of variability between Kaggle models, specifically which molecules each model does well on.
\nScientifically, this is good news as even better models might be achieved by synthesizing insights from the diverse models -- we encourage everyone to make their code and associated datasets public and to publish their writeups as Discussion posts!
", "rawMarkdown": "The length distributions are quite similar. For example, the fraction of sequences with lengths > 200 matches *exactly* between the Public and Private datasets. \n\nBut the molecules' functions and folds are quite diverse within and between the datasets. There appears to be a lot of variability between Kaggle models, specifically which molecules each model does well on. \n\nScientifically, this is good news as even better models might be achieved by synthesizing insights from the diverse models -- we encourage everyone to make their code and associated datasets public and to publish their writeups as Discussion posts!" } ] }, { "id": 3293905, "postDate": "2025-09-25T01:41:06.617Z", "content": "Will there still be opportunities to submit in the future, just to test out our on models for \"fun\"? Or if not, is there a good public script somewhere that will evaluate TM scores in the same manner as you use for your own scoring, so that we can compare where we stand to this competition (even if not on the same test sequences)? I noticed some RNA structures released in the PDB in the past few months that I haven't looked at, so I have my own sequences for blind prediction, but I want to make sure I have the correct evaluation metric (in particular for the C1'-based definition of TM-Score, as opposed to an all-atom score, to test out coarse grained/lattice type models).
\nThanks for linking RNA-Puzzles. Is this a rolling competition or is it a once-a-year thing?
", "rawMarkdown": "Will there still be opportunities to submit in the future, just to test out our on models for \"fun\"? Or if not, is there a good public script somewhere that will evaluate TM scores in the same manner as you use for your own scoring, so that we can compare where we stand to this competition (even if not on the same test sequences)? I noticed some RNA structures released in the PDB in the past few months that I haven't looked at, so I have my own sequences for blind prediction, but I want to make sure I have the correct evaluation metric (in particular for the C1'-based definition of TM-Score, as opposed to an all-atom score, to test out coarse grained/lattice type models).\n\nThanks for linking RNA-Puzzles. Is this a rolling competition or is it a once-a-year thing?\n\n", "replies": [ { "id": 3293923, "postDate": "2025-09-25T02:52:49.237Z", "content": "Thanks for asking the questions!
\nThanks! For me, there is a \"button\" that says \"Late Submission\" but it isn't actually a functioning button, there is no change to my cursor when hovering over it. Maybe I would need to have a notebook ready for submission before it would actually become available?
", "rawMarkdown": "Thanks! For me, there is a \"button\" that says \"Late Submission\" but it isn't actually a functioning button, there is no change to my cursor when hovering over it. Maybe I would need to have a notebook ready for submission before it would actually become available?" } ] } ] }, { "id": 3293885, "postDate": "2025-09-24T22:47:41.923Z", "content": "Thanks very much to Das Lab for hosting this great competition—it was truly exciting. Although the result was quite unexpected for me, and I regret missing out on the gold medal, I will carefully study the test set of the private leaderboard. I look forward to seeing the solutions from the top-ranked kagglers in the future.
", "rawMarkdown": "Thanks very much to Das Lab for hosting this great competition—it was truly exciting. Although the result was quite unexpected for me, and I regret missing out on the gold medal, I will carefully study the test set of the private leaderboard. I look forward to seeing the solutions from the top-ranked kagglers in the future.", "replies": [ { "id": 3293886, "postDate": "2025-09-24T22:53:01.653Z", "content": "Why is the ranking fluctuation so significant after the gold medal zone on the private leaderboard of this competition? It's completely beyond my expectations.
", "rawMarkdown": "Why is the ranking fluctuation so significant after the gold medal zone on the private leaderboard of this competition? It's completely beyond my expectations.", "replies": [ { "id": 3293952, "postDate": "2025-09-25T04:16:20.377Z", "content": "Unbelievable😂
", "rawMarkdown": "Unbelievable😂" } ] } ] }, { "id": 3293814, "postDate": "2025-09-24T17:42:55.207Z", "content": "Amazing results! 🎉 The fact that John, odat, and EIGEN maintained top positions even on the Private Leaderboard shows strong generalization. I’d love to learn more about how combining different modeling approaches could improve overall accuracy.
", "rawMarkdown": "Amazing results! 🎉 The fact that John, odat, and EIGEN maintained top positions even on the Private Leaderboard shows strong generalization. I’d love to learn more about how combining different modeling approaches could improve overall accuracy." } ], "comments": [ { "id": 3293951, "author_name": "C R Suthikshn Kumar", "author_url": "", "post_date": "2025-09-25T04:00:05.167000", "content": "The notebooks selected are not scored however, last two notebooks submitted were scored. Can you please clarify.
\n
We have the same issue. Selected submits: v.8 and v.12 of inference notebook. Scored on LB: v.11 and v.12
\n
Weird, we'll have a look at it.
", "votes": 0, "replies": [] }, { "id": 3294091, "author_name": "Pi", "author_url": "", "post_date": "2025-09-25T10:36:40.927000", "content": "The scored notebooks are the duplicates of the selected notebooks rather than the latest ones. However, I don't know why there is discrepancy in the public scores.
", "votes": 0, "replies": [ { "id": 3294108, "author_name": "DaoHe Liu", "author_url": "", "post_date": "2025-09-25T11:14:56.377000", "content": "And the difference is too big.
", "votes": 0, "replies": [] }, { "id": 3294378, "author_name": "Rhiju Das", "author_url": "", "post_date": "2025-09-26T03:06:50.150000", "content": "For most notebooks, the rerun scores are about 0.01-0.02 different in Public LB. For some notebooks that are fully deterministic, the Public LB scores don't change at all in the rerun.
\nCan you check though? Try to rerun your selected notebooks and let us know how much the score varies.
", "votes": 0, "replies": [] }, { "id": 3294381, "author_name": "", "author_url": "", "post_date": "2025-09-26T03:18:53.417000", "content": "", "votes": 0, "replies": [] }, { "id": 3294382, "author_name": "", "author_url": "", "post_date": "2025-09-26T03:20:14.517000", "content": "", "votes": 0, "replies": [] }, { "id": 3294384, "author_name": "DaoHe Liu", "author_url": "", "post_date": "2025-09-26T03:24:07.440000", "content": "@rhijudas For me, when the ranking was closed in May, my score was 0.41107. To prevent OOM, I also chose a truncated version with a score of 0.40729. The scores after the public ranking was refreshed were:
\n0.41107 - 0.37069 The difference is 0.04038
\n0.40729 - 0.37145 The difference is 0.03584
\nThere are two very strange things here. First, the score dropped drastically. Second, the score of the truncated version is higher than the normal version. So I doubt it's a problem with the notebook, because I have already set the random seed. I just resubmitted my notebook and I'm not sure what happened.
@rhijudas How can I retry the process? The submission has been blocked.
", "votes": 0, "replies": [] }, { "id": 3294389, "author_name": "Rhiju Das", "author_url": "", "post_date": "2025-09-26T03:35:39.620000", "content": "Looking into it. Thanks to you and @alexxanderlarko for reporting!
", "votes": 2, "replies": [] }, { "id": 3294829, "author_name": "Rhiju Das", "author_url": "", "post_date": "2025-09-26T21:30:42.027000", "content": "Please try now!
", "votes": 0, "replies": [] } ] }, { "id": 3296726, "author_name": "Addison Howard", "author_url": "", "post_date": "2025-10-01T14:15:48.500000", "content": "As the competition has already closed, we investigated these reports with the immediate goal of determining if there was any meaningful change to the competition leaderboard, and found none. We're continuing to look into the underlying cause, but are happy to report the leaderboard for this competition won't require an update. Thanks for surfacing!
", "votes": 0, "replies": [] } ] }, { "id": 3293811, "author_name": "Arunodhayan", "author_url": "", "post_date": "2025-09-24T17:28:23.983000", "content": "@rhijudas, one of my notebooks listed a score of 0.80, i don't understand what this score
I'm also very curious about it.
", "votes": 0, "replies": [] }, { "id": 3293818, "author_name": "Rhiju Das", "author_url": "", "post_date": "2025-09-24T17:46:28.723000", "content": "In the middle of the training phase, there was a 'data refresh', where we removed targets from both Public and Private Leaderboards whose structures had become available publicly. The apparent Private Leaderboard scores of >0.8 you're seeing are from before this data refresh.
", "votes": 1, "replies": [] } ] }, { "id": 3293802, "author_name": "Shun Kuraishi", "author_url": "", "post_date": "2025-09-24T16:56:02.723000", "content": "To all the organizers,
\nThis competition has been truly educational and has sparked my deep interest in RNA structure prediction.
\nI plan to continue developing models for structure prediction over time, and I’m very grateful for this opportunity.
\nThank you very much!
One more thing: we have re-opened this Kaggle competition to allow download of the competition data and accept submissions.
\nWe will not be able to publicly release the hidden test sequences or solutions until the experimental labs put out their papers, likely in 2026. However, you can use the scoring feedback (which will be private to you and not posted on the leaderboard) for your own hill climbing campaigns.
\nIf you do that, be careful that the strongest apparent boost you’ll get will be from ‘leakage’!
\nThat is, if you use training data that has become public after this competition’s training phase, you will see your scores go up, but the performance is unlikely to generalize to future targets.
\nYou may want to set May 29, 2025 as a temporal cutoff during your model development to avoid unpleasant surprises in future blind tests in RNA-Puzzles or CASP17.
", "votes": 1, "replies": [ { "id": 3293782, "author_name": "DaoHe Liu", "author_url": "", "post_date": "2025-09-24T16:19:39.943000", "content": "I'm not quite clear. Is the public ranking the same as before? Is it just due to randomness? Why has my ranking dropped by more than three hundred places on the public ranking
", "votes": 0, "replies": [] }, { "id": 3293790, "author_name": "DaoHe Liu", "author_url": "", "post_date": "2025-09-24T16:32:25.117000", "content": "My previous ranking on the public ranking list was 32nd, and now it's 344th on the public ranking list and 718th on the private ranking list. Even more astonishingly, the score of the notebook that I truncated the sequence length to prevent OOM is even better than that of the one that didn't
", "votes": 0, "replies": [ { "id": 3293820, "author_name": "Rhiju Das", "author_url": "", "post_date": "2025-09-24T17:55:02.460000", "content": "Thanks for pointing this out! There were a large number of submissions that were very similar in score in the middle of the leaderboard, and their relative rankings were very sensitive to noise. Only at the top of the leaderboard were rankings largely preserved. I've edited the post to clarify.
", "votes": 0, "replies": [] }, { "id": 3293897, "author_name": "DaoHe Liu", "author_url": "", "post_date": "2025-09-25T00:33:02.823000", "content": "Thank you for your answer. This is the biggest shock I have ever experienced. Haha. Since I only used training data and no external data, I thought this would not happen.😂
", "votes": 0, "replies": [] } ] } ] }, { "id": 3294379, "author_name": "Aleksandr Larko", "author_url": "", "post_date": "2025-09-26T03:11:45.323000", "content": "Hello!
\nThe submit button isn't working.
\nCan this be fixed?
Thanks for reporting. Please try now!
", "votes": 1, "replies": [] } ] }, { "id": 3294027, "author_name": "Rem1210", "author_url": "", "post_date": "2025-09-25T08:14:40.740000", "content": "I find the large variability between the public and private leaderboards in some cases very striking, with some cases showing a considerable increase or decrease in score between the two. This leads me to a question: as far as I know, some models perform considerably better on short sequences than on long ones, or vice versa. Do you know if the proportion of sequence lengths in the private dataset is similar to that in the public dataset? Thanks
", "votes": 0, "replies": [ { "id": 3294386, "author_name": "Rhiju Das", "author_url": "", "post_date": "2025-09-26T03:33:12.130000", "content": "The length distributions are quite similar. For example, the fraction of sequences with lengths > 200 matches exactly between the Public and Private datasets.
\nBut the molecules' functions and folds are quite diverse within and between the datasets. There appears to be a lot of variability between Kaggle models, specifically which molecules each model does well on.
\nScientifically, this is good news as even better models might be achieved by synthesizing insights from the diverse models -- we encourage everyone to make their code and associated datasets public and to publish their writeups as Discussion posts!
", "votes": 0, "replies": [] } ] }, { "id": 3293905, "author_name": "Andrew Rosko", "author_url": "", "post_date": "2025-09-25T01:41:06.617000", "content": "Will there still be opportunities to submit in the future, just to test out our on models for \"fun\"? Or if not, is there a good public script somewhere that will evaluate TM scores in the same manner as you use for your own scoring, so that we can compare where we stand to this competition (even if not on the same test sequences)? I noticed some RNA structures released in the PDB in the past few months that I haven't looked at, so I have my own sequences for blind prediction, but I want to make sure I have the correct evaluation metric (in particular for the C1'-based definition of TM-Score, as opposed to an all-atom score, to test out coarse grained/lattice type models).
\nThanks for linking RNA-Puzzles. Is this a rolling competition or is it a once-a-year thing?
", "votes": 0, "replies": [ { "id": 3293923, "author_name": "Rhiju Das", "author_url": "", "post_date": "2025-09-25T02:52:49.237000", "content": "Thanks for asking the questions!
\nThanks! For me, there is a \"button\" that says \"Late Submission\" but it isn't actually a functioning button, there is no change to my cursor when hovering over it. Maybe I would need to have a notebook ready for submission before it would actually become available?
", "votes": 0, "replies": [] } ] } ] }, { "id": 3293885, "author_name": "Timmy Juicehouse", "author_url": "", "post_date": "2025-09-24T22:47:41.923000", "content": "Thanks very much to Das Lab for hosting this great competition—it was truly exciting. Although the result was quite unexpected for me, and I regret missing out on the gold medal, I will carefully study the test set of the private leaderboard. I look forward to seeing the solutions from the top-ranked kagglers in the future.
", "votes": 0, "replies": [ { "id": 3293886, "author_name": "Timmy Juicehouse", "author_url": "", "post_date": "2025-09-24T22:53:01.653000", "content": "Why is the ranking fluctuation so significant after the gold medal zone on the private leaderboard of this competition? It's completely beyond my expectations.
", "votes": 0, "replies": [ { "id": 3293952, "author_name": "DaoHe Liu", "author_url": "", "post_date": "2025-09-25T04:16:20.377000", "content": "Unbelievable😂
", "votes": 0, "replies": [] } ] } ] }, { "id": 3293814, "author_name": "Prince Rajak", "author_url": "", "post_date": "2025-09-24T17:42:55.207000", "content": "Amazing results! 🎉 The fact that John, odat, and EIGEN maintained top positions even on the Private Leaderboard shows strong generalization. I’d love to learn more about how combining different modeling approaches could improve overall accuracy.
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "3293776": "The competition has closed, and the [Private Leaderboard](https://www.kaggle.com/competitions/stanford-rna-3d-folding/leaderboard?tab=private) has posted! \n\n**The final leaderboard**\n\nWe are pleased to announce that the three leading teams from the training phase of the competition are also the top three teams in the final rankings. John @jaejohn, odat @odat1248, and team EIGEN @yekim102 @ryankim99 @ouiqdmw are our leaderboard prize winners!\n\nImportantly, the final Private Leaderboard involves 20 targets with no overlap with the Public Leaderboard used in the training phase.\n\nIn fact many of the Private Leaderboard targets are not yet described publicly but have been made available specially to this Kaggle competition by experimental collaborators to permit rigorous blind evaluation. \n\nThe lack of major shake-up at the top of the leaderboard supports the good generalization of the top models.\n\nIn addition, the score gap between these top Kaggle notebooks and leading prior algorithms, AlphaFold 3 and the trRosettaRNA server (top automated method in CASP16), both posted to the leaderboard as baselines, support the generality of their solutions.\n\nRemarkably, we have evidence that the top teams used *different* strategies to achieve high modeling accuracy, which you can read about in their forum posts. Indeed different teams did well on different targets. Stay tuned for more as hosts collaboratively synthesize these insights into single models that we expect to go into wide use throughout science and medicine.\n\nWe look forward to seeing the write-ups of all Kaggle teams, and we are grateful already for interactions with many teams over the summer.\n\n**Comparison to human experts**\n\nA major goal of this competition was to uncover automated strategies that might be competitive with human experts at RNA structure prediction. \n\nWe have been fortunate to collaborate with the RNA-Puzzles organizers to present 10 of the 20 Private Leaderboard targets as confidential RNA-Puzzles to the prediction community over the recent months. \n\nAnd we are grateful to Shi-Jie Chen’s [Vfold](https://doi.org/10.1002/prot.26856) group at U. Missouri –the top team by a clear margin in last year’s [CASP16 RNA](https://www.biorxiv.org/content/10.1101/2025.05.06.652459v1) category– for sharing their RNA-Puzzle blind predictions as a baseline.\n\nOver the 10 special targets shared between the Kaggle Private Leaderboard and RNA-Puzzles, the mean best-of-5 TM-score (C1’) of the Vfold human expert team is essentially tied with the top Kaggle notebooks:\n\n- 0.5071 `vfold_human_experts` *(corrected from 0.4592)*\n- 0.4907 `john` \n- 0.4250 `EIGEN` \n- 0.4179 `odat` \n- 0.3951 `AF3_rMSA` \n- 0.3400 `trRosettaRNA server` \n\nIn particular, `john`'s notebook and `vfold_human_experts` turn out to be statistically indistinguishable in their scores. So while the top Kaggle notebook is not ‘super-human’, it certainly appears human-competitive. This is a milestone in the RNA structure prediction problem!\n\n**What’s next?**\nThe RNA structure prediction problem isn’t solved yet. For the typical distribution of targets presented in CASP, RNA-Puzzles, and this Kaggle challenge, the maximum achievable TM-score should be about 0.8. We’re still not quite at 0.6. But we’ve certainly made progress in automated RNA structure prediction, and the codes from this Kaggle challenge will likely form the springboard for achieving the solution.\n\nThe hosts are focusing on consolidating the Kaggle results into a scientific paper, synthesizing the Kaggle models into a single, easily useable model, and making these models easy to use. We’re shooting to get this paper out by early 2026 so that the broader computer science and biology community can build on this research by the time of the CASP17 challenge in April 2026. \n\n**Update: the preprint has been posted on [bioRxiv](https://www.biorxiv.org/content/10.64898/2025.12.30.696949v1)!**\n\nFor participants who have become excited about the RNA structure prediction problem and want to develop new models, please consider entering the ongoing [RNA-puzzles](https://www.rnapuzzles.org/) challenges as well as next year’s [CASP17](https://predictioncenter.org/). If you have interest in joining the [Das lab](https://daslab.stanford.edu/) to work on the problem full-time, positions are available both through Stanford University and through the new [AI@HHMI](https://ai.hhmi.org/) initiative. Hope to see you in the RNA world!\n\n*Congratulations to the leaderboard winners and to all who participated!*\n\nFor the hosts\nRhiju Das @rhijudas\n\nWith much thanks to:\n* Kaggle co-hosts @shujun717 and, from Nvidia, @chrismunley@theoviel @youhanlee\n* Members of the Das lab providing new structures and analysis, @rkretsch and @alissahummer\n* CASP16 organizers, @andriyca and John Moult\n* RNA-Puzzles organizers, Chichau Miao and Eric Westhof\n* The global RNA structural biology community providing blind prediction targets, esp. the groups of Koirala (UMBC), Chiu (Stanford), Marcia (Upsalla), Zhang (USTC), Su (Sichuan), Huang/Taylor (Case Western), and Piccirilli (U. Chicago)\n* The RNA RFdiffusion/MPNN team at the Institute of Protein Design, Andrew Favor @andrewfavor, Andrew Kubaney, and David Baker\n* Shi-Jie Chen and the Vfold team for providing Vfold_human_expert baseline predictions.\n* Public leaderboard leaders who contributed code and model descriptions to hosts over summer 2025: @alanchen1115 @johnhsu7 @aypyaypy @yekim102 @ryankim99 @ouiqdmw @alejopaullier @zoushuxian @biancochiu @hiranorm @jaejohn @lihaoweicvch @nguyenhoa @hengck23 @arunodhayan @raulenrique @odat1248 @koooeo @shunkuraishi @sweetyheehee\n\nP.S. No one claimed the Early Sharing Prize! \n\nP.P.S. The Public Leaderboard scores shifted slightly compared to the training phase. That's due to some stochasticity in the re-run and also because scores are now shown only for the two notebooks the team chose as their submissions rather than best over all submitted notebooks (which was the case in the training phase). In any case, focus on the Private Leaderboard!\n\n*[Edited: Jan 5, 2026 to provide link to preprint, and update Vfold score after final evaluation.]*\n\n", "3293951": "The notebooks selected are not scored however, last two notebooks submitted were scored. Can you please clarify.\n", "3293811": "@rhijudas, one of my notebooks listed a score of 0.80, i don't understand what this score ", "3293802": "To all the organizers,\nThis competition has been truly educational and has sparked my deep interest in RNA structure prediction.\nI plan to continue developing models for structure prediction over time, and I’m very grateful for this opportunity.\nThank you very much!", "3293778": "One more thing: we have re-opened this Kaggle competition to allow download of the competition data and accept submissions. \n\nWe will not be able to publicly release the hidden test sequences or solutions until the experimental labs put out their papers, likely in 2026. However, you can use the scoring feedback (which will be private to you and not posted on the leaderboard) for your own hill climbing campaigns.\n\nIf you do that, be careful that the strongest apparent boost you’ll get will be from ‘leakage’!\n\nThat is, if you use training data that has become public after this competition’s training phase, you will see your scores go up, but the performance is unlikely to generalize to future targets. \n\nYou may want to set **May 29, 2025** as a temporal cutoff during your model development to avoid unpleasant surprises in future blind tests in RNA-Puzzles or CASP17. ", "3294379": "Hello!\nThe submit button isn't working.\nCan this be fixed?", "3294027": "I find the large variability between the public and private leaderboards in some cases very striking, with some cases showing a considerable increase or decrease in score between the two. This leads me to a question: as far as I know, some models perform considerably better on short sequences than on long ones, or vice versa. Do you know if the proportion of sequence lengths in the private dataset is similar to that in the public dataset? Thanks", "3293905": "Will there still be opportunities to submit in the future, just to test out our on models for \"fun\"? Or if not, is there a good public script somewhere that will evaluate TM scores in the same manner as you use for your own scoring, so that we can compare where we stand to this competition (even if not on the same test sequences)? I noticed some RNA structures released in the PDB in the past few months that I haven't looked at, so I have my own sequences for blind prediction, but I want to make sure I have the correct evaluation metric (in particular for the C1'-based definition of TM-Score, as opposed to an all-atom score, to test out coarse grained/lattice type models).\n\nThanks for linking RNA-Puzzles. Is this a rolling competition or is it a once-a-year thing?\n\n", "3293885": "Thanks very much to Das Lab for hosting this great competition—it was truly exciting. Although the result was quite unexpected for me, and I regret missing out on the gold medal, I will carefully study the test set of the private leaderboard. I look forward to seeing the solutions from the top-ranked kagglers in the future.", "3293814": "Amazing results! 🎉 The fact that John, odat, and EIGEN maintained top positions even on the Private Leaderboard shows strong generalization. I’d love to learn more about how combining different modeling approaches could improve overall accuracy." } }