@shujun717 could you help clarify some of these things for me? Much appreciated.
\nLet's take an example, say target 1HLX_A, whose sequence is 20-residue long: GGGAUAACUUCGGUUGUCCC
\nFor each of these 20 residues we happen to have 3 coordinates as training labels (rounded below for brevity):
\n| \n | \n | x | \ny | \nz | \n
|---|---|---|---|---|
| 1 | \nG | \n-10.127 | \n-6.587 | \n-7.376 | \n
| 2 | \nG | \n-9.419 | \n-6.704 | \n-1.318 | \n
| 3 | \nG | \n-6.191 | \n-6.382 | \n3.539 | \n
| 4 | \nA | \n-1.379 | \n-5.931 | \n5.581 | \n
| … | \n… | \n… | \n… | \n… | \n
| 20 | \nC | \n-7.853 | \n-13.855 | \n-0.582 | \n
*Looking at the evaluation metric equation, for this target 1HLX_A we have L_ref = 20 residues, and we need to submit five coordinate triplets for each of these 20 residues (for the test set, of course).
\nMy questions (start from (8) to avoid number confusion):
\n(8) Is my interpretation above correct?
\n(9) What is L_align in this case?
\n(10) For each residue, why are we predicting five triplets even though the ground truth seems to consist of only one triplet?
\nLet's pretend the test set has M = 40 targets in total, comprising of say N = 837 residues in grand total.
\n\n\n… your final score will be the average of best-of-5 TM-scores of all targets.
\n
(11) By \"best-of-5\" do you mean that for each of the 837 residues, the evaluation engine will select the one out of the five predictions that is the best, and simply ignore the other four predictions? If so, then this selection process is completely independent across all residues, even for those residues that belong to the same target RNA, right? Or, do all residues that belong to the same target have to have the same selection (same selected one out of the five)?
\n(12) By \"average of … all targets\" do you mean averaged over each result from (11) of all 837 residues (which might or might not be \"fair\" as a target might have more residues than another target), or a stratified/weighted average so that each of the 40 targets has equal contribution?
\n* Could someone please tell me how to add a line break after a table, and how to subscript font? The markdown documentation doesn't help. Much thanks.
@shujun717 could you help clarify some of these things for me? Much appreciated.
\nLet's take an example, say target 1HLX_A, whose sequence is 20-residue long: GGGAUAACUUCGGUUGUCCC
\nFor each of these 20 residues we happen to have 3 coordinates as training labels (rounded below for brevity):
\n| \n | \n | x | \ny | \nz | \n
|---|---|---|---|---|
| 1 | \nG | \n-10.127 | \n-6.587 | \n-7.376 | \n
| 2 | \nG | \n-9.419 | \n-6.704 | \n-1.318 | \n
| 3 | \nG | \n-6.191 | \n-6.382 | \n3.539 | \n
| 4 | \nA | \n-1.379 | \n-5.931 | \n5.581 | \n
| … | \n… | \n… | \n… | \n… | \n
| 20 | \nC | \n-7.853 | \n-13.855 | \n-0.582 | \n
*Looking at the evaluation metric equation, for this target 1HLX_A we have L_ref = 20 residues, and we need to submit five coordinate triplets for each of these 20 residues (for the test set, of course).
\nMy questions (start from (8) to avoid number confusion):
\n(8) Is my interpretation above correct?
\n(9) What is L_align in this case?
\n(10) For each residue, why are we predicting five triplets even though the ground truth seems to consist of only one triplet?
\nLet's pretend the test set has M = 40 targets in total, comprising of say N = 837 residues in grand total.
\n\n\n… your final score will be the average of best-of-5 TM-scores of all targets.
\n
(11) By \"best-of-5\" do you mean that for each of the 837 residues, the evaluation engine will select the one out of the five predictions that is the best, and simply ignore the other four predictions? If so, then this selection process is completely independent across all residues, even for those residues that belong to the same target RNA, right? Or, do all residues that belong to the same target have to have the same selection (same selected one out of the five)?
\n(12) By \"average of … all targets\" do you mean averaged over each result from (11) of all 837 residues (which might or might not be \"fair\" as a target might have more residues than another target), or a stratified/weighted average so that each of the 40 targets has equal contribution?
\n* Could someone please tell me how to add a line break after a table, and how to subscript font? The markdown documentation doesn't help. Much thanks.
Good question. I know the question is not for me, but I also struggled to understand the TM-score calculation. I made a notebook with explanation some things. It's still in progress. Will be much better soon :)
\nhttps://www.kaggle.com/code/igorbashko/tm-score-explanation-for-non-biologists
\nL_align is number of overlapped residues between predicted and ground truth RNA molecules. It is coming from 3d space optimization part of TM-score. We don't see it in the formula but it's essential part of the calculation.
", "rawMarkdown": "Good question. I know the question is not for me, but I also struggled to understand the TM-score calculation. I made a notebook with explanation some things. It's still in progress. Will be much better soon :)\n \nhttps://www.kaggle.com/code/igorbashko/tm-score-explanation-for-non-biologists\n\nL_align is number of overlapped residues between predicted and ground truth RNA molecules. It is coming from 3d space optimization part of TM-score. We don't see it in the formula but it's essential part of the calculation.", "votes": null }, { "id": "3142114", "postDate": "03/06/2025 05:33:25", "content": "Here’s a simple starting point with train test features.
\nWill be granulizing the USalign TM=score metric to multiple tree based ML algos using the starter features in the following notebook which already look promising:
\nhttps://www.kaggle.com/code/jazivxt/tilt-or-fold
", "rawMarkdown": "Here’s a simple starting point with train test features.\n\nWill be granulizing the USalign TM=score metric to multiple tree based ML algos using the starter features in the following notebook which already look promising:\n\nhttps://www.kaggle.com/code/jazivxt/tilt-or-fold", "votes": null }, { "id": "3143026", "postDate": "03/06/2025 21:10:34", "content": "@igorbashko Thank you for your reply. Based on your reply, do you think my interpretation below is correct? (For the 1HLX_A example)
\nOnly some subset S of the 20 residues are aligned. So, 0 <= L_align = |S| <= 20. Only the TM-scores of the residues in S will be calculated, aggregated, and divided by L_ref = 20 to form my score for target 1HLX_A. We don't know what S is, but of course we're welcome to estimate what S is if we want/need to.
\nAnd a direct consequence of above is, whatever predictions I make for those residues not in S will not influence my final score in any way, right?
", "rawMarkdown": "igorbashko Thank you for your reply. Based on your reply, do you think my interpretation below is correct? (For the 1HLX_A example)\n\nOnly some subset S of the 20 residues are aligned. So, 0 <= L_align = |S| <= 20. Only the TM-scores of the residues in S will be calculated, aggregated, and divided by L_ref = 20 to form my score for target 1HLX_A. We don't know what S is, but of course we're welcome to estimate what S is if we want/need to.\n\nAnd a direct consequence of above is, whatever predictions I make for those residues not in S will **not** influence my final score in any way, right?", "votes": null }, { "id": "3143054", "postDate": "03/06/2025 22:23:00", "content": "Hi @revealer,
\nI'll try to add where I can….definitely not an expert here as I don't possess a lot of domain knowledge for this competition, but I believe your interpretations are correct. (#8).
\nFor # 10, I think it's just to give you more chance per sequence, and some of the sequences have multiple structures. One of the things noted in the host's discussion here states:
\n\n\nFor RNA's with multiple structures, are there better ways to generate 5 predictions than just different random seeds?
\n
So these five sets are a deliberate goal of this competition as well.
\nIn terms of linebreak after a table, does the following work?
\n| col 1 | \ncol2 | \n
|---|---|
| a | \nb | \n
Worked! (add linespace after last row, then markdown br, then another linespace…then you should be good)
", "rawMarkdown": "Hi @revealer,\n\nI'll try to add where I can....definitely not an expert here as I don't possess a lot of domain knowledge for this competition, but I believe your interpretations are correct. (#8).\n\nFor # 10, I think it's just to give you more chance per sequence, and some of the sequences have multiple structures. One of the things noted in the host's discussion [here](https://www.kaggle.com/competitions/stanford-rna-3d-folding/discussion/565064) states:\n\n> For RNA's with multiple structures, are there better ways to generate 5 predictions than just different random seeds?\n\nSo these five sets are a deliberate goal of this competition as well.\n\nIn terms of linebreak after a table, does the following work?\n\n\n|col 1|col2|\n|---|---|\n|a|b|\n\nHi @revealer. Sorry for late response. You might have already found the answer, but I hope my comment still will be helpful. As I researched TM-score I figured out that it was developed to compare molecules with different and slightly different RNA and see how similar they are. I guess it comes from biology that 2 RNAs. say 1HLX_A which has residues sequence AGGUH and 1HLX_B which has residues sequence AGGUHGGU can be considered from the same specie even if they have different length. If the majority of the residues(AGGUH) are perfectly aligned with each other and they have high TM-score(above 0.5) this is the same molecule. There is even such thing that we can get different TM-score depending on which molecule we compare to. 1HLX_A to 1HLX_B will have not the same score as 1HLX_B to 1HLX_A because as normalization L_ref will be different and in second case L_ref=L_align. But in the score of this competition we compare the same molecule and the same number of residues always. We are given them as the train labels. Are we ? The only thing which affects the TM-score is residues coordinates which we are predicting from them machine learning algorithm(Neural network or whatever it will be). Now, back to your question. Your statement about that aligned number of residues which is taken into account and divided by L_ref is correct, but we don't need to estimate the S. In the score of this competition S is the length of RNA sequence and L_align=L_ref in all cases. If some residues are not perfectly aligned because of poor predictions coordinates from machine learning model than yes, they will affect the TM-score. It will be lower because L_align<L_ref in this case.
", "rawMarkdown": "Hi @revealer. Sorry for late response. You might have already found the answer, but I hope my comment still will be helpful. As I researched TM-score I figured out that it was developed to compare molecules with different and slightly different RNA and see how similar they are. I guess it comes from biology that 2 RNAs. say 1HLX_A which has residues sequence AGGUH and 1HLX_B which has residues sequence AGGUHGGU can be considered from the same specie even if they have different length. If the majority of the residues(AGGUH) are perfectly aligned with each other and they have high TM-score(above 0.5) this is the same molecule. There is even such thing that we can get different TM-score depending on which molecule we compare to. 1HLX_A to 1HLX_B will have not the same score as 1HLX_B to 1HLX_A because as normalization L_ref will be different and in second case L_ref=L_align. But in the score of this competition we compare the same molecule and the same number of residues always. We are given them as the train labels. Are we ? The only thing which affects the TM-score is residues coordinates which we are predicting from them machine learning algorithm(Neural network or whatever it will be). Now, back to your question. Your statement about that aligned number of residues which is taken into account and divided by L_ref is correct, but we don't need to estimate the S. In the score of this competition S is the length of RNA sequence and L_align=L_ref in all cases. If some residues are not perfectly aligned because of poor predictions coordinates from machine learning model than yes, they will affect the TM-score. It will be lower because L_alignThanks everyone for trying to help.
\nBelow is my updated cartoonish understanding of this competition. Feel free to correct me if anything is still wrong. Let's still consider the example of 20-residue long 1HLX_A:
\n(30) We predict L_ref = 20 triplets: one (x, y, z) for each of the 20 residues.
\n(31) Host puts these 20 triplets into an engine called US-align.
\n(32) US-align does some rotation/translation/magic to best align our predicted 20 triplets to the ground truth.
\n(33) During (32), some of our 20 triplets will end up aligning with each of their corresponding ground truth coordinates pretty well, but the remaining triplets will not. L_align is simply the number of those triplets that end up aligning pretty well. Naturally, we want our L_align to be as close to 20 as possible. And if we're good, of course L_align could be 20. However, at all times we can't know what L_align is as we don't have the ground truth.
\n(34) Our TM-score (for 1HLX_A) is computed from (33) and according to the evaluation equation. Ignore the \"max\" in the equation for now, though.
\n(35) We repeat the above (30) - (34) for five times, each time possibly with a different 20-triplet prediction resulting in a possibly different TM-score. The host will pick the best (this is what the \"max\" is for) of these five TM-scores and call it our TM-score verdict for target 1HLX_A.
\n(36) We repeat (35) for all the targets, and the average will be our final score.
", "rawMarkdown": "Thanks everyone for trying to help.\n\nBelow is my updated cartoonish understanding of this competition. Feel free to correct me if anything is still wrong. Let's still consider the example of 20-residue long 1HLX_A:\n\n(30) We predict L_ref = 20 triplets: one (x, y, z) for each of the 20 residues.\n\n(31) Host puts these 20 triplets into an engine called US-align.\n\n(32) US-align does some rotation/translation/magic to best align our predicted 20 triplets to the ground truth.\n\n(33) During (32), some of our 20 triplets will end up aligning with each of their corresponding ground truth coordinates pretty well, but the remaining triplets will not. L_align is simply the number of those triplets that end up aligning pretty well. Naturally, we want our L_align to be as close to 20 as possible. And if we're good, of course L_align could be 20. However, at all times we can't know what L_align is as we don't have the ground truth.\n\n(34) Our TM-score (for 1HLX_A) is computed from (33) and according to the evaluation equation. Ignore the \"max\" in the equation for now, though.\n\n(35) We repeat the above (30) - (34) for five times, each time possibly with a different 20-triplet prediction resulting in a possibly different TM-score. The host will pick the best (this is what the \"max\" is for) of these five TM-scores and call it our TM-score verdict for target 1HLX_A.\n\n(36) We repeat (35) for all the targets, and the average will be our final score.", "votes": null }, { "id": "3147340", "postDate": "03/12/2025 00:18:19", "content": "\"your final score will be the average of best-of-5 TM-scores of all targets.\"
\nyou should find your answer in the kaggle discussion post.
\nthe metric code is given by the host:
\nhttps://www.kaggle.com/code/metric/ribonanza-tm-score/notebook
Hi @chrisk321
\nalso got questions… I created this NB, maybe it works for you as well?
\nhttps://www.kaggle.com/code/dantheshark/rna-3d-folding-understand-data
\nbest, dan
@dantheshark , danke! There's some concepts in there I will incorporate into my work…thanks.
", "rawMarkdown": "dantheshark , danke! There's some concepts in there I will incorporate into my work...thanks.", "votes": null } ], "comments": [ { "id": 3142106, "author_name": "igorbashko", "author_url": "", "post_date": "03/06/2025 05:12:55", "content": "Good question. I know the question is not for me, but I also struggled to understand the TM-score calculation. I made a notebook with explanation some things. It's still in progress. Will be much better soon :)
\nhttps://www.kaggle.com/code/igorbashko/tm-score-explanation-for-non-biologists
\nL_align is number of overlapped residues between predicted and ground truth RNA molecules. It is coming from 3d space optimization part of TM-score. We don't see it in the formula but it's essential part of the calculation.
", "votes": null, "replies": [ { "id": 3143026, "author_name": "revealer", "author_url": "", "post_date": "03/06/2025 21:10:34", "content": "@igorbashko Thank you for your reply. Based on your reply, do you think my interpretation below is correct? (For the 1HLX_A example)
\nOnly some subset S of the 20 residues are aligned. So, 0 <= L_align = |S| <= 20. Only the TM-scores of the residues in S will be calculated, aggregated, and divided by L_ref = 20 to form my score for target 1HLX_A. We don't know what S is, but of course we're welcome to estimate what S is if we want/need to.
\nAnd a direct consequence of above is, whatever predictions I make for those residues not in S will not influence my final score in any way, right?
", "votes": null, "replies": [ { "id": 3146167, "author_name": "igorbashko", "author_url": "", "post_date": "03/10/2025 15:30:44", "content": "Hi @revealer. Sorry for late response. You might have already found the answer, but I hope my comment still will be helpful. As I researched TM-score I figured out that it was developed to compare molecules with different and slightly different RNA and see how similar they are. I guess it comes from biology that 2 RNAs. say 1HLX_A which has residues sequence AGGUH and 1HLX_B which has residues sequence AGGUHGGU can be considered from the same specie even if they have different length. If the majority of the residues(AGGUH) are perfectly aligned with each other and they have high TM-score(above 0.5) this is the same molecule. There is even such thing that we can get different TM-score depending on which molecule we compare to. 1HLX_A to 1HLX_B will have not the same score as 1HLX_B to 1HLX_A because as normalization L_ref will be different and in second case L_ref=L_align. But in the score of this competition we compare the same molecule and the same number of residues always. We are given them as the train labels. Are we ? The only thing which affects the TM-score is residues coordinates which we are predicting from them machine learning algorithm(Neural network or whatever it will be). Now, back to your question. Your statement about that aligned number of residues which is taken into account and divided by L_ref is correct, but we don't need to estimate the S. In the score of this competition S is the length of RNA sequence and L_align=L_ref in all cases. If some residues are not perfectly aligned because of poor predictions coordinates from machine learning model than yes, they will affect the TM-score. It will be lower because L_align<L_ref in this case.
", "votes": null, "replies": [] }, { "id": 3146172, "author_name": "igorbashko", "author_url": "", "post_date": "03/10/2025 15:33:23", "content": "And yes. Sequence for 1HLX_A and 1HLX_B I created on the fly. Don't consider it as a valid example which you provided. I just was lazy to write down all the 20 residues :)
", "votes": null, "replies": [] } ] } ] }, { "id": 3142114, "author_name": "jazivxt", "author_url": "", "post_date": "03/06/2025 05:33:25", "content": "Here’s a simple starting point with train test features.
\nWill be granulizing the USalign TM=score metric to multiple tree based ML algos using the starter features in the following notebook which already look promising:
\nhttps://www.kaggle.com/code/jazivxt/tilt-or-fold
", "votes": null, "replies": [] }, { "id": 3143054, "author_name": "chrisk321", "author_url": "", "post_date": "03/06/2025 22:23:00", "content": "Hi @revealer,
\nI'll try to add where I can….definitely not an expert here as I don't possess a lot of domain knowledge for this competition, but I believe your interpretations are correct. (#8).
\nFor # 10, I think it's just to give you more chance per sequence, and some of the sequences have multiple structures. One of the things noted in the host's discussion here states:
\n\n\nFor RNA's with multiple structures, are there better ways to generate 5 predictions than just different random seeds?
\n
So these five sets are a deliberate goal of this competition as well.
\nIn terms of linebreak after a table, does the following work?
\n| col 1 | \ncol2 | \n
|---|---|
| a | \nb | \n
Worked! (add linespace after last row, then markdown br, then another linespace…then you should be good)
", "votes": null, "replies": [] }, { "id": 3147261, "author_name": "revealer", "author_url": "", "post_date": "03/11/2025 20:29:54", "content": "Thanks everyone for trying to help.
\nBelow is my updated cartoonish understanding of this competition. Feel free to correct me if anything is still wrong. Let's still consider the example of 20-residue long 1HLX_A:
\n(30) We predict L_ref = 20 triplets: one (x, y, z) for each of the 20 residues.
\n(31) Host puts these 20 triplets into an engine called US-align.
\n(32) US-align does some rotation/translation/magic to best align our predicted 20 triplets to the ground truth.
\n(33) During (32), some of our 20 triplets will end up aligning with each of their corresponding ground truth coordinates pretty well, but the remaining triplets will not. L_align is simply the number of those triplets that end up aligning pretty well. Naturally, we want our L_align to be as close to 20 as possible. And if we're good, of course L_align could be 20. However, at all times we can't know what L_align is as we don't have the ground truth.
\n(34) Our TM-score (for 1HLX_A) is computed from (33) and according to the evaluation equation. Ignore the \"max\" in the equation for now, though.
\n(35) We repeat the above (30) - (34) for five times, each time possibly with a different 20-triplet prediction resulting in a possibly different TM-score. The host will pick the best (this is what the \"max\" is for) of these five TM-scores and call it our TM-score verdict for target 1HLX_A.
\n(36) We repeat (35) for all the targets, and the average will be our final score.
", "votes": null, "replies": [] }, { "id": 3147340, "author_name": "hengck23", "author_url": "", "post_date": "03/12/2025 00:18:19", "content": "\"your final score will be the average of best-of-5 TM-scores of all targets.\"
\nyou should find your answer in the kaggle discussion post.
\nthe metric code is given by the host:
\nhttps://www.kaggle.com/code/metric/ribonanza-tm-score/notebook
Hi @chrisk321
\nalso got questions… I created this NB, maybe it works for you as well?
\nhttps://www.kaggle.com/code/dantheshark/rna-3d-folding-understand-data
\nbest, dan
@dantheshark , danke! There's some concepts in there I will incorporate into my work…thanks.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "3141513": "shujun717 could you help clarify some of these things for me? Much appreciated.\n\nLet's take an example, say target 1HLX_A, whose sequence is 20-residue long: GGGAUAACUUCGGUUGUCCC\n\nFor each of these 20 residues we happen to have 3 coordinates as training labels (rounded below for brevity):\n\n| | | x | y | z |\n| -- | -- |\n| 1 | G | -10.127 | -6.587 | -7.376 |\n| 2 | G | -9.419 | -6.704 | -1.318 |\n| 3 | G | -6.191 | -6.382 | 3.539 |\n| 4 | A | -1.379 | -5.931 | 5.581 |\n| ... | ... | ... | ... | ... |\n| 20 | C | -7.853 | -13.855 | -0.582 | \n\n*Looking at the evaluation metric equation, for this target 1HLX_A we have L_ref = 20 residues, and we need to submit five coordinate triplets for each of these 20 residues (for the test set, of course).\n\nMy questions (start from (8) to avoid number confusion):\n\n(8) Is my interpretation above correct?\n\n(9) What is L_align in this case?\n\n(10) For each residue, why are we predicting five triplets even though the ground truth seems to consist of only one triplet?\n\nLet's pretend the test set has M = 40 targets in total, comprising of say N = 837 residues in **grand total**.\n\n>... your final score will be the average of best-of-5 TM-scores of all targets.\n\n(11) By \"best-of-5\" do you mean that for each of the 837 residues, the evaluation engine will *select* the one out of the five predictions that is the best, and simply ignore the other four predictions? If so, then this *selection* process is completely independent across all residues, even for those residues that belong to the same target RNA, right? Or, do all residues that belong to the same target have to have the same *selection* (same *selected* one out of the five)?\n\n(12) By \"average of ... all targets\" do you mean averaged over each result from (11) of all 837 residues (which might or might not be \"fair\" as a target might have more residues than another target), or a stratified/weighted average so that each of the 40 targets has equal contribution?\n\n`*` Could someone please tell me how to add a line break after a table, and how to subscript font? The markdown documentation doesn't help. Much thanks.", "3142106": "Good question. I know the question is not for me, but I also struggled to understand the TM-score calculation. I made a notebook with explanation some things. It's still in progress. Will be much better soon :)\n \nhttps://www.kaggle.com/code/igorbashko/tm-score-explanation-for-non-biologists\n\nL_align is number of overlapped residues between predicted and ground truth RNA molecules. It is coming from 3d space optimization part of TM-score. We don't see it in the formula but it's essential part of the calculation.", "3142114": "Here’s a simple starting point with train test features.\n\nWill be granulizing the USalign TM=score metric to multiple tree based ML algos using the starter features in the following notebook which already look promising:\n\nhttps://www.kaggle.com/code/jazivxt/tilt-or-fold", "3143026": "igorbashko Thank you for your reply. Based on your reply, do you think my interpretation below is correct? (For the 1HLX_A example)\n\nOnly some subset S of the 20 residues are aligned. So, 0 <= L_align = |S| <= 20. Only the TM-scores of the residues in S will be calculated, aggregated, and divided by L_ref = 20 to form my score for target 1HLX_A. We don't know what S is, but of course we're welcome to estimate what S is if we want/need to.\n\nAnd a direct consequence of above is, whatever predictions I make for those residues not in S will **not** influence my final score in any way, right?", "3143054": "Hi @revealer,\n\nI'll try to add where I can....definitely not an expert here as I don't possess a lot of domain knowledge for this competition, but I believe your interpretations are correct. (#8).\n\nFor # 10, I think it's just to give you more chance per sequence, and some of the sequences have multiple structures. One of the things noted in the host's discussion [here](https://www.kaggle.com/competitions/stanford-rna-3d-folding/discussion/565064) states:\n\n> For RNA's with multiple structures, are there better ways to generate 5 predictions than just different random seeds?\n\nSo these five sets are a deliberate goal of this competition as well.\n\nIn terms of linebreak after a table, does the following work?\n\n\n|col 1|col2|\n|---|---|\n|a|b|\n\n