![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F5091071a931c5eec3b4849717a11c612%2FSelection_052.png?generation=1741999960906042&alt=media\")
Each step will be described in detail below. The focus is on how to check whether your usage is correct. Fortunately, there is an online web server for Rhofold+ that allows verification against our local results
\nhere are the summary:
\n1)
\ni think kaggle MAS already in a3m format. the extension fasta should be a3m
\n2) predict using Rhofold+ using web server and local
\n3) compare the results
\n4) other important note, eg effects of num of msa used, conformations using different msa input
\n3) generate your own msa
Each step will be described in detail below. The focus is on how to check whether your usage is correct. Fortunately, there is an online web server for Rhofold+ that allows verification against our local results
\nhere are the summary:
\n1)
\ni think kaggle MAS already in a3m format. the extension fasta should be a3m
\n2) predict using Rhofold+ using web server and local
\n3) compare the results
\n4) other important note, eg effects of num of msa used, conformations using different msa input
\n3) generate your own msa
good news.
\nwe have another benchmark paper, their a3m files is provided and can be downloaded
Systematic benchmarking of deep-learning methods for tertiary RNA structure prediction
\nhttps://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1012715#sec035
\nhttps://github.com/akashbahai/rna_benchmarking/tree/main/msa/r1108
but this is for rhofold repo (and not rhofold+ repo)
\nbut both rhofold and rhofold+ repot seem to point to the same pretrain model
rhofold+:
\nhttps://github.com/ml4bio/RhoFold?tab=readme-ov-file
rhofold:
\nhttps://github.com/Dharmogata/RhoFold?tab=readme-ov-file
let's \"algorithmically debug\" our implementation.
\nwe use kaggle validation csv (which is casp15).
first we need reference results and we will use rhofold+ paper (supplementary material)
\nhttps://www.nature.com/articles/s41592-024-02487-0#additional-information
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2Fc85b403de5259c2a67a340fff86577fb%2FSelection_051.png?generation=1741996415341156&alt=media\")
if you run my example notebook[1] with max num of MAS you should be getting
\n[1] https://www.kaggle.com/code/hengck23/demo-for-rhofold-plus-with-kaggle-msa
relax\n\n R1107 num_msa= len(seq)= df.shape=(, ) seq = GGGGGCCACA...\n .\n\n R1108 (, ) GGGGGCCACA...\n .\n\n R1116 (, ) CGCCCGGAUA...\n .\n\n R1117v2 (, ) UUGGGUUCCC...\n .\n\n R1149 (, ) GGACACGAGU...\n .\n\n R1156 (, ) GGAGCAUCGU...\n .\nour results are definitely lower!
\nlet's investigate why and improve on yt!
… to be updated …
\nsome quite observation:
\ni can call rhofold server to do MSA.
\nthen we can compare the quality of MSA. will report results later
We got RhoFold LB score for MSAs created from rnacentral.fasta a bit lower than for precomputed MSAs from organizers (0.208 vs 0.214).
", "rawMarkdown": "We got RhoFold LB score for MSAs created from rnacentral.fasta a bit lower than for precomputed MSAs from organizers (0.208 vs 0.214)." }, { "id": 3150242, "postDate": "2025-03-15T08:11:54.673Z", "content": "base on my experiments and \"guess\", for natural rna of length less than 200, SOTA like rhofold+ etc should be able to get tm score 0.40 to 0.50 for casp15/16 and rna puzzle test samples. We are still not getting the implementation correct. the devils are still hiding
", "rawMarkdown": "base on my experiments and \"guess\", for natural rna of length less than 200, SOTA like rhofold+ etc should be able to get tm score 0.40 to 0.50 for casp15/16 and rna puzzle test samples. We are still not getting the implementation correct. the devils are still hiding", "votes": 1 }, { "id": 3150255, "postDate": "2025-03-15T08:43:28.150Z", "content": "Validation scores for RhoFold are not so impressive:
\n\n\n0.256 for single_seq_pred
\n0.295 for rnacentral.fasta MSAs
\n0.286 for kaggle MSAs
\n
validation_sequences.csv (and test_sequences.csv which is the same file) contains all but one sequences <500 nt, so RhoFold inference runs without any issues despite it was trained on shorter RNAs.
", "rawMarkdown": "Validation scores for RhoFold are not so impressive:\n>### 0.256 for single_seq_pred\n### 0.295 for rnacentral.fasta MSAs\n### 0.286 for kaggle MSAs\n\nvalidation_sequences.csv (and test_sequences.csv which is the same file) contains all but one sequences <500 nt, so RhoFold inference runs without any issues despite it was trained on shorter RNAs." }, { "id": 3150345, "postDate": "2025-03-15T11:37:02.120Z", "content": "validation_sequences is casp15. if you check rhofold papers and other benchmark papers, it can get higher
\n(you need to split score for natural and sythetic rna)
the rhoflod code only provide blastN for construct MSA.
\nBut the paper actually uses rMSA and infernal which gives better results
@hengck23 I hope it's possible to make rMSA & infernal work on kaggle!
at least one have to prove that that is the cuase of performance.
\nif the results are really good, we can ask the host to provide precomputed mas again
follow this notebook link to use rhofold+ on kaggle data for 1EIY_C (5 MSA)
\nhttps://www.kaggle.com/code/hengck23/demo-for-rhofold-plus-with-kaggle-msa
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2Fac2110a027e0fea1e42531f8f170febb%2FSelection_049.png?generation=1741941642841589&alt=media\")
render on https://molstar.org/viewer/
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2Fcd2c3b43c290599db28d456144dee661%2FSelection_048.png?generation=1741941699751604&alt=media\")
In the method section of the Rhofold+ paper, the authors state: \"By default, the top 256 MSAs are chosen as input features for predicting the standard structure, which we refer to as standard RhoFold+. RhoFold+ (TopK) refers to the optimal model selected from K different models generated using distinct sampled MSAs.\"
\nDoes this imply that the CASP15 results were selected from multiple attempts? Additionally, Supplementary Table 6 indicates that the MSA depths used in their experiments are deeper than those available in the Kaggle dataset.
", "rawMarkdown": "In the method section of the Rhofold+ paper, the authors state: \"By default, the top 256 MSAs are chosen as input features for predicting the standard structure, which we refer to as standard RhoFold+. RhoFold+ (TopK) refers to the optimal model selected from K different models generated using distinct sampled MSAs.\"\n\nDoes this imply that the CASP15 results were selected from multiple attempts? Additionally, Supplementary Table 6 indicates that the MSA depths used in their experiments are deeper than those available in the Kaggle dataset. ", "replies": [ { "id": 3157217, "postDate": "2025-03-23T06:08:43.727Z", "content": "Msa is not unique process. There are many ways Eg blastn rmsa mmseq etc. not sure what is exactly use. U will need to experiment
", "rawMarkdown": "Msa is not unique process. There are many ways Eg blastn rmsa mmseq etc. not sure what is exactly use. U will need to experiment \n", "replies": [ { "id": 3157232, "postDate": "2025-03-23T06:32:34.887Z", "content": "Yes, I plan to use Infernal to generate deeper MSAs and evaluate whether this can lead Rhofold+ to a higher score.
", "rawMarkdown": "Yes, I plan to use Infernal to generate deeper MSAs and evaluate whether this can lead Rhofold+ to a higher score." }, { "id": 3157394, "postDate": "2025-03-23T11:55:59.290Z", "content": "don't over focus on one method rhofold. try af3, nufold, trRosettaRNA, etc … maybe other authors describe their mas pipline in more details and their results are more reproducible.
", "rawMarkdown": "don't over focus on one method rhofold. try af3, nufold, trRosettaRNA, etc ... maybe other authors describe their mas pipline in more details and their results are more reproducible." }, { "id": 3157425, "postDate": "2025-03-23T12:29:44.873Z", "content": "\n\nevaluate whether this can lead Rhofold+ to a higher score.
\n
It depends on huge datasets:
\n\n", "rawMarkdown": ">evaluate whether this can lead Rhofold+ to a higher score.\n\nIt depends on huge datasets:\n\n>To support MSA construction, 3 sequence databases (RNAcentral, Rfam, and nt) totaling about 900GB need to be downloaded." }, { "id": 3157448, "postDate": "2025-03-23T12:56:43.673Z", "rawMarkdown": "", "isDeleted": true }, { "id": 3157449, "postDate": "2025-03-23T12:57:08.160Z", "content": "To support MSA construction, 3 sequence databases (RNAcentral, Rfam, and nt) totaling about 900GB need to be downloaded.
\n
I agree that my current storage capacity is insufficient for handling such large datasets. I should likely focus on other methods.
", "rawMarkdown": "I agree that my current storage capacity is insufficient for handling such large datasets. I should likely focus on other methods." }, { "id": 3157457, "postDate": "2025-03-23T13:08:32.023Z", "content": "Alternatively, try online msa server. Key is to get pipeline correct. Can get data to local later.
\nTo submit with online server, submit offline predicted xyz, see my other notebook
", "rawMarkdown": "Alternatively, try online msa server. Key is to get pipeline correct. Can get data to local later.\n\nTo submit with online server, submit offline predicted xyz, see my other notebook" } ] } ] }, { "id": 3150555, "postDate": "2025-03-15T16:00:55.957Z", "content": "![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2Ff1b85c9b31b9245c6da161fee2f18001%2FSelection_058.png?generation=1742054324943370&alt=media\")
i need to implement or find online server for blastN, rMSA, infernal to verify if the paper better results is due to better MSA search.
It is interesting to note that server results is better then local for same input.
\n(did the server use a better pretrain model? or is it due to different GPU, python package version)
great work!
", "rawMarkdown": "great work! ", "votes": 1 } ], "comments": [ { "id": 3150011, "author_name": "hengck23", "author_url": "", "post_date": "2025-03-15T00:53:39.307000", "content": "good news.
\nwe have another benchmark paper, their a3m files is provided and can be downloaded
Systematic benchmarking of deep-learning methods for tertiary RNA structure prediction
\nhttps://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1012715#sec035
\nhttps://github.com/akashbahai/rna_benchmarking/tree/main/msa/r1108
but this is for rhofold repo (and not rhofold+ repo)
\nbut both rhofold and rhofold+ repot seem to point to the same pretrain model
rhofold+:
\nhttps://github.com/ml4bio/RhoFold?tab=readme-ov-file
rhofold:
\nhttps://github.com/Dharmogata/RhoFold?tab=readme-ov-file
let's \"algorithmically debug\" our implementation.
\nwe use kaggle validation csv (which is casp15).
first we need reference results and we will use rhofold+ paper (supplementary material)
\nhttps://www.nature.com/articles/s41592-024-02487-0#additional-information
if you run my example notebook[1] with max num of MAS you should be getting
\n[1] https://www.kaggle.com/code/hengck23/demo-for-rhofold-plus-with-kaggle-msa
relax\n\n R1107 num_msa= len(seq)= df.shape=(, ) seq = GGGGGCCACA...\n .\n\n R1108 (, ) GGGGGCCACA...\n .\n\n R1116 (, ) CGCCCGGAUA...\n .\n\n R1117v2 (, ) UUGGGUUCCC...\n .\n\n R1149 (, ) GGACACGAGU...\n .\n\n R1156 (, ) GGAGCAUCGU...\n .\nour results are definitely lower!
\nlet's investigate why and improve on yt!
… to be updated …
\nsome quite observation:
\ni can call rhofold server to do MSA.
\nthen we can compare the quality of MSA. will report results later
We got RhoFold LB score for MSAs created from rnacentral.fasta a bit lower than for precomputed MSAs from organizers (0.208 vs 0.214).
", "votes": 0, "replies": [] }, { "id": 3150242, "author_name": "hengck23", "author_url": "", "post_date": "2025-03-15T08:11:54.673000", "content": "base on my experiments and \"guess\", for natural rna of length less than 200, SOTA like rhofold+ etc should be able to get tm score 0.40 to 0.50 for casp15/16 and rna puzzle test samples. We are still not getting the implementation correct. the devils are still hiding
", "votes": 1, "replies": [] }, { "id": 3150255, "author_name": "Ogurtsov", "author_url": "", "post_date": "2025-03-15T08:43:28.150000", "content": "Validation scores for RhoFold are not so impressive:
\n\n\n0.256 for single_seq_pred
\n0.295 for rnacentral.fasta MSAs
\n0.286 for kaggle MSAs
\n
validation_sequences.csv (and test_sequences.csv which is the same file) contains all but one sequences <500 nt, so RhoFold inference runs without any issues despite it was trained on shorter RNAs.
", "votes": 0, "replies": [] }, { "id": 3150345, "author_name": "hengck23", "author_url": "", "post_date": "2025-03-15T11:37:02.120000", "content": "validation_sequences is casp15. if you check rhofold papers and other benchmark papers, it can get higher
\n(you need to split score for natural and sythetic rna)
the rhoflod code only provide blastN for construct MSA.
\nBut the paper actually uses rMSA and infernal which gives better results
@hengck23 I hope it's possible to make rMSA & infernal work on kaggle!
at least one have to prove that that is the cuase of performance.
\nif the results are really good, we can ask the host to provide precomputed mas again
follow this notebook link to use rhofold+ on kaggle data for 1EIY_C (5 MSA)
\nhttps://www.kaggle.com/code/hengck23/demo-for-rhofold-plus-with-kaggle-msa
render on https://molstar.org/viewer/
\n
In the method section of the Rhofold+ paper, the authors state: \"By default, the top 256 MSAs are chosen as input features for predicting the standard structure, which we refer to as standard RhoFold+. RhoFold+ (TopK) refers to the optimal model selected from K different models generated using distinct sampled MSAs.\"
\nDoes this imply that the CASP15 results were selected from multiple attempts? Additionally, Supplementary Table 6 indicates that the MSA depths used in their experiments are deeper than those available in the Kaggle dataset.
", "votes": 0, "replies": [ { "id": 3157217, "author_name": "hengck23", "author_url": "", "post_date": "2025-03-23T06:08:43.727000", "content": "Msa is not unique process. There are many ways Eg blastn rmsa mmseq etc. not sure what is exactly use. U will need to experiment
", "votes": 0, "replies": [ { "id": 3157232, "author_name": "Bianco Chiu", "author_url": "", "post_date": "2025-03-23T06:32:34.887000", "content": "Yes, I plan to use Infernal to generate deeper MSAs and evaluate whether this can lead Rhofold+ to a higher score.
", "votes": 0, "replies": [] }, { "id": 3157394, "author_name": "hengck23", "author_url": "", "post_date": "2025-03-23T11:55:59.290000", "content": "don't over focus on one method rhofold. try af3, nufold, trRosettaRNA, etc … maybe other authors describe their mas pipline in more details and their results are more reproducible.
", "votes": 0, "replies": [] }, { "id": 3157425, "author_name": "Timmy Juicehouse", "author_url": "", "post_date": "2025-03-23T12:29:44.873000", "content": "\n\nevaluate whether this can lead Rhofold+ to a higher score.
\n
It depends on huge datasets:
\n\n", "votes": 0, "replies": [] }, { "id": 3157448, "author_name": "", "author_url": "", "post_date": "2025-03-23T12:56:43.673000", "content": "", "votes": 0, "replies": [] }, { "id": 3157449, "author_name": "Bianco Chiu", "author_url": "", "post_date": "2025-03-23T12:57:08.160000", "content": "To support MSA construction, 3 sequence databases (RNAcentral, Rfam, and nt) totaling about 900GB need to be downloaded.
\n
I agree that my current storage capacity is insufficient for handling such large datasets. I should likely focus on other methods.
", "votes": 0, "replies": [] }, { "id": 3157457, "author_name": "hengck23", "author_url": "", "post_date": "2025-03-23T13:08:32.023000", "content": "Alternatively, try online msa server. Key is to get pipeline correct. Can get data to local later.
\nTo submit with online server, submit offline predicted xyz, see my other notebook
", "votes": 0, "replies": [] } ] } ] }, { "id": 3150555, "author_name": "hengck23", "author_url": "", "post_date": "2025-03-15T16:00:55.957000", "content": "
i need to implement or find online server for blastN, rMSA, infernal to verify if the paper better results is due to better MSA search.
It is interesting to note that server results is better then local for same input.
\n(did the server use a better pretrain model? or is it due to different GPU, python package version)
great work!
", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "3149222": "Each step will be described in detail below. The focus is on how to check whether your usage is correct. Fortunately, there is an online web server for Rhofold+ that allows verification against our local results\n\nhere are the summary:\n1) ~~understand alignment format: fasta and a3m. Convert to a3m.~~ \ni think kaggle MAS already in a3m format. the extension fasta should be a3m\n2) predict using Rhofold+ using web server and local\n3) compare the results\n4) other important note, eg effects of num of msa used, conformations using different msa input\n3) generate your own msa", "3150011": "good news.\nwe have another benchmark paper, their a3m files is provided and can be downloaded\n\nSystematic benchmarking of deep-learning methods for tertiary RNA structure prediction\nhttps://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1012715#sec035\nhttps://github.com/akashbahai/rna_benchmarking/tree/main/msa/r1108\n\n\n\nbut this is for rhofold repo (and not rhofold+ repo)\nbut both rhofold and rhofold+ repot seem to point to the same pretrain model\n\n\nrhofold+:\nhttps://github.com/ml4bio/RhoFold?tab=readme-ov-file\n\nrhofold:\nhttps://github.com/Dharmogata/RhoFold?tab=readme-ov-file", "3149991": "let's \"algorithmically debug\" our implementation.\nwe use kaggle validation csv (which is casp15).\n\nfirst we need reference results and we will use rhofold+ paper (supplementary material)\nhttps://www.nature.com/articles/s41592-024-02487-0#additional-information\n\n\n\n\nif you run my example notebook[1] with max num of MAS you should be getting\n[1] https://www.kaggle.com/code/hengck23/demo-for-rhofold-plus-with-kaggle-msa \n\n```\nno relax\n\n0 R1107 num_msa=21 len(seq)=69 df.shape=(69, 6) seq = GGGGGCCACA...\ntm_score_unrelax 0.32113\n\n1 R1108 26 69 (69, 6) GGGGGCCACA...\ntm_score_unrelax 0.41223\n\n2 R1116 68 157 (157, 6) CGCCCGGAUA...\ntm_score_unrelax 0.52747\n\n3 R1117v2 1 30 (30, 6) UUGGGUUCCC...\ntm_score_unrelax 0.20945\n\n8 R1149 9 124 (124, 6) GGACACGAGU...\ntm_score_unrelax 0.25613\n\n9 R1156 11 135 (135, 6) GGAGCAUCGU...\ntm_score_unrelax 0.34896\n```\n\nour results are definitely lower!\nlet's investigate why and improve on yt!\n\n... to be updated ...\n\nsome quite observation:\n- is TM-score in paper for backbone or all atoms (i.e. paper TM score is the same as ours?)\n- I also note that the num of MSA is different from the paper. luckliy the paper provide details of max match seq ...)\n- I think the comment from a3m header like \">000|DF967972.1/3772384-3772312/1-73 [subseq from] Longilinea arvoryzae DNA, scaffold: Larv_scaffold_1, strain: KOME-1.\" would help. maybe we can use a LLM model for text analysis for this ", "3149461": "##2) predict using Rhofold+ using local##\n\nfollow this notebook link to use rhofold+ on kaggle data for 1EIY_C (5 MSA)\nhttps://www.kaggle.com/code/hengck23/demo-for-rhofold-plus-with-kaggle-msa\n\n\n\nrender on https://molstar.org/viewer/\n", "3157211": "In the method section of the Rhofold+ paper, the authors state: \"By default, the top 256 MSAs are chosen as input features for predicting the standard structure, which we refer to as standard RhoFold+. RhoFold+ (TopK) refers to the optimal model selected from K different models generated using distinct sampled MSAs.\"\n\nDoes this imply that the CASP15 results were selected from multiple attempts? Additionally, Supplementary Table 6 indicates that the MSA depths used in their experiments are deeper than those available in the Kaggle dataset. ", "3150555": "\n\n1. i need to implement or find online server for blastN, rMSA, infernal to verify if the paper better results is due to better MSA search.\n\n2. It is interesting to note that server results is better then local for same input.\n(did the server use a better pretrain model? or is it due to different GPU, python package version)", "3149224": "", "3207667": "great work! " } }