\nIt's showing error 404 after accessing the link.
![\"SS\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F25371292%2F25ea360174dda13e475147a74ad155d1%2FScreenshot%202025-03-22%20155234.png?generation=1742638970630946&alt=media\")
Hi Kagglers,
\nJust a quick note that we have updated the multiple sequence alignment (MSA) files to outputs generated using rMSA (https://github.com/pylelab/rMSA). This method produced deeper MSAs and we used databases with a temporal cutoff more closely aligned with the validation dataset (database versions included below). The rMSA outputs are in the A2M format.
\nWe have retained the MSA/{target_id}.MSA.fasta file naming.
\nrMSA Database Versions/Cutoffs
\n• RNAcentral v20.0 (2022-03-28)
\n• RFam v14.7 (2021-12-09)
\n• NCBI NT (2022-10-03)
[Updated 20250321 - A2M format clarified]
\n[Updated 20250325 - all placeholders have been replaced with rMSA outputs]
Hi Kagglers,
\nJust a quick note that we have updated the multiple sequence alignment (MSA) files to outputs generated using rMSA (https://github.com/pylelab/rMSA). This method produced deeper MSAs and we used databases with a temporal cutoff more closely aligned with the validation dataset (database versions included below). The rMSA outputs are in the A2M format.
\nWe have retained the MSA/{target_id}.MSA.fasta file naming.
\nrMSA Database Versions/Cutoffs
\n• RNAcentral v20.0 (2022-03-28)
\n• RFam v14.7 (2021-12-09)
\n• NCBI NT (2022-10-03)
[Updated 20250321 - A2M format clarified]
\n[Updated 20250325 - all placeholders have been replaced with rMSA outputs]
for the newbies:
\n@alissahummer
\nDo u think u can open source your rmsa script to make the msa?
Our team will be making external data and share with the kaggle community. We will be publishing eg af3 3d prediction. We thinking if we want to publish msa of exeternal data as well
", "rawMarkdown": "@alissahummer \nDo u think u can open source your rmsa script to make the msa?\n\nOur team will be making external data and share with the kaggle community. We will be publishing eg af3 3d prediction. We thinking if we want to publish msa of exeternal data as well ", "votes": 1, "replies": [ { "id": 3156312, "postDate": "2025-03-22T03:12:19.210Z", "content": "Hi @hengck23, I am reposting my response to Zacchaeus' question below (posted around the same time as yours):
\nWe used the rMSA tool with default parameters and database versions/dates as described above. I would recommend looking at the rMSA GitHub repo (https://github.com/pylelab/rMSA), which has helpful information for installation, database downloads, and MSA generation. To generate the MSAs, we ran ./rMSA.pl seq.fasta, as on their GitHub README. We updated the outputs to replace \"T\" nucleotides with \"U\". One thing to be aware of is that the databases for MSA generation are very large (~2 TB) and can take a long time to download/set up (on the order of days).
yes, i spend i week download then realise that orginal NT server ftp.ncbi.nlm.nih.gov/blast/db/ is slow (and downloaded file corrupted).
\nIt is better to look for mirrors in your region (use google)
nt database is not mandatory, for example, RhoFold was trained with MSAs computed on rnacentral+rfam only.
\nAnd it's straightforward to patch ./rMSA.pl this way:
\n", "rawMarkdown": "nt database is not mandatory, for example, RhoFold was trained with MSAs computed on rnacentral+rfam only. \n\nAnd it's straightforward to patch `./rMSA.pl` this way:\n\n>my $db0 =\"$dbdir/Rfam.cm\";\nmy $db1 =\"$dbdir/rnacentral.fasta\";\nmy $db2 =\"$dbdir/rnacentral.fasta\";\nmy $db0to1 =\"$dbdir/rfam_annotations.tsv.gz\";\nmy $db0to2 =\"$dbdir/rfam_annotations.tsv.gz\";", "votes": 3 }, { "id": 3157017, "postDate": "2025-03-22T20:49:00.213Z", "content": "my $db0 =\"$dbdir/Rfam.cm\";
\n
\n my $db1 =\"$dbdir/rnacentral.fasta\";
\n my $db2 =\"$dbdir/rnacentral.fasta\";
\n my $db0to1 =\"$dbdir/rfam_annotations.tsv.gz\";
\n my $db0to2 =\"$dbdir/rfam_annotations.tsv.gz\";
\n\nnt database is not mandatory
\n
Exactly. Not only that, but it is a very redundant database (contrary to its name). Sequences that have >80% identity don't help very much in co-variance efforts as they are too similar to each other.
", "rawMarkdown": "> nt database is not mandatory\n\nExactly. Not only that, but it is a very redundant database (contrary to its name). Sequences that have >80% identity don't help very much in co-variance efforts as they are too similar to each other.", "votes": 4 } ] } ] }, { "id": 3156652, "postDate": "2025-03-22T11:37:48.013Z", "content": "It might be worth considering faster alternatives like JackHMMER and MMseqs2.
", "rawMarkdown": "It might be worth considering faster alternatives like JackHMMER and MMseqs2.\n", "replies": [ { "id": 3158246, "postDate": "2025-03-24T11:10:17.640Z", "content": "Gpu mmseqs2 is very fast
", "rawMarkdown": "Gpu mmseqs2 is very fast", "votes": 2 } ] }, { "id": 3156608, "postDate": "2025-03-22T10:23:11.537Z", "content": "Hello,
\nIt's showing error 404 after accessing the link.
Hi @prnvpwr2612, it looks like the URL you tried includes the \")\" at the end. Could you try reloading the page without the \")\"?
\nhttps://github.com/pylelab/rMSA
", "rawMarkdown": "Hi @prnvpwr2612, it looks like the URL you tried includes the \")\" at the end. Could you try reloading the page without the \")\"?\n\nhttps://github.com/pylelab/rMSA", "votes": 1, "replies": [ { "id": 3157042, "postDate": "2025-03-22T21:40:15.507Z", "content": "Thanks a lot it is working now.
", "rawMarkdown": "Thanks a lot it is working now." } ] } ] }, { "id": 3156305, "postDate": "2025-03-22T02:59:48.620Z", "content": "Would you mind sharing the pipeline generating MSAs? It could be helpful that we adopt same pipeline in our experiments.
", "rawMarkdown": "Would you mind sharing the pipeline generating MSAs? It could be helpful that we adopt same pipeline in our experiments.", "replies": [ { "id": 3156310, "postDate": "2025-03-22T03:08:38.947Z", "content": "Hi @zacchaeus, we used the rMSA tool with default parameters and database versions/dates as described above. I would recommend looking at the rMSA GitHub repo (https://github.com/pylelab/rMSA), which has helpful information for installation, database downloads, and MSA generation. To generate the MSAs, we ran ./rMSA.pl seq.fasta as on their GitHub README. We updated the outputs to replace \"T\" nucleotides with \"U\". One thing to be aware of is that the databases for MSA generation are very large (~2 TB) and can take a long time to download/set up (on the order of days).
@alissahummer Good to have these files, thanks.
\nIt seems that we now have straight-up aligned FASTA files, with no insert states in lowercase letters. If so, that should be specified for those who have scripts expecting A3M files. I don't think what you did with this update fulfills the criterion so that all notebooks that use the files are compatible with the update, without any changes required.
Same question as before: when do you expect that all alignments will be available?
", "rawMarkdown": "@alissahummer Good to have these files, thanks.\n\nIt seems that we now have straight-up aligned FASTA files, with no insert states in lowercase letters. If so, that should be specified for those who have scripts expecting A3M files. I don't think what you did with this update fulfills the criterion `so that all notebooks that use the files are compatible with the update, without any changes required.`\n\nSame question as before: when do you expect that all alignments will be available?", "replies": [ { "id": 3156292, "postDate": "2025-03-22T01:58:52.893Z", "content": "Hi @tilii7, thank you for pointing this out and my apologies for any confusion. I have updated the post to clarify the A2M format.
\nWe hope to have all alignments available next week, but please note that this timeline might change.
", "rawMarkdown": "Hi @tilii7, thank you for pointing this out and my apologies for any confusion. I have updated the post to clarify the A2M format.\n\nWe hope to have all alignments available next week, but please note that this timeline might change.", "votes": 1 }, { "id": 3159516, "postDate": "2025-03-25T17:13:26.750Z", "content": "Hi @tilii7, I wanted to let you know that all placeholders have now been updated with the rMSA outputs.
", "rawMarkdown": "Hi @tilii7, I wanted to let you know that all placeholders have now been updated with the rMSA outputs.", "votes": 2, "replies": [ { "id": 3159525, "postDate": "2025-03-25T17:30:11.743Z", "content": "Thank you, that was fast.
\nFor some of short RNAs there are still no additional sequences, and that can't be because they don't have homologs in the database (e.g., 1AFX and 1ESH). I suspect it is because the search procedure is not optimized to work with very short sequences. I will try to generate MSAs for them on my own and will share them if there is a global solution.
", "rawMarkdown": "Thank you, that was fast.\n\nFor some of short RNAs there are still no additional sequences, and that can't be because they don't have homologs in the database (*e.g.,* 1AFX and 1ESH). I suspect it is because the search procedure is not optimized to work with very short sequences. I will try to generate MSAs for them on my own and will share them if there is a global solution.", "votes": 2 } ] } ] }, { "id": 3156228, "postDate": "2025-03-21T22:55:47.617Z", "content": "Thanks! By the way, what does it mean for some query-only files to be \"placeholders\"? Would these be the ones in the test spplit and they are then replaced with the real MSAs?
", "rawMarkdown": "Thanks! By the way, what does it mean for some query-only files to be \"placeholders\"? Would these be the ones in the test spplit and they are then replaced with the real MSAs?", "replies": [ { "id": 3156287, "postDate": "2025-03-22T01:41:41.033Z", "content": "Hi @asarvazyan, thanks for the question! The placeholder MSAs are only for some targets in the training dataset, where MSA generation is taking a long time to complete. We did not want to hold up the release of the majority of the MSAs waiting for these. We plan to update the placeholders with the rMSA outputs once the jobs have finished.
", "rawMarkdown": "Hi @asarvazyan, thanks for the question! The placeholder MSAs are only for some targets in the training dataset, where MSA generation is taking a long time to complete. We did not want to hold up the release of the majority of the MSAs waiting for these. We plan to update the placeholders with the rMSA outputs once the jobs have finished.", "replies": [ { "id": 3156293, "postDate": "2025-03-22T02:03:07.720Z", "content": "Got it, thanks!
", "rawMarkdown": "Got it, thanks!" } ] } ] }, { "id": 3158169, "postDate": "2025-03-24T09:30:45.823Z", "rawMarkdown": "", "isDeleted": true }, { "id": 3156226, "postDate": "2025-03-21T22:43:48.767Z", "content": "thanks a lot again!
", "rawMarkdown": "thanks a lot again!", "votes": 1 }, { "id": 3158337, "postDate": "2025-03-24T13:09:34.423Z", "content": "Thanks for the update
", "rawMarkdown": "Thanks for the update " } ], "comments": [ { "id": 3156294, "author_name": "hengck23", "author_url": "", "post_date": "2025-03-22T02:05:00.203000", "content": "for the newbies:
\n@alissahummer
\nDo u think u can open source your rmsa script to make the msa?
Our team will be making external data and share with the kaggle community. We will be publishing eg af3 3d prediction. We thinking if we want to publish msa of exeternal data as well
", "votes": 1, "replies": [ { "id": 3156312, "author_name": "Alissa Hummer", "author_url": "", "post_date": "2025-03-22T03:12:19.210000", "content": "Hi @hengck23, I am reposting my response to Zacchaeus' question below (posted around the same time as yours):
\nWe used the rMSA tool with default parameters and database versions/dates as described above. I would recommend looking at the rMSA GitHub repo (https://github.com/pylelab/rMSA), which has helpful information for installation, database downloads, and MSA generation. To generate the MSAs, we ran ./rMSA.pl seq.fasta, as on their GitHub README. We updated the outputs to replace \"T\" nucleotides with \"U\". One thing to be aware of is that the databases for MSA generation are very large (~2 TB) and can take a long time to download/set up (on the order of days).
yes, i spend i week download then realise that orginal NT server ftp.ncbi.nlm.nih.gov/blast/db/ is slow (and downloaded file corrupted).
\nIt is better to look for mirrors in your region (use google)
nt database is not mandatory, for example, RhoFold was trained with MSAs computed on rnacentral+rfam only.
\nAnd it's straightforward to patch ./rMSA.pl this way:
\n", "votes": 3, "replies": [] }, { "id": 3157017, "author_name": "Tilii", "author_url": "", "post_date": "2025-03-22T20:49:00.213000", "content": "my $db0 =\"$dbdir/Rfam.cm\";
\n
\n my $db1 =\"$dbdir/rnacentral.fasta\";
\n my $db2 =\"$dbdir/rnacentral.fasta\";
\n my $db0to1 =\"$dbdir/rfam_annotations.tsv.gz\";
\n my $db0to2 =\"$dbdir/rfam_annotations.tsv.gz\";
\n\nnt database is not mandatory
\n
Exactly. Not only that, but it is a very redundant database (contrary to its name). Sequences that have >80% identity don't help very much in co-variance efforts as they are too similar to each other.
", "votes": 4, "replies": [] } ] } ] }, { "id": 3156652, "author_name": "Ogurtsov", "author_url": "", "post_date": "2025-03-22T11:37:48.013000", "content": "It might be worth considering faster alternatives like JackHMMER and MMseqs2.
", "votes": 0, "replies": [ { "id": 3158246, "author_name": "hengck23", "author_url": "", "post_date": "2025-03-24T11:10:17.640000", "content": "Gpu mmseqs2 is very fast
", "votes": 2, "replies": [] } ] }, { "id": 3156608, "author_name": "Pranav Pawar", "author_url": "", "post_date": "2025-03-22T10:23:11.537000", "content": "Hello,
\nIt's showing error 404 after accessing the link.
Hi @prnvpwr2612, it looks like the URL you tried includes the \")\" at the end. Could you try reloading the page without the \")\"?
\nhttps://github.com/pylelab/rMSA
", "votes": 1, "replies": [ { "id": 3157042, "author_name": "Pranav Pawar", "author_url": "", "post_date": "2025-03-22T21:40:15.507000", "content": "Thanks a lot it is working now.
", "votes": 0, "replies": [] } ] } ] }, { "id": 3156305, "author_name": "Zacchaeus", "author_url": "", "post_date": "2025-03-22T02:59:48.620000", "content": "Would you mind sharing the pipeline generating MSAs? It could be helpful that we adopt same pipeline in our experiments.
", "votes": 0, "replies": [ { "id": 3156310, "author_name": "Alissa Hummer", "author_url": "", "post_date": "2025-03-22T03:08:38.947000", "content": "Hi @zacchaeus, we used the rMSA tool with default parameters and database versions/dates as described above. I would recommend looking at the rMSA GitHub repo (https://github.com/pylelab/rMSA), which has helpful information for installation, database downloads, and MSA generation. To generate the MSAs, we ran ./rMSA.pl seq.fasta as on their GitHub README. We updated the outputs to replace \"T\" nucleotides with \"U\". One thing to be aware of is that the databases for MSA generation are very large (~2 TB) and can take a long time to download/set up (on the order of days).
@alissahummer Good to have these files, thanks.
\nIt seems that we now have straight-up aligned FASTA files, with no insert states in lowercase letters. If so, that should be specified for those who have scripts expecting A3M files. I don't think what you did with this update fulfills the criterion so that all notebooks that use the files are compatible with the update, without any changes required.
Same question as before: when do you expect that all alignments will be available?
", "votes": 0, "replies": [ { "id": 3156292, "author_name": "Alissa Hummer", "author_url": "", "post_date": "2025-03-22T01:58:52.893000", "content": "Hi @tilii7, thank you for pointing this out and my apologies for any confusion. I have updated the post to clarify the A2M format.
\nWe hope to have all alignments available next week, but please note that this timeline might change.
", "votes": 1, "replies": [] }, { "id": 3159516, "author_name": "Alissa Hummer", "author_url": "", "post_date": "2025-03-25T17:13:26.750000", "content": "Hi @tilii7, I wanted to let you know that all placeholders have now been updated with the rMSA outputs.
", "votes": 2, "replies": [ { "id": 3159525, "author_name": "Tilii", "author_url": "", "post_date": "2025-03-25T17:30:11.743000", "content": "Thank you, that was fast.
\nFor some of short RNAs there are still no additional sequences, and that can't be because they don't have homologs in the database (e.g., 1AFX and 1ESH). I suspect it is because the search procedure is not optimized to work with very short sequences. I will try to generate MSAs for them on my own and will share them if there is a global solution.
", "votes": 2, "replies": [] } ] } ] }, { "id": 3156228, "author_name": "asarvazyan", "author_url": "", "post_date": "2025-03-21T22:55:47.617000", "content": "Thanks! By the way, what does it mean for some query-only files to be \"placeholders\"? Would these be the ones in the test spplit and they are then replaced with the real MSAs?
", "votes": 0, "replies": [ { "id": 3156287, "author_name": "Alissa Hummer", "author_url": "", "post_date": "2025-03-22T01:41:41.033000", "content": "Hi @asarvazyan, thanks for the question! The placeholder MSAs are only for some targets in the training dataset, where MSA generation is taking a long time to complete. We did not want to hold up the release of the majority of the MSAs waiting for these. We plan to update the placeholders with the rMSA outputs once the jobs have finished.
", "votes": 0, "replies": [ { "id": 3156293, "author_name": "asarvazyan", "author_url": "", "post_date": "2025-03-22T02:03:07.720000", "content": "Got it, thanks!
", "votes": 0, "replies": [] } ] } ] }, { "id": 3158169, "author_name": "", "author_url": "", "post_date": "2025-03-24T09:30:45.823000", "content": "", "votes": 0, "replies": [] }, { "id": 3156226, "author_name": "hengck23", "author_url": "", "post_date": "2025-03-21T22:43:48.767000", "content": "thanks a lot again!
", "votes": 1, "replies": [] }, { "id": 3158337, "author_name": "", "author_url": "", "post_date": "2025-03-24T13:09:34.423000", "content": "Thanks for the update
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "3156129": "Hi Kagglers,\n\nJust a quick note that we have updated the multiple sequence alignment (MSA) files to outputs generated using rMSA (https://github.com/pylelab/rMSA). This method produced deeper MSAs and we used databases with a temporal cutoff more closely aligned with the validation dataset (database versions included below). The rMSA outputs are in the A2M format.\n\nWe have retained the MSA/{target_id}.MSA.fasta file naming.\n\nrMSA Database Versions/Cutoffs\n• RNAcentral v20.0 (2022-03-28)\n• RFam v14.7 (2021-12-09)\n• NCBI NT (2022-10-03)\n\n[Updated 20250321 - A2M format clarified]\n[Updated 20250325 - all placeholders have been replaced with rMSA outputs]", "3156294": "for the newbies:\n\n1. A2M: Uses uppercase characters for matches, lowercase for insertions (with gaps aligned to insertions shown as periods), and dashes (-) for deletions. \n1. A3M: Similar to A2M, but allows gaps aligned to insertions to be omitted(periods), making the format more compact. ", "3156307": "@alissahummer \nDo u think u can open source your rmsa script to make the msa?\n\nOur team will be making external data and share with the kaggle community. We will be publishing eg af3 3d prediction. We thinking if we want to publish msa of exeternal data as well ", "3156652": "It might be worth considering faster alternatives like JackHMMER and MMseqs2.\n", "3156608": "Hello,\nIt's showing error 404 after accessing the link.", "3156305": "Would you mind sharing the pipeline generating MSAs? It could be helpful that we adopt same pipeline in our experiments.", "3156271": "@alissahummer Good to have these files, thanks.\n\nIt seems that we now have straight-up aligned FASTA files, with no insert states in lowercase letters. If so, that should be specified for those who have scripts expecting A3M files. I don't think what you did with this update fulfills the criterion `so that all notebooks that use the files are compatible with the update, without any changes required.`\n\nSame question as before: when do you expect that all alignments will be available?", "3156228": "Thanks! By the way, what does it mean for some query-only files to be \"placeholders\"? Would these be the ones in the test spplit and they are then replaced with the real MSAs?", "3158169": "", "3156226": "thanks a lot again!", "3158337": "Thanks for the update " } }