![\"Capture-d-cran-2025-05-09-102233\"](\"https://i.ibb.co/cSDfz92n/Capture-d-cran-2025-05-09-102233.png\")
\nVigg Team's transformer architecture which won last year's competition. Link.
This competition can be quite intimidating for someone with little to no experience in RNA folding. However, there's still plenty of time to learn about this Nobel Prize–winning challenge during the final few weeks of the competition !
\nGiven an RNA sequence, predict the 3D structure of its molecule.
\nInput: GGGUGCUCAGUACGAGAGGAACCGCACCC
\nOutput: x, y, z coordinates of each of the “letters” (ribonucleotides) in the sequence
It’s a sequence-to-sequence task where the input is a sequence of ribonucleotides (A, U, C or G), and the output is their position.
\nEach ribonucleotide can be considered as a word (or token), and therefore an RNA chain can be seen as a sentence. The goal is then to translate this sentence into coordinates.
\nThe good news is that we know which approaches work well for processing sentences and solving sequence-to-sequence tasks: transformers!
\nFirst, tokenize your RNA using a simple tokenizer where the tokens are A, U, C and G (and optionally and ). Then, you can use any transformer architecture you like and plug a classification head on top of it to get your coordinates output.
\nSqueezeformer is a good choice when using transformers and was used as a starting point by last years 2nd and 3rd place winners.
Referring to past similar solutions is always a good way to catch up with what Kagglers have done on the topic. Although the target is different from this year’s, the architecture used can most likely be reused. Furthermore, the hosts have combined the findings into a state-of-the-art model called RibonanzaNet.
\nThe key takeaway was to use transformers on the embedded sequences, but not only :
\nFinetuning RibonanzaNet is a great starting point; it achieves LB 0.177 as-is, and LB 0.19 when plugging the invariant point attention (IPA) module for AlphaFold 2 which is well suited to our task. See discussion here
\n
\nVigg Team's transformer architecture which won last year's competition. Link.
The Kaggle competition was in 2023, and the landscape has evolved quite a bit since. DeepMind and Isomorphics Labs released AlphaFold 3 which considerably improved on previous models. The idea of using a pair transformer trunk that leverages both sequence features and pair features is still key, but the key novelty is that a diffusion model (yes, like the ones used for image generation) is added.
\nDownstream to the trunk, a diffusion head is trained to remove a gaussian noise added to the atom coordinates. At test time, the diffusion module iteratively transforms a gaussian noise into the 3D structure by running the denoising several times.
\nThis module can be plugged on whichever RNA model of your choice. @shujun717 has demonstrated how you can use it on top of RibonanzaNet2 to improve it and reach LB 0.3.
\nAlphaFold 3 weights are not licensed for public use, but Boltz-1’s are. Boltz-1 is heavily inspired by AlphaFold 3 and even improves on top of it. Compared to @shujun717’s baselines, the key differences include:
Use the information in the multiple sequence alignments (MSA) which are provided in this competition in the pair representation. This adds signal from amino acids that co-evolved throughout evolution and are similar to the input RNA.
Use a confidence model to predict structure confidence metrics. The confidence head is also a pair transformer, which takes as input the trunk features as well as the structure output at the different stages of diffusion.
\n@youhanlee shared how you can use it for inference here
![\"Capture-d-cran-2025-05-07-123548\"](\"https://i.ibb.co/nJQ007Z/Capture-d-cran-2025-05-07-123548.png\")
\nDiagram of the architecture of Boltz-1. Figure 3 of the Boltz-1 paper.
Thanks for reading, and happy learning!
", "messages": [ { "id": "3198278", "postDate": "05/09/2025 08:27:42", "content": "This competition can be quite intimidating for someone with little to no experience in RNA folding. However, there's still plenty of time to learn about this Nobel Prize–winning challenge during the final few weeks of the competition !
\nGiven an RNA sequence, predict the 3D structure of its molecule.
\nInput: GGGUGCUCAGUACGAGAGGAACCGCACCC
\nOutput: x, y, z coordinates of each of the “letters” (ribonucleotides) in the sequence
It’s a sequence-to-sequence task where the input is a sequence of ribonucleotides (A, U, C or G), and the output is their position.
\nEach ribonucleotide can be considered as a word (or token), and therefore an RNA chain can be seen as a sentence. The goal is then to translate this sentence into coordinates.
\nThe good news is that we know which approaches work well for processing sentences and solving sequence-to-sequence tasks: transformers!
\nFirst, tokenize your RNA using a simple tokenizer where the tokens are A, U, C and G (and optionally and ). Then, you can use any transformer architecture you like and plug a classification head on top of it to get your coordinates output.
\nSqueezeformer is a good choice when using transformers and was used as a starting point by last years 2nd and 3rd place winners.
Referring to past similar solutions is always a good way to catch up with what Kagglers have done on the topic. Although the target is different from this year’s, the architecture used can most likely be reused. Furthermore, the hosts have combined the findings into a state-of-the-art model called RibonanzaNet.
\nThe key takeaway was to use transformers on the embedded sequences, but not only :
\nFinetuning RibonanzaNet is a great starting point; it achieves LB 0.177 as-is, and LB 0.19 when plugging the invariant point attention (IPA) module for AlphaFold 2 which is well suited to our task. See discussion here
\n
\nVigg Team's transformer architecture which won last year's competition. Link.
The Kaggle competition was in 2023, and the landscape has evolved quite a bit since. DeepMind and Isomorphics Labs released AlphaFold 3 which considerably improved on previous models. The idea of using a pair transformer trunk that leverages both sequence features and pair features is still key, but the key novelty is that a diffusion model (yes, like the ones used for image generation) is added.
\nDownstream to the trunk, a diffusion head is trained to remove a gaussian noise added to the atom coordinates. At test time, the diffusion module iteratively transforms a gaussian noise into the 3D structure by running the denoising several times.
\nThis module can be plugged on whichever RNA model of your choice. @shujun717 has demonstrated how you can use it on top of RibonanzaNet2 to improve it and reach LB 0.3.
\nAlphaFold 3 weights are not licensed for public use, but Boltz-1’s are. Boltz-1 is heavily inspired by AlphaFold 3 and even improves on top of it. Compared to @shujun717’s baselines, the key differences include:
Use the information in the multiple sequence alignments (MSA) which are provided in this competition in the pair representation. This adds signal from amino acids that co-evolved throughout evolution and are similar to the input RNA.
Use a confidence model to predict structure confidence metrics. The confidence head is also a pair transformer, which takes as input the trunk features as well as the structure output at the different stages of diffusion.
\n@youhanlee shared how you can use it for inference here
\nDiagram of the architecture of Boltz-1. Figure 3 of the Boltz-1 paper.
Thanks for reading, and happy learning!
", "rawMarkdown": "This competition can be quite intimidating for someone with little to no experience in RNA folding. However, there's still plenty of time to learn about this Nobel Prize–winning challenge during the final few weeks of the competition !\n\n## The task\n\nGiven an RNA sequence, predict the 3D structure of its molecule. \n\n**Input:** GGGUGCUCAGUACGAGAGGAACCGCACCC\n**Output:** x, y, z coordinates of each of the “letters” (ribonucleotides) in the sequence\n\nIt’s a sequence-to-sequence task where the input is a sequence of ribonucleotides (A, U, C or G), and the output is their position. \n\n## Analogy with LLMs and NLP\n\nEach ribonucleotide can be considered as a word (or token), and therefore an RNA chain can be seen as a sentence. The goal is then to translate this sentence into coordinates. \nThe good news is that we know which approaches work well for processing sentences and solving sequence-to-sequence tasks: **transformers**!\nFirst, tokenize your RNA using a simple tokenizer where the tokens are A, U, C and G (and optionally\n*Vigg Team's transformer architecture which won last year's competition. [Link](https://www.kaggle.com/competitions/stanford-ribonanza-rna-folding/discussion/460121).*\n\n## Diffusion models and towards current state-of-the-art \n\nThe Kaggle competition was in 2023, and the landscape has evolved quite a bit since. DeepMind and Isomorphics Labs released AlphaFold 3 which considerably improved on previous models. The idea of using a pair transformer trunk that leverages both sequence features and pair features is still key, but the key novelty is that a diffusion model (yes, like the ones used for image generation) is added.\nDownstream to the trunk, a diffusion head is trained to remove a gaussian noise added to the atom coordinates. At test time, the diffusion module iteratively transforms a gaussian noise into the 3D structure by running the denoising several times. \nThis module can be plugged on whichever RNA model of your choice. @shujun717 has demonstrated how you can use it on top of RibonanzaNet2 to improve it and reach LB 0.3. \nAlphaFold 3 weights are not licensed for public use, but [Boltz-1]( https://www.biorxiv.org/content/10.1101/2024.11.19.624167v3.full.pdf)’s are. Boltz-1 is heavily inspired by AlphaFold 3 and even improves on top of it. Compared to @shujun717’s baselines, the key differences include:\n- Use the information in the multiple sequence alignments (MSA) which are provided in this competition in the pair representation. This adds signal from amino acids that co-evolved throughout evolution and are similar to the input RNA.\n- Use a confidence model to predict structure confidence metrics. The confidence head is also a pair transformer, which takes as input the trunk features as well as the structure output at the different stages of diffusion.\n\n @youhanlee shared how you can use it for inference [here]( https://www.kaggle.com/code/youhanlee/boltz-1-inference-submission)\n \n\n*Diagram of the architecture of Boltz-1. Figure 3 of the Boltz-1 paper.*\n\n## Some useful resources\n-\tThe supplementary information for the AlphaFold 3 paper : https://www.nature.com/articles/s41586-024-07487-w#Sec19 which is very detailed and contains a lot of useful information\n-\t@alejopaullier’s notebook which explains the RibonanzaNet implementation https://www.kaggle.com/code/alejopaullier/stanford-rna-3d-ribonanzanet-explained \n\n*Thanks for reading, and happy learning!*",
"votes": null
},
{
"id": "3198456",
"postDate": "05/09/2025 12:55:34",
"content": "It is ccomplex. What cours i shold learn to understand it?
", "rawMarkdown": "It is ccomplex. What cours i shold learn to understand it?", "votes": null }, { "id": "3206717", "postDate": "05/21/2025 17:46:26", "content": "What about protenix model then? It is also inspired from Alphafold3 right?
", "rawMarkdown": "What about protenix model then? It is also inspired from Alphafold3 right?", "votes": null }, { "id": "3207688", "postDate": "05/23/2025 06:10:41", "content": "Protenix baseline code is useful, you can try to combine it with other model that can get a better score
", "rawMarkdown": "Protenix baseline code is useful, you can try to combine it with other model that can get a better score", "votes": null } ], "comments": [ { "id": 3198456, "author_name": "nancywaltonlaur", "author_url": "", "post_date": "05/09/2025 12:55:34", "content": "It is ccomplex. What cours i shold learn to understand it?
", "votes": null, "replies": [ { "id": 3207688, "author_name": "", "author_url": "", "post_date": "05/23/2025 06:10:41", "content": "Protenix baseline code is useful, you can try to combine it with other model that can get a better score
", "votes": null, "replies": [] } ] }, { "id": 3206717, "author_name": "rahuladhi", "author_url": "", "post_date": "05/21/2025 17:46:26", "content": "What about protenix model then? It is also inspired from Alphafold3 right?
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "3198278": "This competition can be quite intimidating for someone with little to no experience in RNA folding. However, there's still plenty of time to learn about this Nobel Prize–winning challenge during the final few weeks of the competition !\n\n## The task\n\nGiven an RNA sequence, predict the 3D structure of its molecule. \n\n**Input:** GGGUGCUCAGUACGAGAGGAACCGCACCC\n**Output:** x, y, z coordinates of each of the “letters” (ribonucleotides) in the sequence\n\nIt’s a sequence-to-sequence task where the input is a sequence of ribonucleotides (A, U, C or G), and the output is their position. \n\n## Analogy with LLMs and NLP\n\nEach ribonucleotide can be considered as a word (or token), and therefore an RNA chain can be seen as a sentence. The goal is then to translate this sentence into coordinates. \nThe good news is that we know which approaches work well for processing sentences and solving sequence-to-sequence tasks: **transformers**!\nFirst, tokenize your RNA using a simple tokenizer where the tokens are A, U, C and G (and optionally\n*Vigg Team's transformer architecture which won last year's competition. [Link](https://www.kaggle.com/competitions/stanford-ribonanza-rna-folding/discussion/460121).*\n\n## Diffusion models and towards current state-of-the-art \n\nThe Kaggle competition was in 2023, and the landscape has evolved quite a bit since. DeepMind and Isomorphics Labs released AlphaFold 3 which considerably improved on previous models. The idea of using a pair transformer trunk that leverages both sequence features and pair features is still key, but the key novelty is that a diffusion model (yes, like the ones used for image generation) is added.\nDownstream to the trunk, a diffusion head is trained to remove a gaussian noise added to the atom coordinates. At test time, the diffusion module iteratively transforms a gaussian noise into the 3D structure by running the denoising several times. \nThis module can be plugged on whichever RNA model of your choice. @shujun717 has demonstrated how you can use it on top of RibonanzaNet2 to improve it and reach LB 0.3. \nAlphaFold 3 weights are not licensed for public use, but [Boltz-1]( https://www.biorxiv.org/content/10.1101/2024.11.19.624167v3.full.pdf)’s are. Boltz-1 is heavily inspired by AlphaFold 3 and even improves on top of it. Compared to @shujun717’s baselines, the key differences include:\n- Use the information in the multiple sequence alignments (MSA) which are provided in this competition in the pair representation. This adds signal from amino acids that co-evolved throughout evolution and are similar to the input RNA.\n- Use a confidence model to predict structure confidence metrics. The confidence head is also a pair transformer, which takes as input the trunk features as well as the structure output at the different stages of diffusion.\n\n @youhanlee shared how you can use it for inference [here]( https://www.kaggle.com/code/youhanlee/boltz-1-inference-submission)\n \n\n*Diagram of the architecture of Boltz-1. Figure 3 of the Boltz-1 paper.*\n\n## Some useful resources\n-\tThe supplementary information for the AlphaFold 3 paper : https://www.nature.com/articles/s41586-024-07487-w#Sec19 which is very detailed and contains a lot of useful information\n-\t@alejopaullier’s notebook which explains the RibonanzaNet implementation https://www.kaggle.com/code/alejopaullier/stanford-rna-3d-ribonanzanet-explained \n\n*Thanks for reading, and happy learning!*",
"3198456": "It is ccomplex. What cours i shold learn to understand it?",
"3206717": "What about protenix model then? It is also inspired from Alphafold3 right?",
"3207688": "Protenix baseline code is useful, you can try to combine it with other model that can get a better score"
},
"source": "meta"
}