{
  "id": 569322,
  "title": "RibonanzaNet 3D finetuning with a structure module lb 0.19",
  "url": "/competitions/stanford-rna-3d-folding/discussion/569322",
  "author_name": "",
  "post_date": "2025-03-21T07:23:51.256639Z",
  "votes": 9,
  "comment_count": 2,
  "views": 0,
  "content": "<p>Hi guys, AF2 uses invariant point attention as their structure module to do iterative refinement, here we can do something similar with RIbonanzaNet. </p>\n<p>I have added a structure module that looks at the distance matrix and makes xyz updates to the finetuning notebook and it improves the score from 0.177 to 0.19.</p>\n<p>Let me know if you have questions! </p>\n<p>finetuning and inference notebooks:<br>\n<a href=\"https://www.kaggle.com/code/shujun717/ribonanzanet-3d-finetune-add-structure-module\" target=\"_blank\">https://www.kaggle.com/code/shujun717/ribonanzanet-3d-finetune-add-structure-module</a><br>\n<a href=\"https://www.kaggle.com/code/shujun717/ribonanzanet-3d-inference-add-structure-module?scriptVersionId=228564059\" target=\"_blank\">https://www.kaggle.com/code/shujun717/ribonanzanet-3d-inference-add-structure-module?scriptVersionId=228564059</a></p>",
  "messages": [
    {
      "id": "3155583",
      "postDate": "03/21/2025 07:23:51",
      "content": "<p>Hi guys, AF2 uses invariant point attention as their structure module to do iterative refinement, here we can do something similar with RIbonanzaNet. </p>\n<p>I have added a structure module that looks at the distance matrix and makes xyz updates to the finetuning notebook and it improves the score from 0.177 to 0.19.</p>\n<p>Let me know if you have questions! </p>\n<p>finetuning and inference notebooks:<br>\n<a href=\"https://www.kaggle.com/code/shujun717/ribonanzanet-3d-finetune-add-structure-module\" target=\"_blank\">https://www.kaggle.com/code/shujun717/ribonanzanet-3d-finetune-add-structure-module</a><br>\n<a href=\"https://www.kaggle.com/code/shujun717/ribonanzanet-3d-inference-add-structure-module?scriptVersionId=228564059\" target=\"_blank\">https://www.kaggle.com/code/shujun717/ribonanzanet-3d-inference-add-structure-module?scriptVersionId=228564059</a></p>",
      "rawMarkdown": "Hi guys, AF2 uses invariant point attention as their structure module to do iterative refinement, here we can do something similar with RIbonanzaNet. \n\nI have added a structure module that looks at the distance matrix and makes xyz updates to the finetuning notebook and it improves the score from 0.177 to 0.19.\n\nLet me know if you have questions! \n\n\nfinetuning and inference notebooks:\nhttps://www.kaggle.com/code/shujun717/ribonanzanet-3d-finetune-add-structure-module\nhttps://www.kaggle.com/code/shujun717/ribonanzanet-3d-inference-add-structure-module?scriptVersionId=228564059",
      "votes": null
    },
    {
      "id": "3155586",
      "postDate": "03/21/2025 07:27:07",
      "content": "<p>do you think triangular attention (node in and node out) is important? this seems to be the bottle neck</p>",
      "rawMarkdown": "do you think triangular attention (node in and node out) is important? this seems to be the bottle neck",
      "votes": null
    },
    {
      "id": "3156054",
      "postDate": "03/21/2025 17:00:50",
      "content": "<p>thanks to the host for implementing IPA (invariant point attention). For beginners, do note the differences below. </p>\n<ol>\n<li>alphafold2: evoformer (extract feature) --&gt; structure IPA(predict rotation+translation of backbone frame(P,C4,N1/9))</li>\n<li>host example: ribonanza_net (extract feature) --&gt; structure  IPA(predict location of backbone atom(C1))</li>\n<li>alphafold3: evoformer (extract feature) --&gt; structure diffusion (IPA has been replaced by diffusion)</li>\n</ol>\n<hr>\n<p>key ideas of IPA is:</p>\n<ul>\n<li>from current point(or frame), make Q,V,K point.  Compute distance matrix LxL from Q,K</li>\n<li>from seq representation, make Q,V,K. compute affinity matrix LxL from Q,K</li>\n<li>from pair representation, compute LxL bias.</li>\n<li>add the above three. do softmax to get attention weight w</li>\n<li>apply the attention w back to point V, seq representation V, pair representation.</li>\n<li>you can now apply last output linear to the weighted V.</li>\n</ul>\n<p>useful:</p>\n<ol>\n<li><p><a href=\"https://www.youtube.com/watch?v=Rmn_DDfpRjc\" target=\"_blank\">https://www.youtube.com/watch?v=Rmn_DDfpRjc</a><br>\nAlphaFold Decoded: Structure Model (Lesson 8)</p></li>\n<li><p><a href=\"https://github.com/lucidrains/invariant-point-attention\" target=\"_blank\">https://github.com/lucidrains/invariant-point-attention</a></p></li>\n</ol>\n<p>3.<br>\nAF2 (IPA part):<br>\n<a href=\"https://piip.co.kr/en/blog/AlphaFold2_Architecture_Improvements\" target=\"_blank\">https://piip.co.kr/en/blog/AlphaFold2_Architecture_Improvements</a></p>\n<p>AF3 (diffusion part):<br>\n<a href=\"https://elanapearl.github.io/blog/2024/the-illustrated-alphafold/\" target=\"_blank\">https://elanapearl.github.io/blog/2024/the-illustrated-alphafold/</a></p>",
      "rawMarkdown": "thanks to the host for implementing IPA (invariant point attention). For beginners, do note the differences below. \n1. alphafold2: evoformer (extract feature) --> structure IPA(predict rotation+translation of backbone frame(P,C4,N1/9))\n2. host example: ribonanza_net (extract feature) --> structure  IPA(predict location of backbone atom(C1))\n3. alphafold3: evoformer (extract feature) --> structure diffusion (IPA has been replaced by diffusion)\n\n---\n\nkey ideas of IPA is:\n- from current point(or frame), make Q,V,K point.  Compute distance matrix LxL from Q,K\n- from seq representation, make Q,V,K. compute affinity matrix LxL from Q,K\n- from pair representation, compute LxL bias.\n- add the above three. do softmax to get attention weight w\n- apply the attention w back to point V, seq representation V, pair representation.\n- you can now apply last output linear to the weighted V.\n\nuseful:\n1. https://www.youtube.com/watch?v=Rmn_DDfpRjc\nAlphaFold Decoded: Structure Model (Lesson 8)\n\n2. https://github.com/lucidrains/invariant-point-attention\n\n3.\nAF2 (IPA part):\nhttps://piip.co.kr/en/blog/AlphaFold2_Architecture_Improvements\n\nAF3 (diffusion part):\nhttps://elanapearl.github.io/blog/2024/the-illustrated-alphafold/",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3155586,
      "author_name": "hengck23",
      "author_url": "",
      "post_date": "03/21/2025 07:27:07",
      "content": "<p>do you think triangular attention (node in and node out) is important? this seems to be the bottle neck</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 3156054,
      "author_name": "hengck23",
      "author_url": "",
      "post_date": "03/21/2025 17:00:50",
      "content": "<p>thanks to the host for implementing IPA (invariant point attention). For beginners, do note the differences below. </p>\n<ol>\n<li>alphafold2: evoformer (extract feature) --&gt; structure IPA(predict rotation+translation of backbone frame(P,C4,N1/9))</li>\n<li>host example: ribonanza_net (extract feature) --&gt; structure  IPA(predict location of backbone atom(C1))</li>\n<li>alphafold3: evoformer (extract feature) --&gt; structure diffusion (IPA has been replaced by diffusion)</li>\n</ol>\n<hr>\n<p>key ideas of IPA is:</p>\n<ul>\n<li>from current point(or frame), make Q,V,K point.  Compute distance matrix LxL from Q,K</li>\n<li>from seq representation, make Q,V,K. compute affinity matrix LxL from Q,K</li>\n<li>from pair representation, compute LxL bias.</li>\n<li>add the above three. do softmax to get attention weight w</li>\n<li>apply the attention w back to point V, seq representation V, pair representation.</li>\n<li>you can now apply last output linear to the weighted V.</li>\n</ul>\n<p>useful:</p>\n<ol>\n<li><p><a href=\"https://www.youtube.com/watch?v=Rmn_DDfpRjc\" target=\"_blank\">https://www.youtube.com/watch?v=Rmn_DDfpRjc</a><br>\nAlphaFold Decoded: Structure Model (Lesson 8)</p></li>\n<li><p><a href=\"https://github.com/lucidrains/invariant-point-attention\" target=\"_blank\">https://github.com/lucidrains/invariant-point-attention</a></p></li>\n</ol>\n<p>3.<br>\nAF2 (IPA part):<br>\n<a href=\"https://piip.co.kr/en/blog/AlphaFold2_Architecture_Improvements\" target=\"_blank\">https://piip.co.kr/en/blog/AlphaFold2_Architecture_Improvements</a></p>\n<p>AF3 (diffusion part):<br>\n<a href=\"https://elanapearl.github.io/blog/2024/the-illustrated-alphafold/\" target=\"_blank\">https://elanapearl.github.io/blog/2024/the-illustrated-alphafold/</a></p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "3155583": "Hi guys, AF2 uses invariant point attention as their structure module to do iterative refinement, here we can do something similar with RIbonanzaNet. \n\nI have added a structure module that looks at the distance matrix and makes xyz updates to the finetuning notebook and it improves the score from 0.177 to 0.19.\n\nLet me know if you have questions! \n\n\nfinetuning and inference notebooks:\nhttps://www.kaggle.com/code/shujun717/ribonanzanet-3d-finetune-add-structure-module\nhttps://www.kaggle.com/code/shujun717/ribonanzanet-3d-inference-add-structure-module?scriptVersionId=228564059",
    "3155586": "do you think triangular attention (node in and node out) is important? this seems to be the bottle neck",
    "3156054": "thanks to the host for implementing IPA (invariant point attention). For beginners, do note the differences below. \n1. alphafold2: evoformer (extract feature) --> structure IPA(predict rotation+translation of backbone frame(P,C4,N1/9))\n2. host example: ribonanza_net (extract feature) --> structure  IPA(predict location of backbone atom(C1))\n3. alphafold3: evoformer (extract feature) --> structure diffusion (IPA has been replaced by diffusion)\n\n---\n\nkey ideas of IPA is:\n- from current point(or frame), make Q,V,K point.  Compute distance matrix LxL from Q,K\n- from seq representation, make Q,V,K. compute affinity matrix LxL from Q,K\n- from pair representation, compute LxL bias.\n- add the above three. do softmax to get attention weight w\n- apply the attention w back to point V, seq representation V, pair representation.\n- you can now apply last output linear to the weighted V.\n\nuseful:\n1. https://www.youtube.com/watch?v=Rmn_DDfpRjc\nAlphaFold Decoded: Structure Model (Lesson 8)\n\n2. https://github.com/lucidrains/invariant-point-attention\n\n3.\nAF2 (IPA part):\nhttps://piip.co.kr/en/blog/AlphaFold2_Architecture_Improvements\n\nAF3 (diffusion part):\nhttps://elanapearl.github.io/blog/2024/the-illustrated-alphafold/"
  },
  "source": "meta"
}