{
  "id": 510644,
  "title": "Next generation seq model: transformer+flash attention2, mamba1/2, xlstm, Griffin ...",
  "url": "/competitions/leash-BELKA/discussion/510644",
  "author_name": "",
  "post_date": "2024-06-07T03:11:33.750730Z",
  "votes": 16,
  "comment_count": 5,
  "views": 0,
  "content": "<p>Sample code here at: <a href=\"https://www.kaggle.com/code/hengck23/lb0-428-single-fold-transformer\" target=\"_blank\">https://www.kaggle.com/code/hengck23/lb0-428-single-fold-transformer</a></p>\n<p>all code are cuda accelerated. Seems that for small model, transformer+flash attention2 is still the most efficient</p>",
  "messages": [
    {
      "id": "2859424",
      "postDate": "06/07/2024 03:11:33",
      "content": "<p>Sample code here at: <a href=\"https://www.kaggle.com/code/hengck23/lb0-428-single-fold-transformer\" target=\"_blank\">https://www.kaggle.com/code/hengck23/lb0-428-single-fold-transformer</a></p>\n<p>all code are cuda accelerated. Seems that for small model, transformer+flash attention2 is still the most efficient</p>",
      "rawMarkdown": "Sample code here at: https://www.kaggle.com/code/hengck23/lb0-428-single-fold-transformer\n\nall code are cuda accelerated. Seems that for small model, transformer+flash attention2 is still the most efficient",
      "votes": null
    },
    {
      "id": "2860886",
      "postDate": "06/07/2024 20:54:02",
      "content": "<p>i realized i can train both test and train molecules together</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2Fd18c3d9432330350a82034e44ace28a2%2FSelection_175.png?generation=1717794494932695&amp;alt=media\"></p>",
      "rawMarkdown": "i realized i can train both test and train molecules together\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2Fd18c3d9432330350a82034e44ace28a2%2FSelection_175.png?generation=1717794494932695&alt=media)",
      "votes": null
    },
    {
      "id": "2862482",
      "postDate": "06/08/2024 19:12:52",
      "content": "<p>mamba1 single fold score is lb0.432</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F7a652f0737f0fee736c4cdda5984a599%2FSelection_002.png?generation=1718073289828614&amp;alt=media\"></p>",
      "rawMarkdown": "mamba1 single fold score is lb0.432\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F7a652f0737f0fee736c4cdda5984a599%2FSelection_002.png?generation=1718073289828614&alt=media)",
      "votes": null
    },
    {
      "id": "2862501",
      "postDate": "06/08/2024 19:24:43",
      "content": "<p>Mamba graph will be something interesting</p>",
      "rawMarkdown": "Mamba graph will be something interesting",
      "votes": null
    },
    {
      "id": "2867837",
      "postDate": "06/12/2024 05:26:46",
      "content": "<p>I've been studying mamba architecture recently.<br>\nNice to see this in the Kaggle competition!<br>\nI think it will be very interesting to see it in the winning solution.</p>",
      "rawMarkdown": "I've been studying mamba architecture recently.\nNice to see this in the Kaggle competition!\nI think it will be very interesting to see it in the winning solution.",
      "votes": null
    },
    {
      "id": "2871598",
      "postDate": "06/14/2024 09:33:20",
      "content": "<p>Great to see Mamba here!</p>",
      "rawMarkdown": "Great to see Mamba here!",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2860886,
      "author_name": "hengck23",
      "author_url": "",
      "post_date": "06/07/2024 20:54:02",
      "content": "<p>i realized i can train both test and train molecules together</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2Fd18c3d9432330350a82034e44ace28a2%2FSelection_175.png?generation=1717794494932695&amp;alt=media\"></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 2862482,
      "author_name": "hengck23",
      "author_url": "",
      "post_date": "06/08/2024 19:12:52",
      "content": "<p>mamba1 single fold score is lb0.432</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F7a652f0737f0fee736c4cdda5984a599%2FSelection_002.png?generation=1718073289828614&amp;alt=media\"></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 2862501,
      "author_name": "hengck23",
      "author_url": "",
      "post_date": "06/08/2024 19:24:43",
      "content": "<p>Mamba graph will be something interesting</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 2867837,
      "author_name": "yuuniekiri",
      "author_url": "",
      "post_date": "06/12/2024 05:26:46",
      "content": "<p>I've been studying mamba architecture recently.<br>\nNice to see this in the Kaggle competition!<br>\nI think it will be very interesting to see it in the winning solution.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 2871598,
      "author_name": "nithinskantha10",
      "author_url": "",
      "post_date": "06/14/2024 09:33:20",
      "content": "<p>Great to see Mamba here!</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2859424": "Sample code here at: https://www.kaggle.com/code/hengck23/lb0-428-single-fold-transformer\n\nall code are cuda accelerated. Seems that for small model, transformer+flash attention2 is still the most efficient",
    "2860886": "i realized i can train both test and train molecules together\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2Fd18c3d9432330350a82034e44ace28a2%2FSelection_175.png?generation=1717794494932695&alt=media)",
    "2862482": "mamba1 single fold score is lb0.432\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F7a652f0737f0fee736c4cdda5984a599%2FSelection_002.png?generation=1718073289828614&alt=media)",
    "2862501": "Mamba graph will be something interesting",
    "2867837": "I've been studying mamba architecture recently.\nNice to see this in the Kaggle competition!\nI think it will be very interesting to see it in the winning solution.",
    "2871598": "Great to see Mamba here!"
  },
  "source": "meta"
}