{
  "id": 243131,
  "title": "a vision transformer basline",
  "url": "/competitions/bms-molecular-translation/discussion/243131",
  "author_name": "",
  "post_date": "2021-06-01T09:37:54.909683700Z",
  "votes": 3,
  "comment_count": 6,
  "views": 0,
  "content": "<p>Even thought this competition is ver near end </p>\n<p>I finally come up with a vision transformer baseline </p>\n<p>link: <a href=\"https://www.kaggle.com/drzhuzhe/bms-vision-transformer\" target=\"_blank\">https://www.kaggle.com/drzhuzhe/bms-vision-transformer</a></p>\n<p>maybe we can </p>\n<ol>\n<li>training more epoch</li>\n<li>fine tune via full MAX_LEN rather than HALF-LENGTH</li>\n<li>use VIT16-base </li>\n<li>use CNN to patch</li>\n<li>tuning my learning rate schedule </li>\n</ol>\n<p>I known it is too late … for me</p>\n<p>But still hope it useful and if you find something config wrong please tell me</p>",
  "messages": [
    {
      "id": "1331121",
      "postDate": "06/01/2021 09:37:54",
      "content": "<p>Even thought this competition is ver near end </p>\n<p>I finally come up with a vision transformer baseline </p>\n<p>link: <a href=\"https://www.kaggle.com/drzhuzhe/bms-vision-transformer\" target=\"_blank\">https://www.kaggle.com/drzhuzhe/bms-vision-transformer</a></p>\n<p>maybe we can </p>\n<ol>\n<li>training more epoch</li>\n<li>fine tune via full MAX_LEN rather than HALF-LENGTH</li>\n<li>use VIT16-base </li>\n<li>use CNN to patch</li>\n<li>tuning my learning rate schedule </li>\n</ol>\n<p>I known it is too late … for me</p>\n<p>But still hope it useful and if you find something config wrong please tell me</p>",
      "rawMarkdown": "Even thought this competition is ver near end \n\nI finally come up with a vision transformer baseline \n\nlink: https://www.kaggle.com/drzhuzhe/bms-vision-transformer\n\nmaybe we can \n\n1. training more epoch\n2. fine tune via full MAX_LEN rather than HALF-LENGTH\n3. use VIT16-base \n4. use CNN to patch\n5. tuning my learning rate schedule \n\nI known it is too late ... for me\n\nBut still hope it useful and if you find something config wrong please tell me",
      "votes": null
    },
    {
      "id": "1332293",
      "postDate": "06/02/2021 03:13:51",
      "content": "<p>validation statistic takes too long time! </p>",
      "rawMarkdown": "validation statistic takes too long time!",
      "votes": null
    },
    {
      "id": "1332486",
      "postDate": "06/02/2021 06:33:08",
      "content": "<p>do you have any idea dealing with it？</p>",
      "rawMarkdown": "do you have any idea dealing with it？",
      "votes": null
    },
    {
      "id": "1332560",
      "postDate": "06/02/2021 07:24:18",
      "content": "<p>I will have a look. The Transformer  notebook in your reference does not cost that much time for validation LSD. <br>\nbtw,  I  stopped a few times and keep retraining, the LOSS/ACC fluctuation happens again.</p>\n<p>Perhaps like the way for iteration number like in Heng's script, a huge number like 80000x1000?</p>\n<p>That would need a local good machine to run. </p>",
      "rawMarkdown": "I will have a look. The Transformer  notebook in your reference does not cost that much time for validation LSD. \nbtw,  I  stopped a few times and keep retraining, the LOSS/ACC fluctuation happens again.\n\nPerhaps like the way for iteration number like in Heng's script, a huge number like 80000x1000?\n\nThat would need a local good machine to run.",
      "votes": null
    },
    {
      "id": "1332572",
      "postDate": "06/02/2021 07:37:34",
      "content": "<blockquote>\n  <p>I will have a look. The Transformer notebook in your reference does not cost that much time for validation LSD.</p>\n</blockquote>\n<p>which one ？ this <a href=\"https://www.kaggle.com/dschettler8845/3-71-cv-bms-efficientnetv2-transformer-e2e\" target=\"_blank\">https://www.kaggle.com/dschettler8845/3-71-cv-bms-efficientnetv2-transformer-e2e</a>  ？？？</p>\n<p>it's very slow to validate too</p>",
      "rawMarkdown": "> I will have a look. The Transformer notebook in your reference does not cost that much time for validation LSD.\n\nwhich one ？ this https://www.kaggle.com/dschettler8845/3-71-cv-bms-efficientnetv2-transformer-e2e  ？？？\n\nit's very slow to validate too",
      "votes": null
    },
    {
      "id": "1332597",
      "postDate": "06/02/2021 07:52:27",
      "content": "<p>I think this maybe </p>\n<p>if you valid lsd in very earlier get_levenshtein_distance will be very slow if you get a low lsd score , because it may reach the worst case of algorithm (BIG O) </p>\n<p><a href=\"https://en.wikipedia.org/wiki/Levenshtein_distance#Computational_complexity\" target=\"_blank\">https://en.wikipedia.org/wiki/Levenshtein_distance#Computational_complexity</a></p>",
      "rawMarkdown": "I think this maybe \n\nif you valid lsd in very earlier get_levenshtein_distance will be very slow if you get a low lsd score , because it may reach the worst case of algorithm (BIG O) \n\nhttps://en.wikipedia.org/wiki/Levenshtein_distance#Computational_complexity",
      "votes": null
    },
    {
      "id": "1333066",
      "postDate": "06/02/2021 13:20:04",
      "content": "<p>I  loaded pretrained encoder in that notebook, that is probably the reason It is not that slow. BTW, transformer can not load the saved weights. I will try to save encoder and decoder.</p>",
      "rawMarkdown": "I  loaded pretrained encoder in that notebook, that is probably the reason It is not that slow. BTW, transformer can not load the saved weights. I will try to save encoder and decoder.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1332293,
      "author_name": "dragonzhang",
      "author_url": "",
      "post_date": "06/02/2021 03:13:51",
      "content": "<p>validation statistic takes too long time! </p>",
      "votes": null,
      "replies": [
        {
          "id": 1332486,
          "author_name": "drzhuzhe",
          "author_url": "",
          "post_date": "06/02/2021 06:33:08",
          "content": "<p>do you have any idea dealing with it？</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1332560,
          "author_name": "dragonzhang",
          "author_url": "",
          "post_date": "06/02/2021 07:24:18",
          "content": "<p>I will have a look. The Transformer  notebook in your reference does not cost that much time for validation LSD. <br>\nbtw,  I  stopped a few times and keep retraining, the LOSS/ACC fluctuation happens again.</p>\n<p>Perhaps like the way for iteration number like in Heng's script, a huge number like 80000x1000?</p>\n<p>That would need a local good machine to run. </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1332572,
          "author_name": "drzhuzhe",
          "author_url": "",
          "post_date": "06/02/2021 07:37:34",
          "content": "<blockquote>\n  <p>I will have a look. The Transformer notebook in your reference does not cost that much time for validation LSD.</p>\n</blockquote>\n<p>which one ？ this <a href=\"https://www.kaggle.com/dschettler8845/3-71-cv-bms-efficientnetv2-transformer-e2e\" target=\"_blank\">https://www.kaggle.com/dschettler8845/3-71-cv-bms-efficientnetv2-transformer-e2e</a>  ？？？</p>\n<p>it's very slow to validate too</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1332597,
          "author_name": "drzhuzhe",
          "author_url": "",
          "post_date": "06/02/2021 07:52:27",
          "content": "<p>I think this maybe </p>\n<p>if you valid lsd in very earlier get_levenshtein_distance will be very slow if you get a low lsd score , because it may reach the worst case of algorithm (BIG O) </p>\n<p><a href=\"https://en.wikipedia.org/wiki/Levenshtein_distance#Computational_complexity\" target=\"_blank\">https://en.wikipedia.org/wiki/Levenshtein_distance#Computational_complexity</a></p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1333066,
          "author_name": "dragonzhang",
          "author_url": "",
          "post_date": "06/02/2021 13:20:04",
          "content": "<p>I  loaded pretrained encoder in that notebook, that is probably the reason It is not that slow. BTW, transformer can not load the saved weights. I will try to save encoder and decoder.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1331121": "Even thought this competition is ver near end \n\nI finally come up with a vision transformer baseline \n\nlink: https://www.kaggle.com/drzhuzhe/bms-vision-transformer\n\nmaybe we can \n\n1. training more epoch\n2. fine tune via full MAX_LEN rather than HALF-LENGTH\n3. use VIT16-base \n4. use CNN to patch\n5. tuning my learning rate schedule \n\nI known it is too late ... for me\n\nBut still hope it useful and if you find something config wrong please tell me",
    "1332293": "validation statistic takes too long time!",
    "1332486": "do you have any idea dealing with it？",
    "1332560": "I will have a look. The Transformer  notebook in your reference does not cost that much time for validation LSD. \nbtw,  I  stopped a few times and keep retraining, the LOSS/ACC fluctuation happens again.\n\nPerhaps like the way for iteration number like in Heng's script, a huge number like 80000x1000?\n\nThat would need a local good machine to run.",
    "1332572": "> I will have a look. The Transformer notebook in your reference does not cost that much time for validation LSD.\n\nwhich one ？ this https://www.kaggle.com/dschettler8845/3-71-cv-bms-efficientnetv2-transformer-e2e  ？？？\n\nit's very slow to validate too",
    "1332597": "I think this maybe \n\nif you valid lsd in very earlier get_levenshtein_distance will be very slow if you get a low lsd score , because it may reach the worst case of algorithm (BIG O) \n\nhttps://en.wikipedia.org/wiki/Levenshtein_distance#Computational_complexity",
    "1333066": "I  loaded pretrained encoder in that notebook, that is probably the reason It is not that slow. BTW, transformer can not load the saved weights. I will try to save encoder and decoder."
  },
  "source": "meta"
}