{
  "id": 271501,
  "title": "A starter using the Vision Transformer (ViT)",
  "url": "/competitions/g2net-gravitational-wave-detection/discussion/271501",
  "author_name": "",
  "post_date": "2021-09-10T19:47:25.166939Z",
  "votes": 5,
  "comment_count": 3,
  "views": 0,
  "content": "<p><a href=\"https://www.kaggle.com/scaomath/g2net-vision-transformer-starter\" target=\"_blank\">https://www.kaggle.com/scaomath/g2net-vision-transformer-starter</a></p>\n<p>ViT's idea is pretty simple, first dividing the image into patches, then use the Attention mechanism to capture the global relations between the embedding of each patch.</p>\n<p>The AUC is about 0.04 better than the baseline <a href=\"https://www.kaggle.com/xhlulu\" target=\"_blank\">@xhlulu</a> provided. </p>\n<p>I will work on a PyTorch version later.</p>",
  "messages": [
    {
      "id": "1509075",
      "postDate": "09/10/2021 19:47:25",
      "content": "<p><a href=\"https://www.kaggle.com/scaomath/g2net-vision-transformer-starter\" target=\"_blank\">https://www.kaggle.com/scaomath/g2net-vision-transformer-starter</a></p>\n<p>ViT's idea is pretty simple, first dividing the image into patches, then use the Attention mechanism to capture the global relations between the embedding of each patch.</p>\n<p>The AUC is about 0.04 better than the baseline <a href=\"https://www.kaggle.com/xhlulu\" target=\"_blank\">@xhlulu</a> provided. </p>\n<p>I will work on a PyTorch version later.</p>",
      "rawMarkdown": "https://www.kaggle.com/scaomath/g2net-vision-transformer-starter\n\nViT's idea is pretty simple, first dividing the image into patches, then use the Attention mechanism to capture the global relations between the embedding of each patch.\n\nThe AUC is about 0.04 better than the baseline @xhlulu provided. \n\nI will work on a PyTorch version later.",
      "votes": null
    },
    {
      "id": "1514191",
      "postDate": "09/15/2021 18:53:07",
      "content": "<p>PyTorch version updated: <a href=\"https://www.kaggle.com/scaomath/g2net-vision-transformer-pytorch-baseline\" target=\"_blank\">https://www.kaggle.com/scaomath/g2net-vision-transformer-pytorch-baseline</a></p>",
      "rawMarkdown": "PyTorch version updated: https://www.kaggle.com/scaomath/g2net-vision-transformer-pytorch-baseline",
      "votes": null
    },
    {
      "id": "1514564",
      "postDate": "09/16/2021 07:41:28",
      "content": "<p>one may have interesting results if you use the transformer code to:</p>\n<ol>\n<li>self supervised learning with unlabelled public test data<br>\n(eg : <a href=\"https://github.com/microsoft/unilm/tree/master/beit\" target=\"_blank\">https://github.com/microsoft/unilm/tree/master/beit</a>)</li>\n<li>consistency loss with distillation token with wave 1d CNN</li>\n</ol>",
      "rawMarkdown": "one may have interesting results if you use the transformer code to:\n1. self supervised learning with unlabelled public test data\n(eg : https://github.com/microsoft/unilm/tree/master/beit)\n2. consistency loss with distillation token with wave 1d CNN",
      "votes": null
    },
    {
      "id": "1559717",
      "postDate": "10/27/2021 07:09:02",
      "content": "<p>Hey All,</p>\n<p>Thank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey <a href=\"https://forms.gle/QP9L16niPexozyhu5\" target=\"_blank\">https://forms.gle/QP9L16niPexozyhu5</a>.</p>\n<p>Thank you all,</p>\n<p>Regards,<br>\nChris</p>",
      "rawMarkdown": "Hey All,\n\nThank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.\n\nThank you all,\n\nRegards,\nChris",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1514191,
      "author_name": "scaomath",
      "author_url": "",
      "post_date": "09/15/2021 18:53:07",
      "content": "<p>PyTorch version updated: <a href=\"https://www.kaggle.com/scaomath/g2net-vision-transformer-pytorch-baseline\" target=\"_blank\">https://www.kaggle.com/scaomath/g2net-vision-transformer-pytorch-baseline</a></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1514564,
      "author_name": "hengck23",
      "author_url": "",
      "post_date": "09/16/2021 07:41:28",
      "content": "<p>one may have interesting results if you use the transformer code to:</p>\n<ol>\n<li>self supervised learning with unlabelled public test data<br>\n(eg : <a href=\"https://github.com/microsoft/unilm/tree/master/beit\" target=\"_blank\">https://github.com/microsoft/unilm/tree/master/beit</a>)</li>\n<li>consistency loss with distillation token with wave 1d CNN</li>\n</ol>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1559717,
      "author_name": "zerafachris",
      "author_url": "",
      "post_date": "10/27/2021 07:09:02",
      "content": "<p>Hey All,</p>\n<p>Thank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey <a href=\"https://forms.gle/QP9L16niPexozyhu5\" target=\"_blank\">https://forms.gle/QP9L16niPexozyhu5</a>.</p>\n<p>Thank you all,</p>\n<p>Regards,<br>\nChris</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1509075": "https://www.kaggle.com/scaomath/g2net-vision-transformer-starter\n\nViT's idea is pretty simple, first dividing the image into patches, then use the Attention mechanism to capture the global relations between the embedding of each patch.\n\nThe AUC is about 0.04 better than the baseline @xhlulu provided. \n\nI will work on a PyTorch version later.",
    "1514191": "PyTorch version updated: https://www.kaggle.com/scaomath/g2net-vision-transformer-pytorch-baseline",
    "1514564": "one may have interesting results if you use the transformer code to:\n1. self supervised learning with unlabelled public test data\n(eg : https://github.com/microsoft/unilm/tree/master/beit)\n2. consistency loss with distillation token with wave 1d CNN",
    "1559717": "Hey All,\n\nThank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.\n\nThank you all,\n\nRegards,\nChris"
  },
  "source": "meta"
}