{
  "id": 70722,
  "title": "BERT from Google AI?",
  "url": "/competitions/quora-insincere-questions-classification/discussion/70722",
  "author_name": "",
  "post_date": "2018-11-06T19:31:24.112532400Z",
  "votes": 23,
  "comment_count": 10,
  "views": 0,
  "content": "<p>Would anyone want to try BERT? Seems it is very cool but I am not sure how to use it in kernel: <a href=\"https://github.com/google-research/bert#sentence-and-sentence-pair-classification-tasks\">https://github.com/google-research/bert#sentence-and-sentence-pair-classification-tasks</a></p>",
  "messages": [
    {
      "id": "416515",
      "postDate": "11/06/2018 19:31:24",
      "content": "<p>Would anyone want to try BERT? Seems it is very cool but I am not sure how to use it in kernel: <a href=\"https://github.com/google-research/bert#sentence-and-sentence-pair-classification-tasks\">https://github.com/google-research/bert#sentence-and-sentence-pair-classification-tasks</a></p>",
      "rawMarkdown": "Would anyone want to try BERT? Seems it is very cool but I am not sure how to use it in kernel: https://github.com/google-research/bert#sentence-and-sentence-pair-classification-tasks",
      "votes": null
    },
    {
      "id": "416529",
      "postDate": "11/06/2018 20:29:09",
      "content": "<p>I am afraid external models and data other than the whitelisted embeddings attached in the data section are not allowed</p>",
      "rawMarkdown": "I am afraid external models and data other than the whitelisted embeddings attached in the data section are not allowed",
      "votes": null
    },
    {
      "id": "416535",
      "postDate": "11/06/2018 20:45:33",
      "content": "<p>Yeah, external data is not allowed. I wonder whether we can train that from scratch inside the kernel.</p>",
      "rawMarkdown": "Yeah, external data is not allowed. I wonder whether we can train that from scratch inside the kernel.",
      "votes": null
    },
    {
      "id": "416575",
      "postDate": "11/06/2018 22:39:39",
      "content": "<p>With 2 hours of K80 GPU time this seems unlikely</p>",
      "rawMarkdown": "With 2 hours of K80 GPU time this seems unlikely",
      "votes": null
    },
    {
      "id": "416610",
      "postDate": "11/07/2018 01:08:48",
      "content": "<p>bert is too expensive to train</p>",
      "rawMarkdown": "bert is too expensive to train",
      "votes": null
    },
    {
      "id": "416629",
      "postDate": "11/07/2018 02:21:48",
      "content": "<p>$3 * 64 TPUs * 24 hrs * 4 days.... Yeah, very expensive</p>",
      "rawMarkdown": "$3 * 64 TPUs * 24 hrs * 4 days.... Yeah, very expensive",
      "votes": null
    },
    {
      "id": "416746",
      "postDate": "11/07/2018 08:03:33",
      "content": "<p>That's for training from scratch. You can fine tune in a lot less time.</p>",
      "rawMarkdown": "That's for training from scratch. You can fine tune in a lot less time.",
      "votes": null
    },
    {
      "id": "416809",
      "postDate": "11/07/2018 10:00:11",
      "content": "<p>One could probably train such a model from scratch with just the target data. Training time is very limited, but most gain comes in beginning of the training and later epochs are lot smaller fine-tuning.</p>\n\n<p>Would be interesting to see how good score such can reach.</p>",
      "rawMarkdown": "One could probably train such a model from scratch with just the target data. Training time is very limited, but most gain comes in beginning of the training and later epochs are lot smaller fine-tuning.\n\nWould be interesting to see how good score such can reach.",
      "votes": null
    },
    {
      "id": "416842",
      "postDate": "11/07/2018 11:08:36",
      "content": "<p>BET you can not load that</p>",
      "rawMarkdown": "BET you can not load that",
      "votes": null
    },
    {
      "id": "459717",
      "postDate": "01/22/2019 08:34:29",
      "content": "<p>Did you try this from the pre-trained models? Understand the competition does not allow pre-trained, but more from a performance curiosity, I'm particularly interested :)  </p>",
      "rawMarkdown": "Did you try this from the pre-trained models? Understand the competition does not allow pre-trained, but more from a performance curiosity, I'm particularly interested :)",
      "votes": null
    },
    {
      "id": "1037997",
      "postDate": "10/05/2020 13:20:54",
      "content": "<p>After reviewing all the answers, I have a question: Does any one have a BERT kernel?<br>\nThanks.</p>",
      "rawMarkdown": "After reviewing all the answers, I have a question: Does any one have a BERT kernel?\nThanks.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1037997,
      "author_name": "ianalyticsgeek",
      "author_url": "",
      "post_date": "10/05/2020 13:20:54",
      "content": "<p>After reviewing all the answers, I have a question: Does any one have a BERT kernel?<br>\nThanks.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 416529,
      "author_name": "christofhenkel",
      "author_url": "",
      "post_date": "11/06/2018 20:29:09",
      "content": "<p>I am afraid external models and data other than the whitelisted embeddings attached in the data section are not allowed</p>",
      "votes": null,
      "replies": [
        {
          "id": 416535,
          "author_name": "shujian",
          "author_url": "",
          "post_date": "11/06/2018 20:45:33",
          "content": "<p>Yeah, external data is not allowed. I wonder whether we can train that from scratch inside the kernel.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 416575,
          "author_name": "borowis",
          "author_url": "",
          "post_date": "11/06/2018 22:39:39",
          "content": "<p>With 2 hours of K80 GPU time this seems unlikely</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 416809,
          "author_name": "jannen",
          "author_url": "",
          "post_date": "11/07/2018 10:00:11",
          "content": "<p>One could probably train such a model from scratch with just the target data. Training time is very limited, but most gain comes in beginning of the training and later epochs are lot smaller fine-tuning.</p>\n\n<p>Would be interesting to see how good score such can reach.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 416610,
      "author_name": "alexyung757",
      "author_url": "",
      "post_date": "11/07/2018 01:08:48",
      "content": "<p>bert is too expensive to train</p>",
      "votes": null,
      "replies": [
        {
          "id": 416629,
          "author_name": "shujian",
          "author_url": "",
          "post_date": "11/07/2018 02:21:48",
          "content": "<p>$3 * 64 TPUs * 24 hrs * 4 days.... Yeah, very expensive</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 416746,
          "author_name": "nlothian",
          "author_url": "",
          "post_date": "11/07/2018 08:03:33",
          "content": "<p>That's for training from scratch. You can fine tune in a lot less time.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 416842,
      "author_name": "chenyangh",
      "author_url": "",
      "post_date": "11/07/2018 11:08:36",
      "content": "<p>BET you can not load that</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 459717,
      "author_name": "sebastianvermaas",
      "author_url": "",
      "post_date": "01/22/2019 08:34:29",
      "content": "<p>Did you try this from the pre-trained models? Understand the competition does not allow pre-trained, but more from a performance curiosity, I'm particularly interested :)  </p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "416515": "Would anyone want to try BERT? Seems it is very cool but I am not sure how to use it in kernel: https://github.com/google-research/bert#sentence-and-sentence-pair-classification-tasks",
    "416529": "I am afraid external models and data other than the whitelisted embeddings attached in the data section are not allowed",
    "416535": "Yeah, external data is not allowed. I wonder whether we can train that from scratch inside the kernel.",
    "416575": "With 2 hours of K80 GPU time this seems unlikely",
    "416610": "bert is too expensive to train",
    "416629": "$3 * 64 TPUs * 24 hrs * 4 days.... Yeah, very expensive",
    "416746": "That's for training from scratch. You can fine tune in a lot less time.",
    "416809": "One could probably train such a model from scratch with just the target data. Training time is very limited, but most gain comes in beginning of the training and later epochs are lot smaller fine-tuning.\n\nWould be interesting to see how good score such can reach.",
    "416842": "BET you can not load that",
    "459717": "Did you try this from the pre-trained models? Understand the competition does not allow pre-trained, but more from a performance curiosity, I'm particularly interested :)",
    "1037997": "After reviewing all the answers, I have a question: Does any one have a BERT kernel?\nThanks."
  },
  "source": "meta"
}