{
  "id": 143932,
  "title": "Best score using Bert Multilingual",
  "url": "/competitions/jigsaw-multilingual-toxic-comment-classification/discussion/143932",
  "author_name": "",
  "post_date": "2020-04-16T21:19:57.209488400Z",
  "votes": 6,
  "comment_count": 5,
  "views": 0,
  "content": "<p>Hello everyone, </p>\n\n<p>Just a quick poll. I'd like to know what are your best scores on LB using Bert Multilingual. I have a hard time finetuning the hyperparameters (epochs, learning rate, scheduler...).</p>\n\n<p>For me:\n- Without training on validation set: 0.9188\n- With training on validation set: 0.9248</p>\n\n<p>Does anybody have great articles or tutorials for fine tuning Bert hyperparameters ? </p>\n\n<p>I've spent a while trying to finetune then but I can't manage to cross 0.93 on LB...</p>",
  "messages": [
    {
      "id": "810312",
      "postDate": "04/16/2020 21:19:57",
      "content": "<p>Hello everyone, </p>\n\n<p>Just a quick poll. I'd like to know what are your best scores on LB using Bert Multilingual. I have a hard time finetuning the hyperparameters (epochs, learning rate, scheduler...).</p>\n\n<p>For me:\n- Without training on validation set: 0.9188\n- With training on validation set: 0.9248</p>\n\n<p>Does anybody have great articles or tutorials for fine tuning Bert hyperparameters ? </p>\n\n<p>I've spent a while trying to finetune then but I can't manage to cross 0.93 on LB...</p>",
      "rawMarkdown": "Hello everyone, \n\nJust a quick poll. I'd like to know what are your best scores on LB using Bert Multilingual. I have a hard time finetuning the hyperparameters (epochs, learning rate, scheduler...).\n\nFor me:\n- Without training on validation set: 0.9188\n- With training on validation set: 0.9248\n\nDoes anybody have great articles or tutorials for fine tuning Bert hyperparameters ? \n\nI've spent a while trying to finetune then but I can't manage to cross 0.93 on LB...",
      "votes": null
    },
    {
      "id": "812138",
      "postDate": "04/18/2020 14:36:20",
      "content": "<p><a href=\"/rftexas\">@rftexas</a> , for the scores you mentioned, are they obtained with translated training dataset or without? I haven't tried translation yet, but with bert, i never get something like your scores.</p>",
      "rawMarkdown": "rftexas , for the scores you mentioned, are they obtained with translated training dataset or without? I haven't tried translation yet, but with bert, i never get something like your scores.",
      "votes": null
    },
    {
      "id": "812191",
      "postDate": "04/18/2020 15:16:30",
      "content": "<p>auc=0.90 on validation set ('validation.csv') after training on english training set. But the score does not seem to be consistent with PL score - from 0.88 to 0.90, but never has it been above.</p>",
      "rawMarkdown": "auc=0.90 on validation set ('validation.csv') after training on english training set. But the score does not seem to be consistent with PL score - from 0.88 to 0.90, but never has it been above.",
      "votes": null
    },
    {
      "id": "813261",
      "postDate": "04/19/2020 14:19:30",
      "content": "<p>Translated datasets ;)</p>",
      "rawMarkdown": "Translated datasets ;)",
      "votes": null
    },
    {
      "id": "823190",
      "postDate": "04/27/2020 13:34:12",
      "content": "<p>For me only training data with validation on validation set gave around 90.7\nAfter training on validation data I got close to 91.\nI am using pytorch xla and having a tough time fine-tuning my model. Could you explain your procedure/pipepline?\nAlso a weird thing to notice is that my lb score goes down as the number of epochs I am training on increases..\nAny help would be much appreciated</p>",
      "rawMarkdown": "For me only training data with validation on validation set gave around 90.7\nAfter training on validation data I got close to 91.\nI am using pytorch xla and having a tough time fine-tuning my model. Could you explain your procedure/pipepline?\nAlso a weird thing to notice is that my lb score goes down as the number of epochs I am training on increases..\nAny help would be much appreciated",
      "votes": null
    },
    {
      "id": "824732",
      "postDate": "04/28/2020 15:02:31",
      "content": "<p>Trained with all of the translated dataset, auc on validation set is 0.94+. Trained one more epoch on validation set, the lb score is 0.9325.</p>",
      "rawMarkdown": "Trained with all of the translated dataset, auc on validation set is 0.94+. Trained one more epoch on validation set, the lb score is 0.9325.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 812138,
      "author_name": "yihdarshieh",
      "author_url": "",
      "post_date": "04/18/2020 14:36:20",
      "content": "<p><a href=\"/rftexas\">@rftexas</a> , for the scores you mentioned, are they obtained with translated training dataset or without? I haven't tried translation yet, but with bert, i never get something like your scores.</p>",
      "votes": null,
      "replies": [
        {
          "id": 813261,
          "author_name": "rftexas",
          "author_url": "",
          "post_date": "04/19/2020 14:19:30",
          "content": "<p>Translated datasets ;)</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 812191,
      "author_name": "isakev",
      "author_url": "",
      "post_date": "04/18/2020 15:16:30",
      "content": "<p>auc=0.90 on validation set ('validation.csv') after training on english training set. But the score does not seem to be consistent with PL score - from 0.88 to 0.90, but never has it been above.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 823190,
      "author_name": "amoghjrules",
      "author_url": "",
      "post_date": "04/27/2020 13:34:12",
      "content": "<p>For me only training data with validation on validation set gave around 90.7\nAfter training on validation data I got close to 91.\nI am using pytorch xla and having a tough time fine-tuning my model. Could you explain your procedure/pipepline?\nAlso a weird thing to notice is that my lb score goes down as the number of epochs I am training on increases..\nAny help would be much appreciated</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 824732,
      "author_name": "tonyxu",
      "author_url": "",
      "post_date": "04/28/2020 15:02:31",
      "content": "<p>Trained with all of the translated dataset, auc on validation set is 0.94+. Trained one more epoch on validation set, the lb score is 0.9325.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "810312": "Hello everyone, \n\nJust a quick poll. I'd like to know what are your best scores on LB using Bert Multilingual. I have a hard time finetuning the hyperparameters (epochs, learning rate, scheduler...).\n\nFor me:\n- Without training on validation set: 0.9188\n- With training on validation set: 0.9248\n\nDoes anybody have great articles or tutorials for fine tuning Bert hyperparameters ? \n\nI've spent a while trying to finetune then but I can't manage to cross 0.93 on LB...",
    "812138": "rftexas , for the scores you mentioned, are they obtained with translated training dataset or without? I haven't tried translation yet, but with bert, i never get something like your scores.",
    "812191": "auc=0.90 on validation set ('validation.csv') after training on english training set. But the score does not seem to be consistent with PL score - from 0.88 to 0.90, but never has it been above.",
    "813261": "Translated datasets ;)",
    "823190": "For me only training data with validation on validation set gave around 90.7\nAfter training on validation data I got close to 91.\nI am using pytorch xla and having a tough time fine-tuning my model. Could you explain your procedure/pipepline?\nAlso a weird thing to notice is that my lb score goes down as the number of epochs I am training on increases..\nAny help would be much appreciated",
    "824732": "Trained with all of the translated dataset, auc on validation set is 0.94+. Trained one more epoch on validation set, the lb score is 0.9325."
  },
  "source": "meta"
}