{
  "id": 124158,
  "title": "Best Single Model Score",
  "url": "/competitions/tensorflow2-question-answering/discussion/124158",
  "author_name": "",
  "post_date": "2020-01-02T10:00:09.717004Z",
  "votes": 11,
  "comment_count": 25,
  "views": 0,
  "content": "<p>I just wanted to get an idea of how other top solutions are performing. My current best is single Update: Bert Large. CV: 0.645 LB: 0.62.</p>\n\n<p>What are the best single model scores of others?</p>",
  "messages": [
    {
      "id": "708443",
      "postDate": "01/02/2020 10:00:09",
      "content": "<p>I just wanted to get an idea of how other top solutions are performing. My current best is single Update: Bert Large. CV: 0.645 LB: 0.62.</p>\n\n<p>What are the best single model scores of others?</p>",
      "rawMarkdown": "I just wanted to get an idea of how other top solutions are performing. My current best is single Update: Bert Large. CV: 0.645 LB: 0.62.\n\nWhat are the best single model scores of others?",
      "votes": null
    },
    {
      "id": "708454",
      "postDate": "01/02/2020 10:10:25",
      "content": "<p>Thanks for starting this thread!\nSingle Bert Large. CV: N/A LB: 0.60.</p>",
      "rawMarkdown": "Thanks for starting this thread!\nSingle Bert Large. CV: N/A LB: 0.60.",
      "votes": null
    },
    {
      "id": "708572",
      "postDate": "01/02/2020 12:47:36",
      "content": "<p>Single Bert Base. CV: N/A LB:0.56.\nNo machine to train the large version.\nSo sad...</p>",
      "rawMarkdown": "Single Bert Base. CV: N/A LB:0.56.\nNo machine to train the large version.\nSo sad...",
      "votes": null
    },
    {
      "id": "708981",
      "postDate": "01/02/2020 22:16:51",
      "content": "<p>Barebone BERT-joint: dev 0.6, LB 0.62</p>",
      "rawMarkdown": "Barebone BERT-joint: dev 0.6, LB 0.62",
      "votes": null
    },
    {
      "id": "709062",
      "postDate": "01/03/2020 02:23:31",
      "content": "<p>I'm a little worried about the possibility of timeout in the final run of all test sets with large version. How to solve this problem?</p>",
      "rawMarkdown": "I'm a little worried about the possibility of timeout in the final run of all test sets with large version. How to solve this problem?",
      "votes": null
    },
    {
      "id": "709127",
      "postDate": "01/03/2020 04:21:20",
      "content": "<p><a href=\"/renxingkai\">@renxingkai</a> if you submitted using kernel and it ran successfully that means your solution will work on private LB too. This is synchronous kernel contest where your submission is ran on whole private test set when you submit. If it would have failed on larger dataset you wouldn't get score</p>",
      "rawMarkdown": "renxingkai if you submitted using kernel and it ran successfully that means your solution will work on private LB too. This is synchronous kernel contest where your submission is ran on whole private test set when you submit. If it would have failed on larger dataset you wouldn't get score",
      "votes": null
    },
    {
      "id": "709128",
      "postDate": "01/03/2020 04:22:08",
      "content": "<p><a href=\"/kashnitsky\">@kashnitsky</a> as far as I have tried in my experiments my CV is always &gt; LB score. Any idea what could be different with your solution? </p>",
      "rawMarkdown": "kashnitsky as far as I have tried in my experiments my CV is always &gt; LB score. Any idea what could be different with your solution?",
      "votes": null
    },
    {
      "id": "709135",
      "postDate": "01/03/2020 04:29:51",
      "content": "<p>how many epochs have you run?</p>",
      "rawMarkdown": "how many epochs have you run?",
      "votes": null
    },
    {
      "id": "709265",
      "postDate": "01/03/2020 09:47:50",
      "content": "<p>I don't do cross-validation, it's just a holdout score on dev set</p>",
      "rawMarkdown": "I don't do cross-validation, it's just a holdout score on dev set",
      "votes": null
    },
    {
      "id": "709401",
      "postDate": "01/03/2020 13:15:19",
      "content": "<p>Thanks for clarification~</p>",
      "rawMarkdown": "Thanks for clarification~",
      "votes": null
    },
    {
      "id": "709467",
      "postDate": "01/03/2020 14:45:38",
      "content": "<p>Single model LB 0.58</p>",
      "rawMarkdown": "Single model LB 0.58",
      "votes": null
    },
    {
      "id": "710635",
      "postDate": "01/05/2020 02:38:04",
      "content": "<p>About 1/2 epoch.</p>",
      "rawMarkdown": "About 1/2 epoch.",
      "votes": null
    },
    {
      "id": "711022",
      "postDate": "01/05/2020 15:10:18",
      "content": "<p>You could have to use TPU nodes, it can train all large architectures ..</p>",
      "rawMarkdown": "You could have to use TPU nodes, it can train all large architectures ..",
      "votes": null
    },
    {
      "id": "711303",
      "postDate": "01/05/2020 22:49:51",
      "content": "<p>Just scored 0.15 with a test run of QA-Net. It's no LB killer...but it is likely one of the few models where training from scratch and inference for submission can be done in less then 3 hours.</p>",
      "rawMarkdown": "Just scored 0.15 with a test run of QA-Net. It's no LB killer...but it is likely one of the few models where training from scratch and inference for submission can be done in less then 3 hours.",
      "votes": null
    },
    {
      "id": "711633",
      "postDate": "01/06/2020 10:37:21",
      "content": "<p>val 0.66 lb 0.67，but never improve on lb even val become 0.69，so sad story</p>",
      "rawMarkdown": "val 0.66 lb 0.67，but never improve on lb even val become 0.69，so sad story",
      "votes": null
    },
    {
      "id": "711667",
      "postDate": "01/06/2020 11:24:37",
      "content": "<p><a href=\"/zhaomeng1126\">@zhaomeng1126</a> my best val with new model is around ~0.64, I guess there's lot of scope to improve</p>",
      "rawMarkdown": "zhaomeng1126 my best val with new model is around ~0.64, I guess there's lot of scope to improve",
      "votes": null
    },
    {
      "id": "711672",
      "postDate": "01/06/2020 11:31:53",
      "content": "<p>do you use pytorch or tensorflow?</p>",
      "rawMarkdown": "do you use pytorch or tensorflow?",
      "votes": null
    },
    {
      "id": "711677",
      "postDate": "01/06/2020 11:37:28",
      "content": "<p>I'm using PyTorch LB 0.65 CV N/A.</p>",
      "rawMarkdown": "I'm using PyTorch LB 0.65 CV N/A.",
      "votes": null
    },
    {
      "id": "711804",
      "postDate": "01/06/2020 14:47:21",
      "content": "<p>I'm using TF</p>",
      "rawMarkdown": "I'm using TF",
      "votes": null
    },
    {
      "id": "712725",
      "postDate": "01/07/2020 14:35:30",
      "content": "<p>I'm almost like you. CV is a little lower than LB</p>",
      "rawMarkdown": "I'm almost like you. CV is a little lower than LB",
      "votes": null
    },
    {
      "id": "712761",
      "postDate": "01/07/2020 15:05:38",
      "content": "<p>Are you guys training a new model or just searching post processing parameters like people in open kernels?</p>",
      "rawMarkdown": "Are you guys training a new model or just searching post processing parameters like people in open kernels?",
      "votes": null
    },
    {
      "id": "712806",
      "postDate": "01/07/2020 15:50:02",
      "content": "<p>I spend most of my time searching post processing parameters. But I think it is just overfitting to public test dataset 😭</p>",
      "rawMarkdown": "I spend most of my time searching post processing parameters. But I think it is just overfitting to public test dataset 😭",
      "votes": null
    },
    {
      "id": "713939",
      "postDate": "01/08/2020 20:39:32",
      "content": "",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "714116",
      "postDate": "01/09/2020 04:36:49",
      "content": "<p>Same here  :P</p>",
      "rawMarkdown": "Same here  :P",
      "votes": null
    },
    {
      "id": "717052",
      "postDate": "01/12/2020 16:59:15",
      "content": "<p>For me, score on LB is not stable. dev score 0.71+ got only 0.67 for LB.  While dev score 0.69+ got 0.69 for LB.</p>",
      "rawMarkdown": "For me, score on LB is not stable. dev score 0.71+ got only 0.67 for LB.  While dev score 0.69+ got 0.69 for LB.",
      "votes": null
    },
    {
      "id": "722417",
      "postDate": "01/18/2020 14:54:00",
      "content": "<p><a href=\"/fryzito\">@fryzito</a> seems like many of the kagglers disagree to your opinion.🤒 </p>",
      "rawMarkdown": "fryzito seems like many of the kagglers disagree to your opinion.🤒",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 708454,
      "author_name": "higepon",
      "author_url": "",
      "post_date": "01/02/2020 10:10:25",
      "content": "<p>Thanks for starting this thread!\nSingle Bert Large. CV: N/A LB: 0.60.</p>",
      "votes": null,
      "replies": [
        {
          "id": 709062,
          "author_name": "renxingkai",
          "author_url": "",
          "post_date": "01/03/2020 02:23:31",
          "content": "<p>I'm a little worried about the possibility of timeout in the final run of all test sets with large version. How to solve this problem?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 709127,
          "author_name": "axel81",
          "author_url": "",
          "post_date": "01/03/2020 04:21:20",
          "content": "<p><a href=\"/renxingkai\">@renxingkai</a> if you submitted using kernel and it ran successfully that means your solution will work on private LB too. This is synchronous kernel contest where your submission is ran on whole private test set when you submit. If it would have failed on larger dataset you wouldn't get score</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 709401,
          "author_name": "renxingkai",
          "author_url": "",
          "post_date": "01/03/2020 13:15:19",
          "content": "<p>Thanks for clarification~</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 708572,
      "author_name": "renxingkai",
      "author_url": "",
      "post_date": "01/02/2020 12:47:36",
      "content": "<p>Single Bert Base. CV: N/A LB:0.56.\nNo machine to train the large version.\nSo sad...</p>",
      "votes": null,
      "replies": [
        {
          "id": 711022,
          "author_name": "hakeem",
          "author_url": "",
          "post_date": "01/05/2020 15:10:18",
          "content": "<p>You could have to use TPU nodes, it can train all large architectures ..</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 708981,
      "author_name": "kashnitsky",
      "author_url": "",
      "post_date": "01/02/2020 22:16:51",
      "content": "<p>Barebone BERT-joint: dev 0.6, LB 0.62</p>",
      "votes": null,
      "replies": [
        {
          "id": 709128,
          "author_name": "axel81",
          "author_url": "",
          "post_date": "01/03/2020 04:22:08",
          "content": "<p><a href=\"/kashnitsky\">@kashnitsky</a> as far as I have tried in my experiments my CV is always &gt; LB score. Any idea what could be different with your solution? </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 709265,
          "author_name": "kashnitsky",
          "author_url": "",
          "post_date": "01/03/2020 09:47:50",
          "content": "<p>I don't do cross-validation, it's just a holdout score on dev set</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 712725,
          "author_name": "wochidadonggua",
          "author_url": "",
          "post_date": "01/07/2020 14:35:30",
          "content": "<p>I'm almost like you. CV is a little lower than LB</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 712761,
          "author_name": "igormunizims",
          "author_url": "",
          "post_date": "01/07/2020 15:05:38",
          "content": "<p>Are you guys training a new model or just searching post processing parameters like people in open kernels?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 712806,
          "author_name": "wochidadonggua",
          "author_url": "",
          "post_date": "01/07/2020 15:50:02",
          "content": "<p>I spend most of my time searching post processing parameters. But I think it is just overfitting to public test dataset 😭</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 714116,
          "author_name": "axel81",
          "author_url": "",
          "post_date": "01/09/2020 04:36:49",
          "content": "<p>Same here  :P</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 717052,
          "author_name": "tonyxu",
          "author_url": "",
          "post_date": "01/12/2020 16:59:15",
          "content": "<p>For me, score on LB is not stable. dev score 0.71+ got only 0.67 for LB.  While dev score 0.69+ got 0.69 for LB.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 709135,
      "author_name": "",
      "author_url": "",
      "post_date": "01/03/2020 04:29:51",
      "content": "<p>how many epochs have you run?</p>",
      "votes": null,
      "replies": [
        {
          "id": 710635,
          "author_name": "higepon",
          "author_url": "",
          "post_date": "01/05/2020 02:38:04",
          "content": "<p>About 1/2 epoch.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 709467,
      "author_name": "rohitagarwal",
      "author_url": "",
      "post_date": "01/03/2020 14:45:38",
      "content": "<p>Single model LB 0.58</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 711303,
      "author_name": "rsmits",
      "author_url": "",
      "post_date": "01/05/2020 22:49:51",
      "content": "<p>Just scored 0.15 with a test run of QA-Net. It's no LB killer...but it is likely one of the few models where training from scratch and inference for submission can be done in less then 3 hours.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 711633,
      "author_name": "zhaomeng1126",
      "author_url": "",
      "post_date": "01/06/2020 10:37:21",
      "content": "<p>val 0.66 lb 0.67，but never improve on lb even val become 0.69，so sad story</p>",
      "votes": null,
      "replies": [
        {
          "id": 711667,
          "author_name": "axel81",
          "author_url": "",
          "post_date": "01/06/2020 11:24:37",
          "content": "<p><a href=\"/zhaomeng1126\">@zhaomeng1126</a> my best val with new model is around ~0.64, I guess there's lot of scope to improve</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 711672,
      "author_name": "abhishek",
      "author_url": "",
      "post_date": "01/06/2020 11:31:53",
      "content": "<p>do you use pytorch or tensorflow?</p>",
      "votes": null,
      "replies": [
        {
          "id": 711677,
          "author_name": "higepon",
          "author_url": "",
          "post_date": "01/06/2020 11:37:28",
          "content": "<p>I'm using PyTorch LB 0.65 CV N/A.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 711804,
          "author_name": "axel81",
          "author_url": "",
          "post_date": "01/06/2020 14:47:21",
          "content": "<p>I'm using TF</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 713939,
          "author_name": "fryzito",
          "author_url": "",
          "post_date": "01/08/2020 20:39:32",
          "content": "",
          "votes": null,
          "replies": []
        },
        {
          "id": 722417,
          "author_name": "shahules",
          "author_url": "",
          "post_date": "01/18/2020 14:54:00",
          "content": "<p><a href=\"/fryzito\">@fryzito</a> seems like many of the kagglers disagree to your opinion.🤒 </p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "708443": "I just wanted to get an idea of how other top solutions are performing. My current best is single Update: Bert Large. CV: 0.645 LB: 0.62.\n\nWhat are the best single model scores of others?",
    "708454": "Thanks for starting this thread!\nSingle Bert Large. CV: N/A LB: 0.60.",
    "708572": "Single Bert Base. CV: N/A LB:0.56.\nNo machine to train the large version.\nSo sad...",
    "708981": "Barebone BERT-joint: dev 0.6, LB 0.62",
    "709062": "I'm a little worried about the possibility of timeout in the final run of all test sets with large version. How to solve this problem?",
    "709127": "renxingkai if you submitted using kernel and it ran successfully that means your solution will work on private LB too. This is synchronous kernel contest where your submission is ran on whole private test set when you submit. If it would have failed on larger dataset you wouldn't get score",
    "709128": "kashnitsky as far as I have tried in my experiments my CV is always &gt; LB score. Any idea what could be different with your solution?",
    "709135": "how many epochs have you run?",
    "709265": "I don't do cross-validation, it's just a holdout score on dev set",
    "709401": "Thanks for clarification~",
    "709467": "Single model LB 0.58",
    "710635": "About 1/2 epoch.",
    "711022": "You could have to use TPU nodes, it can train all large architectures ..",
    "711303": "Just scored 0.15 with a test run of QA-Net. It's no LB killer...but it is likely one of the few models where training from scratch and inference for submission can be done in less then 3 hours.",
    "711633": "val 0.66 lb 0.67，but never improve on lb even val become 0.69，so sad story",
    "711667": "zhaomeng1126 my best val with new model is around ~0.64, I guess there's lot of scope to improve",
    "711672": "do you use pytorch or tensorflow?",
    "711677": "I'm using PyTorch LB 0.65 CV N/A.",
    "711804": "I'm using TF",
    "712725": "I'm almost like you. CV is a little lower than LB",
    "712761": "Are you guys training a new model or just searching post processing parameters like people in open kernels?",
    "712806": "I spend most of my time searching post processing parameters. But I think it is just overfitting to public test dataset 😭",
    "713939": "",
    "714116": "Same here  :P",
    "717052": "For me, score on LB is not stable. dev score 0.71+ got only 0.67 for LB.  While dev score 0.69+ got 0.69 for LB.",
    "722417": "fryzito seems like many of the kagglers disagree to your opinion.🤒"
  },
  "source": "meta"
}