{
  "id": 120931,
  "title": "We're going to be updating the leaderboard...",
  "url": "/competitions/tensorflow2-question-answering/discussion/120931",
  "author_name": "",
  "post_date": "2019-12-10T01:10:48.407060700Z",
  "votes": 7,
  "comment_count": 16,
  "views": 0,
  "content": "<p>Hi all -</p>\n\n<p>Thanks for all of your feedback! Thanks to help from @christofhenkel, @boliu0, and many others, we've identified a bug in the metric. False positives were not being fully accounted for.</p>\n\n<p>We've updated the metric, and we're going to update the leaderboard tonight. I'll post a message when the process is complete. In the meantime, any submissions you make will be scored using the updated metric.</p>",
  "messages": [
    {
      "id": "691313",
      "postDate": "12/10/2019 01:10:48",
      "content": "<p>Hi all -</p>\n\n<p>Thanks for all of your feedback! Thanks to help from @christofhenkel, @boliu0, and many others, we've identified a bug in the metric. False positives were not being fully accounted for.</p>\n\n<p>We've updated the metric, and we're going to update the leaderboard tonight. I'll post a message when the process is complete. In the meantime, any submissions you make will be scored using the updated metric.</p>",
      "rawMarkdown": "Hi all -\n\nThanks for all of your feedback! Thanks to help from @christofhenkel, @boliu0, and many others, we've identified a bug in the metric. False positives were not being fully accounted for.\n\nWe've updated the metric, and we're going to update the leaderboard tonight. I'll post a message when the process is complete. In the meantime, any submissions you make will be scored using the updated metric.",
      "votes": null
    },
    {
      "id": "691400",
      "postDate": "12/10/2019 05:07:54",
      "content": "<p>Hey Phil, thanks for the update!</p>\n\n<p>Would you and your team be able to share the metric with the competitors in order to be able to work on better models for this competition? Or maybe rock-solid documentation (without any ambiguity) on the metric?</p>\n\n<p>Thanks,\nMasoud</p>",
      "rawMarkdown": "Hey Phil, thanks for the update!\n\nWould you and your team be able to share the metric with the competitors in order to be able to work on better models for this competition? Or maybe rock-solid documentation (without any ambiguity) on the metric?\n\nThanks,\nMasoud",
      "votes": null
    },
    {
      "id": "691520",
      "postDate": "12/10/2019 08:15:30",
      "content": "<p><a href=\"https://www.kaggle.com/c/tensorflow2-question-answering/discussion/120030#\">A bit confused</a>\nThis no longer looks confused</p>",
      "rawMarkdown": "[A bit confused](https://www.kaggle.com/c/tensorflow2-question-answering/discussion/120030#)\nThis no longer looks confused",
      "votes": null
    },
    {
      "id": "691963",
      "postDate": "12/10/2019 17:43:38",
      "content": "<p>I almost got a heartbreak until I realized the rescoring wasn't completed yet... :D</p>",
      "rawMarkdown": "I almost got a heartbreak until I realized the rescoring wasn't completed yet... :D",
      "votes": null
    },
    {
      "id": "692575",
      "postDate": "12/11/2019 13:44:42",
      "content": "<p>1M to go, changing the metric. 😂 </p>",
      "rawMarkdown": "1M to go, changing the metric. 😂",
      "votes": null
    },
    {
      "id": "692582",
      "postDate": "12/11/2019 13:47:45",
      "content": "<p>The metric wasn't changed, the private LB was. Now the LB match the metric.  </p>",
      "rawMarkdown": "The metric wasn't changed, the private LB was. Now the LB match the metric.",
      "votes": null
    },
    {
      "id": "692596",
      "postDate": "12/11/2019 14:10:18",
      "content": "<p>That's true. The description is correct where I should create my score without matching the exam (LB). That's life.🤓 Thank you for replying.</p>",
      "rawMarkdown": "That's true. The description is correct where I should create my score without matching the exam (LB). That's life.🤓 Thank you for replying.",
      "votes": null
    },
    {
      "id": "692675",
      "postDate": "12/11/2019 15:59:33",
      "content": "<p>Hi Phil, new submissions and all my old submissions are reflecting the correct metrics now. But it looks like LB update has not been finished for some teams? When do you expect it to be done?</p>\n\n<p>Thanks</p>",
      "rawMarkdown": "Hi Phil, new submissions and all my old submissions are reflecting the correct metrics now. But it looks like LB update has not been finished for some teams? When do you expect it to be done?\n\nThanks",
      "votes": null
    },
    {
      "id": "693600",
      "postDate": "12/12/2019 14:57:21",
      "content": "<p><a href=\"/yuval6967\">@yuval6967</a> </p>\n\n<blockquote>\n  <p>Now the LB match the metric</p>\n</blockquote>\n\n<p>how do you know?</p>",
      "rawMarkdown": "yuval6967 \n&gt; Now the LB match the metric\n\nhow do you know?",
      "votes": null
    },
    {
      "id": "693658",
      "postDate": "12/12/2019 16:33:54",
      "content": "<p><a href=\"/christofhenkel\">@christofhenkel</a> A few weeks ago I considered entering this competition. As I always do, I started by building a scoring function for CV. When I tried a simple model I could only get CV ~0.48 while everyone got 0.7 on LB. I suspected either my model has a bug, or something isn't right with the competition. Didn't have time to dig in then and ended up in another competition. But now I see the common base line kernel gets LB 0.48, which makes sense (don't know about the highest ranking LBs).</p>",
      "rawMarkdown": "christofhenkel A few weeks ago I considered entering this competition. As I always do, I started by building a scoring function for CV. When I tried a simple model I could only get CV ~0.48 while everyone got 0.7 on LB. I suspected either my model has a bug, or something isn't right with the competition. Didn't have time to dig in then and ended up in another competition. But now I see the common base line kernel gets LB 0.48, which makes sense (don't know about the highest ranking LBs).",
      "votes": null
    },
    {
      "id": "693688",
      "postDate": "12/12/2019 17:13:35",
      "content": "<p><a href=\"/philculliton\">@philculliton</a> </p>",
      "rawMarkdown": "philculliton",
      "votes": null
    },
    {
      "id": "693896",
      "postDate": "12/12/2019 22:39:37",
      "content": "<p><a href=\"/philculliton\">@philculliton</a> </p>",
      "rawMarkdown": "philculliton",
      "votes": null
    },
    {
      "id": "693957",
      "postDate": "12/13/2019 01:44:03",
      "content": "<p>Because 0.7+ doesn't update until now</p>",
      "rawMarkdown": "Because 0.7+ doesn't update until now",
      "votes": null
    },
    {
      "id": "694116",
      "postDate": "12/13/2019 07:57:39",
      "content": "<p>The update is still in progress. We will update this thread when it is complete. Thank you for your patience &amp; sorry for the inconvenience.</p>",
      "rawMarkdown": "The update is still in progress. We will update this thread when it is complete. Thank you for your patience &amp; sorry for the inconvenience.",
      "votes": null
    },
    {
      "id": "694126",
      "postDate": "12/13/2019 08:10:39",
      "content": "<p>🤓 thank you!</p>",
      "rawMarkdown": "🤓 thank you!",
      "votes": null
    },
    {
      "id": "694519",
      "postDate": "12/13/2019 18:27:51",
      "content": "<p>seems to be complete now</p>",
      "rawMarkdown": "seems to be complete now",
      "votes": null
    },
    {
      "id": "694525",
      "postDate": "12/13/2019 18:32:23",
      "content": "<p>yep</p>",
      "rawMarkdown": "yep",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 691400,
      "author_name": "hoveidar",
      "author_url": "",
      "post_date": "12/10/2019 05:07:54",
      "content": "<p>Hey Phil, thanks for the update!</p>\n\n<p>Would you and your team be able to share the metric with the competitors in order to be able to work on better models for this competition? Or maybe rock-solid documentation (without any ambiguity) on the metric?</p>\n\n<p>Thanks,\nMasoud</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 691520,
      "author_name": "zhaomeng1126",
      "author_url": "",
      "post_date": "12/10/2019 08:15:30",
      "content": "<p><a href=\"https://www.kaggle.com/c/tensorflow2-question-answering/discussion/120030#\">A bit confused</a>\nThis no longer looks confused</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 691963,
      "author_name": "cpmpml",
      "author_url": "",
      "post_date": "12/10/2019 17:43:38",
      "content": "<p>I almost got a heartbreak until I realized the rescoring wasn't completed yet... :D</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 692575,
      "author_name": "xincuimath",
      "author_url": "",
      "post_date": "12/11/2019 13:44:42",
      "content": "<p>1M to go, changing the metric. 😂 </p>",
      "votes": null,
      "replies": [
        {
          "id": 692582,
          "author_name": "yuval6967",
          "author_url": "",
          "post_date": "12/11/2019 13:47:45",
          "content": "<p>The metric wasn't changed, the private LB was. Now the LB match the metric.  </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 692596,
          "author_name": "xincuimath",
          "author_url": "",
          "post_date": "12/11/2019 14:10:18",
          "content": "<p>That's true. The description is correct where I should create my score without matching the exam (LB). That's life.🤓 Thank you for replying.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 693600,
          "author_name": "christofhenkel",
          "author_url": "",
          "post_date": "12/12/2019 14:57:21",
          "content": "<p><a href=\"/yuval6967\">@yuval6967</a> </p>\n\n<blockquote>\n  <p>Now the LB match the metric</p>\n</blockquote>\n\n<p>how do you know?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 693658,
          "author_name": "yuval6967",
          "author_url": "",
          "post_date": "12/12/2019 16:33:54",
          "content": "<p><a href=\"/christofhenkel\">@christofhenkel</a> A few weeks ago I considered entering this competition. As I always do, I started by building a scoring function for CV. When I tried a simple model I could only get CV ~0.48 while everyone got 0.7 on LB. I suspected either my model has a bug, or something isn't right with the competition. Didn't have time to dig in then and ended up in another competition. But now I see the common base line kernel gets LB 0.48, which makes sense (don't know about the highest ranking LBs).</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 693957,
          "author_name": "zhaomeng1126",
          "author_url": "",
          "post_date": "12/13/2019 01:44:03",
          "content": "<p>Because 0.7+ doesn't update until now</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 692675,
      "author_name": "boliu0",
      "author_url": "",
      "post_date": "12/11/2019 15:59:33",
      "content": "<p>Hi Phil, new submissions and all my old submissions are reflecting the correct metrics now. But it looks like LB update has not been finished for some teams? When do you expect it to be done?</p>\n\n<p>Thanks</p>",
      "votes": null,
      "replies": [
        {
          "id": 693688,
          "author_name": "boliu0",
          "author_url": "",
          "post_date": "12/12/2019 17:13:35",
          "content": "<p><a href=\"/philculliton\">@philculliton</a> </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 693896,
          "author_name": "xincuimath",
          "author_url": "",
          "post_date": "12/12/2019 22:39:37",
          "content": "<p><a href=\"/philculliton\">@philculliton</a> </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 694116,
          "author_name": "juliaelliott",
          "author_url": "",
          "post_date": "12/13/2019 07:57:39",
          "content": "<p>The update is still in progress. We will update this thread when it is complete. Thank you for your patience &amp; sorry for the inconvenience.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 694126,
          "author_name": "xincuimath",
          "author_url": "",
          "post_date": "12/13/2019 08:10:39",
          "content": "<p>🤓 thank you!</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 694519,
          "author_name": "christofhenkel",
          "author_url": "",
          "post_date": "12/13/2019 18:27:51",
          "content": "<p>seems to be complete now</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 694525,
          "author_name": "boliu0",
          "author_url": "",
          "post_date": "12/13/2019 18:32:23",
          "content": "<p>yep</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "691313": "Hi all -\n\nThanks for all of your feedback! Thanks to help from @christofhenkel, @boliu0, and many others, we've identified a bug in the metric. False positives were not being fully accounted for.\n\nWe've updated the metric, and we're going to update the leaderboard tonight. I'll post a message when the process is complete. In the meantime, any submissions you make will be scored using the updated metric.",
    "691400": "Hey Phil, thanks for the update!\n\nWould you and your team be able to share the metric with the competitors in order to be able to work on better models for this competition? Or maybe rock-solid documentation (without any ambiguity) on the metric?\n\nThanks,\nMasoud",
    "691520": "[A bit confused](https://www.kaggle.com/c/tensorflow2-question-answering/discussion/120030#)\nThis no longer looks confused",
    "691963": "I almost got a heartbreak until I realized the rescoring wasn't completed yet... :D",
    "692575": "1M to go, changing the metric. 😂",
    "692582": "The metric wasn't changed, the private LB was. Now the LB match the metric.",
    "692596": "That's true. The description is correct where I should create my score without matching the exam (LB). That's life.🤓 Thank you for replying.",
    "692675": "Hi Phil, new submissions and all my old submissions are reflecting the correct metrics now. But it looks like LB update has not been finished for some teams? When do you expect it to be done?\n\nThanks",
    "693600": "yuval6967 \n&gt; Now the LB match the metric\n\nhow do you know?",
    "693658": "christofhenkel A few weeks ago I considered entering this competition. As I always do, I started by building a scoring function for CV. When I tried a simple model I could only get CV ~0.48 while everyone got 0.7 on LB. I suspected either my model has a bug, or something isn't right with the competition. Didn't have time to dig in then and ended up in another competition. But now I see the common base line kernel gets LB 0.48, which makes sense (don't know about the highest ranking LBs).",
    "693688": "philculliton",
    "693896": "philculliton",
    "693957": "Because 0.7+ doesn't update until now",
    "694116": "The update is still in progress. We will update this thread when it is complete. Thank you for your patience &amp; sorry for the inconvenience.",
    "694126": "🤓 thank you!",
    "694519": "seems to be complete now",
    "694525": "yep"
  },
  "source": "meta"
}