{
  "id": 235458,
  "title": "Looking to Get Started: Statistics",
  "url": "/competitions/birdclef-2021/discussion/235458",
  "author_name": "",
  "post_date": "2021-04-29T17:07:07.470868900Z",
  "votes": 1,
  "comment_count": 1,
  "views": 0,
  "content": "<p>Hello, <br>\nI'm relatively new to data science, and was wondering if any of the more knowledgeable/advanced kaggler's would be willing to share good resources for the statistics they use to evaluate models. Currently, I'm trying to use a basic inverse binomial confidence interval and test statistic, but I feel there is room for improvement. Any suggestions, resources, and additional tips would be greatly appreciated! Thanks!</p>\n<p>Note: I understand that binomial doesn't work due to the fact that this is qualitative data, with multiple possible answers, and that the data isn't entirely independent (for example, if a crow makes a loud call, with a hawk in the background, that could mess up the ability to hear a hawk in the background, so lack of independence could be a stickler).</p>",
  "messages": [
    {
      "id": "1288089",
      "postDate": "04/29/2021 17:07:07",
      "content": "<p>Hello, <br>\nI'm relatively new to data science, and was wondering if any of the more knowledgeable/advanced kaggler's would be willing to share good resources for the statistics they use to evaluate models. Currently, I'm trying to use a basic inverse binomial confidence interval and test statistic, but I feel there is room for improvement. Any suggestions, resources, and additional tips would be greatly appreciated! Thanks!</p>\n<p>Note: I understand that binomial doesn't work due to the fact that this is qualitative data, with multiple possible answers, and that the data isn't entirely independent (for example, if a crow makes a loud call, with a hawk in the background, that could mess up the ability to hear a hawk in the background, so lack of independence could be a stickler).</p>",
      "rawMarkdown": "Hello, \nI'm relatively new to data science, and was wondering if any of the more knowledgeable/advanced kaggler's would be willing to share good resources for the statistics they use to evaluate models. Currently, I'm trying to use a basic inverse binomial confidence interval and test statistic, but I feel there is room for improvement. Any suggestions, resources, and additional tips would be greatly appreciated! Thanks!\n\nNote: I understand that binomial doesn't work due to the fact that this is qualitative data, with multiple possible answers, and that the data isn't entirely independent (for example, if a crow makes a loud call, with a hawk in the background, that could mess up the ability to hear a hawk in the background, so lack of independence could be a stickler).",
      "votes": null
    },
    {
      "id": "1288181",
      "postDate": "04/29/2021 18:27:24",
      "content": "<p>I generally use the competition metric applied to out of fold predictions in a cross validation.  If you use anything else than the competition metric then you may think you are improving your model, when you optimize the wrong objective function.</p>\n<p>In case you are not familiar with cross validation (excuse me if you are) then it is rather simple.  You split your training data in, say, 3 folds (I usually use 5 or 7, but 3 is simpler to explain).  Training data is now divided in 3 folds A, B, C.  Then you train 3 models:</p>\n<p>mA trained on B + C<br>\nmB trained on A + C<br>\nmC trained on A + B</p>\n<p>You use the exact same algorithm and hyper parameters for the 3 models.  Then you use them to predict on the fold that was not used to train them:</p>\n<p>mA predictions on A<br>\nmB predictions on B<br>\nmC predictions on C</p>\n<p>and you apply the competition metric on these predictions and the ground truth for the folds A, B, C.</p>",
      "rawMarkdown": "I generally use the competition metric applied to out of fold predictions in a cross validation.  If you use anything else than the competition metric then you may think you are improving your model, when you optimize the wrong objective function.\n\nIn case you are not familiar with cross validation (excuse me if you are) then it is rather simple.  You split your training data in, say, 3 folds (I usually use 5 or 7, but 3 is simpler to explain).  Training data is now divided in 3 folds A, B, C.  Then you train 3 models:\n\nmA trained on B + C\nmB trained on A + C\nmC trained on A + B\n\nYou use the exact same algorithm and hyper parameters for the 3 models.  Then you use them to predict on the fold that was not used to train them:\n\nmA predictions on A\nmB predictions on B\nmC predictions on C\n\nand you apply the competition metric on these predictions and the ground truth for the folds A, B, C.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1288181,
      "author_name": "cpmpml",
      "author_url": "",
      "post_date": "04/29/2021 18:27:24",
      "content": "<p>I generally use the competition metric applied to out of fold predictions in a cross validation.  If you use anything else than the competition metric then you may think you are improving your model, when you optimize the wrong objective function.</p>\n<p>In case you are not familiar with cross validation (excuse me if you are) then it is rather simple.  You split your training data in, say, 3 folds (I usually use 5 or 7, but 3 is simpler to explain).  Training data is now divided in 3 folds A, B, C.  Then you train 3 models:</p>\n<p>mA trained on B + C<br>\nmB trained on A + C<br>\nmC trained on A + B</p>\n<p>You use the exact same algorithm and hyper parameters for the 3 models.  Then you use them to predict on the fold that was not used to train them:</p>\n<p>mA predictions on A<br>\nmB predictions on B<br>\nmC predictions on C</p>\n<p>and you apply the competition metric on these predictions and the ground truth for the folds A, B, C.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1288089": "Hello, \nI'm relatively new to data science, and was wondering if any of the more knowledgeable/advanced kaggler's would be willing to share good resources for the statistics they use to evaluate models. Currently, I'm trying to use a basic inverse binomial confidence interval and test statistic, but I feel there is room for improvement. Any suggestions, resources, and additional tips would be greatly appreciated! Thanks!\n\nNote: I understand that binomial doesn't work due to the fact that this is qualitative data, with multiple possible answers, and that the data isn't entirely independent (for example, if a crow makes a loud call, with a hawk in the background, that could mess up the ability to hear a hawk in the background, so lack of independence could be a stickler).",
    "1288181": "I generally use the competition metric applied to out of fold predictions in a cross validation.  If you use anything else than the competition metric then you may think you are improving your model, when you optimize the wrong objective function.\n\nIn case you are not familiar with cross validation (excuse me if you are) then it is rather simple.  You split your training data in, say, 3 folds (I usually use 5 or 7, but 3 is simpler to explain).  Training data is now divided in 3 folds A, B, C.  Then you train 3 models:\n\nmA trained on B + C\nmB trained on A + C\nmC trained on A + B\n\nYou use the exact same algorithm and hyper parameters for the 3 models.  Then you use them to predict on the fold that was not used to train them:\n\nmA predictions on A\nmB predictions on B\nmC predictions on C\n\nand you apply the competition metric on these predictions and the ground truth for the folds A, B, C."
  },
  "source": "meta"
}