{
  "id": 20206,
  "title": "Accuracy - Logloss Comparison",
  "url": "/competitions/state-farm-distracted-driver-detection/discussion/20206",
  "author_name": "",
  "post_date": "2016-04-17T19:58:39.367Z",
  "votes": null,
  "comment_count": 3,
  "views": 775,
  "content": "<p>To have a more practical understanding of the Leaderboard, what's the accuracy of a top 50 model?</p>",
  "messages": [
    {
      "id": "115295",
      "postDate": "04/17/2016 19:58:39",
      "content": "<p>To have a more practical understanding of the Leaderboard, what's the accuracy of a top 50 model?</p>",
      "rawMarkdown": "To have a more practical understanding of the Leaderboard, what's the accuracy of a top 50 model?",
      "votes": null
    },
    {
      "id": "115298",
      "postDate": "04/17/2016 20:27:32",
      "content": "<p>Note I'm no longer Top 50 material, but was a couple of days ago . . . </p>\n\n<p>The accuracy I was seeing varies wildly, and had only a very loose connection to logloss. I believe that to be because it is easy for my model to be very confident in some of its mis-categorisations (increasing logloss significantly, but no change to accuracy).</p>\n\n<p>The accuracy also varied a lot depending on which drivers were in CV set.</p>\n\n<p>My &quot;typical&quot; accuracy for a score of 1.13 on the public LB, was 50-60%. In CV, some drivers were 20% or lower though, and a couple were 70-80%.</p>",
      "rawMarkdown": "Note I'm no longer Top 50 material, but was a couple of days ago . . . \r\n\r\nThe accuracy I was seeing varies wildly, and had only a very loose connection to logloss. I believe that to be because it is easy for my model to be very confident in some of its mis-categorisations (increasing logloss significantly, but no change to accuracy).\r\n\r\nThe accuracy also varied a lot depending on which drivers were in CV set.\r\n\r\nMy \"typical\" accuracy for a score of 1.13 on the public LB, was 50-60%. In CV, some drivers were 20% or lower though, and a couple were 70-80%.",
      "votes": null
    },
    {
      "id": "115305",
      "postDate": "04/17/2016 21:56:47",
      "content": "<p>Hey Neil\nI ended up with a log_loss score of 0.138 after running a 10fold cross validation on image input dimensions of (48, 64).\nAny recommendations as to how I can improve the accuracy further while lowering the log_loss score further?  I'm running all this on my mac CPU so it definitely takes a while too. </p>",
      "rawMarkdown": "Hey Neil\r\nI ended up with a log_loss score of 0.138 after running a 10fold cross validation on image input dimensions of (48, 64).\r\nAny recommendations as to how I can improve the accuracy further while lowering the log_loss score further?  I'm running all this on my mac CPU so it definitely takes a while too.",
      "votes": null
    },
    {
      "id": "115354",
      "postDate": "04/18/2016 10:19:42",
      "content": "<p>[quote=Mustyy;115305]</p>\n\n<p>Hey Neil\nI ended up with a log_loss score of 0.138 after running a 10fold cross validation on image input dimensions of (48, 64).\nAny recommendations as to how I can improve the accuracy further while lowering the log_loss score further?  I'm running all this on my mac CPU so it definitely takes a while too. </p>\n\n<p>[/quote]</p>\n\n<p>I guess you mean 1.38? </p>\n\n<p>You can make some simple gains by increasing regularisation and things that bias your model (the latter also usually makes it run faster). Start by <em>reducing</em> the input image size. Also, consider increasing max-pooling pool size from 2 to 3, and see if you can create a deeper network with smaller layer sizes.</p>\n\n<p>I think this approach is a bit of a dead end, it won't win the competition but it looks possible to tune a CPU-based approach like this below a log loss of 1.0 (once the different fold predictions are averaged), using ZFTurbo's starter script. I am also bound to a CPU-only approach, and agree it is frustrating, but so far not got enough experience with deep nets to justify spending money to get them running faster.</p>",
      "rawMarkdown": "[quote=Mustyy;115305]\r\n\r\nHey Neil\r\nI ended up with a log_loss score of 0.138 after running a 10fold cross validation on image input dimensions of (48, 64).\r\nAny recommendations as to how I can improve the accuracy further while lowering the log_loss score further?  I'm running all this on my mac CPU so it definitely takes a while too. \r\n\r\n[/quote]\r\n\r\nI guess you mean 1.38? \r\n\r\nYou can make some simple gains by increasing regularisation and things that bias your model (the latter also usually makes it run faster). Start by *reducing* the input image size. Also, consider increasing max-pooling pool size from 2 to 3, and see if you can create a deeper network with smaller layer sizes.\r\n\r\nI think this approach is a bit of a dead end, it won't win the competition but it looks possible to tune a CPU-based approach like this below a log loss of 1.0 (once the different fold predictions are averaged), using ZFTurbo's starter script. I am also bound to a CPU-only approach, and agree it is frustrating, but so far not got enough experience with deep nets to justify spending money to get them running faster.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 115298,
      "author_name": "slobo777",
      "author_url": "",
      "post_date": "04/17/2016 20:27:32",
      "content": "<p>Note I'm no longer Top 50 material, but was a couple of days ago . . . </p>\n\n<p>The accuracy I was seeing varies wildly, and had only a very loose connection to logloss. I believe that to be because it is easy for my model to be very confident in some of its mis-categorisations (increasing logloss significantly, but no change to accuracy).</p>\n\n<p>The accuracy also varied a lot depending on which drivers were in CV set.</p>\n\n<p>My &quot;typical&quot; accuracy for a score of 1.13 on the public LB, was 50-60%. In CV, some drivers were 20% or lower though, and a couple were 70-80%.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 115305,
      "author_name": "mustyy",
      "author_url": "",
      "post_date": "04/17/2016 21:56:47",
      "content": "<p>Hey Neil\nI ended up with a log_loss score of 0.138 after running a 10fold cross validation on image input dimensions of (48, 64).\nAny recommendations as to how I can improve the accuracy further while lowering the log_loss score further?  I'm running all this on my mac CPU so it definitely takes a while too. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 115354,
      "author_name": "slobo777",
      "author_url": "",
      "post_date": "04/18/2016 10:19:42",
      "content": "<p>[quote=Mustyy;115305]</p>\n\n<p>Hey Neil\nI ended up with a log_loss score of 0.138 after running a 10fold cross validation on image input dimensions of (48, 64).\nAny recommendations as to how I can improve the accuracy further while lowering the log_loss score further?  I'm running all this on my mac CPU so it definitely takes a while too. </p>\n\n<p>[/quote]</p>\n\n<p>I guess you mean 1.38? </p>\n\n<p>You can make some simple gains by increasing regularisation and things that bias your model (the latter also usually makes it run faster). Start by <em>reducing</em> the input image size. Also, consider increasing max-pooling pool size from 2 to 3, and see if you can create a deeper network with smaller layer sizes.</p>\n\n<p>I think this approach is a bit of a dead end, it won't win the competition but it looks possible to tune a CPU-based approach like this below a log loss of 1.0 (once the different fold predictions are averaged), using ZFTurbo's starter script. I am also bound to a CPU-only approach, and agree it is frustrating, but so far not got enough experience with deep nets to justify spending money to get them running faster.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "115295": "To have a more practical understanding of the Leaderboard, what's the accuracy of a top 50 model?",
    "115298": "Note I'm no longer Top 50 material, but was a couple of days ago . . . \r\n\r\nThe accuracy I was seeing varies wildly, and had only a very loose connection to logloss. I believe that to be because it is easy for my model to be very confident in some of its mis-categorisations (increasing logloss significantly, but no change to accuracy).\r\n\r\nThe accuracy also varied a lot depending on which drivers were in CV set.\r\n\r\nMy \"typical\" accuracy for a score of 1.13 on the public LB, was 50-60%. In CV, some drivers were 20% or lower though, and a couple were 70-80%.",
    "115305": "Hey Neil\r\nI ended up with a log_loss score of 0.138 after running a 10fold cross validation on image input dimensions of (48, 64).\r\nAny recommendations as to how I can improve the accuracy further while lowering the log_loss score further?  I'm running all this on my mac CPU so it definitely takes a while too.",
    "115354": "[quote=Mustyy;115305]\r\n\r\nHey Neil\r\nI ended up with a log_loss score of 0.138 after running a 10fold cross validation on image input dimensions of (48, 64).\r\nAny recommendations as to how I can improve the accuracy further while lowering the log_loss score further?  I'm running all this on my mac CPU so it definitely takes a while too. \r\n\r\n[/quote]\r\n\r\nI guess you mean 1.38? \r\n\r\nYou can make some simple gains by increasing regularisation and things that bias your model (the latter also usually makes it run faster). Start by *reducing* the input image size. Also, consider increasing max-pooling pool size from 2 to 3, and see if you can create a deeper network with smaller layer sizes.\r\n\r\nI think this approach is a bit of a dead end, it won't win the competition but it looks possible to tune a CPU-based approach like this below a log loss of 1.0 (once the different fold predictions are averaged), using ZFTurbo's starter script. I am also bound to a CPU-only approach, and agree it is frustrating, but so far not got enough experience with deep nets to justify spending money to get them running faster."
  },
  "source": "meta"
}