{
  "id": 48473,
  "title": "Upgrading to Tensorflow 1.5",
  "url": "/competitions/sp-society-camera-model-identification/discussion/48473",
  "author_name": "",
  "post_date": "2018-01-28T11:45:20.786080700Z",
  "votes": 6,
  "comment_count": 10,
  "views": 0,
  "content": "<p>I decided to to upgrade to TF 1.5 (released few days ago... fresh from the oven) so there I went...</p>\n\n<p>I trained a few networks (for this comp) from scratch yesterday night and hit the ceiling for training accuracy @ 0.77ish... and loss fluctuated a lot... whereas the day before (w/ TF 1.4.1) things were much more stable (training accuracy would plateau @ 0.98ish).</p>\n\n<p>I've been checking the code for code changes that could explain the regression and found nothing, went back to TF 1.4.1 and things are working again smoothly.</p>\n\n<p>So just a heads up if it happens to you and if you upgrade to TF 1.5 and get no issues also please comment.</p>",
  "messages": [
    {
      "id": "275131",
      "postDate": "01/28/2018 11:45:20",
      "content": "<p>I decided to to upgrade to TF 1.5 (released few days ago... fresh from the oven) so there I went...</p>\n\n<p>I trained a few networks (for this comp) from scratch yesterday night and hit the ceiling for training accuracy @ 0.77ish... and loss fluctuated a lot... whereas the day before (w/ TF 1.4.1) things were much more stable (training accuracy would plateau @ 0.98ish).</p>\n\n<p>I've been checking the code for code changes that could explain the regression and found nothing, went back to TF 1.4.1 and things are working again smoothly.</p>\n\n<p>So just a heads up if it happens to you and if you upgrade to TF 1.5 and get no issues also please comment.</p>",
      "rawMarkdown": "I decided to to upgrade to TF 1.5 (released few days ago... fresh from the oven) so there I went...\n\nI trained a few networks (for this comp) from scratch yesterday night and hit the ceiling for training accuracy @ 0.77ish... and loss fluctuated a lot... whereas the day before (w/ TF 1.4.1) things were much more stable (training accuracy would plateau @ 0.98ish).\n\nI've been checking the code for code changes that could explain the regression and found nothing, went back to TF 1.4.1 and things are working again smoothly.\n\nSo just a heads up if it happens to you and if you upgrade to TF 1.5 and get no issues also please comment.",
      "votes": null
    },
    {
      "id": "276105",
      "postDate": "01/30/2018 17:55:25",
      "content": "<p>I haven't tried TF 1.5, but I got a quick question: is your training accuracy still based on the same validation set that you posted in your Github repository?</p>",
      "rawMarkdown": "I haven't tried TF 1.5, but I got a quick question: is your training accuracy still based on the same validation set that you posted in your Github repository?",
      "votes": null
    },
    {
      "id": "276114",
      "postDate": "01/30/2018 18:20:49",
      "content": "<p>Yes.</p>\n\n<p>Im working in adjusting augmentation (some augmentations do not make sense given some characteristics of the training set distribution), convenience functions and some inference improvements for test inference.</p>\n\n<p>The two things I want to do is: \n- have a much bigger training set (working on it but still the organizer @inversion has not responded to some key questions)\n- mixup</p>\n\n<p>Hopefully I will have both personal time and GPU time.</p>",
      "rawMarkdown": "Yes.\n\nIm working in adjusting augmentation (some augmentations do not make sense given some characteristics of the training set distribution), convenience functions and some inference improvements for test inference.\n\nThe two things I want to do is: \n- have a much bigger training set (working on it but still the organizer @inversion has not responded to some key questions)\n- mixup\n\nHopefully I will have both personal time and GPU time.",
      "votes": null
    },
    {
      "id": "276116",
      "postDate": "01/30/2018 18:28:30",
      "content": "<p>Thanks, that helps. I've been hitting the ceiling with the validation accuracy with this validation set (high 0.96ish) regardless of what I do with the architecture, and wanted to make sure that it's at least in principle possible to go beyond that. </p>",
      "rawMarkdown": "Thanks, that helps. I've been hitting the ceiling with the validation accuracy with this validation set (high 0.96ish) regardless of what I do with the architecture, and wanted to make sure that it's at least in principle possible to go beyond that.",
      "votes": null
    },
    {
      "id": "276125",
      "postDate": "01/30/2018 18:51:02",
      "content": "<p>Cryptic advice: look at the data, both training and your predictions.</p>",
      "rawMarkdown": "Cryptic advice: look at the data, both training and your predictions.",
      "votes": null
    },
    {
      "id": "276399",
      "postDate": "01/31/2018 12:08:12",
      "content": "<p>Does it get some speed up using 1.5 version?</p>",
      "rawMarkdown": "Does it get some speed up using 1.5 version?",
      "votes": null
    },
    {
      "id": "277115",
      "postDate": "02/02/2018 06:56:04",
      "content": "<p>Did you use CUDA 9.0 and CuDNN 7?</p>",
      "rawMarkdown": "Did you use CUDA 9.0 and CuDNN 7?",
      "votes": null
    },
    {
      "id": "277120",
      "postDate": "02/02/2018 07:02:22",
      "content": "<p>Yes. Did you experience the same?</p>",
      "rawMarkdown": "Yes. Did you experience the same?",
      "votes": null
    },
    {
      "id": "279866",
      "postDate": "02/08/2018 21:13:11",
      "content": "<p>Hey Andres, am curious if you still face the same issue with tf1.5</p>",
      "rawMarkdown": "Hey Andres, am curious if you still face the same issue with tf1.5",
      "votes": null
    },
    {
      "id": "279873",
      "postDate": "02/08/2018 21:33:29",
      "content": "<p>There is already TF 1.6 release candidate. What a speed time flies...</p>",
      "rawMarkdown": "There is already TF 1.6 release candidate. What a speed time flies...",
      "votes": null
    },
    {
      "id": "279906",
      "postDate": "02/08/2018 22:55:59",
      "content": "<p>I haven't tried. I'll upgrade to TF 1.5 after the competition. </p>",
      "rawMarkdown": "I haven't tried. I'll upgrade to TF 1.5 after the competition.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 276105,
      "author_name": "tunguz",
      "author_url": "",
      "post_date": "01/30/2018 17:55:25",
      "content": "<p>I haven't tried TF 1.5, but I got a quick question: is your training accuracy still based on the same validation set that you posted in your Github repository?</p>",
      "votes": null,
      "replies": [
        {
          "id": 276114,
          "author_name": "antorsae",
          "author_url": "",
          "post_date": "01/30/2018 18:20:49",
          "content": "<p>Yes.</p>\n\n<p>Im working in adjusting augmentation (some augmentations do not make sense given some characteristics of the training set distribution), convenience functions and some inference improvements for test inference.</p>\n\n<p>The two things I want to do is: \n- have a much bigger training set (working on it but still the organizer @inversion has not responded to some key questions)\n- mixup</p>\n\n<p>Hopefully I will have both personal time and GPU time.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 276116,
          "author_name": "tunguz",
          "author_url": "",
          "post_date": "01/30/2018 18:28:30",
          "content": "<p>Thanks, that helps. I've been hitting the ceiling with the validation accuracy with this validation set (high 0.96ish) regardless of what I do with the architecture, and wanted to make sure that it's at least in principle possible to go beyond that. </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 276125,
          "author_name": "antorsae",
          "author_url": "",
          "post_date": "01/30/2018 18:51:02",
          "content": "<p>Cryptic advice: look at the data, both training and your predictions.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 276399,
      "author_name": "mrlzla",
      "author_url": "",
      "post_date": "01/31/2018 12:08:12",
      "content": "<p>Does it get some speed up using 1.5 version?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 277115,
      "author_name": "famazon",
      "author_url": "",
      "post_date": "02/02/2018 06:56:04",
      "content": "<p>Did you use CUDA 9.0 and CuDNN 7?</p>",
      "votes": null,
      "replies": [
        {
          "id": 277120,
          "author_name": "antorsae",
          "author_url": "",
          "post_date": "02/02/2018 07:02:22",
          "content": "<p>Yes. Did you experience the same?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 279866,
          "author_name": "yipcma",
          "author_url": "",
          "post_date": "02/08/2018 21:13:11",
          "content": "<p>Hey Andres, am curious if you still face the same issue with tf1.5</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 279906,
          "author_name": "antorsae",
          "author_url": "",
          "post_date": "02/08/2018 22:55:59",
          "content": "<p>I haven't tried. I'll upgrade to TF 1.5 after the competition. </p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 279873,
      "author_name": "ceperaang",
      "author_url": "",
      "post_date": "02/08/2018 21:33:29",
      "content": "<p>There is already TF 1.6 release candidate. What a speed time flies...</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "275131": "I decided to to upgrade to TF 1.5 (released few days ago... fresh from the oven) so there I went...\n\nI trained a few networks (for this comp) from scratch yesterday night and hit the ceiling for training accuracy @ 0.77ish... and loss fluctuated a lot... whereas the day before (w/ TF 1.4.1) things were much more stable (training accuracy would plateau @ 0.98ish).\n\nI've been checking the code for code changes that could explain the regression and found nothing, went back to TF 1.4.1 and things are working again smoothly.\n\nSo just a heads up if it happens to you and if you upgrade to TF 1.5 and get no issues also please comment.",
    "276105": "I haven't tried TF 1.5, but I got a quick question: is your training accuracy still based on the same validation set that you posted in your Github repository?",
    "276114": "Yes.\n\nIm working in adjusting augmentation (some augmentations do not make sense given some characteristics of the training set distribution), convenience functions and some inference improvements for test inference.\n\nThe two things I want to do is: \n- have a much bigger training set (working on it but still the organizer @inversion has not responded to some key questions)\n- mixup\n\nHopefully I will have both personal time and GPU time.",
    "276116": "Thanks, that helps. I've been hitting the ceiling with the validation accuracy with this validation set (high 0.96ish) regardless of what I do with the architecture, and wanted to make sure that it's at least in principle possible to go beyond that.",
    "276125": "Cryptic advice: look at the data, both training and your predictions.",
    "276399": "Does it get some speed up using 1.5 version?",
    "277115": "Did you use CUDA 9.0 and CuDNN 7?",
    "277120": "Yes. Did you experience the same?",
    "279866": "Hey Andres, am curious if you still face the same issue with tf1.5",
    "279873": "There is already TF 1.6 release candidate. What a speed time flies...",
    "279906": "I haven't tried. I'll upgrade to TF 1.5 after the competition."
  },
  "source": "meta"
}