{
  "id": 71712,
  "title": "How to get reproducible results with CuDNN layers",
  "url": "/competitions/quora-insincere-questions-classification/discussion/71712",
  "author_name": "",
  "post_date": "2018-11-15T21:14:35.806899900Z",
  "votes": 1,
  "comment_count": 1,
  "views": 0,
  "content": "<p>Every time when I'm fitting the model that contains CuDNN layers in Keras, I get completely different results. Anybody has this problem too?</p>\n\n<p>I tried: </p>\n\n<ol>\n<li>Setting seed for Numpy, Random and Tensorflow</li>\n<li>Adding these lines for Tensorflow config</li>\n</ol>\n\n<p><code>session_conf = tf.ConfigProto(intra_op_parallelism_threads=1, inter_op_parallelism_threads=1)</code></p>\n\n<p><code>sess = tf.Session(graph=tf.get_default_graph(), config=session_conf)</code></p>\n\n<p><code>K.set_session(sess)</code></p>\n\n<p>But still getting different results for each run. \nAny ideas?</p>",
  "messages": [
    {
      "id": "422157",
      "postDate": "11/15/2018 21:14:35",
      "content": "<p>Every time when I'm fitting the model that contains CuDNN layers in Keras, I get completely different results. Anybody has this problem too?</p>\n\n<p>I tried: </p>\n\n<ol>\n<li>Setting seed for Numpy, Random and Tensorflow</li>\n<li>Adding these lines for Tensorflow config</li>\n</ol>\n\n<p><code>session_conf = tf.ConfigProto(intra_op_parallelism_threads=1, inter_op_parallelism_threads=1)</code></p>\n\n<p><code>sess = tf.Session(graph=tf.get_default_graph(), config=session_conf)</code></p>\n\n<p><code>K.set_session(sess)</code></p>\n\n<p>But still getting different results for each run. \nAny ideas?</p>",
      "rawMarkdown": "Every time when I'm fitting the model that contains CuDNN layers in Keras, I get completely different results. Anybody has this problem too?\n\nI tried: \n\n 1. Setting seed for Numpy, Random and Tensorflow\n 2.  Adding these lines for Tensorflow config\n\n`session_conf = tf.ConfigProto(intra_op_parallelism_threads=1, inter_op_parallelism_threads=1)`\n\n`sess = tf.Session(graph=tf.get_default_graph(), config=session_conf)`\n\n`K.set_session(sess)`\n\n\nBut still getting different results for each run. \nAny ideas?",
      "votes": null
    },
    {
      "id": "422176",
      "postDate": "11/15/2018 21:52:26",
      "content": "<p>I don't think you can: <a href=\"https://docs.nvidia.com/deeplearning/sdk/cudnn-developer-guide/index.html#reproducibility\">https://docs.nvidia.com/deeplearning/sdk/cudnn-developer-guide/index.html#reproducibility</a></p>\n\n<blockquote>\n  <p>2.6. Reproducibility (determinism)\n  By design, most of cuDNN's routines from a given version generate the same bit-wise results across runs when executed on GPUs with the same architecture and the same number of SMs. However, bit-wise reproducibility is not guaranteed across versions, as the implementation of a given routine may change.</p>\n</blockquote>",
      "rawMarkdown": "I don't think you can: https://docs.nvidia.com/deeplearning/sdk/cudnn-developer-guide/index.html#reproducibility\n\n&gt; 2.6. Reproducibility (determinism)\nBy design, most of cuDNN's routines from a given version generate the same bit-wise results across runs when executed on GPUs with the same architecture and the same number of SMs. However, bit-wise reproducibility is not guaranteed across versions, as the implementation of a given routine may change.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 422176,
      "author_name": "shujian",
      "author_url": "",
      "post_date": "11/15/2018 21:52:26",
      "content": "<p>I don't think you can: <a href=\"https://docs.nvidia.com/deeplearning/sdk/cudnn-developer-guide/index.html#reproducibility\">https://docs.nvidia.com/deeplearning/sdk/cudnn-developer-guide/index.html#reproducibility</a></p>\n\n<blockquote>\n  <p>2.6. Reproducibility (determinism)\n  By design, most of cuDNN's routines from a given version generate the same bit-wise results across runs when executed on GPUs with the same architecture and the same number of SMs. However, bit-wise reproducibility is not guaranteed across versions, as the implementation of a given routine may change.</p>\n</blockquote>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "422157": "Every time when I'm fitting the model that contains CuDNN layers in Keras, I get completely different results. Anybody has this problem too?\n\nI tried: \n\n 1. Setting seed for Numpy, Random and Tensorflow\n 2.  Adding these lines for Tensorflow config\n\n`session_conf = tf.ConfigProto(intra_op_parallelism_threads=1, inter_op_parallelism_threads=1)`\n\n`sess = tf.Session(graph=tf.get_default_graph(), config=session_conf)`\n\n`K.set_session(sess)`\n\n\nBut still getting different results for each run. \nAny ideas?",
    "422176": "I don't think you can: https://docs.nvidia.com/deeplearning/sdk/cudnn-developer-guide/index.html#reproducibility\n\n&gt; 2.6. Reproducibility (determinism)\nBy design, most of cuDNN's routines from a given version generate the same bit-wise results across runs when executed on GPUs with the same architecture and the same number of SMs. However, bit-wise reproducibility is not guaranteed across versions, as the implementation of a given routine may change."
  },
  "source": "meta"
}