{
  "id": 230293,
  "title": "Having deterministic result with Keras",
  "url": "/competitions/indoor-location-navigation/discussion/230293",
  "author_name": "",
  "post_date": "2021-04-02T23:55:02.798030500Z",
  "votes": 8,
  "comment_count": 5,
  "views": 0,
  "content": "<p>I am struggling dealing with reproducibility using Keras. </p>\n<h3>Solution for the GPU</h3>\n<p>I found a way when we use the GPU.<br>\nInstall <a href=\"https://github.com/NVIDIA/framework-determinism\" target=\"_blank\">framework-determinism</a> and set environment valuable. <br>\nThen we can see complete reproducibility. <br>\nYou can try it with the following code. </p>\n<p><code>!pip install tensorflow-determinism</code><br>\n<code>os.environ['TF_DETERMINISTIC_OPS'] = '1'</code></p>\n<p>It works perfectly on the GPU, but not on the CPU.</p>\n<h3>The problem that I'm facing</h3>\n<p>I can't find a way for complete reproducibility with Keras on <strong>the CPU</strong>. (There are reasons that I use the CPU, not the GPU. <br>\nAs I'm using Keras on the CPU, the result is always different even though I set random seeds using the following code. <br>\nAnd the fluctuation is too big, it is not neglectable. </p>\n<pre><code>def set_seed(seed=42):\n    random.seed(seed)\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    np.random.seed(seed)\n    tf.random.set_seed(seed)\n    #torch.manual_seed(seed)\n    session_conf = tf.compat.v1.ConfigProto(\n        intra_op_parallelism_threads=1,\n        inter_op_parallelism_threads=1\n    )\n    sess = tf.compat.v1.Session(graph=tf.compat.v1.get_default_graph(), config=session_conf)\n    tf.compat.v1.keras.backend.set_session(sess)\n</code></pre>\n<p>Does anybody have any tip for this? Thank you! </p>",
  "messages": [
    {
      "id": "1261319",
      "postDate": "04/02/2021 23:55:02",
      "content": "<p>I am struggling dealing with reproducibility using Keras. </p>\n<h3>Solution for the GPU</h3>\n<p>I found a way when we use the GPU.<br>\nInstall <a href=\"https://github.com/NVIDIA/framework-determinism\" target=\"_blank\">framework-determinism</a> and set environment valuable. <br>\nThen we can see complete reproducibility. <br>\nYou can try it with the following code. </p>\n<p><code>!pip install tensorflow-determinism</code><br>\n<code>os.environ['TF_DETERMINISTIC_OPS'] = '1'</code></p>\n<p>It works perfectly on the GPU, but not on the CPU.</p>\n<h3>The problem that I'm facing</h3>\n<p>I can't find a way for complete reproducibility with Keras on <strong>the CPU</strong>. (There are reasons that I use the CPU, not the GPU. <br>\nAs I'm using Keras on the CPU, the result is always different even though I set random seeds using the following code. <br>\nAnd the fluctuation is too big, it is not neglectable. </p>\n<pre><code>def set_seed(seed=42):\n    random.seed(seed)\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    np.random.seed(seed)\n    tf.random.set_seed(seed)\n    #torch.manual_seed(seed)\n    session_conf = tf.compat.v1.ConfigProto(\n        intra_op_parallelism_threads=1,\n        inter_op_parallelism_threads=1\n    )\n    sess = tf.compat.v1.Session(graph=tf.compat.v1.get_default_graph(), config=session_conf)\n    tf.compat.v1.keras.backend.set_session(sess)\n</code></pre>\n<p>Does anybody have any tip for this? Thank you! </p>",
      "rawMarkdown": "I am struggling dealing with reproducibility using Keras. \n\n### Solution for the GPU\n\nI found a way when we use the GPU.\nInstall [framework-determinism](https://github.com/NVIDIA/framework-determinism) and set environment valuable. \nThen we can see complete reproducibility. \nYou can try it with the following code. \n\n`!pip install tensorflow-determinism`\n`os.environ['TF_DETERMINISTIC_OPS'] = '1'`\n\nIt works perfectly on the GPU, but not on the CPU.\n\n\n### The problem that I'm facing\n\nI can't find a way for complete reproducibility with Keras on **the CPU**. (There are reasons that I use the CPU, not the GPU. \nAs I'm using Keras on the CPU, the result is always different even though I set random seeds using the following code. \nAnd the fluctuation is too big, it is not neglectable. \n\n```\ndef set_seed(seed=42):\n    random.seed(seed)\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    np.random.seed(seed)\n    tf.random.set_seed(seed)\n    #torch.manual_seed(seed)\n    session_conf = tf.compat.v1.ConfigProto(\n        intra_op_parallelism_threads=1,\n        inter_op_parallelism_threads=1\n    )\n    sess = tf.compat.v1.Session(graph=tf.compat.v1.get_default_graph(), config=session_conf)\n    tf.compat.v1.keras.backend.set_session(sess)\n```\n\nDoes anybody have any tip for this? Thank you!",
      "votes": null
    },
    {
      "id": "1261398",
      "postDate": "04/03/2021 03:39:22",
      "content": "<p>I think this <a href=\"https://stackoverflow.com/questions/42449635/why-does-my-keras-neural-network-model-output-different-values-on-a-different-ma\" target=\"_blank\">stackoverflow</a> might help. Basically, call <code>numpy.random.seed(seed_no)</code> before importing any keras libraries. </p>",
      "rawMarkdown": "I think this [stackoverflow](https://stackoverflow.com/questions/42449635/why-does-my-keras-neural-network-model-output-different-values-on-a-different-ma) might help. Basically, call `numpy.random.seed(seed_no)` before importing any keras libraries.",
      "votes": null
    },
    {
      "id": "1261472",
      "postDate": "04/03/2021 05:14:04",
      "content": "<p>Somehow I overlooked the thread. Thank you for sharing the link!<br>\nI put <code>numpy.random.seed(seed_no)</code> right before using any keras libraries, even before importing them, but it doesn't work… </p>\n<p>Thank you for the information, anyway! </p>",
      "rawMarkdown": "Somehow I overlooked the thread. Thank you for sharing the link!\nI put `numpy.random.seed(seed_no)` right before using any keras libraries, even before importing them, but it doesn't work... \n\nThank you for the information, anyway!",
      "votes": null
    },
    {
      "id": "1262764",
      "postDate": "04/04/2021 17:28:58",
      "content": "<p><a href=\"https://www.kaggle.com/kouki\" target=\"_blank\">@kouki</a> how long is it taking you to run a full run through the data? I'd love to look into this as well but it's taking me a whole day to just run models. </p>",
      "rawMarkdown": "kouki how long is it taking you to run a full run through the data? I'd love to look into this as well but it's taking me a whole day to just run models.",
      "votes": null
    },
    {
      "id": "1262861",
      "postDate": "04/04/2021 19:47:43",
      "content": "<p>For my latest model, an epoch takes approximately 500 sec and converges in around 80 epochs with 10 folds. Meaning it takes around 111 hours.<br>\nIt is not realistic running this with a single notebook, I use some notebooks to finish it within a day. </p>",
      "rawMarkdown": "For my latest model, an epoch takes approximately 500 sec and converges in around 80 epochs with 10 folds. Meaning it takes around 111 hours.\nIt is not realistic running this with a single notebook, I use some notebooks to finish it within a day.",
      "votes": null
    },
    {
      "id": "1263215",
      "postDate": "04/05/2021 07:34:59",
      "content": "<p>In my environment, the result using GPU is different every time even if the seed is fixed.<br>\nI am using cuDNN.</p>",
      "rawMarkdown": "In my environment, the result using GPU is different every time even if the seed is fixed.\nI am using cuDNN.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1261398,
      "author_name": "andy1010",
      "author_url": "",
      "post_date": "04/03/2021 03:39:22",
      "content": "<p>I think this <a href=\"https://stackoverflow.com/questions/42449635/why-does-my-keras-neural-network-model-output-different-values-on-a-different-ma\" target=\"_blank\">stackoverflow</a> might help. Basically, call <code>numpy.random.seed(seed_no)</code> before importing any keras libraries. </p>",
      "votes": null,
      "replies": [
        {
          "id": 1261472,
          "author_name": "kokitanisaka",
          "author_url": "",
          "post_date": "04/03/2021 05:14:04",
          "content": "<p>Somehow I overlooked the thread. Thank you for sharing the link!<br>\nI put <code>numpy.random.seed(seed_no)</code> right before using any keras libraries, even before importing them, but it doesn't work… </p>\n<p>Thank you for the information, anyway! </p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1262764,
      "author_name": "crained",
      "author_url": "",
      "post_date": "04/04/2021 17:28:58",
      "content": "<p><a href=\"https://www.kaggle.com/kouki\" target=\"_blank\">@kouki</a> how long is it taking you to run a full run through the data? I'd love to look into this as well but it's taking me a whole day to just run models. </p>",
      "votes": null,
      "replies": [
        {
          "id": 1262861,
          "author_name": "kokitanisaka",
          "author_url": "",
          "post_date": "04/04/2021 19:47:43",
          "content": "<p>For my latest model, an epoch takes approximately 500 sec and converges in around 80 epochs with 10 folds. Meaning it takes around 111 hours.<br>\nIt is not realistic running this with a single notebook, I use some notebooks to finish it within a day. </p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1263215,
      "author_name": "minakamikichi",
      "author_url": "",
      "post_date": "04/05/2021 07:34:59",
      "content": "<p>In my environment, the result using GPU is different every time even if the seed is fixed.<br>\nI am using cuDNN.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1261319": "I am struggling dealing with reproducibility using Keras. \n\n### Solution for the GPU\n\nI found a way when we use the GPU.\nInstall [framework-determinism](https://github.com/NVIDIA/framework-determinism) and set environment valuable. \nThen we can see complete reproducibility. \nYou can try it with the following code. \n\n`!pip install tensorflow-determinism`\n`os.environ['TF_DETERMINISTIC_OPS'] = '1'`\n\nIt works perfectly on the GPU, but not on the CPU.\n\n\n### The problem that I'm facing\n\nI can't find a way for complete reproducibility with Keras on **the CPU**. (There are reasons that I use the CPU, not the GPU. \nAs I'm using Keras on the CPU, the result is always different even though I set random seeds using the following code. \nAnd the fluctuation is too big, it is not neglectable. \n\n```\ndef set_seed(seed=42):\n    random.seed(seed)\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    np.random.seed(seed)\n    tf.random.set_seed(seed)\n    #torch.manual_seed(seed)\n    session_conf = tf.compat.v1.ConfigProto(\n        intra_op_parallelism_threads=1,\n        inter_op_parallelism_threads=1\n    )\n    sess = tf.compat.v1.Session(graph=tf.compat.v1.get_default_graph(), config=session_conf)\n    tf.compat.v1.keras.backend.set_session(sess)\n```\n\nDoes anybody have any tip for this? Thank you!",
    "1261398": "I think this [stackoverflow](https://stackoverflow.com/questions/42449635/why-does-my-keras-neural-network-model-output-different-values-on-a-different-ma) might help. Basically, call `numpy.random.seed(seed_no)` before importing any keras libraries.",
    "1261472": "Somehow I overlooked the thread. Thank you for sharing the link!\nI put `numpy.random.seed(seed_no)` right before using any keras libraries, even before importing them, but it doesn't work... \n\nThank you for the information, anyway!",
    "1262764": "kouki how long is it taking you to run a full run through the data? I'd love to look into this as well but it's taking me a whole day to just run models.",
    "1262861": "For my latest model, an epoch takes approximately 500 sec and converges in around 80 epochs with 10 folds. Meaning it takes around 111 hours.\nIt is not realistic running this with a single notebook, I use some notebooks to finish it within a day.",
    "1263215": "In my environment, the result using GPU is different every time even if the seed is fixed.\nI am using cuDNN."
  },
  "source": "meta"
}