When trying to decide if something is an improvement or in the right direction, we must be able to reproduce results in identical runs.\nI have tried:
\n\nimport numpy as np\nnp.random.seed(42)\nimport random\nrandom.seed(42)\nbut to no avail. Even two consecutive commits without changing the code produce different results.\nDid anyone manage to do that?
", "messages": [ { "id": 446328, "postDate": "2018-12-27T22:07:45.267Z", "content": "after some more digging, thiis before anything else improves the situation somewhat:
\n\nimport numpy as np # linear algebra\nnp.random.seed(42)\nimport random\nrandom.seed(42)\nimport os\nos.environ['PYTHONHASHSEED'] = '0'\nimport tensorflow as tf\ntf.set_random_seed(2019)\nfrom keras import backend as K\nsess = tf.Session(graph=tf.get_default_graph())\nK.set_session(sess)\nWhen I started with this competition I also researched this. I was using pretrained Resnet50 with Keras and Tensorflow. It seems that Tensorflow is not fully deterministic.
\n\nI then tried Resnet50 with Keras and Microsoft CNTK. That turned out to be fully deterministic..you can force it to run that way even when using the GPU....also make sure that any other library (train/test split, augmentation etc..) supports it.\nYou need to specify the following settings:
\n\n# Set All Random Stuff\nfrom _cntk_py import set_fixed_random_seed, force_deterministic_algorithms\nSEED=4249\ncntk.try_set_default_device(cntk.device.gpu(0))\nnp.random.seed(SEED)\nrandom.seed(SEED)\ncntk.cntk_py.set_fixed_random_seed(SEED)\ncntk.cntk_py.force_deterministic_algorithms()\nThe speed for my setup was however 2 to 3 times slower than with Keras + Tensorflow. So even if it is possible....with the speed achieved I didn't further use it. I prefer to do 2 to 3 random runs and than average them.
", "rawMarkdown": "When I started with this competition I also researched this. I was using pretrained Resnet50 with Keras and Tensorflow. It seems that Tensorflow is not fully deterministic. \n\nI then tried Resnet50 with Keras and Microsoft CNTK. That turned out to be fully deterministic..you can force it to run that way even when using the GPU....also make sure that any other library (train/test split, augmentation etc..) supports it.\nYou need to specify the following settings:\n\n # Set All Random Stuff\n from _cntk_py import set_fixed_random_seed, force_deterministic_algorithms\n SEED=4249\n cntk.try_set_default_device(cntk.device.gpu(0))\n np.random.seed(SEED)\n random.seed(SEED)\n cntk.cntk_py.set_fixed_random_seed(SEED)\n cntk.cntk_py.force_deterministic_algorithms()\n\nThe speed for my setup was however 2 to 3 times slower than with Keras + Tensorflow. So even if it is possible....with the speed achieved I didn't further use it. I prefer to do 2 to 3 random runs and than average them.\n" }, { "id": 448347, "postDate": "2018-12-31T20:44:05.947Z", "content": "Thank you! It is a pain, isn't it... especially since it seems that the convergence space is \"wild\", so you sometimes have a lucky model but can't fine tune it.\nI agree, 2 to 3 times reduction makes this non practical.
", "rawMarkdown": "Thank you! It is a pain, isn't it... especially since it seems that the convergence space is \"wild\", so you sometimes have a lucky model but can't fine tune it.\nI agree, 2 to 3 times reduction makes this non practical." } ] }, { "id": 446305, "postDate": "2018-12-27T20:56:09.380Z", "content": "There was a fastai thread on this and the seeds that needed setting were documented. I tried that and some things were reproducible (like train/test split) but I was never able to reproduce any run.
", "rawMarkdown": "There was a fastai thread on this and the seeds that needed setting were documented. I tried that and some things were reproducible (like train/test split) but I was never able to reproduce any run. " }, { "id": 446178, "postDate": "2018-12-27T16:47:11.347Z", "content": "Moshel there is an inherent randomness when using the gpu which I don't think you can avoid, even if you set all sorts of random seeds for pytorch. I expect this randomness can cause the f1 score to vary even more depending on whether a rare label is predicted as being present or absent due to randomness. That's probably why there is so much fluctuation in the f1 score.
", "rawMarkdown": "Moshel there is an inherent randomness when using the gpu which I don't think you can avoid, even if you set all sorts of random seeds for pytorch. I expect this randomness can cause the f1 score to vary even more depending on whether a rare label is predicted as being present or absent due to randomness. That's probably why there is so much fluctuation in the f1 score." }, { "id": 445949, "postDate": "2018-12-27T09:03:10.743Z", "content": "When trying to decide if something is an improvement or in the right direction, we must be able to reproduce results in identical runs.\nI have tried:
\n\nimport numpy as np\nnp.random.seed(42)\nimport random\nrandom.seed(42)\nbut to no avail. Even two consecutive commits without changing the code produce different results.\nDid anyone manage to do that?
", "rawMarkdown": "When trying to decide if something is an improvement or in the right direction, we must be able to reproduce results in identical runs.\nI have tried: \n\n import numpy as np\n np.random.seed(42)\n import random\n random.seed(42)\n\nbut to no avail. Even two consecutive commits without changing the code produce different results.\nDid anyone manage to do that?\n\n" }, { "id": 446114, "postDate": "2018-12-27T14:28:10.080Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 446328, "author_name": "Moshel", "author_url": "", "post_date": "2018-12-27T22:07:45.267000", "content": "after some more digging, thiis before anything else improves the situation somewhat:
\n\nimport numpy as np # linear algebra\nnp.random.seed(42)\nimport random\nrandom.seed(42)\nimport os\nos.environ['PYTHONHASHSEED'] = '0'\nimport tensorflow as tf\ntf.set_random_seed(2019)\nfrom keras import backend as K\nsess = tf.Session(graph=tf.get_default_graph())\nK.set_session(sess)\nWhen I started with this competition I also researched this. I was using pretrained Resnet50 with Keras and Tensorflow. It seems that Tensorflow is not fully deterministic.
\n\nI then tried Resnet50 with Keras and Microsoft CNTK. That turned out to be fully deterministic..you can force it to run that way even when using the GPU....also make sure that any other library (train/test split, augmentation etc..) supports it.\nYou need to specify the following settings:
\n\n# Set All Random Stuff\nfrom _cntk_py import set_fixed_random_seed, force_deterministic_algorithms\nSEED=4249\ncntk.try_set_default_device(cntk.device.gpu(0))\nnp.random.seed(SEED)\nrandom.seed(SEED)\ncntk.cntk_py.set_fixed_random_seed(SEED)\ncntk.cntk_py.force_deterministic_algorithms()\nThe speed for my setup was however 2 to 3 times slower than with Keras + Tensorflow. So even if it is possible....with the speed achieved I didn't further use it. I prefer to do 2 to 3 random runs and than average them.
", "votes": 0, "replies": [] }, { "id": 448347, "author_name": "Moshel", "author_url": "", "post_date": "2018-12-31T20:44:05.947000", "content": "Thank you! It is a pain, isn't it... especially since it seems that the convergence space is \"wild\", so you sometimes have a lucky model but can't fine tune it.\nI agree, 2 to 3 times reduction makes this non practical.
", "votes": 0, "replies": [] } ] }, { "id": 446305, "author_name": "pete", "author_url": "", "post_date": "2018-12-27T20:56:09.380000", "content": "There was a fastai thread on this and the seeds that needed setting were documented. I tried that and some things were reproducible (like train/test split) but I was never able to reproduce any run.
", "votes": 0, "replies": [] }, { "id": 446178, "author_name": "Robert", "author_url": "", "post_date": "2018-12-27T16:47:11.347000", "content": "Moshel there is an inherent randomness when using the gpu which I don't think you can avoid, even if you set all sorts of random seeds for pytorch. I expect this randomness can cause the f1 score to vary even more depending on whether a rare label is predicted as being present or absent due to randomness. That's probably why there is so much fluctuation in the f1 score.
", "votes": 0, "replies": [] }, { "id": 446114, "author_name": "", "author_url": "", "post_date": "2018-12-27T14:28:10.080000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "446328": "after some more digging, thiis before anything else improves the situation somewhat: \n\n import numpy as np # linear algebra\n np.random.seed(42)\n import random\n random.seed(42)\n import os\n os.environ['PYTHONHASHSEED'] = '0'\n import tensorflow as tf\n tf.set_random_seed(2019)\n from keras import backend as K\n sess = tf.Session(graph=tf.get_default_graph())\n K.set_session(sess)", "446305": "There was a fastai thread on this and the seeds that needed setting were documented. I tried that and some things were reproducible (like train/test split) but I was never able to reproduce any run. ", "446178": "Moshel there is an inherent randomness when using the gpu which I don't think you can avoid, even if you set all sorts of random seeds for pytorch. I expect this randomness can cause the f1 score to vary even more depending on whether a rare label is predicted as being present or absent due to randomness. That's probably why there is so much fluctuation in the f1 score.", "445949": "When trying to decide if something is an improvement or in the right direction, we must be able to reproduce results in identical runs.\nI have tried: \n\n import numpy as np\n np.random.seed(42)\n import random\n random.seed(42)\n\nbut to no avail. Even two consecutive commits without changing the code produce different results.\nDid anyone manage to do that?\n\n", "446114": "" } }