I have a kernel of LB score 0.710 (commit to competition few days ago)\nbut when I commit to competition today with this kernel, the score much lower, only got 0.647\nthen i recommit some other previous kernels to competition, it turned out that all LB score is declined.\ndose public test dataset changed? \nanyone has encountered this problem?
", "messages": [ { "id": "588240", "postDate": "07/30/2019 10:17:44", "content": "I have a kernel of LB score 0.710 (commit to competition few days ago)\nbut when I commit to competition today with this kernel, the score much lower, only got 0.647\nthen i recommit some other previous kernels to competition, it turned out that all LB score is declined.\ndose public test dataset changed? \nanyone has encountered this problem?
", "rawMarkdown": "I have a kernel of LB score 0.710 (commit to competition few days ago)\nbut when I commit to competition today with this kernel, the score much lower, only got 0.647\nthen i recommit some other previous kernels to competition, it turned out that all LB score is declined.\ndose public test dataset changed? \nanyone has encountered this problem?", "votes": null }, { "id": "588242", "postDate": "07/30/2019 10:21:11", "content": "In case you haven't set random seed for everything than it is supposed to differ.
", "rawMarkdown": "In case you haven't set random seed for everything than it is supposed to differ.", "votes": null }, { "id": "588304", "postDate": "07/30/2019 12:00:58", "content": "i use kernel pipeline to commit my predictions. that is : training the model in one kernel, upload the model as dataset, and using another kernel to load that trained model and predict. \nso there should be no random seed problem, cos they load exactly the same model, and preprocessing is deterministic(just resize)\nbut they got quite different score. that puzzled me.
", "rawMarkdown": "i use kernel pipeline to commit my predictions. that is : training the model in one kernel, upload the model as dataset, and using another kernel to load that trained model and predict. \nso there should be no random seed problem, cos they load exactly the same model, and preprocessing is deterministic(just resize)\nbut they got quite different score. that puzzled me.", "votes": null }, { "id": "588578", "postDate": "07/30/2019 18:59:27", "content": "as Val an said, you need to explicitly set the seed to get reproducible results. You can find more details in this nice blog: https://machinelearningmastery.com/randomness-in-machine-learning/
\n\nin PyTorch, I use the below snippet to set the seed (credits to multiple Kagglers who posted the same in other threads)
\n\n\n np.random.seed(seedValue) # cpu vars\n torch.manual_seed(seedValue) # cpu vars\n random.seed(seedValue) # Python\n os.environ['PYTHONHASHSEED'] = str(seedValue)\n torch.cuda.manual_seed(seedValue)\n torch.cuda.manual_seed_all(seedValue) # gpu vars\n torch.backends.cudnn.deterministic = True #needed\n torch.backends.cudnn.benchmark = False\n\nApparrently Keras with GPUs does not support deterministic results. Hard luck if you are using Keras with GPU. This is one of the reasons I moved from Keras to PyTorch.
thank you, learned a lot. unfortunately, I use keras now, I will try PyTorch if I have time.
", "rawMarkdown": "thank you, learned a lot. unfortunately, I use keras now, I will try PyTorch if I have time.", "votes": null } ], "comments": [ { "id": 588242, "author_name": "valanm", "author_url": "", "post_date": "07/30/2019 10:21:11", "content": "In case you haven't set random seed for everything than it is supposed to differ.
", "votes": null, "replies": [ { "id": 588304, "author_name": "frank518", "author_url": "", "post_date": "07/30/2019 12:00:58", "content": "i use kernel pipeline to commit my predictions. that is : training the model in one kernel, upload the model as dataset, and using another kernel to load that trained model and predict. \nso there should be no random seed problem, cos they load exactly the same model, and preprocessing is deterministic(just resize)\nbut they got quite different score. that puzzled me.
", "votes": null, "replies": [] }, { "id": 588578, "author_name": "ravivadapalli", "author_url": "", "post_date": "07/30/2019 18:59:27", "content": "as Val an said, you need to explicitly set the seed to get reproducible results. You can find more details in this nice blog: https://machinelearningmastery.com/randomness-in-machine-learning/
\n\nin PyTorch, I use the below snippet to set the seed (credits to multiple Kagglers who posted the same in other threads)
\n\n\n np.random.seed(seedValue) # cpu vars\n torch.manual_seed(seedValue) # cpu vars\n random.seed(seedValue) # Python\n os.environ['PYTHONHASHSEED'] = str(seedValue)\n torch.cuda.manual_seed(seedValue)\n torch.cuda.manual_seed_all(seedValue) # gpu vars\n torch.backends.cudnn.deterministic = True #needed\n torch.backends.cudnn.benchmark = False\n\nApparrently Keras with GPUs does not support deterministic results. Hard luck if you are using Keras with GPU. This is one of the reasons I moved from Keras to PyTorch.
thank you, learned a lot. unfortunately, I use keras now, I will try PyTorch if I have time.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "588240": "I have a kernel of LB score 0.710 (commit to competition few days ago)\nbut when I commit to competition today with this kernel, the score much lower, only got 0.647\nthen i recommit some other previous kernels to competition, it turned out that all LB score is declined.\ndose public test dataset changed? \nanyone has encountered this problem?", "588242": "In case you haven't set random seed for everything than it is supposed to differ.", "588304": "i use kernel pipeline to commit my predictions. that is : training the model in one kernel, upload the model as dataset, and using another kernel to load that trained model and predict. \nso there should be no random seed problem, cos they load exactly the same model, and preprocessing is deterministic(just resize)\nbut they got quite different score. that puzzled me.", "588578": "as Val an said, you need to explicitly set the seed to get reproducible results. You can find more details in this nice blog: https://machinelearningmastery.com/randomness-in-machine-learning/\n\nin PyTorch, I use the below snippet to set the seed (credits to multiple Kagglers who posted the same in other threads)\n\n```\n np.random.seed(seedValue) # cpu vars\n torch.manual_seed(seedValue) # cpu vars\n random.seed(seedValue) # Python\n os.environ['PYTHONHASHSEED'] = str(seedValue)\n torch.cuda.manual_seed(seedValue)\n torch.cuda.manual_seed_all(seedValue) # gpu vars\n torch.backends.cudnn.deterministic = True #needed\n torch.backends.cudnn.benchmark = False\n```\nApparrently Keras with GPUs does not support deterministic results. Hard luck if you are using Keras with GPU. This is one of the reasons I moved from Keras to PyTorch.", "588765": "thank you, learned a lot. unfortunately, I use keras now, I will try PyTorch if I have time." }, "source": "meta" }