Hi ! \nI have been trying to submit the exact same kernel (same version) 3 times in a row and I get different scores that range from 0.52 to 0.65... I get that my model is retrained every time but still it seems like a big variation ! Do you know where this might come from ?
\nFor info my kernel is very similar to this one https://www.kaggle.com/dimitreoliveira/aptos-blindness-detection-eda-and-keras-resnet50
Thank you !
", "messages": [ { "id": 566829, "postDate": "2019-07-02T16:32:24.207Z", "content": "It's tensorflow features/bugs(it's hard to say what is it really). You can obtain the same results in two independent running only if would not be used parallelism (used 1 core of cpu only), but and this not guaranteed the same results. I can advise you use pytorch if you want have the same results.\nP.s. This video can help you understand why it's happening.
", "rawMarkdown": "It's tensorflow features/bugs(it's hard to say what is it really). You can obtain the same results in two independent running only if would not be used parallelism (used 1 core of cpu only), but and this not guaranteed the same results. I can advise you use pytorch if you want have the same results.\nP.s. This [video](https://youtu.be/Ys8ofBeR2kA) can help you understand why it's happening.", "votes": 5, "replies": [ { "id": 567069, "postDate": "2019-07-03T02:39:34.863Z", "content": "Great video. Thanks.
", "rawMarkdown": "Great video. Thanks." }, { "id": 567480, "postDate": "2019-07-03T15:28:46.647Z", "content": "Thank you very much for that ! I'm just impressed by the amount of variation I can get !
", "rawMarkdown": "Thank you very much for that ! I'm just impressed by the amount of variation I can get ! " } ] }, { "id": 566700, "postDate": "2019-07-02T13:39:42.350Z", "content": "Hi ! \nI have been trying to submit the exact same kernel (same version) 3 times in a row and I get different scores that range from 0.52 to 0.65... I get that my model is retrained every time but still it seems like a big variation ! Do you know where this might come from ?
\nFor info my kernel is very similar to this one https://www.kaggle.com/dimitreoliveira/aptos-blindness-detection-eda-and-keras-resnet50
Thank you !
", "rawMarkdown": "Hi ! \nI have been trying to submit the exact same kernel (same version) 3 times in a row and I get different scores that range from 0.52 to 0.65... I get that my model is retrained every time but still it seems like a big variation ! Do you know where this might come from ? \nFor info my kernel is very similar to this one https://www.kaggle.com/dimitreoliveira/aptos-blindness-detection-eda-and-keras-resnet50\n\nThank you !", "votes": 5 }, { "id": 567022, "postDate": "2019-07-03T00:31:25.020Z", "content": "Hi Benoit,
\n\nThat is my kernel, to be honest, to be honest, I'm also having a hard time getting the same results, once I got 0.702 then using the same parameters I got less than 0.650, I'm even using seeds to reduce this variance but it seems to have little effect, I wouldn't really care about this variance now, because once you start using bagging your modes, this will reduce.
", "rawMarkdown": "Hi Benoit, \n\nThat is my kernel, to be honest, to be honest, I'm also having a hard time getting the same results, once I got 0.702 then using the same parameters I got less than 0.650, I'm even using seeds to reduce this variance but it seems to have little effect, I wouldn't really care about this variance now, because once you start using bagging your modes, this will reduce.", "votes": 1, "replies": [ { "id": 567483, "postDate": "2019-07-03T15:29:37.940Z", "content": "Yeah I tried to use seeds also, I still get it... Thanks !
", "rawMarkdown": "Yeah I tried to use seeds also, I still get it... Thanks !", "votes": 1 } ] }, { "id": 568775, "postDate": "2019-07-05T12:37:03.340Z", "content": "Just one more thing that may be leading to this kind of variance, the data split (train/validation) and the data augmentation, if you check the number of trained epochs and the resulting loss, it's probably changing from one kernel to the other, the data split point could be fixed using \"train_test_split\" from SkLearn with seeds, but the data augmentation I'm not sure if can seeded.
", "rawMarkdown": "Just one more thing that may be leading to this kind of variance, the data split (train/validation) and the data augmentation, if you check the number of trained epochs and the resulting loss, it's probably changing from one kernel to the other, the data split point could be fixed using \"train_test_split\" from SkLearn with seeds, but the data augmentation I'm not sure if can seeded." }, { "id": 567812, "postDate": "2019-07-04T03:23:06.390Z", "content": "Same here
", "rawMarkdown": "Same here" }, { "id": 567131, "postDate": "2019-07-03T05:10:11.340Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 566829, "author_name": "Welf Crozzo", "author_url": "", "post_date": "2019-07-02T16:32:24.207000", "content": "It's tensorflow features/bugs(it's hard to say what is it really). You can obtain the same results in two independent running only if would not be used parallelism (used 1 core of cpu only), but and this not guaranteed the same results. I can advise you use pytorch if you want have the same results.\nP.s. This video can help you understand why it's happening.
", "votes": 5, "replies": [ { "id": 567069, "author_name": "vbookshelf", "author_url": "", "post_date": "2019-07-03T02:39:34.863000", "content": "Great video. Thanks.
", "votes": 0, "replies": [] }, { "id": 567480, "author_name": "Benoit Charmettant", "author_url": "", "post_date": "2019-07-03T15:28:46.647000", "content": "Thank you very much for that ! I'm just impressed by the amount of variation I can get !
", "votes": 0, "replies": [] } ] }, { "id": 567022, "author_name": "DimitreOliveira", "author_url": "", "post_date": "2019-07-03T00:31:25.020000", "content": "Hi Benoit,
\n\nThat is my kernel, to be honest, to be honest, I'm also having a hard time getting the same results, once I got 0.702 then using the same parameters I got less than 0.650, I'm even using seeds to reduce this variance but it seems to have little effect, I wouldn't really care about this variance now, because once you start using bagging your modes, this will reduce.
", "votes": 1, "replies": [ { "id": 567483, "author_name": "Benoit Charmettant", "author_url": "", "post_date": "2019-07-03T15:29:37.940000", "content": "Yeah I tried to use seeds also, I still get it... Thanks !
", "votes": 1, "replies": [] } ] }, { "id": 568775, "author_name": "DimitreOliveira", "author_url": "", "post_date": "2019-07-05T12:37:03.340000", "content": "Just one more thing that may be leading to this kind of variance, the data split (train/validation) and the data augmentation, if you check the number of trained epochs and the resulting loss, it's probably changing from one kernel to the other, the data split point could be fixed using \"train_test_split\" from SkLearn with seeds, but the data augmentation I'm not sure if can seeded.
", "votes": 0, "replies": [] }, { "id": 567812, "author_name": "Prashant Kikani", "author_url": "", "post_date": "2019-07-04T03:23:06.390000", "content": "Same here
", "votes": 0, "replies": [] }, { "id": 567131, "author_name": "", "author_url": "", "post_date": "2019-07-03T05:10:11.340000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "566829": "It's tensorflow features/bugs(it's hard to say what is it really). You can obtain the same results in two independent running only if would not be used parallelism (used 1 core of cpu only), but and this not guaranteed the same results. I can advise you use pytorch if you want have the same results.\nP.s. This [video](https://youtu.be/Ys8ofBeR2kA) can help you understand why it's happening.", "566700": "Hi ! \nI have been trying to submit the exact same kernel (same version) 3 times in a row and I get different scores that range from 0.52 to 0.65... I get that my model is retrained every time but still it seems like a big variation ! Do you know where this might come from ? \nFor info my kernel is very similar to this one https://www.kaggle.com/dimitreoliveira/aptos-blindness-detection-eda-and-keras-resnet50\n\nThank you !", "567022": "Hi Benoit, \n\nThat is my kernel, to be honest, to be honest, I'm also having a hard time getting the same results, once I got 0.702 then using the same parameters I got less than 0.650, I'm even using seeds to reduce this variance but it seems to have little effect, I wouldn't really care about this variance now, because once you start using bagging your modes, this will reduce.", "568775": "Just one more thing that may be leading to this kind of variance, the data split (train/validation) and the data augmentation, if you check the number of trained epochs and the resulting loss, it's probably changing from one kernel to the other, the data split point could be fixed using \"train_test_split\" from SkLearn with seeds, but the data augmentation I'm not sure if can seeded.", "567812": "Same here", "567131": "" } }