I meet this problem of unreproducible result between kaggle and AWS Sagemaker when training. On kaggle, the log loss quickly get under 0.69314, but on sagemaker, the log loss is always above 0.69314 and predict the same value for every image. I suspect that it is due to tensorflow version difference(kaggle:2.1.0, sagemaker:1.4), but I cannot upgrade it because it requires to upgrade cudNN.
", "messages": [ { "id": "739569", "postDate": "02/08/2020 01:39:33", "content": "I meet this problem of unreproducible result between kaggle and AWS Sagemaker when training. On kaggle, the log loss quickly get under 0.69314, but on sagemaker, the log loss is always above 0.69314 and predict the same value for every image. I suspect that it is due to tensorflow version difference(kaggle:2.1.0, sagemaker:1.4), but I cannot upgrade it because it requires to upgrade cudNN.
", "rawMarkdown": "I meet this problem of unreproducible result between kaggle and AWS Sagemaker when training. On kaggle, the log loss quickly get under 0.69314, but on sagemaker, the log loss is always above 0.69314 and predict the same value for every image. I suspect that it is due to tensorflow version difference(kaggle:2.1.0, sagemaker:1.4), but I cannot upgrade it because it requires to upgrade cudNN.", "votes": null }, { "id": "739576", "postDate": "02/08/2020 02:11:27", "content": "Yeah that's a tough one. Looking at the AWS deeplearning AMI's to see which TF version is included but not seeing TF 2.1.0.
", "rawMarkdown": "Yeah that's a tough one. Looking at the AWS deeplearning AMI's to see which TF version is included but not seeing TF 2.1.0.", "votes": null }, { "id": "739578", "postDate": "02/08/2020 02:16:15", "content": "Getting closer. Deeplearning AMI has TF2.0 \nhttps://aws.amazon.com/marketplace/pp/Amazon-Web-Services-Deep-Learning-AMI-Amazon-Linux/B077GF11NF
", "rawMarkdown": "Getting closer. Deeplearning AMI has TF2.0 \nhttps://aws.amazon.com/marketplace/pp/Amazon-Web-Services-Deep-Learning-AMI-Amazon-Linux/B077GF11NF", "votes": null }, { "id": "739580", "postDate": "02/08/2020 02:18:49", "content": "but not TF2.1.0
", "rawMarkdown": "but not TF2.1.0", "votes": null }, { "id": "739581", "postDate": "02/08/2020 02:24:29", "content": "True. But can you upgrade from 2.0 to 2.1 without needing to change the cudnn?
", "rawMarkdown": "True. But can you upgrade from 2.0 to 2.1 without needing to change the cudnn?", "votes": null }, { "id": "739583", "postDate": "02/08/2020 02:29:30", "content": "I'm not really sure how to change from tf 1.4 to tf 2.0. Do you change it in IAM? Thanks for you're help btw.
", "rawMarkdown": "I'm not really sure how to change from tf 1.4 to tf 2.0. Do you change it in IAM? Thanks for you're help btw.", "votes": null }, { "id": "739591", "postDate": "02/08/2020 03:04:37", "content": "I believe with pip or conda
", "rawMarkdown": "I believe with pip or conda", "votes": null }, { "id": "740332", "postDate": "02/09/2020 09:20:21", "content": "I haven't noticed a discrepancy between the log loss, but I had the same issue with the tensorflow version because I required TF 2 at some point. In the end, I shifted away from Sagemaker towards plain EC2. There, you can install jupyter too, making it similar, although it's a bit a pain, as usual. In EC2, I used the Ubuntu flavor of the \"Deep learning\" AMI shown above. I didn't miss anything in particular from 2.1 and the jupyter lab is a bit more responsive than through Sagemaker.
", "rawMarkdown": "I haven't noticed a discrepancy between the log loss, but I had the same issue with the tensorflow version because I required TF 2 at some point. In the end, I shifted away from Sagemaker towards plain EC2. There, you can install jupyter too, making it similar, although it's a bit a pain, as usual. In EC2, I used the Ubuntu flavor of the \"Deep learning\" AMI shown above. I didn't miss anything in particular from 2.1 and the jupyter lab is a bit more responsive than through Sagemaker.", "votes": null } ], "comments": [ { "id": 739576, "author_name": "sciarrilli", "author_url": "", "post_date": "02/08/2020 02:11:27", "content": "Yeah that's a tough one. Looking at the AWS deeplearning AMI's to see which TF version is included but not seeing TF 2.1.0.
", "votes": null, "replies": [] }, { "id": 739578, "author_name": "sciarrilli", "author_url": "", "post_date": "02/08/2020 02:16:15", "content": "Getting closer. Deeplearning AMI has TF2.0 \nhttps://aws.amazon.com/marketplace/pp/Amazon-Web-Services-Deep-Learning-AMI-Amazon-Linux/B077GF11NF
", "votes": null, "replies": [ { "id": 739580, "author_name": "unkownhihi", "author_url": "", "post_date": "02/08/2020 02:18:49", "content": "but not TF2.1.0
", "votes": null, "replies": [] }, { "id": 739581, "author_name": "sciarrilli", "author_url": "", "post_date": "02/08/2020 02:24:29", "content": "True. But can you upgrade from 2.0 to 2.1 without needing to change the cudnn?
", "votes": null, "replies": [] }, { "id": 739583, "author_name": "unkownhihi", "author_url": "", "post_date": "02/08/2020 02:29:30", "content": "I'm not really sure how to change from tf 1.4 to tf 2.0. Do you change it in IAM? Thanks for you're help btw.
", "votes": null, "replies": [] }, { "id": 739591, "author_name": "sciarrilli", "author_url": "", "post_date": "02/08/2020 03:04:37", "content": "I believe with pip or conda
", "votes": null, "replies": [] } ] }, { "id": 740332, "author_name": "dagnelies", "author_url": "", "post_date": "02/09/2020 09:20:21", "content": "I haven't noticed a discrepancy between the log loss, but I had the same issue with the tensorflow version because I required TF 2 at some point. In the end, I shifted away from Sagemaker towards plain EC2. There, you can install jupyter too, making it similar, although it's a bit a pain, as usual. In EC2, I used the Ubuntu flavor of the \"Deep learning\" AMI shown above. I didn't miss anything in particular from 2.1 and the jupyter lab is a bit more responsive than through Sagemaker.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "739569": "I meet this problem of unreproducible result between kaggle and AWS Sagemaker when training. On kaggle, the log loss quickly get under 0.69314, but on sagemaker, the log loss is always above 0.69314 and predict the same value for every image. I suspect that it is due to tensorflow version difference(kaggle:2.1.0, sagemaker:1.4), but I cannot upgrade it because it requires to upgrade cudNN.", "739576": "Yeah that's a tough one. Looking at the AWS deeplearning AMI's to see which TF version is included but not seeing TF 2.1.0.", "739578": "Getting closer. Deeplearning AMI has TF2.0 \nhttps://aws.amazon.com/marketplace/pp/Amazon-Web-Services-Deep-Learning-AMI-Amazon-Linux/B077GF11NF", "739580": "but not TF2.1.0", "739581": "True. But can you upgrade from 2.0 to 2.1 without needing to change the cudnn?", "739583": "I'm not really sure how to change from tf 1.4 to tf 2.0. Do you change it in IAM? Thanks for you're help btw.", "739591": "I believe with pip or conda", "740332": "I haven't noticed a discrepancy between the log loss, but I had the same issue with the tensorflow version because I required TF 2 at some point. In the end, I shifted away from Sagemaker towards plain EC2. There, you can install jupyter too, making it similar, although it's a bit a pain, as usual. In EC2, I used the Ubuntu flavor of the \"Deep learning\" AMI shown above. I didn't miss anything in particular from 2.1 and the jupyter lab is a bit more responsive than through Sagemaker." }, "source": "meta" }