The code is based on Andres solution with changes made to improve performance and readability. With the original approach used as is I've managed to complete epoch in ~17 min with batch size 8 on GTX1080. After the rewrite the training time was reduced to ~4.5 while being able to fit larger batch of size 16 inside GPU memory.
\n\nhttps://github.com/irrmnv/pytorch-ieee-cmi
\n\nfor training:\npython3 train.py -uiw -x
\n\nfor predict:\npython3 train.py -uiw -x -t -tta -m models/ResNet50_132.pth
", "messages": [ { "id": "276406", "postDate": "01/31/2018 12:33:39", "content": "The code is based on Andres solution with changes made to improve performance and readability. With the original approach used as is I've managed to complete epoch in ~17 min with batch size 8 on GTX1080. After the rewrite the training time was reduced to ~4.5 while being able to fit larger batch of size 16 inside GPU memory.
\n\nhttps://github.com/irrmnv/pytorch-ieee-cmi
\n\nfor training:\npython3 train.py -uiw -x
\n\nfor predict:\npython3 train.py -uiw -x -t -tta -m models/ResNet50_132.pth
", "rawMarkdown": "The code is based on Andres solution with changes made to improve performance and readability. With the original approach used as is I've managed to complete epoch in ~17 min with batch size 8 on GTX1080. After the rewrite the training time was reduced to ~4.5 while being able to fit larger batch of size 16 inside GPU memory.\n\nhttps://github.com/irrmnv/pytorch-ieee-cmi\n\nfor training:\npython3 train.py -uiw -x\n\nfor predict:\npython3 train.py -uiw -x -t -tta -m models/ResNet50_132.pth", "votes": null }, { "id": "276439", "postDate": "01/31/2018 13:36:36", "content": "Great!\nThe changes to readability are obvious. \nWould you like to share your improvements regarding performance?
", "rawMarkdown": "Great!\nThe changes to readability are obvious. \nWould you like to share your improvements regarding performance?", "votes": null }, { "id": "276440", "postDate": "01/31/2018 13:36:58", "content": "If you end up trying DenseNet, consider using a memory-efficient version for PyTorch: https://github.com/gpleiss/efficient_densenet_pytorch
", "rawMarkdown": "If you end up trying DenseNet, consider using a memory-efficient version for PyTorch: https://github.com/gpleiss/efficient_densenet_pytorch", "votes": null }, { "id": "276615", "postDate": "01/31/2018 22:29:01", "content": "Ivan, pretty nice!
\n\nI'll study the code after competition b/c I want to move from Keras to Pytorch after competition (faster code in my limited experience but importantly flexibility and ready access to gradients and variables which you can just print and see... whereas in TF you need to be a neural surgeon to peek into gradients or do fancy loss functions, etc.).
Very nice! Can you also upload a LICENSE file?
", "rawMarkdown": "Very nice! Can you also upload a LICENSE file?", "votes": null }, { "id": "277089", "postDate": "02/02/2018 05:23:59", "content": "Hi guys,
\n\nI'm trying to install the libjpeg-turbo library in windows. Do you guys know how to install it for windows.
\n\nThanks.
", "rawMarkdown": "Hi guys,\n\nI'm trying to install the libjpeg-turbo library in windows. Do you guys know how to install it for windows.\n\nThanks.", "votes": null }, { "id": "277292", "postDate": "02/02/2018 18:11:03", "content": "easiest way use conda. Then conda install libjpeg-turbo
FileNotFoundError: [Errno 2] No such file or directory: 'flickr_images/./nexus_5x/34827215284_f48dc8a64b_o.jpg'
\n\nHow fix this?\nThank you very much!
", "rawMarkdown": "FileNotFoundError: [Errno 2] No such file or directory: 'flickr_images/./nexus_5x/34827215284_f48dc8a64b_o.jpg'\n\nHow fix this?\nThank you very much!", "votes": null }, { "id": "277508", "postDate": "02/03/2018 11:34:32", "content": "Would probably be better to actually get that image, but you could add os.path.exists check:\nids_train = [id_train for id_train in ids_train if os.path.exists(id_train)] and similarly for ids_val
", "rawMarkdown": "Would probably be better to actually get that image, but you could add os.path.exists check:\nids_train = [id_train for id_train in ids_train if os.path.exists(id_train)] and similarly for ids_val", "votes": null }, { "id": "277510", "postDate": "02/03/2018 11:39:33", "content": "Thanks, I'll have a try.
", "rawMarkdown": "Thanks, I'll have a try.", "votes": null }, { "id": "277689", "postDate": "02/03/2018 23:07:28", "content": "That doesn't work for me on windows 10. It's ok, I might try to use a different imaging library.
\n\nThanks for the help.
", "rawMarkdown": "That doesn't work for me on windows 10. It's ok, I might try to use a different imaging library.\n\nThanks for the help.", "votes": null }, { "id": "277882", "postDate": "02/04/2018 17:57:34", "content": "Bo, note that windows is not officially supported by pytorch.
", "rawMarkdown": "Bo, note that windows is not officially supported by pytorch.", "votes": null }, { "id": "277886", "postDate": "02/04/2018 18:11:03", "content": "Ivan, this is cool! I tried to run this, and it appears that it references imgs_train.txt and good_imgs_val.txt. Where are they located?
", "rawMarkdown": "Ivan, this is cool! I tried to run this, and it appears that it references imgs_train.txt and good_imgs_val.txt. Where are they located?", "votes": null }, { "id": "277902", "postDate": "02/04/2018 19:16:16", "content": "https://github.com/irrmnv/pytorch-ieee-cmi/blob/master/good_imgs_train.txt\nhttps://github.com/irrmnv/pytorch-ieee-cmi/blob/master/good_imgs_val.txt
", "rawMarkdown": "https://github.com/irrmnv/pytorch-ieee-cmi/blob/master/good_imgs_train.txt\nhttps://github.com/irrmnv/pytorch-ieee-cmi/blob/master/good_imgs_val.txt", "votes": null }, { "id": "278034", "postDate": "02/05/2018 07:12:27", "content": "I have download the extra-dataset three times , while it's not work with \"-x\".\nFinally, I delete the error extra file.
", "rawMarkdown": "I have download the extra-dataset three times , while it's not work with \"-x\".\nFinally, I delete the error extra file.", "votes": null }, { "id": "278396", "postDate": "02/06/2018 04:02:52", "content": "Hi Ananth,
\n\nPytorch works for me in windows. I just had trouble installing libjpeg-turbo that was needed for the jpeg4py package. It's ok, I could just try to use another library.
\n\nAppreciate the help.
", "rawMarkdown": "Hi Ananth,\n\nPytorch works for me in windows. I just had trouble installing libjpeg-turbo that was needed for the jpeg4py package. It's ok, I could just try to use another library. \n\nAppreciate the help.", "votes": null }, { "id": "278695", "postDate": "02/06/2018 18:54:08", "content": "for libjpeg-turbo you need to download from https://libjpeg-turbo.org/
", "rawMarkdown": "for libjpeg-turbo you need to download from https://libjpeg-turbo.org/", "votes": null }, { "id": "278876", "postDate": "02/07/2018 02:11:49", "content": "Hi Andrew,
\n\nI tried that but the library still didn't load. It's ok. I moved on and am using a different library.
\n\nThank you so much for your help.
", "rawMarkdown": "Hi Andrew, \n\nI tried that but the library still didn't load. It's ok. I moved on and am using a different library. \n\nThank you so much for your help.", "votes": null }, { "id": "279397", "postDate": "02/08/2018 00:09:45", "content": "Thank you.
", "rawMarkdown": "Thank you.", "votes": null }, { "id": "279779", "postDate": "02/08/2018 17:30:32", "content": "I faced similar issues. I think it's about setting the path to the datasets in some text files.
", "rawMarkdown": "I faced similar issues. I think it's about setting the path to the datasets in some text files.", "votes": null } ], "comments": [ { "id": 276439, "author_name": "fabsta", "author_url": "", "post_date": "01/31/2018 13:36:36", "content": "Great!\nThe changes to readability are obvious. \nWould you like to share your improvements regarding performance?
", "votes": null, "replies": [] }, { "id": 276440, "author_name": "kleinsmith", "author_url": "", "post_date": "01/31/2018 13:36:58", "content": "If you end up trying DenseNet, consider using a memory-efficient version for PyTorch: https://github.com/gpleiss/efficient_densenet_pytorch
", "votes": null, "replies": [] }, { "id": 276615, "author_name": "antorsae", "author_url": "", "post_date": "01/31/2018 22:29:01", "content": "Ivan, pretty nice!
\n\nI'll study the code after competition b/c I want to move from Keras to Pytorch after competition (faster code in my limited experience but importantly flexibility and ready access to gradients and variables which you can just print and see... whereas in TF you need to be a neural surgeon to peek into gradients or do fancy loss functions, etc.).
Very nice! Can you also upload a LICENSE file?
", "votes": null, "replies": [] }, { "id": 277089, "author_name": "bopengiowa", "author_url": "", "post_date": "02/02/2018 05:23:59", "content": "Hi guys,
\n\nI'm trying to install the libjpeg-turbo library in windows. Do you guys know how to install it for windows.
\n\nThanks.
", "votes": null, "replies": [ { "id": 277292, "author_name": "kirk86", "author_url": "", "post_date": "02/02/2018 18:11:03", "content": "easiest way use conda. Then conda install libjpeg-turbo
That doesn't work for me on windows 10. It's ok, I might try to use a different imaging library.
\n\nThanks for the help.
", "votes": null, "replies": [] }, { "id": 277882, "author_name": "eananth", "author_url": "", "post_date": "02/04/2018 17:57:34", "content": "Bo, note that windows is not officially supported by pytorch.
", "votes": null, "replies": [] }, { "id": 278396, "author_name": "bopengiowa", "author_url": "", "post_date": "02/06/2018 04:02:52", "content": "Hi Ananth,
\n\nPytorch works for me in windows. I just had trouble installing libjpeg-turbo that was needed for the jpeg4py package. It's ok, I could just try to use another library.
\n\nAppreciate the help.
", "votes": null, "replies": [] }, { "id": 278695, "author_name": "yipcma", "author_url": "", "post_date": "02/06/2018 18:54:08", "content": "for libjpeg-turbo you need to download from https://libjpeg-turbo.org/
", "votes": null, "replies": [] }, { "id": 278876, "author_name": "bopengiowa", "author_url": "", "post_date": "02/07/2018 02:11:49", "content": "Hi Andrew,
\n\nI tried that but the library still didn't load. It's ok. I moved on and am using a different library.
\n\nThank you so much for your help.
", "votes": null, "replies": [] } ] }, { "id": 277504, "author_name": "yyll008", "author_url": "", "post_date": "02/03/2018 11:23:37", "content": "FileNotFoundError: [Errno 2] No such file or directory: 'flickr_images/./nexus_5x/34827215284_f48dc8a64b_o.jpg'
\n\nHow fix this?\nThank you very much!
", "votes": null, "replies": [ { "id": 277508, "author_name": "bobutis", "author_url": "", "post_date": "02/03/2018 11:34:32", "content": "Would probably be better to actually get that image, but you could add os.path.exists check:\nids_train = [id_train for id_train in ids_train if os.path.exists(id_train)] and similarly for ids_val
", "votes": null, "replies": [] }, { "id": 277510, "author_name": "yyll008", "author_url": "", "post_date": "02/03/2018 11:39:33", "content": "Thanks, I'll have a try.
", "votes": null, "replies": [] } ] }, { "id": 277886, "author_name": "eananth", "author_url": "", "post_date": "02/04/2018 18:11:03", "content": "Ivan, this is cool! I tried to run this, and it appears that it references imgs_train.txt and good_imgs_val.txt. Where are they located?
", "votes": null, "replies": [ { "id": 277902, "author_name": "irrmnv", "author_url": "", "post_date": "02/04/2018 19:16:16", "content": "https://github.com/irrmnv/pytorch-ieee-cmi/blob/master/good_imgs_train.txt\nhttps://github.com/irrmnv/pytorch-ieee-cmi/blob/master/good_imgs_val.txt
", "votes": null, "replies": [] }, { "id": 279397, "author_name": "eananth", "author_url": "", "post_date": "02/08/2018 00:09:45", "content": "Thank you.
", "votes": null, "replies": [] } ] }, { "id": 278034, "author_name": "yyll008", "author_url": "", "post_date": "02/05/2018 07:12:27", "content": "I have download the extra-dataset three times , while it's not work with \"-x\".\nFinally, I delete the error extra file.
", "votes": null, "replies": [ { "id": 279779, "author_name": "yipcma", "author_url": "", "post_date": "02/08/2018 17:30:32", "content": "I faced similar issues. I think it's about setting the path to the datasets in some text files.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "276406": "The code is based on Andres solution with changes made to improve performance and readability. With the original approach used as is I've managed to complete epoch in ~17 min with batch size 8 on GTX1080. After the rewrite the training time was reduced to ~4.5 while being able to fit larger batch of size 16 inside GPU memory.\n\nhttps://github.com/irrmnv/pytorch-ieee-cmi\n\nfor training:\npython3 train.py -uiw -x\n\nfor predict:\npython3 train.py -uiw -x -t -tta -m models/ResNet50_132.pth", "276439": "Great!\nThe changes to readability are obvious. \nWould you like to share your improvements regarding performance?", "276440": "If you end up trying DenseNet, consider using a memory-efficient version for PyTorch: https://github.com/gpleiss/efficient_densenet_pytorch", "276615": "Ivan, pretty nice!\n\nI'll study the code after competition b/c I want to move from Keras to Pytorch after competition (faster code in my limited experience but importantly flexibility and ready access to gradients and variables which you can just `print` and see... whereas in TF you need to be a neural surgeon to peek into gradients or do fancy loss functions, etc.).", "276743": "Very nice! Can you also upload a LICENSE file?", "277089": "Hi guys,\n\nI'm trying to install the libjpeg-turbo library in windows. Do you guys know how to install it for windows.\n\nThanks.", "277292": "easiest way use conda. Then `conda install libjpeg-turbo`", "277504": "FileNotFoundError: [Errno 2] No such file or directory: 'flickr_images/./nexus_5x/34827215284_f48dc8a64b_o.jpg'\n\nHow fix this?\nThank you very much!", "277508": "Would probably be better to actually get that image, but you could add os.path.exists check:\nids_train = [id_train for id_train in ids_train if os.path.exists(id_train)] and similarly for ids_val", "277510": "Thanks, I'll have a try.", "277689": "That doesn't work for me on windows 10. It's ok, I might try to use a different imaging library.\n\nThanks for the help.", "277882": "Bo, note that windows is not officially supported by pytorch.", "277886": "Ivan, this is cool! I tried to run this, and it appears that it references imgs_train.txt and good_imgs_val.txt. Where are they located?", "277902": "https://github.com/irrmnv/pytorch-ieee-cmi/blob/master/good_imgs_train.txt\nhttps://github.com/irrmnv/pytorch-ieee-cmi/blob/master/good_imgs_val.txt", "278034": "I have download the extra-dataset three times , while it's not work with \"-x\".\nFinally, I delete the error extra file.", "278396": "Hi Ananth,\n\nPytorch works for me in windows. I just had trouble installing libjpeg-turbo that was needed for the jpeg4py package. It's ok, I could just try to use another library. \n\nAppreciate the help.", "278695": "for libjpeg-turbo you need to download from https://libjpeg-turbo.org/", "278876": "Hi Andrew, \n\nI tried that but the library still didn't load. It's ok. I moved on and am using a different library. \n\nThank you so much for your help.", "279397": "Thank you.", "279779": "I faced similar issues. I think it's about setting the path to the datasets in some text files." }, "source": "meta" }