{ "id": 175461, "title": "Winners have made my belief in Pytorch even stronger", "url": "/competitions/siim-isic-melanoma-classification/discussion/175461", "author_name": "", "post_date": "2020-08-18T08:59:58.311635300Z", "votes": 5, "comment_count": 14, "views": 0, "content": "

Still i am not able to figure out why Tensorflow models did so well on public LB. Any light on that will be helpful.

", "messages": [ { "id": "975356", "postDate": "08/18/2020 08:59:58", "content": "

Still i am not able to figure out why Tensorflow models did so well on public LB. Any light on that will be helpful.

", "rawMarkdown": "Still i am not able to figure out why Tensorflow models did so well on public LB. Any light on that will be helpful.", "votes": null }, { "id": "975363", "postDate": "08/18/2020 09:02:54", "content": "

I think the main reason is that there only was a strong tensorflow baseline but not a strong pytorch baseline. What I still dont get is why the CV was so bad on that one.

", "rawMarkdown": "I think the main reason is that there only was a strong tensorflow baseline but not a strong pytorch baseline. What I still dont get is why the CV was so bad on that one.", "votes": null }, { "id": "975365", "postDate": "08/18/2020 09:04:00", "content": "

I think Chris very popular starter kernel, was just lucky on public LB and most people built on that.

", "rawMarkdown": "I think Chris very popular starter kernel, was just lucky on public LB and most people built on that.", "votes": null }, { "id": "975368", "postDate": "08/18/2020 09:06:23", "content": "

We have also trained around 140 models in Pytorch itself with different setting but won't able to get LB above 0.95, for Cv we are able to get 0.957 with ensemble of pytorch only models.

", "rawMarkdown": "We have also trained around 140 models in Pytorch itself with different setting but won't able to get LB above 0.95, for Cv we are able to get 0.957 with ensemble of pytorch only models.", "votes": null }, { "id": "975410", "postDate": "08/18/2020 09:30:47", "content": "

As a beginner, i was so sad for not getting a baseline like @cdeotte 's kernel. But Alex's kernel was so good for Pytorch . Sadly we discovered it 5 days prior to ending the competition. So it didn't help that much

", "rawMarkdown": "As a beginner, i was so sad for not getting a baseline like @cdeotte 's kernel. But Alex's kernel was so good for Pytorch . Sadly we discovered it 5 days prior to ending the competition. So it didn't help that much", "votes": null }, { "id": "975413", "postDate": "08/18/2020 09:33:36", "content": "

So why should pytoch score better than tensorflow? With both modules we can de the exact same, just different implementations, right?

", "rawMarkdown": "So why should pytoch score better than tensorflow? With both modules we can de the exact same, just different implementations, right?", "votes": null }, { "id": "975427", "postDate": "08/18/2020 09:44:31", "content": "

In ideal situation both should work same, but due to difference in implementation and a lot of moving parts, we can expect a little bit of difference, but what we have seen in this comp. is huge difference. Even with same augmentations as in the @cdeotte Tf kernel, we are not able to replicate the same results.

", "rawMarkdown": "In ideal situation both should work same, but due to difference in implementation and a lot of moving parts, we can expect a little bit of difference, but what we have seen in this comp. is huge difference. Even with same augmentations as in the @cdeotte Tf kernel, we are not able to replicate the same results.", "votes": null }, { "id": "975439", "postDate": "08/18/2020 09:49:58", "content": "

It's difficult to do great Data augmentation on tensorflow.

\n

I tried TF in the last weeks to blend with my Pytorch models, but the augmentation was very basic.

", "rawMarkdown": "It's difficult to do great Data augmentation on tensorflow. \n\nI tried TF in the last weeks to blend with my Pytorch models, but the augmentation was very basic.", "votes": null }, { "id": "975450", "postDate": "08/18/2020 09:54:46", "content": "

in our case, TF models performed slightly better than pytocrch both in cv and (much more ) in LB and that was consistent in both public and private. But as of late, I normally see the reverse (e.g pytorch performing better)

", "rawMarkdown": "in our case, TF models performed slightly better than pytocrch both in cv and (much more ) in LB and that was consistent in both public and private. But as of late, I normally see the reverse (e.g pytorch performing better)", "votes": null }, { "id": "975463", "postDate": "08/18/2020 10:00:42", "content": "

The use of TPU is random and supports fewer enhancement methods.This makes me distressed.

", "rawMarkdown": "The use of TPU is random and supports fewer enhancement methods.This makes me distressed.", "votes": null }, { "id": "975507", "postDate": "08/18/2020 10:22:05", "content": "

how are you sure you used the same augmentation?

", "rawMarkdown": "how are you sure you used the same augmentation?", "votes": null }, { "id": "975523", "postDate": "08/18/2020 10:35:03", "content": "

In my case TF performed better. Still this competition was not really about building sophisticated DL models - but about finding proper validation schema and properly ensemble (personal opinion) :)

", "rawMarkdown": "In my case TF performed better. Still this competition was not really about building sophisticated DL models - but about finding proper validation schema and properly ensemble (personal opinion) :)", "votes": null }, { "id": "975530", "postDate": "08/18/2020 10:42:01", "content": "

By same augmentation i mean i tried rotation, zoom and shear augmentations as used in chrish's kernel, you are right pytorch implementation of these augs can be different. My apologies.

", "rawMarkdown": "By same augmentation i mean i tried `rotation`, `zoom` and `shear` augmentations as used in chrish's kernel, you are right pytorch implementation of these augs can be different. My apologies.", "votes": null }, { "id": "976141", "postDate": "08/18/2020 16:52:12", "content": "

I think @cpmpml had a theory about it.
\nThat we saw is that TF has consistently higher LB and lower CV than Pytorch models we trained. One thing that I had in my mind was that Pytorch models we had are using information from corners and periphery to make predictions, as shown by these CAM prepared by @rohitsingh9990 \"\".
\nSo in the train set there could be some consistency about lightening and darker periphery for images with and without melanoma, which is cached by our Pytorch models, and which may be not present in the test data. It's just a hypothesis we had that TF models somehow may be picking different features base more on Melanoma rather than image lightening and tone, so the CV is lower while LB is higher.

", "rawMarkdown": "I think @cpmpml had a theory about it.\nThat we saw is that TF has consistently higher LB and lower CV than Pytorch models we trained. One thing that I had in my mind was that Pytorch models we had are using information from corners and periphery to make predictions, as shown by these CAM prepared by @rohitsingh9990 ![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1212661%2Fdadc3c3e355e9239a42aefce93d24610%2Fscreenshot_2020-08-13_at_4.23.28_am.png?generation=1597768798753222&alt=media).\nSo in the train set there could be some consistency about lightening and darker periphery for images with and without melanoma, which is cached by our Pytorch models, and which may be not present in the test data. It's just a hypothesis we had that TF models somehow may be picking different features base more on Melanoma rather than image lightening and tone, so the CV is lower while LB is higher.", "votes": null }, { "id": "976223", "postDate": "08/18/2020 18:18:35", "content": "

I think PyTorch supports stronger augmentations which leads to better CV and generalization. Why Chris' TF baseline showed a big gap in CV and LB while CV and private LB was much smaller? I think its the augmentations. When I added heavy augmentations/upsampling through 512x512 center crops on 1024x1024 of the entire train, my CV shot way up (from 0.91 to 0.935+, but LB came down and Private LB matched CV and the lowered LB. Additionally, on another model I made with even heavier augmentations, I managed to get 0.9466 private LB and 0.9524 LB with 0.9449 ensemble CV.

", "rawMarkdown": "I think PyTorch supports stronger augmentations which leads to better CV and generalization. Why Chris' TF baseline showed a big gap in CV and LB while CV and private LB was much smaller? I think its the augmentations. When I added [heavy augmentations/upsampling](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/175372) through 512x512 center crops on 1024x1024 of the entire train, my CV shot way up (from 0.91 to 0.935+, but LB came down and Private LB matched CV and the lowered LB. Additionally, on another model I made with even heavier augmentations, I managed to get 0.9466 private LB and 0.9524 LB with 0.9449 ensemble CV.", "votes": null } ], "comments": [ { "id": 975363, "author_name": "philippsinger", "author_url": "", "post_date": "08/18/2020 09:02:54", "content": "

I think the main reason is that there only was a strong tensorflow baseline but not a strong pytorch baseline. What I still dont get is why the CV was so bad on that one.

", "votes": null, "replies": [ { "id": 975368, "author_name": "rohitsingh9990", "author_url": "", "post_date": "08/18/2020 09:06:23", "content": "

We have also trained around 140 models in Pytorch itself with different setting but won't able to get LB above 0.95, for Cv we are able to get 0.957 with ensemble of pytorch only models.

", "votes": null, "replies": [] } ] }, { "id": 975365, "author_name": "christofhenkel", "author_url": "", "post_date": "08/18/2020 09:04:00", "content": "

I think Chris very popular starter kernel, was just lucky on public LB and most people built on that.

", "votes": null, "replies": [] }, { "id": 975410, "author_name": "tawheedrony", "author_url": "", "post_date": "08/18/2020 09:30:47", "content": "

As a beginner, i was so sad for not getting a baseline like @cdeotte 's kernel. But Alex's kernel was so good for Pytorch . Sadly we discovered it 5 days prior to ending the competition. So it didn't help that much

", "votes": null, "replies": [] }, { "id": 975413, "author_name": "lukereijnen", "author_url": "", "post_date": "08/18/2020 09:33:36", "content": "

So why should pytoch score better than tensorflow? With both modules we can de the exact same, just different implementations, right?

", "votes": null, "replies": [ { "id": 975427, "author_name": "rohitsingh9990", "author_url": "", "post_date": "08/18/2020 09:44:31", "content": "

In ideal situation both should work same, but due to difference in implementation and a lot of moving parts, we can expect a little bit of difference, but what we have seen in this comp. is huge difference. Even with same augmentations as in the @cdeotte Tf kernel, we are not able to replicate the same results.

", "votes": null, "replies": [] }, { "id": 975507, "author_name": "christofhenkel", "author_url": "", "post_date": "08/18/2020 10:22:05", "content": "

how are you sure you used the same augmentation?

", "votes": null, "replies": [] }, { "id": 975530, "author_name": "rohitsingh9990", "author_url": "", "post_date": "08/18/2020 10:42:01", "content": "

By same augmentation i mean i tried rotation, zoom and shear augmentations as used in chrish's kernel, you are right pytorch implementation of these augs can be different. My apologies.

", "votes": null, "replies": [] } ] }, { "id": 975439, "author_name": "serigne", "author_url": "", "post_date": "08/18/2020 09:49:58", "content": "

It's difficult to do great Data augmentation on tensorflow.

\n

I tried TF in the last weeks to blend with my Pytorch models, but the augmentation was very basic.

", "votes": null, "replies": [] }, { "id": 975450, "author_name": "kazanova", "author_url": "", "post_date": "08/18/2020 09:54:46", "content": "

in our case, TF models performed slightly better than pytocrch both in cv and (much more ) in LB and that was consistent in both public and private. But as of late, I normally see the reverse (e.g pytorch performing better)

", "votes": null, "replies": [] }, { "id": 975463, "author_name": "mnk812", "author_url": "", "post_date": "08/18/2020 10:00:42", "content": "

The use of TPU is random and supports fewer enhancement methods.This makes me distressed.

", "votes": null, "replies": [] }, { "id": 975523, "author_name": "gpamoukoff", "author_url": "", "post_date": "08/18/2020 10:35:03", "content": "

In my case TF performed better. Still this competition was not really about building sophisticated DL models - but about finding proper validation schema and properly ensemble (personal opinion) :)

", "votes": null, "replies": [] }, { "id": 976141, "author_name": "iafoss", "author_url": "", "post_date": "08/18/2020 16:52:12", "content": "

I think @cpmpml had a theory about it.
\nThat we saw is that TF has consistently higher LB and lower CV than Pytorch models we trained. One thing that I had in my mind was that Pytorch models we had are using information from corners and periphery to make predictions, as shown by these CAM prepared by @rohitsingh9990 \"\".
\nSo in the train set there could be some consistency about lightening and darker periphery for images with and without melanoma, which is cached by our Pytorch models, and which may be not present in the test data. It's just a hypothesis we had that TF models somehow may be picking different features base more on Melanoma rather than image lightening and tone, so the CV is lower while LB is higher.

", "votes": null, "replies": [] }, { "id": 976223, "author_name": "teeyee314", "author_url": "", "post_date": "08/18/2020 18:18:35", "content": "

I think PyTorch supports stronger augmentations which leads to better CV and generalization. Why Chris' TF baseline showed a big gap in CV and LB while CV and private LB was much smaller? I think its the augmentations. When I added heavy augmentations/upsampling through 512x512 center crops on 1024x1024 of the entire train, my CV shot way up (from 0.91 to 0.935+, but LB came down and Private LB matched CV and the lowered LB. Additionally, on another model I made with even heavier augmentations, I managed to get 0.9466 private LB and 0.9524 LB with 0.9449 ensemble CV.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "975356": "Still i am not able to figure out why Tensorflow models did so well on public LB. Any light on that will be helpful.", "975363": "I think the main reason is that there only was a strong tensorflow baseline but not a strong pytorch baseline. What I still dont get is why the CV was so bad on that one.", "975365": "I think Chris very popular starter kernel, was just lucky on public LB and most people built on that.", "975368": "We have also trained around 140 models in Pytorch itself with different setting but won't able to get LB above 0.95, for Cv we are able to get 0.957 with ensemble of pytorch only models.", "975410": "As a beginner, i was so sad for not getting a baseline like @cdeotte 's kernel. But Alex's kernel was so good for Pytorch . Sadly we discovered it 5 days prior to ending the competition. So it didn't help that much", "975413": "So why should pytoch score better than tensorflow? With both modules we can de the exact same, just different implementations, right?", "975427": "In ideal situation both should work same, but due to difference in implementation and a lot of moving parts, we can expect a little bit of difference, but what we have seen in this comp. is huge difference. Even with same augmentations as in the @cdeotte Tf kernel, we are not able to replicate the same results.", "975439": "It's difficult to do great Data augmentation on tensorflow. \n\nI tried TF in the last weeks to blend with my Pytorch models, but the augmentation was very basic.", "975450": "in our case, TF models performed slightly better than pytocrch both in cv and (much more ) in LB and that was consistent in both public and private. But as of late, I normally see the reverse (e.g pytorch performing better)", "975463": "The use of TPU is random and supports fewer enhancement methods.This makes me distressed.", "975507": "how are you sure you used the same augmentation?", "975523": "In my case TF performed better. Still this competition was not really about building sophisticated DL models - but about finding proper validation schema and properly ensemble (personal opinion) :)", "975530": "By same augmentation i mean i tried `rotation`, `zoom` and `shear` augmentations as used in chrish's kernel, you are right pytorch implementation of these augs can be different. My apologies.", "976141": "I think @cpmpml had a theory about it.\nThat we saw is that TF has consistently higher LB and lower CV than Pytorch models we trained. One thing that I had in my mind was that Pytorch models we had are using information from corners and periphery to make predictions, as shown by these CAM prepared by @rohitsingh9990 ![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1212661%2Fdadc3c3e355e9239a42aefce93d24610%2Fscreenshot_2020-08-13_at_4.23.28_am.png?generation=1597768798753222&alt=media).\nSo in the train set there could be some consistency about lightening and darker periphery for images with and without melanoma, which is cached by our Pytorch models, and which may be not present in the test data. It's just a hypothesis we had that TF models somehow may be picking different features base more on Melanoma rather than image lightening and tone, so the CV is lower while LB is higher.", "976223": "I think PyTorch supports stronger augmentations which leads to better CV and generalization. Why Chris' TF baseline showed a big gap in CV and LB while CV and private LB was much smaller? I think its the augmentations. When I added [heavy augmentations/upsampling](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/175372) through 512x512 center crops on 1024x1024 of the entire train, my CV shot way up (from 0.91 to 0.935+, but LB came down and Private LB matched CV and the lowered LB. Additionally, on another model I made with even heavier augmentations, I managed to get 0.9466 private LB and 0.9524 LB with 0.9449 ensemble CV." }, "source": "meta" }