{ "id": 175372, "title": "Hacking super-resolution images", "url": "/competitions/siim-isic-melanoma-classification/discussion/175372", "author_name": "", "post_date": "2020-08-18T02:51:01.114793900Z", "votes": 6, "comment_count": 13, "views": 0, "content": "

Hacking Super-resolution Images

\n

First, thank you to the organizers and Kaggle for this competition. Thank you to everyone who contributed to making this image based competition fun.

\n

Special thanks to @cdeotte for all of his notebooks and discussion. I would not have been able to create my tfrecords and explore this idea without his work.

\n

Congrats to the winners and everyone who achieved a medal!

\n

Intro

\n

Image resolutions were all over the place, but a large majority of the images were downsampled from resolutions greater than 1024x1024, thus a major consideration was how to extract the most out of super-resolution images while being able to train a reasonably well performing model with the highest resolution given limited time and resources.

\n

Approach

\n

Toward the last week of the competition, I wanted to run 1024x1024 but with limited time and resources I came up with a hack: center crop 1024x1024 resolution sizes of 512x512. To my surprise, by including the entire upsample center crop in train, there was no leakage. Somehow the 4x increased pixel density and zoom contributed to generalizing better without leakage.

\n

While this approach appears to be a mix of data augmentation and upsampling both positive and negatives, I believe it was not. Images that were originally super-resolution were downsampled (resized to a lower resolution) just to fit within a pre-trained model. Downsampled images have lost so much information which is precisely why image sizes mattered so much in this competition.

\n

Edit: to clarify - I trained on both 2020 512x512 resolutions and 512x512 super resolution crops from 1024x1024. I validated on only 2020 original 512x512 resolutions.

\n

Final Single Model Results

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
CVLBPrivate LB
0.93790.93940.9325
\n

B6 512x512 15 fold took over 22 hours to train on TPU with 75000 images per epoch. While the LB was somewhat low, I felt confident that private LB would be close judging by how close CV was to LB.

\n

Edit: Using this approach, I was able to increase CV by 0.02-0.03, which was huge!

\n

Extra

\n

This same approach was briefly explored with 2018 + 2017 images in addition to 2020 with some success but I ran out of time to get conclusive results.

\n

Side note:

\n", "messages": [ { "id": "974709", "postDate": "08/18/2020 02:51:01", "content": "

Hacking Super-resolution Images

\n

First, thank you to the organizers and Kaggle for this competition. Thank you to everyone who contributed to making this image based competition fun.

\n

Special thanks to @cdeotte for all of his notebooks and discussion. I would not have been able to create my tfrecords and explore this idea without his work.

\n

Congrats to the winners and everyone who achieved a medal!

\n

Intro

\n

Image resolutions were all over the place, but a large majority of the images were downsampled from resolutions greater than 1024x1024, thus a major consideration was how to extract the most out of super-resolution images while being able to train a reasonably well performing model with the highest resolution given limited time and resources.

\n

Approach

\n

Toward the last week of the competition, I wanted to run 1024x1024 but with limited time and resources I came up with a hack: center crop 1024x1024 resolution sizes of 512x512. To my surprise, by including the entire upsample center crop in train, there was no leakage. Somehow the 4x increased pixel density and zoom contributed to generalizing better without leakage.

\n

While this approach appears to be a mix of data augmentation and upsampling both positive and negatives, I believe it was not. Images that were originally super-resolution were downsampled (resized to a lower resolution) just to fit within a pre-trained model. Downsampled images have lost so much information which is precisely why image sizes mattered so much in this competition.

\n

Edit: to clarify - I trained on both 2020 512x512 resolutions and 512x512 super resolution crops from 1024x1024. I validated on only 2020 original 512x512 resolutions.

\n

Final Single Model Results

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
CVLBPrivate LB
0.93790.93940.9325
\n

B6 512x512 15 fold took over 22 hours to train on TPU with 75000 images per epoch. While the LB was somewhat low, I felt confident that private LB would be close judging by how close CV was to LB.

\n

Edit: Using this approach, I was able to increase CV by 0.02-0.03, which was huge!

\n

Extra

\n

This same approach was briefly explored with 2018 + 2017 images in addition to 2020 with some success but I ran out of time to get conclusive results.

\n

Side note:

\n", "rawMarkdown": "### Hacking Super-resolution Images\n\nFirst, thank you to the organizers and Kaggle for this competition. Thank you to everyone who contributed to making this image based competition fun. \n\nSpecial thanks to @cdeotte for all of his notebooks and discussion. I would not have been able to create my tfrecords and explore this idea without his work.\n\nCongrats to the winners and everyone who achieved a medal!\n\n### Intro\n\nImage resolutions were all over the place, but a large majority of the images were downsampled from resolutions greater than 1024x1024, thus a major consideration was how to extract the most out of super-resolution images while being able to train a reasonably well performing model with the highest resolution given limited time and resources.\n\n### Approach \n\nToward the last week of the competition, I wanted to run 1024x1024 but with limited time and resources I came up with a hack: **center crop 1024x1024 resolution sizes of 512x512**. To my surprise, by including the entire upsample center crop in train, there was no leakage. Somehow the 4x increased pixel density and zoom contributed to generalizing better without leakage. \n\nWhile this approach appears to be a mix of data augmentation and upsampling both positive and negatives, I believe it was not. Images that were originally super-resolution were downsampled (resized to a lower resolution) just to fit within a pre-trained model. Downsampled images have lost so much information which is precisely why image sizes mattered so much in this competition.\n\nEdit: to clarify - I trained on both 2020 512x512 resolutions **and** 512x512 super resolution crops from 1024x1024. I validated on only 2020 original 512x512 resolutions.\n\n### Final Single Model Results \n\n| CV | LB | Private LB |\n| --- | --- | --- |\n| 0.9379 | 0.9394 | 0.9325\n\nB6 512x512 15 fold took over 22 hours to train on TPU with 75000 images per epoch. While the LB was somewhat low, I felt confident that private LB would be close judging by how close CV was to LB.\n\nEdit: Using this approach, I was able to increase CV by 0.02-0.03, which was huge!\n\n### Extra\n\nThis same approach was briefly explored with 2018 + 2017 images in addition to 2020 with some success but I ran out of time to get conclusive results. \n\nSide note: \n- I refer to upsampling as the approach to increase the size of a training dataset. \n- On the other hand, I refer to downsampled resolutions as a process by which pixel information is lost due to resizing a larger image to a smaller one.", "votes": null }, { "id": "974728", "postDate": "08/18/2020 03:10:17", "content": "

Nice trick Tim. Congrats on a great solo silver finish. By starting with 1024x1024 and then cropping to 512x512, you're getting all the information of the larger images but get to train at the speed of smaller images. I'm confused about the following:

\n
\n

To my surprise, by including the entire upsample center crop in train, there was no leakage.

\n
\n

Are you saying that you included all the extra malignant images including the ones in your validation fold? Also did you both center crop train and validation? So aren't the exact same malignant images now in both train and val?

", "rawMarkdown": "Nice trick Tim. Congrats on a great solo silver finish. By starting with 1024x1024 and then cropping to 512x512, you're getting all the information of the larger images but get to train at the speed of smaller images. I'm confused about the following:\n\n> To my surprise, by including the entire upsample center crop in train, there was no leakage.\n\nAre you saying that you included all the extra malignant images including the ones in your validation fold? Also did you both center crop train and validation? So aren't the exact same malignant images now in both train and val?", "votes": null }, { "id": "974738", "postDate": "08/18/2020 03:17:07", "content": "

So I included the center cropped images for the entire 2020 (positive/negative) as train for each of the 15 folds. In all likelihood there may have been a slight leakage at best. Certainly I managed to get away with it even though it was generally not a recommended approach.

\n

I was also editing this idea in real-time since I had to split the notebook up into 15 individual runs. So actually about half had the same images (albeit augmented) in both train and validation and the other half did not.

", "rawMarkdown": "So I included the center cropped images for the entire 2020 (positive/negative) as train for each of the 15 folds. In all likelihood there may have been a slight leakage at best. Certainly I managed to get away with it even though it was generally not a recommended approach.\n\nI was also editing this idea in real-time since I had to split the notebook up into 15 individual runs. So actually about half had the same images (albeit augmented) in both train and validation and the other half did not.", "votes": null }, { "id": "974817", "postDate": "08/18/2020 04:10:37", "content": "
\n

B6 512x512 15 fold took over 22 hours to train on TPU with 75000 images per epoch. While the LB was somewhat low, I felt confident that private LB would be close judging by how close CV was to LB.

\n
\n

After removed all duplicates from all sources, there're only ~57k images left.

", "rawMarkdown": "> B6 512x512 15 fold took over 22 hours to train on TPU with 75000 images per epoch. While the LB was somewhat low, I felt confident that private LB would be close judging by how close CV was to LB.\n\nAfter removed all duplicates from all sources, there're only ~57k images left.", "votes": null }, { "id": "974852", "postDate": "08/18/2020 04:28:01", "content": "

So there here's the breakdown of the 75000 training images:

\n\n

32692 + 12040 + 30512 = 75224

", "rawMarkdown": "So there here's the breakdown of the 75000 training images: \n\n- 32692 images in 2020 with duplicates removed (center cropped 512)\n- 12900 2018 images * 14/15 = 12040\n- 32692 2020 original images * 14/15 = 30512 \n\n32692 + 12040 + 30512 = 75224", "votes": null }, { "id": "974868", "postDate": "08/18/2020 04:34:02", "content": "

how about method that uses:

\n
    \n
  1. self supervised to train on crops
    \n2.freeze pretrained self supervised model. add a classification head and train head on large images?
    2. \n
\n

self-super tasks could be like predicting one part of image from other part (like GPT3, wavtovec) or clustering or contrastive loss.

", "rawMarkdown": "how about method that uses:\n1. self supervised to train on crops\n2.freeze pretrained self supervised model. add a classification head and train head on large images?\n\nself-super tasks could be like predicting one part of image from other part (like GPT3, wavtovec) or clustering or contrastive loss.", "votes": null }, { "id": "974931", "postDate": "08/18/2020 05:01:10", "content": "

Oh so you added center cropped 512 images. I don't know if this could be consider a leakage??

", "rawMarkdown": "Oh so you added center cropped 512 images. I don't know if this could be consider a leakage??", "votes": null }, { "id": "974939", "postDate": "08/18/2020 05:05:43", "content": "

You can try it. My dataset melanoma008 is 2020 center crop 512 with no meta data. Let me know if you notice leakage. Based on what I've seen, I don't think there's much leakage if any. On the safe side, you can easily just include only the same folds as train.

", "rawMarkdown": "You can try it. My dataset melanoma008 is 2020 center crop 512 with no meta data. Let me know if you notice leakage. Based on what I've seen, I don't think there's much leakage if any. On the safe side, you can easily just include only the same folds as train.", "votes": null }, { "id": "974949", "postDate": "08/18/2020 05:09:39", "content": "

I'm still a little confused. Did you use both 512 full images and 512 center cropped (from 1024) to train with?

", "rawMarkdown": "I'm still a little confused. Did you use both 512 full images and 512 center cropped (from 1024) to train with?", "votes": null }, { "id": "974957", "postDate": "08/18/2020 05:12:34", "content": "

Yes! I added a full upsample of 512 center crop of 1024x1024 to 512 2020 original train. It added 0.2x to CV.

", "rawMarkdown": "Yes! I added a full upsample of 512 center crop of 1024x1024 to 512 2020 original train. It added 0.2x to CV.", "votes": null }, { "id": "974966", "postDate": "08/18/2020 05:16:35", "content": "

You get the benefit of seeing the original image at 512x512, and the benefit of zoomed in super-resolution crops at 1024x1024. Validation is only 2020 original 512x512.

", "rawMarkdown": "You get the benefit of seeing the original image at 512x512, **and** the benefit of zoomed in super-resolution crops at 1024x1024. Validation is only 2020 original 512x512.", "votes": null }, { "id": "974970", "postDate": "08/18/2020 05:19:00", "content": "

Wow, that's very creative. I need to try this. I thought about doing something like this and then forgot, haha.

", "rawMarkdown": "Wow, that's very creative. I need to try this. I thought about doing something like this and then forgot, haha.", "votes": null }, { "id": "974991", "postDate": "08/18/2020 05:28:18", "content": "

Interestingly, I trained 2018 512x512 original + 512x512 center crop 1024x1024 + 2020 512x512 center crop 1024x1024 and validated on the entire 2020 training data - all 32692 images and was able to achieve 0.895 CV. That's how I knew there was no leakage. If there was the entire 2020 validation would be way higher.

", "rawMarkdown": "Interestingly, I trained 2018 512x512 original + 512x512 center crop 1024x1024 + 2020 512x512 center crop 1024x1024 and validated on the entire 2020 training data - all 32692 images and was able to achieve 0.895 CV. That's how I knew there was no leakage. If there was the entire 2020 validation would be way higher.", "votes": null }, { "id": "975943", "postDate": "08/18/2020 14:45:20", "content": "

Congratulations ! This is one solution that put smile on my face. I would not have thought of purposefully putting center cropped images for the entire 2020 (positive/negative) as train given that this would cause a leak. Tells me how assumptions fail.

", "rawMarkdown": "Congratulations ! This is one solution that put smile on my face. I would not have thought of purposefully putting center cropped images for the entire 2020 (positive/negative) as train given that this would cause a leak. Tells me how assumptions fail.", "votes": null } ], "comments": [ { "id": 974728, "author_name": "cdeotte", "author_url": "", "post_date": "08/18/2020 03:10:17", "content": "

Nice trick Tim. Congrats on a great solo silver finish. By starting with 1024x1024 and then cropping to 512x512, you're getting all the information of the larger images but get to train at the speed of smaller images. I'm confused about the following:

\n
\n

To my surprise, by including the entire upsample center crop in train, there was no leakage.

\n
\n

Are you saying that you included all the extra malignant images including the ones in your validation fold? Also did you both center crop train and validation? So aren't the exact same malignant images now in both train and val?

", "votes": null, "replies": [ { "id": 974738, "author_name": "teeyee314", "author_url": "", "post_date": "08/18/2020 03:17:07", "content": "

So I included the center cropped images for the entire 2020 (positive/negative) as train for each of the 15 folds. In all likelihood there may have been a slight leakage at best. Certainly I managed to get away with it even though it was generally not a recommended approach.

\n

I was also editing this idea in real-time since I had to split the notebook up into 15 individual runs. So actually about half had the same images (albeit augmented) in both train and validation and the other half did not.

", "votes": null, "replies": [] } ] }, { "id": 974817, "author_name": "quandapro", "author_url": "", "post_date": "08/18/2020 04:10:37", "content": "
\n

B6 512x512 15 fold took over 22 hours to train on TPU with 75000 images per epoch. While the LB was somewhat low, I felt confident that private LB would be close judging by how close CV was to LB.

\n
\n

After removed all duplicates from all sources, there're only ~57k images left.

", "votes": null, "replies": [ { "id": 974852, "author_name": "teeyee314", "author_url": "", "post_date": "08/18/2020 04:28:01", "content": "

So there here's the breakdown of the 75000 training images:

\n\n

32692 + 12040 + 30512 = 75224

", "votes": null, "replies": [] }, { "id": 974931, "author_name": "quandapro", "author_url": "", "post_date": "08/18/2020 05:01:10", "content": "

Oh so you added center cropped 512 images. I don't know if this could be consider a leakage??

", "votes": null, "replies": [] }, { "id": 974939, "author_name": "teeyee314", "author_url": "", "post_date": "08/18/2020 05:05:43", "content": "

You can try it. My dataset melanoma008 is 2020 center crop 512 with no meta data. Let me know if you notice leakage. Based on what I've seen, I don't think there's much leakage if any. On the safe side, you can easily just include only the same folds as train.

", "votes": null, "replies": [] }, { "id": 974949, "author_name": "cdeotte", "author_url": "", "post_date": "08/18/2020 05:09:39", "content": "

I'm still a little confused. Did you use both 512 full images and 512 center cropped (from 1024) to train with?

", "votes": null, "replies": [] }, { "id": 974957, "author_name": "teeyee314", "author_url": "", "post_date": "08/18/2020 05:12:34", "content": "

Yes! I added a full upsample of 512 center crop of 1024x1024 to 512 2020 original train. It added 0.2x to CV.

", "votes": null, "replies": [] }, { "id": 974966, "author_name": "teeyee314", "author_url": "", "post_date": "08/18/2020 05:16:35", "content": "

You get the benefit of seeing the original image at 512x512, and the benefit of zoomed in super-resolution crops at 1024x1024. Validation is only 2020 original 512x512.

", "votes": null, "replies": [] }, { "id": 974970, "author_name": "cdeotte", "author_url": "", "post_date": "08/18/2020 05:19:00", "content": "

Wow, that's very creative. I need to try this. I thought about doing something like this and then forgot, haha.

", "votes": null, "replies": [] }, { "id": 974991, "author_name": "teeyee314", "author_url": "", "post_date": "08/18/2020 05:28:18", "content": "

Interestingly, I trained 2018 512x512 original + 512x512 center crop 1024x1024 + 2020 512x512 center crop 1024x1024 and validated on the entire 2020 training data - all 32692 images and was able to achieve 0.895 CV. That's how I knew there was no leakage. If there was the entire 2020 validation would be way higher.

", "votes": null, "replies": [] } ] }, { "id": 974868, "author_name": "hengck23", "author_url": "", "post_date": "08/18/2020 04:34:02", "content": "

how about method that uses:

\n
    \n
  1. self supervised to train on crops
    \n2.freeze pretrained self supervised model. add a classification head and train head on large images?
    2. \n
\n

self-super tasks could be like predicting one part of image from other part (like GPT3, wavtovec) or clustering or contrastive loss.

", "votes": null, "replies": [] }, { "id": 975943, "author_name": "realsid", "author_url": "", "post_date": "08/18/2020 14:45:20", "content": "

Congratulations ! This is one solution that put smile on my face. I would not have thought of purposefully putting center cropped images for the entire 2020 (positive/negative) as train given that this would cause a leak. Tells me how assumptions fail.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "974709": "### Hacking Super-resolution Images\n\nFirst, thank you to the organizers and Kaggle for this competition. Thank you to everyone who contributed to making this image based competition fun. \n\nSpecial thanks to @cdeotte for all of his notebooks and discussion. I would not have been able to create my tfrecords and explore this idea without his work.\n\nCongrats to the winners and everyone who achieved a medal!\n\n### Intro\n\nImage resolutions were all over the place, but a large majority of the images were downsampled from resolutions greater than 1024x1024, thus a major consideration was how to extract the most out of super-resolution images while being able to train a reasonably well performing model with the highest resolution given limited time and resources.\n\n### Approach \n\nToward the last week of the competition, I wanted to run 1024x1024 but with limited time and resources I came up with a hack: **center crop 1024x1024 resolution sizes of 512x512**. To my surprise, by including the entire upsample center crop in train, there was no leakage. Somehow the 4x increased pixel density and zoom contributed to generalizing better without leakage. \n\nWhile this approach appears to be a mix of data augmentation and upsampling both positive and negatives, I believe it was not. Images that were originally super-resolution were downsampled (resized to a lower resolution) just to fit within a pre-trained model. Downsampled images have lost so much information which is precisely why image sizes mattered so much in this competition.\n\nEdit: to clarify - I trained on both 2020 512x512 resolutions **and** 512x512 super resolution crops from 1024x1024. I validated on only 2020 original 512x512 resolutions.\n\n### Final Single Model Results \n\n| CV | LB | Private LB |\n| --- | --- | --- |\n| 0.9379 | 0.9394 | 0.9325\n\nB6 512x512 15 fold took over 22 hours to train on TPU with 75000 images per epoch. While the LB was somewhat low, I felt confident that private LB would be close judging by how close CV was to LB.\n\nEdit: Using this approach, I was able to increase CV by 0.02-0.03, which was huge!\n\n### Extra\n\nThis same approach was briefly explored with 2018 + 2017 images in addition to 2020 with some success but I ran out of time to get conclusive results. \n\nSide note: \n- I refer to upsampling as the approach to increase the size of a training dataset. \n- On the other hand, I refer to downsampled resolutions as a process by which pixel information is lost due to resizing a larger image to a smaller one.", "974728": "Nice trick Tim. Congrats on a great solo silver finish. By starting with 1024x1024 and then cropping to 512x512, you're getting all the information of the larger images but get to train at the speed of smaller images. I'm confused about the following:\n\n> To my surprise, by including the entire upsample center crop in train, there was no leakage.\n\nAre you saying that you included all the extra malignant images including the ones in your validation fold? Also did you both center crop train and validation? So aren't the exact same malignant images now in both train and val?", "974738": "So I included the center cropped images for the entire 2020 (positive/negative) as train for each of the 15 folds. In all likelihood there may have been a slight leakage at best. Certainly I managed to get away with it even though it was generally not a recommended approach.\n\nI was also editing this idea in real-time since I had to split the notebook up into 15 individual runs. So actually about half had the same images (albeit augmented) in both train and validation and the other half did not.", "974817": "> B6 512x512 15 fold took over 22 hours to train on TPU with 75000 images per epoch. While the LB was somewhat low, I felt confident that private LB would be close judging by how close CV was to LB.\n\nAfter removed all duplicates from all sources, there're only ~57k images left.", "974852": "So there here's the breakdown of the 75000 training images: \n\n- 32692 images in 2020 with duplicates removed (center cropped 512)\n- 12900 2018 images * 14/15 = 12040\n- 32692 2020 original images * 14/15 = 30512 \n\n32692 + 12040 + 30512 = 75224", "974868": "how about method that uses:\n1. self supervised to train on crops\n2.freeze pretrained self supervised model. add a classification head and train head on large images?\n\nself-super tasks could be like predicting one part of image from other part (like GPT3, wavtovec) or clustering or contrastive loss.", "974931": "Oh so you added center cropped 512 images. I don't know if this could be consider a leakage??", "974939": "You can try it. My dataset melanoma008 is 2020 center crop 512 with no meta data. Let me know if you notice leakage. Based on what I've seen, I don't think there's much leakage if any. On the safe side, you can easily just include only the same folds as train.", "974949": "I'm still a little confused. Did you use both 512 full images and 512 center cropped (from 1024) to train with?", "974957": "Yes! I added a full upsample of 512 center crop of 1024x1024 to 512 2020 original train. It added 0.2x to CV.", "974966": "You get the benefit of seeing the original image at 512x512, **and** the benefit of zoomed in super-resolution crops at 1024x1024. Validation is only 2020 original 512x512.", "974970": "Wow, that's very creative. I need to try this. I thought about doing something like this and then forgot, haha.", "974991": "Interestingly, I trained 2018 512x512 original + 512x512 center crop 1024x1024 + 2020 512x512 center crop 1024x1024 and validated on the entire 2020 training data - all 32692 images and was able to achieve 0.895 CV. That's how I knew there was no leakage. If there was the entire 2020 validation would be way higher.", "975943": "Congratulations ! This is one solution that put smile on my face. I would not have thought of purposefully putting center cropped images for the entire 2020 (positive/negative) as train given that this would cause a leak. Tells me how assumptions fail." }, "source": "meta" }