In this (and pretty much any other) competition it's very important to use TTA (test-time augmentation) if you want to get to the top. I would like to share how to do TTA with PyTorch, because there are not many tutorials on this (for instance, in original documentation there is only an example with 10-crop augmentation)
\n\nFirst, select desired augmentations. Say, we want horizontal flip and vertical flip (in case of other augmentations everything is exactly the same).
\n\nSecond, we create transform that later we will pass to Dataset. Since we want to save several versions of one image, we will use torch.stack that allows us to concatenates sequence of tensors along a new dimension. In other words, if previously you data patch in terms of shape looked like this: [batch size, number of channels, height, width], after applying torch.stack for TTA it will look like this: [batch size, number of augmented images, number of channels, height, width]. In other words, +1 dimension.
\n\nSo, we need a list of tensors. To do this, we do the following:
\n\n\ntransforms.Lambda\n(lambda image: torch.stack([OUR LIST OF TENSORS]))\n\nAnd we create list of tensors as follows: \n\ntransforms.ToTensor(transforms.HorizolntalFlip(p=1)(image)), \ntransforms.ToTensor(transforms.VerticalFlip(p=1)(image)), \ntransforms.ToTensor(image)\n\nUsually, we also normalize images. In order to do that, here is what you should do:\n```
transforms.Lambda\n(lambda images: torch.stack([normalize(image) for image in images]))\n```
\n\nThird, we need to calculate a prediction. You can do it as follows:\n\nbatch_size, n_crops, c, h, w = data.size()\ndata = data.view(-1, c, h, w)\noutput = model(data)\noutput = output.view(batch_size, n_crops, -1).mean(1)\n\nSo, first we extract dimensions. Then, change the shape of the tensor and calculate the output. And then we take a mean of the outputs for all augmented versions of the original image.
If you have any questions, just let me know.
", "messages": [ { "id": "490197", "postDate": "03/14/2019 11:26:47", "content": "In this (and pretty much any other) competition it's very important to use TTA (test-time augmentation) if you want to get to the top. I would like to share how to do TTA with PyTorch, because there are not many tutorials on this (for instance, in original documentation there is only an example with 10-crop augmentation)
\n\nFirst, select desired augmentations. Say, we want horizontal flip and vertical flip (in case of other augmentations everything is exactly the same).
\n\nSecond, we create transform that later we will pass to Dataset. Since we want to save several versions of one image, we will use torch.stack that allows us to concatenates sequence of tensors along a new dimension. In other words, if previously you data patch in terms of shape looked like this: [batch size, number of channels, height, width], after applying torch.stack for TTA it will look like this: [batch size, number of augmented images, number of channels, height, width]. In other words, +1 dimension.
\n\nSo, we need a list of tensors. To do this, we do the following:
\n\n\ntransforms.Lambda\n(lambda image: torch.stack([OUR LIST OF TENSORS]))\n\nAnd we create list of tensors as follows: \n\ntransforms.ToTensor(transforms.HorizolntalFlip(p=1)(image)), \ntransforms.ToTensor(transforms.VerticalFlip(p=1)(image)), \ntransforms.ToTensor(image)\n\nUsually, we also normalize images. In order to do that, here is what you should do:\n```
transforms.Lambda\n(lambda images: torch.stack([normalize(image) for image in images]))\n```
\n\nThird, we need to calculate a prediction. You can do it as follows:\n\nbatch_size, n_crops, c, h, w = data.size()\ndata = data.view(-1, c, h, w)\noutput = model(data)\noutput = output.view(batch_size, n_crops, -1).mean(1)\n\nSo, first we extract dimensions. Then, change the shape of the tensor and calculate the output. And then we take a mean of the outputs for all augmented versions of the original image.
If you have any questions, just let me know.
", "rawMarkdown": "In this (and pretty much any other) competition it's very important to use TTA (test-time augmentation) if you want to get to the top. I would like to share how to do TTA with PyTorch, because there are not many tutorials on this (for instance, in original documentation there is only an example with 10-crop augmentation) \n\nFirst, select desired augmentations. Say, we want horizontal flip and vertical flip (in case of other augmentations everything is exactly the same). \n\nSecond, we create transform that later we will pass to Dataset. Since we want to save several versions of one image, we will use torch.stack that allows us to concatenates sequence of tensors along a new dimension. In other words, if previously you data patch in terms of shape looked like this: [batch size, number of channels, height, width], after applying torch.stack for TTA it will look like this: [batch size, number of augmented images, number of channels, height, width]. In other words, +1 dimension. \n\nSo, we need a list of tensors. To do this, we do the following:\n\n```\ntransforms.Lambda\n(lambda image: torch.stack([OUR LIST OF TENSORS]))\n```\nAnd we create list of tensors as follows: \n```\ntransforms.ToTensor(transforms.HorizolntalFlip(p=1)(image)), \ntransforms.ToTensor(transforms.VerticalFlip(p=1)(image)), \ntransforms.ToTensor(image)\n```\nUsually, we also normalize images. In order to do that, here is what you should do:\n```\n\ntransforms.Lambda\n(lambda images: torch.stack([normalize(image) for image in images]))\n```\n\nThird, we need to calculate a prediction. You can do it as follows:\n```\nbatch_size, n_crops, c, h, w = data.size()\ndata = data.view(-1, c, h, w)\noutput = model(data)\noutput = output.view(batch_size, n_crops, -1).mean(1)\n```\nSo, first we extract dimensions. Then, change the shape of the tensor and calculate the output. And then we take a mean of the outputs for all augmented versions of the original image. \n\nIf you have any questions, just let me know.", "votes": null }, { "id": "490637", "postDate": "03/14/2019 18:15:11", "content": "Thanks for your work!! I just predict with augmented test data and do k iterations then take mean before, it costs me lots of time. But it seems like you set batch_size x n_crops as the first channel while predicting, would batch_size x n_crops increases the cuda memory that we need than we set batch_size as first channel without TTA?
", "rawMarkdown": "Thanks for your work!! I just predict with augmented test data and do k iterations then take mean before, it costs me lots of time. But it seems like you set batch_size x n_crops as the first channel while predicting, would batch_size x n_crops increases the cuda memory that we need than we set batch_size as first channel without TTA?", "votes": null }, { "id": "490708", "postDate": "03/14/2019 19:19:54", "content": "Yeah, the way I see it, it does take more cuda memory. Of course, we could run this not for the whole batch, but using a for loop, but it would run longer since we wouldn't take advantage of vectorization.
\n\nI took a different approach. I used a lower batch size. For instance, during training my batch size is 150, but during cv and test it's only 10. Otherwise, I simply run out of cuda memory.
", "rawMarkdown": "Yeah, the way I see it, it does take more cuda memory. Of course, we could run this not for the whole batch, but using a for loop, but it would run longer since we wouldn't take advantage of vectorization. \n\nI took a different approach. I used a lower batch size. For instance, during training my batch size is 150, but during cv and test it's only 10. Otherwise, I simply run out of cuda memory.", "votes": null }, { "id": "490805", "postDate": "03/14/2019 22:03:10", "content": "Got it and thanks!! Through my experiments, I simply set n_crops from {8, 16, 32}, and I found that the performance of my models (Densenet121, Densenet169, Nasnet-Mobile) will increase if I use larger n_crops. Maybe you could choose a larger n_crops too.
", "rawMarkdown": "Got it and thanks!! Through my experiments, I simply set n_crops from {8, 16, 32}, and I found that the performance of my models (Densenet121, Densenet169, Nasnet-Mobile) will increase if I use larger n_crops. Maybe you could choose a larger n_crops too.", "votes": null }, { "id": "490816", "postDate": "03/14/2019 22:15:30", "content": "Well, I use about 15 crops (this is actually kind of a typo. I used n_crops, when I was working with the 10-crop method. Meaning that you take 10 crops and average predictions for them). The parameter that is changing (in case of my model) is batch size. The bigger the number of crops, the lower the batch size.
\n\nAnyway, with the right choice of augmentations, getting more augmented versions of the original image should definitely help. Of course, it makes the whole process more computationally expensive. For instance, in my case cross validation runs in about 30 minutes, just like training. The difference is that training set is like >10 times bigger :)
", "rawMarkdown": "Well, I use about 15 crops (this is actually kind of a typo. I used n_crops, when I was working with the 10-crop method. Meaning that you take 10 crops and average predictions for them). The parameter that is changing (in case of my model) is batch size. The bigger the number of crops, the lower the batch size. \n\nAnyway, with the right choice of augmentations, getting more augmented versions of the original image should definitely help. Of course, it makes the whole process more computationally expensive. For instance, in my case cross validation runs in about 30 minutes, just like training. The difference is that training set is like >10 times bigger :)", "votes": null }, { "id": "490839", "postDate": "03/14/2019 22:47:39", "content": "I see, right now for one model I have done 32 times predictions just like use 32 models, it's so computationally expensive--(one round test time) x 32 x (number of models that I used to ensemble). I need to improve the performance of my single model to make my method more realistic, using too many models in the real world is not practical, lol.
", "rawMarkdown": "I see, right now for one model I have done 32 times predictions just like use 32 models, it's so computationally expensive--(one round test time) x 32 x (number of models that I used to ensemble). I need to improve the performance of my single model to make my method more realistic, using too many models in the real world is not practical, lol.", "votes": null }, { "id": "490842", "postDate": "03/14/2019 22:54:19", "content": "Well, when it comes to ensembles, the approaches that we use on Kaggle are very often very-very unpractical. I mean come on, this whole competition is about getting 99.7-99.8 on AUROC metric instead of 99.5 :)
\n\nBtw, what kind of TTA do you use?
\n\nP.S. Would you like to take this to Telegram? Would love to exchange ideas regarding this competition.
", "rawMarkdown": "Well, when it comes to ensembles, the approaches that we use on Kaggle are very often very-very unpractical. I mean come on, this whole competition is about getting 99.7-99.8 on AUROC metric instead of 99.5 :) \n\nBtw, what kind of TTA do you use? \n\nP.S. Would you like to take this to Telegram? Would love to exchange ideas regarding this competition.", "votes": null }, { "id": "490860", "postDate": "03/14/2019 23:28:32", "content": "Of course ! I never use Telegram before, I have just signed in a new account, my user name is jionie. This the first time I tried TTA and I don't know whether I'm right but the results are pretty good, I just use the same transform that I used for training data to make sure the augmented test data follows the same rule of training data; also while training, same transform for validation data.
", "rawMarkdown": "Of course ! I never use Telegram before, I have just signed in a new account, my user name is jionie. This the first time I tried TTA and I don't know whether I'm right but the results are pretty good, I just use the same transform that I used for training data to make sure the augmented test data follows the same rule of training data; also while training, same transform for validation data.", "votes": null }, { "id": "604259", "postDate": "08/21/2019 07:37:31", "content": "Thank you for this wonderful method! =)
", "rawMarkdown": "Thank you for this wonderful method! =)", "votes": null }, { "id": "1162781", "postDate": "01/21/2021 10:43:45", "content": "is there a way to implement TTA in albumentations library. Is so could you please share the code snippet?
Sure. You can check the great pytorch-toolbelt from BloodAxe https://github.com/BloodAxe/pytorch-toolbelt
", "rawMarkdown": "Sure. You can check the great pytorch-toolbelt from BloodAxe https://github.com/BloodAxe/pytorch-toolbelt", "votes": null } ], "comments": [ { "id": 1162781, "author_name": "aravindpadman", "author_url": "", "post_date": "01/21/2021 10:43:45", "content": "is there a way to implement TTA in albumentations library. Is so could you please share the code snippet?
Sure. You can check the great pytorch-toolbelt from BloodAxe https://github.com/BloodAxe/pytorch-toolbelt
", "votes": null, "replies": [] } ] }, { "id": 490637, "author_name": "jionie", "author_url": "", "post_date": "03/14/2019 18:15:11", "content": "Thanks for your work!! I just predict with augmented test data and do k iterations then take mean before, it costs me lots of time. But it seems like you set batch_size x n_crops as the first channel while predicting, would batch_size x n_crops increases the cuda memory that we need than we set batch_size as first channel without TTA?
", "votes": null, "replies": [ { "id": 490708, "author_name": "ivanpan", "author_url": "", "post_date": "03/14/2019 19:19:54", "content": "Yeah, the way I see it, it does take more cuda memory. Of course, we could run this not for the whole batch, but using a for loop, but it would run longer since we wouldn't take advantage of vectorization.
\n\nI took a different approach. I used a lower batch size. For instance, during training my batch size is 150, but during cv and test it's only 10. Otherwise, I simply run out of cuda memory.
", "votes": null, "replies": [] }, { "id": 490805, "author_name": "jionie", "author_url": "", "post_date": "03/14/2019 22:03:10", "content": "Got it and thanks!! Through my experiments, I simply set n_crops from {8, 16, 32}, and I found that the performance of my models (Densenet121, Densenet169, Nasnet-Mobile) will increase if I use larger n_crops. Maybe you could choose a larger n_crops too.
", "votes": null, "replies": [] }, { "id": 490816, "author_name": "ivanpan", "author_url": "", "post_date": "03/14/2019 22:15:30", "content": "Well, I use about 15 crops (this is actually kind of a typo. I used n_crops, when I was working with the 10-crop method. Meaning that you take 10 crops and average predictions for them). The parameter that is changing (in case of my model) is batch size. The bigger the number of crops, the lower the batch size.
\n\nAnyway, with the right choice of augmentations, getting more augmented versions of the original image should definitely help. Of course, it makes the whole process more computationally expensive. For instance, in my case cross validation runs in about 30 minutes, just like training. The difference is that training set is like >10 times bigger :)
", "votes": null, "replies": [] }, { "id": 490839, "author_name": "jionie", "author_url": "", "post_date": "03/14/2019 22:47:39", "content": "I see, right now for one model I have done 32 times predictions just like use 32 models, it's so computationally expensive--(one round test time) x 32 x (number of models that I used to ensemble). I need to improve the performance of my single model to make my method more realistic, using too many models in the real world is not practical, lol.
", "votes": null, "replies": [] }, { "id": 490842, "author_name": "ivanpan", "author_url": "", "post_date": "03/14/2019 22:54:19", "content": "Well, when it comes to ensembles, the approaches that we use on Kaggle are very often very-very unpractical. I mean come on, this whole competition is about getting 99.7-99.8 on AUROC metric instead of 99.5 :)
\n\nBtw, what kind of TTA do you use?
\n\nP.S. Would you like to take this to Telegram? Would love to exchange ideas regarding this competition.
", "votes": null, "replies": [] }, { "id": 490860, "author_name": "jionie", "author_url": "", "post_date": "03/14/2019 23:28:32", "content": "Of course ! I never use Telegram before, I have just signed in a new account, my user name is jionie. This the first time I tried TTA and I don't know whether I'm right but the results are pretty good, I just use the same transform that I used for training data to make sure the augmented test data follows the same rule of training data; also while training, same transform for validation data.
", "votes": null, "replies": [] } ] }, { "id": 604259, "author_name": "yousof9", "author_url": "", "post_date": "08/21/2019 07:37:31", "content": "Thank you for this wonderful method! =)
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "490197": "In this (and pretty much any other) competition it's very important to use TTA (test-time augmentation) if you want to get to the top. I would like to share how to do TTA with PyTorch, because there are not many tutorials on this (for instance, in original documentation there is only an example with 10-crop augmentation) \n\nFirst, select desired augmentations. Say, we want horizontal flip and vertical flip (in case of other augmentations everything is exactly the same). \n\nSecond, we create transform that later we will pass to Dataset. Since we want to save several versions of one image, we will use torch.stack that allows us to concatenates sequence of tensors along a new dimension. In other words, if previously you data patch in terms of shape looked like this: [batch size, number of channels, height, width], after applying torch.stack for TTA it will look like this: [batch size, number of augmented images, number of channels, height, width]. In other words, +1 dimension. \n\nSo, we need a list of tensors. To do this, we do the following:\n\n```\ntransforms.Lambda\n(lambda image: torch.stack([OUR LIST OF TENSORS]))\n```\nAnd we create list of tensors as follows: \n```\ntransforms.ToTensor(transforms.HorizolntalFlip(p=1)(image)), \ntransforms.ToTensor(transforms.VerticalFlip(p=1)(image)), \ntransforms.ToTensor(image)\n```\nUsually, we also normalize images. In order to do that, here is what you should do:\n```\n\ntransforms.Lambda\n(lambda images: torch.stack([normalize(image) for image in images]))\n```\n\nThird, we need to calculate a prediction. You can do it as follows:\n```\nbatch_size, n_crops, c, h, w = data.size()\ndata = data.view(-1, c, h, w)\noutput = model(data)\noutput = output.view(batch_size, n_crops, -1).mean(1)\n```\nSo, first we extract dimensions. Then, change the shape of the tensor and calculate the output. And then we take a mean of the outputs for all augmented versions of the original image. \n\nIf you have any questions, just let me know.", "490637": "Thanks for your work!! I just predict with augmented test data and do k iterations then take mean before, it costs me lots of time. But it seems like you set batch_size x n_crops as the first channel while predicting, would batch_size x n_crops increases the cuda memory that we need than we set batch_size as first channel without TTA?", "490708": "Yeah, the way I see it, it does take more cuda memory. Of course, we could run this not for the whole batch, but using a for loop, but it would run longer since we wouldn't take advantage of vectorization. \n\nI took a different approach. I used a lower batch size. For instance, during training my batch size is 150, but during cv and test it's only 10. Otherwise, I simply run out of cuda memory.", "490805": "Got it and thanks!! Through my experiments, I simply set n_crops from {8, 16, 32}, and I found that the performance of my models (Densenet121, Densenet169, Nasnet-Mobile) will increase if I use larger n_crops. Maybe you could choose a larger n_crops too.", "490816": "Well, I use about 15 crops (this is actually kind of a typo. I used n_crops, when I was working with the 10-crop method. Meaning that you take 10 crops and average predictions for them). The parameter that is changing (in case of my model) is batch size. The bigger the number of crops, the lower the batch size. \n\nAnyway, with the right choice of augmentations, getting more augmented versions of the original image should definitely help. Of course, it makes the whole process more computationally expensive. For instance, in my case cross validation runs in about 30 minutes, just like training. The difference is that training set is like >10 times bigger :)", "490839": "I see, right now for one model I have done 32 times predictions just like use 32 models, it's so computationally expensive--(one round test time) x 32 x (number of models that I used to ensemble). I need to improve the performance of my single model to make my method more realistic, using too many models in the real world is not practical, lol.", "490842": "Well, when it comes to ensembles, the approaches that we use on Kaggle are very often very-very unpractical. I mean come on, this whole competition is about getting 99.7-99.8 on AUROC metric instead of 99.5 :) \n\nBtw, what kind of TTA do you use? \n\nP.S. Would you like to take this to Telegram? Would love to exchange ideas regarding this competition.", "490860": "Of course ! I never use Telegram before, I have just signed in a new account, my user name is jionie. This the first time I tried TTA and I don't know whether I'm right but the results are pretty good, I just use the same transform that I used for training data to make sure the augmented test data follows the same rule of training data; also while training, same transform for validation data.", "604259": "Thank you for this wonderful method! =)", "1162781": "is there a way to implement TTA in ``` albumentations ``` library. Is so could you please share the code snippet?", "1162791": "Sure. You can check the great pytorch-toolbelt from BloodAxe https://github.com/BloodAxe/pytorch-toolbelt" }, "source": "meta" }