This is a summery of \"Yuval and nosound\" model (27th place)
\nYou can find more in:
\nOur solution is based on two step model + Ensemble:
\nAs base model we used a models from the EfficientNet family[6]:
\nAll models were pre-trained on imagenet using noisy student algorithm. The models and weights are from gen-efficientnet-pytorch[2].
\nThe input to these model is the image and meta-data such as age, sex, and anatomic Site. The meta-data is processed by a small fully connected network and it’s output is concatenated to the input of the classification layer of the original EfficientNet network. This vector is going through a linear layer with output size of 256 to create the “features”, and then after an activation layer to the final linear classification layer.
\nThis network has 8 outputs and tries to classify the diagnosis label (there are actually more than 8 possible diagnoses, but some don’t have enough examples).
\nThe input to the Transformer models are a stack of features from all images belonging to the same patient + the metadata for these images.
\nThe transformer is a stack of 4 transformer encoder layers with self attention as described in Attention Is All You Need [1]. Each transformer encoder layer uses 4 self attention heads.
\nThe output of the transformer is a N*C, where N is the number of input feature vectors (the number of images) and C is the number of classes (8 in this case). Hence, the transformer predicts the class of each feature vector simultaneously, using the information from all other feature vectors.
\nThe metadata is added using a “transformer style”, i.e. each parameter is transformed to a vector (size 256) using an embedding matrix and then added to the feature vector. for continuous values (like age) the embedding matrix was replaced by a 2 layer fully connected network.
\nThe data was split to 3 folds, 3 times (using 3 different seeds for splitting), and the inference was done using 16 (or 12) TTAs. giving 144 predictions from each model. These were averaged and then the outputs of all the models were averaged. All averaging was done on the outputs before softmax and there form it is actually geometric averaging.
\nThe heavy lifting was the training and inference of the base models. This was done on a server with 2 GPUs – Tesla V100, Titan RTX that worked in parallel on different tasks. Training one fold of one model took 3H (B3) to 11H (B7, B6 large images) on the Tesla and 20% more on the Titan, this sums up to about one day for B3 and 3.5 days for B7. Inferencing all the training data for 12 TTA’s + test data for 16 TTA’s to get the features for the next level took another 4h - 14h. The transformer training took less than 1H for the full model (3 folds*3seed).
\nThe total time it took to train all models and folds is about 2.5W for one Tesla (~1.5W using the 2 GPUs).
\nAs base models we tried various types of models (pre-trained on Imagenet):
\nAt the end we used EfficientNet as it was best when judging accuracy/time
\nThe noisy student version performed better than the normal one.
\nWe also tried different image sizes and ended up using a 400*600 images in most cases, except one were we used 600*900 with the B6 network.
As was described above the metadata was processed by a small fully connected nn and its output was concatenated to the output of the EfficientNet network (after removing the original top layer).
\nWe also tried a network without metadata, and used the metadata as targets, i.e. this network predicted the diagnosis, but also the sex, age and anatomic site. The final predictions (including transformer) when using this approach weren't as good as the metadata as input approach.
\nAlthough the task at hand is to predict melanoma yes/no, it is better to let the network choose the diagnosis among a few possible options. This lets the network “understand” more about the image. The final prediction is the value of the Melanoma output after doing softmax on the output vector.
\nThe final layer in this model is a linear layer with 256 inputs and 8 outputs, we use the input to this layer as features.
\nThe following augmentations where used while training and inference:
\nRandom resize + crop
\nRandom rotation
\nRandom flip
\nRandom color jitter (brightness, contrast, saturation, hue)
\nCutout[3] - erasing a small rectangle in the image
\nHair - Randomly adding “hair like” lines to the image
\nMetadata augmentation - adding random noise to the metadata as was done in the 1st place solution in ISIC 2019 challenge [7].
\nFor inference each image was augmented differently 16 times and the final prediction was the average. These augmentations were also used for extracting 16 different features vectors per test image.
\nThe same was done to extract 12 features vectors for the train images (12 and not 16 because of time limits).
\nThe input to the transformer network is the features from all the images from one patient.
\nThe inspiration for this kind of model came from a previous competition in which we participated in RSNA Intracranial Hemorrhage Detection. In that competition, all the top solutions (including ours) used a two stage network approach (although none of them were transformers).
\nUsing a transformer seems appropriate in this case because transformers are built to seek relationships between embedding (feature) vectors in it’s input.
\nAs this is not a full seq2seq task, we only used encoder layers. The transformer is a stack of 4 encoder layers with 4 attention heads in each layer. (we also tested higher numbers of layers and attention heads - no performance improvement).
\nThe metadata was incorporated in the network by adding “metadata vectors” to the input vectors - each value was transformed to a vector size 256 and added. The discrete values’ transformation was done using a trainable embedding matrix and the continuous values using a small nn.
\nThe output of this network is a matrix of size N*C where N - the number of images, C - the number of classes. Which means it decides on all the images of the patient at once.
\nA transformer can be trained on different number of feature vectors, by using padding. But when the range of numbers is very large, from a couple of hundred images for some patients to a handful for others, this may cause some implementation issues (like in calculating the loss). To simplify these issues, we limited N to 24 feature vectors, and for each patient we randomly divided the images to groups of size up to 24.
\nThis might degrade the prediction as the most “similar” images might accidentally fall into different groups, but as we use TTA, this issue is almost solved.
\nFrom the base model we extract a number of feature vectors (12 for train and 16 for test) using different augmentation for the images and metadata. In the training and inference steps of the transformer model we randomly choose one of these vectors.
\nAnother augmentation is the random grouping as stated above.
\nThe original 2020 competition data is highly unbalanced, there are only 2-3% of positive targets in the train and test data. Although we were able to train the base model using uneven sampling, the best way to get good training was to add the data from ISIC 2019 competition which has a much higher percentage of melanoma images.
\nWe split the training data to 3 folds keeping all the images from the same patient in the same fold, and making sure each fold has a similar number of patients with melanoma. The ISIC2019’s data was also split evenly between the folds. The same folds were kept for the base and the transformer models.
\nTo get more diversity we had 3 different splits using 3 seeds
\nAll images were resized to an aspect ratio of 1:1.5, which was the most popular aspect ratio of the images in the original dataset. We prepared 3 image datasets of sizes 300*450, 400*600, 600*900. Most of the models were trained using the 400*600 dataset, as 300*450 gave inferior results and the 600*900 didn’t improve the results enough.
For the metadata we had to set the same terminology for the 2020 and 2019 datasets.
\nThe loss function we used was cross entropy. Although the task is to predict only melanoma we found it is better to predict the diagnosis which was split to 8 different classes one of which was melanoma. The final prediction was the value for the melanoma class, after a softmax function on all classes. We also tried a binary cross entropy on the melanoma class alone and a combination between the two, but using cross entropy gave the best results.
\nThe same loss was used for the base model and the transformer, but in the transformer we needed to regularize for the different number of predictions in each batch resulting from the different number of images for each patient.
\nWe also tried using focal loss which didn’t improve the results, but we left one of the transformer models which was trained with focal loss in the ensemble (A model with cross entropy loss gave the similar CV and LB).
\nThe transformer model was trained in two steps. For the first step we used the data from both competitions (2019, 2020). For the 2019 competition we don’t have information about the patient, and each image got a different dummy patient_id, meaning the transformer didn’t learn much from these images. In the 2nd stage we fine-tuned the transformer using only the 2020 competition’s data.
\nIn both steps we used a sampler that over sampled the larger groups.
\nAs stated above, the inference was done using TTA. For the base model we used 12-16 different augmentations and for the transformer model 32.
\nFor our final submissions we used 2 ensembles:
\nAll the “without context” model +
\nAlthough the number of images in the competition was very large, the number of different patients wasn’t large enough to give a stable and reliable CV and LB. And the correlation between the two was low. At the end we trusted neither and we submitted the models we felt were the most robust.
\nThis model can be much simpler if we won’t ensemble and take only one base model, the EfficientNet B5 is probably the best compromise.
\nIn the real world scenario the images from previous years will probably be tagged already. In that case we can use a full transformer with encoder and decoder which will perform a seq2seq operation.
", "messages": [ { "id": "1005162", "postDate": "09/10/2020 09:45:59", "content": "This is a summery of \"Yuval and nosound\" model (27th place)
\nYou can find more in:
\nOur solution is based on two step model + Ensemble:
\nAs base model we used a models from the EfficientNet family[6]:
\nAll models were pre-trained on imagenet using noisy student algorithm. The models and weights are from gen-efficientnet-pytorch[2].
\nThe input to these model is the image and meta-data such as age, sex, and anatomic Site. The meta-data is processed by a small fully connected network and it’s output is concatenated to the input of the classification layer of the original EfficientNet network. This vector is going through a linear layer with output size of 256 to create the “features”, and then after an activation layer to the final linear classification layer.
\nThis network has 8 outputs and tries to classify the diagnosis label (there are actually more than 8 possible diagnoses, but some don’t have enough examples).
\nThe input to the Transformer models are a stack of features from all images belonging to the same patient + the metadata for these images.
\nThe transformer is a stack of 4 transformer encoder layers with self attention as described in Attention Is All You Need [1]. Each transformer encoder layer uses 4 self attention heads.
\nThe output of the transformer is a N*C, where N is the number of input feature vectors (the number of images) and C is the number of classes (8 in this case). Hence, the transformer predicts the class of each feature vector simultaneously, using the information from all other feature vectors.
\nThe metadata is added using a “transformer style”, i.e. each parameter is transformed to a vector (size 256) using an embedding matrix and then added to the feature vector. for continuous values (like age) the embedding matrix was replaced by a 2 layer fully connected network.
\nThe data was split to 3 folds, 3 times (using 3 different seeds for splitting), and the inference was done using 16 (or 12) TTAs. giving 144 predictions from each model. These were averaged and then the outputs of all the models were averaged. All averaging was done on the outputs before softmax and there form it is actually geometric averaging.
\nThe heavy lifting was the training and inference of the base models. This was done on a server with 2 GPUs – Tesla V100, Titan RTX that worked in parallel on different tasks. Training one fold of one model took 3H (B3) to 11H (B7, B6 large images) on the Tesla and 20% more on the Titan, this sums up to about one day for B3 and 3.5 days for B7. Inferencing all the training data for 12 TTA’s + test data for 16 TTA’s to get the features for the next level took another 4h - 14h. The transformer training took less than 1H for the full model (3 folds*3seed).
\nThe total time it took to train all models and folds is about 2.5W for one Tesla (~1.5W using the 2 GPUs).
\nAs base models we tried various types of models (pre-trained on Imagenet):
\nAt the end we used EfficientNet as it was best when judging accuracy/time
\nThe noisy student version performed better than the normal one.
\nWe also tried different image sizes and ended up using a 400*600 images in most cases, except one were we used 600*900 with the B6 network.
As was described above the metadata was processed by a small fully connected nn and its output was concatenated to the output of the EfficientNet network (after removing the original top layer).
\nWe also tried a network without metadata, and used the metadata as targets, i.e. this network predicted the diagnosis, but also the sex, age and anatomic site. The final predictions (including transformer) when using this approach weren't as good as the metadata as input approach.
\nAlthough the task at hand is to predict melanoma yes/no, it is better to let the network choose the diagnosis among a few possible options. This lets the network “understand” more about the image. The final prediction is the value of the Melanoma output after doing softmax on the output vector.
\nThe final layer in this model is a linear layer with 256 inputs and 8 outputs, we use the input to this layer as features.
\nThe following augmentations where used while training and inference:
\nRandom resize + crop
\nRandom rotation
\nRandom flip
\nRandom color jitter (brightness, contrast, saturation, hue)
\nCutout[3] - erasing a small rectangle in the image
\nHair - Randomly adding “hair like” lines to the image
\nMetadata augmentation - adding random noise to the metadata as was done in the 1st place solution in ISIC 2019 challenge [7].
\nFor inference each image was augmented differently 16 times and the final prediction was the average. These augmentations were also used for extracting 16 different features vectors per test image.
\nThe same was done to extract 12 features vectors for the train images (12 and not 16 because of time limits).
\nThe input to the transformer network is the features from all the images from one patient.
\nThe inspiration for this kind of model came from a previous competition in which we participated in RSNA Intracranial Hemorrhage Detection. In that competition, all the top solutions (including ours) used a two stage network approach (although none of them were transformers).
\nUsing a transformer seems appropriate in this case because transformers are built to seek relationships between embedding (feature) vectors in it’s input.
\nAs this is not a full seq2seq task, we only used encoder layers. The transformer is a stack of 4 encoder layers with 4 attention heads in each layer. (we also tested higher numbers of layers and attention heads - no performance improvement).
\nThe metadata was incorporated in the network by adding “metadata vectors” to the input vectors - each value was transformed to a vector size 256 and added. The discrete values’ transformation was done using a trainable embedding matrix and the continuous values using a small nn.
\nThe output of this network is a matrix of size N*C where N - the number of images, C - the number of classes. Which means it decides on all the images of the patient at once.
\nA transformer can be trained on different number of feature vectors, by using padding. But when the range of numbers is very large, from a couple of hundred images for some patients to a handful for others, this may cause some implementation issues (like in calculating the loss). To simplify these issues, we limited N to 24 feature vectors, and for each patient we randomly divided the images to groups of size up to 24.
\nThis might degrade the prediction as the most “similar” images might accidentally fall into different groups, but as we use TTA, this issue is almost solved.
\nFrom the base model we extract a number of feature vectors (12 for train and 16 for test) using different augmentation for the images and metadata. In the training and inference steps of the transformer model we randomly choose one of these vectors.
\nAnother augmentation is the random grouping as stated above.
\nThe original 2020 competition data is highly unbalanced, there are only 2-3% of positive targets in the train and test data. Although we were able to train the base model using uneven sampling, the best way to get good training was to add the data from ISIC 2019 competition which has a much higher percentage of melanoma images.
\nWe split the training data to 3 folds keeping all the images from the same patient in the same fold, and making sure each fold has a similar number of patients with melanoma. The ISIC2019’s data was also split evenly between the folds. The same folds were kept for the base and the transformer models.
\nTo get more diversity we had 3 different splits using 3 seeds
\nAll images were resized to an aspect ratio of 1:1.5, which was the most popular aspect ratio of the images in the original dataset. We prepared 3 image datasets of sizes 300*450, 400*600, 600*900. Most of the models were trained using the 400*600 dataset, as 300*450 gave inferior results and the 600*900 didn’t improve the results enough.
For the metadata we had to set the same terminology for the 2020 and 2019 datasets.
\nThe loss function we used was cross entropy. Although the task is to predict only melanoma we found it is better to predict the diagnosis which was split to 8 different classes one of which was melanoma. The final prediction was the value for the melanoma class, after a softmax function on all classes. We also tried a binary cross entropy on the melanoma class alone and a combination between the two, but using cross entropy gave the best results.
\nThe same loss was used for the base model and the transformer, but in the transformer we needed to regularize for the different number of predictions in each batch resulting from the different number of images for each patient.
\nWe also tried using focal loss which didn’t improve the results, but we left one of the transformer models which was trained with focal loss in the ensemble (A model with cross entropy loss gave the similar CV and LB).
\nThe transformer model was trained in two steps. For the first step we used the data from both competitions (2019, 2020). For the 2019 competition we don’t have information about the patient, and each image got a different dummy patient_id, meaning the transformer didn’t learn much from these images. In the 2nd stage we fine-tuned the transformer using only the 2020 competition’s data.
\nIn both steps we used a sampler that over sampled the larger groups.
\nAs stated above, the inference was done using TTA. For the base model we used 12-16 different augmentations and for the transformer model 32.
\nFor our final submissions we used 2 ensembles:
\nAll the “without context” model +
\nAlthough the number of images in the competition was very large, the number of different patients wasn’t large enough to give a stable and reliable CV and LB. And the correlation between the two was low. At the end we trusted neither and we submitted the models we felt were the most robust.
\nThis model can be much simpler if we won’t ensemble and take only one base model, the EfficientNet B5 is probably the best compromise.
\nIn the real world scenario the images from previous years will probably be tagged already. In that case we can use a full transformer with encoder and decoder which will perform a seq2seq operation.
", "rawMarkdown": "This is a summery of \"Yuval and nosound\" model (27th place)\n\nYou can find more in:\n* [our github repository](https://github.com/yuval6957/SIIM-Transformer) \n* [Paper](https://github.com/yuval6957/SIIM-Transformer/blob/master/SIIM%20paper.pdf)\n* [Presentation](https://github.com/yuval6957/SIIM-Transformer/blob/master/SIIM%20presentation.pdf)\n* [video](https://www.youtube.com/watch?v=qa6zimKQcno&t=15s)\n\n## 1. Summary\n\nOur solution is based on two step model + Ensemble:\n\n\n\n1. Base model for feature extraction per image\n2. Transformer model - combining all the output features from a patient and predict per image. \n3. The 2nd stage also included some post - processing and ensembling.\n\n\n### Base Model:\n\nAs base model we used a models from the [EfficientNet](https://arxiv.org/abs/1905.11946) family[6]:\n\n\n\n* EfficientNet b3 \n* EfficientNet b4 \n* EfficientNet b5 \n* EfficientNet b6 \n* EfficientNet b7 \n\nAll models were pre-trained on imagenet using noisy student algorithm. The models and weights are from [gen-efficientnet-pytorch](https://github.com/rwightman/gen-efficientnet-pytorch)[2].\n\nThe input to these model is the image and meta-data such as age, sex, and anatomic Site. The meta-data is processed by a small fully connected network and it’s output is concatenated to the input of the classification layer of the original EfficientNet network. This vector is going through a linear layer with output size of 256 to create the “features”, and then after an activation layer to the final linear classification layer. \n\nThis network has 8 outputs and tries to classify the diagnosis label (there are actually more than 8 possible diagnoses, but some don’t have enough examples). \n\n\n### Transformer Models:\n\nThe input to the Transformer models are a stack of features from all images belonging to the same patient + the metadata for these images.\n\nThe transformer is a stack of 4 transformer encoder layers with self attention as described in [Attention Is All You Need](https://arxiv.org/abs/1706.03762) [1]. Each transformer encoder layer uses 4 self attention heads. \n\nThe output of the transformer is a N*C, where N is the number of input feature vectors (the number of images) and C is the number of classes (8 in this case). Hence, the transformer predicts the class of each feature vector simultaneously, using the information from all other feature vectors.\n\nThe metadata is added using a “transformer style”, i.e. each parameter is transformed to a vector (size 256) using an embedding matrix and then added to the feature vector. for continuous values (like age) the embedding matrix was replaced by a 2 layer fully connected network. \n\n\n### Ensembling the output of all networks:\n\nThe data was split to 3 folds, 3 times (using 3 different seeds for splitting), and the inference was done using 16 (or 12) TTAs. giving 144 predictions from each model. These were averaged and then the outputs of all the models were averaged. All averaging was done on the outputs before softmax and there form it is actually geometric averaging.\n\n\n### Training\n\nThe heavy lifting was the training and inference of the base models. This was done on a server with 2 GPUs – Tesla V100, Titan RTX that worked in parallel on different tasks. Training one fold of one model took 3H (B3) to 11H (B7, B6 large images) on the Tesla and 20% more on the Titan, this sums up to about one day for B3 and 3.5 days for B7. Inferencing all the training data for 12 TTA’s + test data for 16 TTA’s to get the features for the next level took another 4h - 14h. The transformer training took less than 1H for the full model (3 folds*3seed). \n\nThe total time it took to train all models and folds is about 2.5W for one Tesla (~1.5W using the 2 GPUs).\n\n\n## 2. Models and features\n\n\n### Base models\n\nAs base models we tried various types of models (pre-trained on Imagenet):\n\n\n\n* Densenets – 121, 161, 169, 201\n* EfficientNet B0, B3 , B4, B5, B6, B7 with and with noisy student pre-training and with normal pretraining \n* ResNet 101\n* Xception \n\nAt the end we used EfficientNet as it was best when judging accuracy/time\n\nThe noisy student version performed better than the normal one.\n\nWe also tried different image sizes and ended up using a `400*600` images in most cases, except one were we used `600*900` with the B6 network.\n\n\n\n#### Metadata\n\nAs was described above the metadata was processed by a small fully connected nn and its output was concatenated to the output of the EfficientNet network (after removing the original top layer).\n\nWe also tried a network without metadata, and used the metadata as targets, i.e. this network predicted the diagnosis, but also the sex, age and anatomic site. The final predictions (including transformer) when using this approach weren't as good as the metadata as input approach. \n\n\n#### Model’s output\n\nAlthough the task at hand is to predict melanoma yes/no, it is better to let the network choose the diagnosis among a few possible options. This lets the network “understand” more about the image. The final prediction is the value of the Melanoma output after doing softmax on the output vector. \n\n\n#### Features\n\nThe final layer in this model is a linear layer with 256 inputs and 8 outputs, we use the input to this layer as features. \n\n\n#### Augmentation\n\nThe following augmentations where used while training and inference:\n\nRandom resize + crop\n\nRandom rotation\n\nRandom flip\n\nRandom color jitter (brightness, contrast, saturation, hue)\n\n[Cutout](https://arxiv.org/abs/1708.04552)[3] - erasing a small rectangle in the image\n\nHair - Randomly adding “hair like” lines to the image\n\nMetadata augmentation - adding random noise to the metadata as was done in the [1st place solution in ISIC 2019 challenge](https://www.sciencedirect.com/science/article/pii/S2215016120300832?via%3Dihub) [7].\n\n\n#### TTA\n\nFor inference each image was augmented differently 16 times and the final prediction was the average. These augmentations were also used for extracting 16 different features vectors per test image.\n\nThe same was done to extract 12 features vectors for the train images (12 and not 16 because of time limits).\n\n\n### Transformer Network\n\nThe input to the transformer network is the features from all the images from one patient.\n\nThe inspiration for this kind of model came from a previous competition in which we participated in RSNA[ Intracranial Hemorrhage Detection](https://www.kaggle.com/c/rsna-intracranial-hemorrhage-detection). In that competition, all the top solutions (including ours) used a two stage network approach (although none of them were transformers).\n\nUsing a transformer seems appropriate in this case because transformers are built to seek relationships between embedding (feature) vectors in it’s input.\n\nAs this is not a full seq2seq task, we only used encoder layers. The transformer is a stack of 4 encoder layers with 4 attention heads in each layer. (we also tested higher numbers of layers and attention heads - no performance improvement).\n\n\n#### metadata\n\nThe metadata was incorporated in the network by adding “metadata vectors” to the input vectors - each value was transformed to a vector size 256 and added. The discrete values’ transformation was done using a trainable embedding matrix and the continuous values using a small nn.\n\n\n#### output\n\nThe output of this network is a matrix of size N*C where N - the number of images, C - the number of classes. Which means it decides on all the images of the patient at once. \n\n\n#### Limit the input size\n\nA transformer can be trained on different number of feature vectors, by using padding. But when the range of numbers is very large, from a couple of hundred images for some patients to a handful for others, this may cause some implementation issues (like in calculating the loss). To simplify these issues, we limited N to 24 feature vectors, and for each patient we randomly divided the images to groups of size up to 24. \n\nThis might degrade the prediction as the most “similar” images might accidentally fall into different groups, but as we use TTA, this issue is almost solved. \n\n\n#### Augmentation\n\nFrom the base model we extract a number of feature vectors (12 for train and 16 for test) using different augmentation for the images and metadata. In the training and inference steps of the transformer model we randomly choose one of these vectors.\n\nAnother augmentation is the random grouping as stated above.\n\n\n## 3. Training and Inferencing\n\nThe original 2020 competition data is highly unbalanced, there are only 2-3% of positive targets in the train and test data. Although we were able to train the base model using uneven sampling, the best way to get good training was to add the data from ISIC 2019 competition which has a much higher percentage of melanoma images. \n\nWe split the training data to 3 folds keeping all the images from the same patient in the same fold, and making sure each fold has a similar number of patients with melanoma. The ISIC2019’s data was also split evenly between the folds. The same folds were kept for the base and the transformer models. \n\nTo get more diversity we had 3 different splits using 3 seeds \n\n\n### Preprocessing\n\nAll images were resized to an aspect ratio of 1:1.5, which was the most popular aspect ratio of the images in the original dataset. We prepared 3 image datasets of sizes `300*450`, `400*600`, `600*900`. Most of the models were trained using the `400*600` dataset, as 300*450 gave inferior results and the `600*900` didn’t improve the results enough.\n\nFor the metadata we had to set the same terminology for the 2020 and 2019 datasets.\n\n\n### Loss Function\n\nThe loss function we used was cross entropy. Although the task is to predict only melanoma we found it is better to predict the diagnosis which was split to 8 different classes one of which was melanoma. The final prediction was the value for the melanoma class, after a softmax function on all classes. We also tried a binary cross entropy on the melanoma class alone and a combination between the two, but using cross entropy gave the best results.\n\nThe same loss was used for the base model and the transformer, but in the transformer we needed to regularize for the different number of predictions in each batch resulting from the different number of images for each patient. \n\nWe also tried using focal loss which didn’t improve the results, but we left one of the transformer models which was trained with focal loss in the ensemble (A model with cross entropy loss gave the similar CV and LB).\n\n\n#### Training the transformer model \n\nThe transformer model was trained in two steps. For the first step we used the data from both competitions (2019, 2020). For the 2019 competition we don’t have information about the patient, and each image got a different dummy patient_id, meaning the transformer didn’t learn much from these images. In the 2nd stage we fine-tuned the transformer using only the 2020 competition’s data.\n\nIn both steps we used a sampler that over sampled the larger groups.\n\n\n#### Inference \n\nAs stated above, the inference was done using TTA. For the base model we used 12-16 different augmentations and for the transformer model 32.\n\n\n### Ensembling\n\nFor our final submissions we used 2 ensembles:\n\n\n##### Without Context Submission:\n\n\n\n1. EfficientNet B3 noisy student image size 400*600\n2. EfficientNet B4 noisy student image size 400*600\n3. EfficientNet B5 noisy student image size 400*600\n4. EfficientNet B6 noisy student image size **600*900**\n5. EfficientNet B7 noisy student image size 400*600\n\n\n##### With Context\n\nAll the “without context” model +\n\n\n\n* Transformer on features from A.\n* Transformer on features from B.\n* Transformer on features from C using focal loss\n* Transformer on features from D.\n* Transformer on features from E.\n\n\n## 4. Interesting findings\n\n\n### CV and LB\n\nAlthough the number of images in the competition was very large, the number of different patients wasn’t large enough to give a stable and reliable CV and LB. And the correlation between the two was low. At the end we trusted neither and we submitted the models we felt were the most robust.\n\n\n### What didn’t work\n\n\n\n1. Using different sizes of images as suggested in [CNN Input Size Explained](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/160147) didn’t show any improvement.\n2. Using Mixup[5] and MixCut[4] augmentation didn’t work\n3. Larger transformers didn’t improve the results. \n\n\n## 5. Better “Real World” model\n\nThis model can be much simpler if we won’t ensemble and take only one base model, the EfficientNet B5 is probably the best compromise.\n\nIn the real world scenario the images from previous years will probably be tagged already. In that case we can use a full transformer with encoder and decoder which will perform a seq2seq operation.", "votes": null }, { "id": "1005643", "postDate": "09/10/2020 16:12:45", "content": "Grats and big thanks for publishing!!!
\nGives me, as a beginner, tons of stuff to learn from!
Grats and big thanks for publishing!!!
\nGives me, as a beginner, tons of stuff to learn from!