![\"image\"](\"https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Jun-2022/hub1.png\")
Thank you Kaggle and HuBMAP for hosting a fun competition. The challenges in this competition were designing a train data loader to work with large images and writing a successful inference notebook to work with large images.
\nMy train data loader each epoch selects 128 random crops from each of the 15 train images. Therefore it balances the contribution from each train image (it does not give more attention to larger train images) since we don't know what the images in private test look like.
\nAdditionally, the train image is first reduced by either 2x, 3x, or 4x using a Numpy trick, image = image[::2, ::2, ], image = image[::3, ::3, ] etc. Lastly, the data loader guarantees that each crop of 1024x1024 contains at least 1 segmentation label. (If not, randomly select again).
Afterward, Albumentations augmentations are applied on the 1024x1024 crop
\ncomposition = albu.Compose([\n albu.HorizontalFlip(p=0.5),\n albu.VerticalFlip(p=0.5),\n albu.ShiftScaleRotate(rotate_limit=25, scale_limit=0.15, shift_limit=0, p=0.75),\n albu.CoarseDropout(max_holes=16, max_height=64 ,max_width=64 ,p=0.5),\n albu.ColorJitter(brightness=0.25, contrast=0.25, saturation=0.25, hue=0.25, p=0.75),\n albu.GridDistortion(num_steps=5, distort_limit=0.3, interpolation=1, p=0.5),\n albu.RandomCrop(height=768, width=768, p=1.0)\n ])\nAnd finally, the data loader extracts a 768x768 random crop from the augmented 1024x1024 random crop.
\n![\"image\"](\"https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Jun-2022/hub2.png\")
The model is FPN with backbone EfficientNetB2 and bce jaccard loss:
\nimport segmentation_models as sm\ndef build_model():\n inp = tf.keras.Input(shape=(None,None,3))\n base = sm.FPN('efficientnetb2', encoder_weights='imagenet', \n classes=1, activation='sigmoid')\n x = base(inp)\n\n opt = tf.keras.optimizers.Adam()\n model = tf.keras.Model(inputs=inp, outputs=x)\n model.compile(\n optimizer=opt,\n loss=sm.losses.bce_jaccard_loss,\n metrics=[sm.metrics.f1_score]\n )\n return model\nUsing four Nvidia V100 32GB, the model is trained 50 epochs with batch size 24, image size 768x768.
\nmodel.fit(train_gen(batch_size=24, image_size=(768,768)), epochs=50, \n callbacks=[lr], use_multiprocessing=True, workers=4)\nmodel.save_weights('model_weights.h5')\nEach epoch, the data loader randomly selects 128 crops from each of the train images. Training uses the following learning rate schedule. Training is a few minutes per epoch.
\n![\"image\"](\"https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Jun-2022/hub3.png\")
There are no folds. The model is trained with 100% train data. One model is trained with 2x reduce images, one model is trained with 3x reduce images, and one model with 4x reduce images. Afterward this is repeated using EfficientNetB0 backbone (image size 1280x1280 reduced to 1024x1024 and batch size 16). The 6 models are ensembled and achieve public LB 0.920 and private LB 0.947! (and an unseletected submission had 8th place Gold private LB 0.9487)
\nWe can check CV score by training 5 folds of 12 train images and inferring the other out-of-fold 3 images. When doing this, CV score averages 0.945 on 14 train images and gets a low 0.890 on train image c68fe75ea due to very light colored image in bottom right.
Thank you Kaggle and HuBMAP for hosting a fun competition. The challenges in this competition were designing a train data loader to work with large images and writing a successful inference notebook to work with large images.
\nMy train data loader each epoch selects 128 random crops from each of the 15 train images. Therefore it balances the contribution from each train image (it does not give more attention to larger train images) since we don't know what the images in private test look like.
\nAdditionally, the train image is first reduced by either 2x, 3x, or 4x using a Numpy trick, image = image[::2, ::2, ], image = image[::3, ::3, ] etc. Lastly, the data loader guarantees that each crop of 1024x1024 contains at least 1 segmentation label. (If not, randomly select again).
Afterward, Albumentations augmentations are applied on the 1024x1024 crop
\ncomposition = albu.Compose([\n albu.HorizontalFlip(p=0.5),\n albu.VerticalFlip(p=0.5),\n albu.ShiftScaleRotate(rotate_limit=25, scale_limit=0.15, shift_limit=0, p=0.75),\n albu.CoarseDropout(max_holes=16, max_height=64 ,max_width=64 ,p=0.5),\n albu.ColorJitter(brightness=0.25, contrast=0.25, saturation=0.25, hue=0.25, p=0.75),\n albu.GridDistortion(num_steps=5, distort_limit=0.3, interpolation=1, p=0.5),\n albu.RandomCrop(height=768, width=768, p=1.0)\n ])\nAnd finally, the data loader extracts a 768x768 random crop from the augmented 1024x1024 random crop.
\n
The model is FPN with backbone EfficientNetB2 and bce jaccard loss:
\nimport segmentation_models as sm\ndef build_model():\n inp = tf.keras.Input(shape=(None,None,3))\n base = sm.FPN('efficientnetb2', encoder_weights='imagenet', \n classes=1, activation='sigmoid')\n x = base(inp)\n\n opt = tf.keras.optimizers.Adam()\n model = tf.keras.Model(inputs=inp, outputs=x)\n model.compile(\n optimizer=opt,\n loss=sm.losses.bce_jaccard_loss,\n metrics=[sm.metrics.f1_score]\n )\n return model\nUsing four Nvidia V100 32GB, the model is trained 50 epochs with batch size 24, image size 768x768.
\nmodel.fit(train_gen(batch_size=24, image_size=(768,768)), epochs=50, \n callbacks=[lr], use_multiprocessing=True, workers=4)\nmodel.save_weights('model_weights.h5')\nEach epoch, the data loader randomly selects 128 crops from each of the train images. Training uses the following learning rate schedule. Training is a few minutes per epoch.
\n
There are no folds. The model is trained with 100% train data. One model is trained with 2x reduce images, one model is trained with 3x reduce images, and one model with 4x reduce images. Afterward this is repeated using EfficientNetB0 backbone (image size 1280x1280 reduced to 1024x1024 and batch size 16). The 6 models are ensembled and achieve public LB 0.920 and private LB 0.947! (and an unseletected submission had 8th place Gold private LB 0.9487)
\nWe can check CV score by training 5 folds of 12 train images and inferring the other out-of-fold 3 images. When doing this, CV score averages 0.945 on 14 train images and gets a low 0.890 on train image c68fe75ea due to very light colored image in bottom right.
@cdeotte which code snippet did you use for lr_schedule?
", "rawMarkdown": "cdeotte which code snippet did you use for **lr_schedule**?", "votes": null }, { "id": "1303044", "postDate": "05/11/2021 21:42:32", "content": "It's one that was posted in Flower Comp here then modified by me:
\nLR_START = 1e-5\nLR_MAX = 1e-3\nLR_RAMPUP_EPOCHS = 2\nLR_SUSTAIN_EPOCHS = 0\nLR_STEP_DECAY = 0.316\nLR_STEP_EPOCHS = 5\n\ndef lrfn(epoch):\n if epoch < LR_RAMPUP_EPOCHS:\n lr = (LR_MAX - LR_START) / LR_RAMPUP_EPOCHS * epoch + LR_START\n elif epoch < LR_RAMPUP_EPOCHS + LR_SUSTAIN_EPOCHS:\n lr = LR_MAX\n else:\n lr = LR_MAX * LR_STEP_DECAY**((epoch - LR_RAMPUP_EPOCHS - LR_SUSTAIN_EPOCHS)// LR_STEP_EPOCHS)\n return lr\n\nlr = tf.keras.callbacks.LearningRateScheduler(lrfn, verbose = True)\n\nplt.figure(figsize=(20,5))\nrng = [i for i in range(50)]\ny = [lrfn(x) for x in rng]\nplt.plot(rng, np.log10(y),'-o'); \nplt.xlabel('epoch',size=14); plt.ylabel('log10 learning rate',size=14)\nplt.title('Training Schedule',size=16); plt.show()\nThanks for sharing the details and congratulations on the position (even though you suffered some shakeup). I like the different illustrations, they make the process clear. I also envy the 4 x V100 GPUs. :D
\nIn the Ensemble section, you mention that you don't use folds. I guess the reason is time limitations?
", "rawMarkdown": "Thanks for sharing the details and congratulations on the position (even though you suffered some shakeup). I like the different illustrations, they make the process clear. I also envy the 4 x V100 GPUs. :D\n\nIn the Ensemble section, you mention that you don't use folds. I guess the reason is time limitations?", "votes": null }, { "id": "1304057", "postDate": "05/12/2021 11:54:44", "content": "@cdeotte we used a similar strategy for 'smart stochastic' sampling or 'positive sampling' (mask postive).
\nHowever I see different concerns with such an approach:
\nOur way to deal with this was to have a percentage of completely random tiles shown during training (10 %to 1%).
\nI think next step would be to do 'positive' and 'negative' sampling (for example showing hard false positives tiles more often).
Have you tried any other sampling strategy?
", "rawMarkdown": "cdeotte we used a similar strategy for 'smart stochastic' sampling or 'positive sampling' (mask postive). \n\nHowever I see different concerns with such an approach:\n- what if your model never learns what to do on full white or full black areas during training?\n- what if there exists similar patterns that you do not want to segment (like FC gloms for example). The model never see examples to differentiate them.\n\nOur way to deal with this was to have a percentage of completely random tiles shown during training (10 %to 1%).\nI think next step would be to do 'positive' and 'negative' sampling (for example showing hard false positives tiles more often).\n\nHave you tried any other sampling strategy?", "votes": null }, { "id": "1304306", "postDate": "05/12/2021 14:46:32", "content": "Those are great points and will most likely improve the data loader.
\nIn my case, i didn't try to improve my model. I spent my entire time hand labeling, pseudo labeling, and hill climbing ensembling to maximize public LB score and acquire the best public LB labels I could. When building ensembles, I downloaded every public notebook submission.csv and mixed those in using genetic algorithms to maximize public LB. I also randomly added and removed labeled gloms and found many annotator pattens and errors that were not discussed in the forums.
As a result, i had very accurate public LB labels that contained many patterns that were not present in the train data. I was hoping that private LB contained the same new patterns that public LB had. If that were the case then only a model trained with overfitted public LB labels would find them.
\nAs it turns out, the private LB consisted of only easy images similar to the easiest train images. I'm surprised that my untuned model did so well. One submission of my model without training on public test labels actually achieved 8th place Gold private LB 9487. In retrospect, i should have just improved my model CV and ignored acquiring public LB labels.
", "rawMarkdown": "Those are great points and will most likely improve the data loader.\n\nIn my case, i didn't try to improve my model. I spent my entire time hand labeling, pseudo labeling, and hill climbing ensembling to maximize public LB score and acquire the best public LB labels I could. When building ensembles, I downloaded every public notebook `submission.csv` and mixed those in using genetic algorithms to maximize public LB. I also randomly added and removed labeled gloms and found many annotator pattens and errors that were not discussed in the forums.\n\nAs a result, i had very accurate public LB labels that contained many patterns that were not present in the train data. I was hoping that private LB contained the same new patterns that public LB had. If that were the case then only a model trained with overfitted public LB labels would find them.\n\nAs it turns out, the private LB consisted of only easy images similar to the easiest train images. I'm surprised that my untuned model did so well. One submission of my model without training on public test labels actually achieved 8th place Gold private LB 9487. In retrospect, i should have just improved my model CV and ignored acquiring public LB labels.", "votes": null }, { "id": "1304336", "postDate": "05/12/2021 15:03:06", "content": "To further clarify, if the private test data contained one image like public test d488c759a it would not matter if our model had a good or bad CV. The winning team would be the team with the best labeled d488c759a image and used it to train with.
", "rawMarkdown": "To further clarify, if the private test data contained one image like public test d488c759a it **would not matter** if our model had a good or bad CV. The winning team would be the team with the best labeled d488c759a image and used it to train with.", "votes": null }, { "id": "1304349", "postDate": "05/12/2021 15:09:36", "content": "This is our best-performing sub that we didn't select. But I still can't seem to understand why this performed so well in the private dataset, I simply used 5 folds cross-validation w/o any post-processing ☹️.
\n![\"my-sbu\"](\"https://i.ibb.co/3pTh2Mt/my-sbu.png\")
",
"votes": null
},
{
"id": "1304406",
"postDate": "05/12/2021 15:51:12",
"content": "Did you use image size reduction?
", "rawMarkdown": "Did you use image size reduction?", "votes": null }, { "id": "1304416", "postDate": "05/12/2021 15:53:33", "content": "yes, 2x & 4x
", "rawMarkdown": "yes, **2x** & **4x**", "votes": null }, { "id": "1304619", "postDate": "05/12/2021 18:31:18", "content": "Wow, private score 0.9499, great model
", "rawMarkdown": "Wow, private score 0.9499, great model", "votes": null }, { "id": "1304740", "postDate": "05/12/2021 20:31:42", "content": "This stochastic dataloader is a great idea, but I wonder if you used rasterio to do this and how much performance penalty it incurs during training (in terms of training speed)? I also thought about doing something like this, but in my experience separating preprocessing and training as much as possible always resulted in much faster training and thus experimentation
", "rawMarkdown": "This stochastic dataloader is a great idea, but I wonder if you used rasterio to do this and how much performance penalty it incurs during training (in terms of training speed)? I also thought about doing something like this, but in my experience separating preprocessing and training as much as possible always resulted in much faster training and thus experimentation", "votes": null }, { "id": "1304764", "postDate": "05/12/2021 21:09:32", "content": "@awsaf49 It's really strange that many people have high private score on the model that has very low on public lb. My best private model was my second submission. Just a single fold on reduce 4 (256x256). Really strange indeed.
", "rawMarkdown": "awsaf49 It's really strange that many people have high private score on the model that has very low on public lb. My best private model was my second submission. Just a single fold on reduce 4 (256x256). Really strange indeed.", "votes": null }, { "id": "1304811", "postDate": "05/12/2021 22:41:28", "content": "Training was fast because i stored all the complete train images in memory, I did not read from disk during training which is slow.
\nAt full resolution, the 15 train images combined is 41.6GB. At 2x scale reduction, all 15 images combined is 10.4GB. At 3x scale reduction, all 15 images are 4.6GB. At 4x reduction, all 15 train images are 2.6GB.
", "rawMarkdown": "Training was fast because i stored all the complete train images in memory, I did not read from disk during training which is slow. \n\nAt full resolution, the 15 train images combined is **41.6GB**. At 2x scale reduction, all 15 images combined is **10.4GB**. At 3x scale reduction, all 15 images are **4.6GB**. At 4x reduction, all 15 train images are **2.6GB**.", "votes": null }, { "id": "1304848", "postDate": "05/13/2021 00:08:28", "content": "I see. That makes a lot of sense! I guess I overestimated how much memory this type of method would require
", "rawMarkdown": "I see. That makes a lot of sense! I guess I overestimated how much memory this type of method would require", "votes": null }, { "id": "1307890", "postDate": "05/14/2021 18:25:26", "content": "\n\nIn the Ensemble section, you mention that you don't use folds.
\n
I joined the competition late and my plan was to find the best masks for the infamous d488c759a image. So once i got my model running and it outperformed public notebooks, i didn't spend time computing CV nor optimizing my model anymore.
\nInstead, i iIteratively trained my model on hand label and pseudo labels. Each time it predicted better masks for d488c759a. I also used genetic algorithms to ensemble my model with public submission.csv files and made random changes to the mask. Eventually, i got my public LB up to 0.944.
For my final submission, i added the 5 test images with their 0.944 public test masks to my balanced stochastic data loader. My final submission had public LB 0.9431 and private LB 0.9476. If private test contained a strange image like d488c759a, then my private test score would still be high and other teams would be lower.
Thank you for always posting good solutions in every competition! I'm curious about creating a data loader randomly with Stochastic Sampling, can I see how you wrote the code?
\nCongratulations on 23rd place!!
", "rawMarkdown": "Thank you for always posting good solutions in every competition! I'm curious about creating a data loader randomly with Stochastic Sampling, can I see how you wrote the code?\n\nCongratulations on 23rd place!!", "votes": null }, { "id": "1326556", "postDate": "05/28/2021 14:39:50", "content": "Thank you. First i read all the train images and masks into memory
\ntrain_images = []\ntrain_masks = []\ntrain_sizes = []\n\nfor k in range(15):\n\n name = train.iloc[k,0]\n print(name,', ',end='')\n\n img = np.squeeze( tiff.imread('/raid/Kaggle/hubmap/train/'+name+'.tiff') )\n if img.shape[0]==3: img = np.transpose(img,[1,2,0])\n train_images.append(img)\n train_sizes.append(img.shape[:2])\n\n rle = train.iloc[k,1]\n mask = rle2mask(rle, shape=(img.shape[1],img.shape[0]))\n train_masks.append(mask)`\nNext, here is my TF Keras dataloader
\nIMG_SIZE = 768\nIMG_SIZE2 = 1024\nSHRINK_SIZE = 3\n\nclass DataGenerator(tf.keras.utils.Sequence):\n 'Generates data for Keras'\n def __init__(self, imgs, masks, batch_size=24, shuffle=False, augment=False,\n crops=128, size=IMG_SIZE, size2=IMG_SIZE2, shrink=SHRINK_SIZE): \n\n self.imgs = imgs\n self.masks = masks\n self.batch_size = batch_size\n self.crops = crops\n self.size = size\n self.size2 = size2\n self.shrink = shrink\n self.shuffle = shuffle\n self.augment = augment\n self.on_epoch_end()\n\n def __len__(self):\n 'Denotes the number of batches per epoch'\n ct = int( np.ceil(self.crops * len( self.imgs ) / self.batch_size ) )\n return ct\n\n def __getitem__(self, index):\n 'Generate one batch of data'\n indexes = self.indexes[index*self.batch_size:(index+1)*self.batch_size]\n X, y = self.__data_generation(indexes)\n if self.augment: \n X2 = np.zeros((len(indexes),self.size,self.size,3),dtype='float32')\n y2 = np.zeros((len(indexes),self.size,self.size,1),dtype='float32')\n X2, y2 = self.__augment_batch(X, y)\n else:\n X2 = X\n y2 = y\n\n return X2, y2\n\n def on_epoch_end(self):\n 'Updates indexes after each epoch'\n self.indexes = np.arange( self.crops * len( self.imgs ) )\n if self.shuffle: np.random.shuffle(self.indexes)\n\n def __data_generation(self, indexes):\n 'Generates data containing batch_size samples' \n\n X = np.zeros((len(indexes),self.size2,self.size2,3),dtype='float32')\n y = np.zeros((len(indexes),self.size2,self.size2,1),dtype='float32')\n\n for k in range(len(indexes)):\n i = np.random.randint(0,len(self.imgs))\n img = self.imgs[i]\n mask = self.masks[i]\n\n sm = 0; ct = 0\n while (sm==0)&(ct<25):\n a = np.random.randint(0,img.shape[0]-self.size2*self.shrink)\n b = np.random.randint(0,img.shape[1]-self.size2*self.shrink)\n sm = np.sum(mask[a:a+self.size*self.shrink,b:b+self.size*self.shrink])\n ct += 1\n\n X[k,] = img[a:a+self.size2*self.shrink,b:b+self.size2*self.shrink,][::self.shrink,::self.shrink,]/255.\n y[k,:,:,0] = mask[a:a+self.size2*self.shrink,b:b+self.size2*self.shrink][::self.shrink,::self.shrink]\n\n return X,y\n\n def __random_transform(self, img, msk):\n composition = albu.Compose([\n albu.HorizontalFlip(p=0.5),\n albu.VerticalFlip(p=0.5),\n albu.ShiftScaleRotate(rotate_limit=25, scale_limit=0.15, shift_limit=0, p=0.75),\n albu.CoarseDropout(max_holes=16, max_height=64 ,max_width=64 ,p=0.5),\n albu.ColorJitter(brightness=0.25, contrast=0.25, saturation=0.25, hue=0.25, p=0.75),\n albu.GridDistortion(num_steps=5, distort_limit=0.3, interpolation=1, p=0.5),\n albu.RandomCrop(height=self.size, width=self.size, p=1.0)\n ])\n tmp = composition(image=img, mask=msk)\n return tmp['image'], tmp['mask']\n\n def __augment_batch(self, img_batch, mask_batch):\n img_batch2 = np.zeros((len(img_batch),self.size,self.size,3))\n mask_batch2 = np.zeros((len(mask_batch),self.size,self.size,1))\n for i in range(img_batch.shape[0]):\n img_batch2[i, ],mask_batch2[i, ] = self.__random_transform(img_batch[i, ],mask_batch[i, ])\n return img_batch2,mask_batch2\nThank you for your kind reply always!!
", "rawMarkdown": "Thank you for your kind reply always!!", "votes": null } ], "comments": [ { "id": 1303042, "author_name": "awsaf49", "author_url": "", "post_date": "05/11/2021 21:40:28", "content": "@cdeotte which code snippet did you use for lr_schedule?
", "votes": null, "replies": [ { "id": 1303044, "author_name": "cdeotte", "author_url": "", "post_date": "05/11/2021 21:42:32", "content": "It's one that was posted in Flower Comp here then modified by me:
\nLR_START = 1e-5\nLR_MAX = 1e-3\nLR_RAMPUP_EPOCHS = 2\nLR_SUSTAIN_EPOCHS = 0\nLR_STEP_DECAY = 0.316\nLR_STEP_EPOCHS = 5\n\ndef lrfn(epoch):\n if epoch < LR_RAMPUP_EPOCHS:\n lr = (LR_MAX - LR_START) / LR_RAMPUP_EPOCHS * epoch + LR_START\n elif epoch < LR_RAMPUP_EPOCHS + LR_SUSTAIN_EPOCHS:\n lr = LR_MAX\n else:\n lr = LR_MAX * LR_STEP_DECAY**((epoch - LR_RAMPUP_EPOCHS - LR_SUSTAIN_EPOCHS)// LR_STEP_EPOCHS)\n return lr\n\nlr = tf.keras.callbacks.LearningRateScheduler(lrfn, verbose = True)\n\nplt.figure(figsize=(20,5))\nrng = [i for i in range(50)]\ny = [lrfn(x) for x in rng]\nplt.plot(rng, np.log10(y),'-o'); \nplt.xlabel('epoch',size=14); plt.ylabel('log10 learning rate',size=14)\nplt.title('Training Schedule',size=16); plt.show()\nThanks for sharing the details and congratulations on the position (even though you suffered some shakeup). I like the different illustrations, they make the process clear. I also envy the 4 x V100 GPUs. :D
\nIn the Ensemble section, you mention that you don't use folds. I guess the reason is time limitations?
", "votes": null, "replies": [ { "id": 1307890, "author_name": "cdeotte", "author_url": "", "post_date": "05/14/2021 18:25:26", "content": "\n\nIn the Ensemble section, you mention that you don't use folds.
\n
I joined the competition late and my plan was to find the best masks for the infamous d488c759a image. So once i got my model running and it outperformed public notebooks, i didn't spend time computing CV nor optimizing my model anymore.
\nInstead, i iIteratively trained my model on hand label and pseudo labels. Each time it predicted better masks for d488c759a. I also used genetic algorithms to ensemble my model with public submission.csv files and made random changes to the mask. Eventually, i got my public LB up to 0.944.
For my final submission, i added the 5 test images with their 0.944 public test masks to my balanced stochastic data loader. My final submission had public LB 0.9431 and private LB 0.9476. If private test contained a strange image like d488c759a, then my private test score would still be high and other teams would be lower.
@cdeotte we used a similar strategy for 'smart stochastic' sampling or 'positive sampling' (mask postive).
\nHowever I see different concerns with such an approach:
\nOur way to deal with this was to have a percentage of completely random tiles shown during training (10 %to 1%).
\nI think next step would be to do 'positive' and 'negative' sampling (for example showing hard false positives tiles more often).
Have you tried any other sampling strategy?
", "votes": null, "replies": [ { "id": 1304306, "author_name": "cdeotte", "author_url": "", "post_date": "05/12/2021 14:46:32", "content": "Those are great points and will most likely improve the data loader.
\nIn my case, i didn't try to improve my model. I spent my entire time hand labeling, pseudo labeling, and hill climbing ensembling to maximize public LB score and acquire the best public LB labels I could. When building ensembles, I downloaded every public notebook submission.csv and mixed those in using genetic algorithms to maximize public LB. I also randomly added and removed labeled gloms and found many annotator pattens and errors that were not discussed in the forums.
As a result, i had very accurate public LB labels that contained many patterns that were not present in the train data. I was hoping that private LB contained the same new patterns that public LB had. If that were the case then only a model trained with overfitted public LB labels would find them.
\nAs it turns out, the private LB consisted of only easy images similar to the easiest train images. I'm surprised that my untuned model did so well. One submission of my model without training on public test labels actually achieved 8th place Gold private LB 9487. In retrospect, i should have just improved my model CV and ignored acquiring public LB labels.
", "votes": null, "replies": [] }, { "id": 1304336, "author_name": "cdeotte", "author_url": "", "post_date": "05/12/2021 15:03:06", "content": "To further clarify, if the private test data contained one image like public test d488c759a it would not matter if our model had a good or bad CV. The winning team would be the team with the best labeled d488c759a image and used it to train with.
", "votes": null, "replies": [] } ] }, { "id": 1304349, "author_name": "awsaf49", "author_url": "", "post_date": "05/12/2021 15:09:36", "content": "This is our best-performing sub that we didn't select. But I still can't seem to understand why this performed so well in the private dataset, I simply used 5 folds cross-validation w/o any post-processing ☹️.
\n
Did you use image size reduction?
", "votes": null, "replies": [] }, { "id": 1304416, "author_name": "awsaf49", "author_url": "", "post_date": "05/12/2021 15:53:33", "content": "yes, 2x & 4x
", "votes": null, "replies": [] }, { "id": 1304619, "author_name": "cdeotte", "author_url": "", "post_date": "05/12/2021 18:31:18", "content": "Wow, private score 0.9499, great model
", "votes": null, "replies": [] }, { "id": 1304764, "author_name": "tom88jerry", "author_url": "", "post_date": "05/12/2021 21:09:32", "content": "@awsaf49 It's really strange that many people have high private score on the model that has very low on public lb. My best private model was my second submission. Just a single fold on reduce 4 (256x256). Really strange indeed.
", "votes": null, "replies": [] } ] }, { "id": 1304740, "author_name": "shujun717", "author_url": "", "post_date": "05/12/2021 20:31:42", "content": "This stochastic dataloader is a great idea, but I wonder if you used rasterio to do this and how much performance penalty it incurs during training (in terms of training speed)? I also thought about doing something like this, but in my experience separating preprocessing and training as much as possible always resulted in much faster training and thus experimentation
", "votes": null, "replies": [ { "id": 1304811, "author_name": "cdeotte", "author_url": "", "post_date": "05/12/2021 22:41:28", "content": "Training was fast because i stored all the complete train images in memory, I did not read from disk during training which is slow.
\nAt full resolution, the 15 train images combined is 41.6GB. At 2x scale reduction, all 15 images combined is 10.4GB. At 3x scale reduction, all 15 images are 4.6GB. At 4x reduction, all 15 train images are 2.6GB.
", "votes": null, "replies": [] }, { "id": 1304848, "author_name": "shujun717", "author_url": "", "post_date": "05/13/2021 00:08:28", "content": "I see. That makes a lot of sense! I guess I overestimated how much memory this type of method would require
", "votes": null, "replies": [] } ] }, { "id": 1326444, "author_name": "chocozzz", "author_url": "", "post_date": "05/28/2021 13:57:19", "content": "Thank you for always posting good solutions in every competition! I'm curious about creating a data loader randomly with Stochastic Sampling, can I see how you wrote the code?
\nCongratulations on 23rd place!!
", "votes": null, "replies": [ { "id": 1326556, "author_name": "cdeotte", "author_url": "", "post_date": "05/28/2021 14:39:50", "content": "Thank you. First i read all the train images and masks into memory
\ntrain_images = []\ntrain_masks = []\ntrain_sizes = []\n\nfor k in range(15):\n\n name = train.iloc[k,0]\n print(name,', ',end='')\n\n img = np.squeeze( tiff.imread('/raid/Kaggle/hubmap/train/'+name+'.tiff') )\n if img.shape[0]==3: img = np.transpose(img,[1,2,0])\n train_images.append(img)\n train_sizes.append(img.shape[:2])\n\n rle = train.iloc[k,1]\n mask = rle2mask(rle, shape=(img.shape[1],img.shape[0]))\n train_masks.append(mask)`\nNext, here is my TF Keras dataloader
\nIMG_SIZE = 768\nIMG_SIZE2 = 1024\nSHRINK_SIZE = 3\n\nclass DataGenerator(tf.keras.utils.Sequence):\n 'Generates data for Keras'\n def __init__(self, imgs, masks, batch_size=24, shuffle=False, augment=False,\n crops=128, size=IMG_SIZE, size2=IMG_SIZE2, shrink=SHRINK_SIZE): \n\n self.imgs = imgs\n self.masks = masks\n self.batch_size = batch_size\n self.crops = crops\n self.size = size\n self.size2 = size2\n self.shrink = shrink\n self.shuffle = shuffle\n self.augment = augment\n self.on_epoch_end()\n\n def __len__(self):\n 'Denotes the number of batches per epoch'\n ct = int( np.ceil(self.crops * len( self.imgs ) / self.batch_size ) )\n return ct\n\n def __getitem__(self, index):\n 'Generate one batch of data'\n indexes = self.indexes[index*self.batch_size:(index+1)*self.batch_size]\n X, y = self.__data_generation(indexes)\n if self.augment: \n X2 = np.zeros((len(indexes),self.size,self.size,3),dtype='float32')\n y2 = np.zeros((len(indexes),self.size,self.size,1),dtype='float32')\n X2, y2 = self.__augment_batch(X, y)\n else:\n X2 = X\n y2 = y\n\n return X2, y2\n\n def on_epoch_end(self):\n 'Updates indexes after each epoch'\n self.indexes = np.arange( self.crops * len( self.imgs ) )\n if self.shuffle: np.random.shuffle(self.indexes)\n\n def __data_generation(self, indexes):\n 'Generates data containing batch_size samples' \n\n X = np.zeros((len(indexes),self.size2,self.size2,3),dtype='float32')\n y = np.zeros((len(indexes),self.size2,self.size2,1),dtype='float32')\n\n for k in range(len(indexes)):\n i = np.random.randint(0,len(self.imgs))\n img = self.imgs[i]\n mask = self.masks[i]\n\n sm = 0; ct = 0\n while (sm==0)&(ct<25):\n a = np.random.randint(0,img.shape[0]-self.size2*self.shrink)\n b = np.random.randint(0,img.shape[1]-self.size2*self.shrink)\n sm = np.sum(mask[a:a+self.size*self.shrink,b:b+self.size*self.shrink])\n ct += 1\n\n X[k,] = img[a:a+self.size2*self.shrink,b:b+self.size2*self.shrink,][::self.shrink,::self.shrink,]/255.\n y[k,:,:,0] = mask[a:a+self.size2*self.shrink,b:b+self.size2*self.shrink][::self.shrink,::self.shrink]\n\n return X,y\n\n def __random_transform(self, img, msk):\n composition = albu.Compose([\n albu.HorizontalFlip(p=0.5),\n albu.VerticalFlip(p=0.5),\n albu.ShiftScaleRotate(rotate_limit=25, scale_limit=0.15, shift_limit=0, p=0.75),\n albu.CoarseDropout(max_holes=16, max_height=64 ,max_width=64 ,p=0.5),\n albu.ColorJitter(brightness=0.25, contrast=0.25, saturation=0.25, hue=0.25, p=0.75),\n albu.GridDistortion(num_steps=5, distort_limit=0.3, interpolation=1, p=0.5),\n albu.RandomCrop(height=self.size, width=self.size, p=1.0)\n ])\n tmp = composition(image=img, mask=msk)\n return tmp['image'], tmp['mask']\n\n def __augment_batch(self, img_batch, mask_batch):\n img_batch2 = np.zeros((len(img_batch),self.size,self.size,3))\n mask_batch2 = np.zeros((len(mask_batch),self.size,self.size,1))\n for i in range(img_batch.shape[0]):\n img_batch2[i, ],mask_batch2[i, ] = self.__random_transform(img_batch[i, ],mask_batch[i, ])\n return img_batch2,mask_batch2\nThank you for your kind reply always!!
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1303029": "Thank you Kaggle and HuBMAP for hosting a fun competition. The challenges in this competition were designing a train data loader to work with large images and writing a successful inference notebook to work with large images.\n\n# Train Data Loader\nMy train data loader each epoch selects 128 **random** crops from each of the 15 train images. Therefore it **balances** the contribution from each train image (it does not give more attention to larger train images) since we don't know what the images in private test look like.\n\nAdditionally, the train image is first reduced by either 2x, 3x, or 4x using a Numpy trick, `image = image[::2, ::2, ]`, `image = image[::3, ::3, ]` etc. Lastly, the data loader guarantees that each crop of 1024x1024 contains **at least 1 segmentation label**. (If not, randomly select again).\n\n\n\nAfterward, Albumentations augmentations are applied on the 1024x1024 crop\n\n composition = albu.Compose([\n albu.HorizontalFlip(p=0.5),\n albu.VerticalFlip(p=0.5),\n albu.ShiftScaleRotate(rotate_limit=25, scale_limit=0.15, shift_limit=0, p=0.75),\n albu.CoarseDropout(max_holes=16, max_height=64 ,max_width=64 ,p=0.5),\n albu.ColorJitter(brightness=0.25, contrast=0.25, saturation=0.25, hue=0.25, p=0.75),\n albu.GridDistortion(num_steps=5, distort_limit=0.3, interpolation=1, p=0.5),\n albu.RandomCrop(height=768, width=768, p=1.0)\n ])\n\nAnd finally, the data loader extracts a 768x768 random crop from the augmented 1024x1024 random crop.\n\n\n\nThe model is FPN with backbone EfficientNetB2 and bce jaccard loss:\n\n import segmentation_models as sm\n def build_model():\n inp = tf.keras.Input(shape=(None,None,3))\n base = sm.FPN('efficientnetb2', encoder_weights='imagenet', \n classes=1, activation='sigmoid')\n x = base(inp)\n\n opt = tf.keras.optimizers.Adam()\n model = tf.keras.Model(inputs=inp, outputs=x)\n model.compile(\n optimizer=opt,\n loss=sm.losses.bce_jaccard_loss,\n metrics=[sm.metrics.f1_score]\n )\n return model\n\n# Nvidia 4xV100 32GB\nUsing four Nvidia V100 32GB, the model is trained 50 epochs with batch size 24, image size 768x768. \n\n model.fit(train_gen(batch_size=24, image_size=(768,768)), epochs=50, \n callbacks=[lr], use_multiprocessing=True, workers=4)\n model.save_weights('model_weights.h5')\n\nEach epoch, the data loader randomly selects 128 crops from each of the train images. Training uses the following learning rate schedule. Training is a few minutes per epoch.\n\n\n\n# Ensemble\nThere are no folds. The model is trained with 100% train data. One model is trained with 2x reduce images, one model is trained with 3x reduce images, and one model with 4x reduce images. Afterward this is repeated using EfficientNetB0 backbone (image size 1280x1280 reduced to 1024x1024 and batch size 16). The 6 models are ensembled and achieve **public LB 0.920 and private LB 0.947**! (and an unseletected submission had **8th place Gold private** LB 0.9487)\n\nWe can check CV score by training 5 folds of 12 train images and inferring the other out-of-fold 3 images. When doing this, **CV score averages 0.945** on 14 train images and gets a low 0.890 on train image `c68fe75ea` due to very light colored image in bottom right.", "1303042": "cdeotte which code snippet did you use for **lr_schedule**?", "1303044": "It's one that was posted in Flower Comp [here][1] then modified by me:\n\n LR_START = 1e-5\n LR_MAX = 1e-3\n LR_RAMPUP_EPOCHS = 2\n LR_SUSTAIN_EPOCHS = 0\n LR_STEP_DECAY = 0.316\n LR_STEP_EPOCHS = 5\n\n def lrfn(epoch):\n if epoch < LR_RAMPUP_EPOCHS:\n lr = (LR_MAX - LR_START) / LR_RAMPUP_EPOCHS * epoch + LR_START\n elif epoch < LR_RAMPUP_EPOCHS + LR_SUSTAIN_EPOCHS:\n lr = LR_MAX\n else:\n lr = LR_MAX * LR_STEP_DECAY**((epoch - LR_RAMPUP_EPOCHS - LR_SUSTAIN_EPOCHS)// LR_STEP_EPOCHS)\n return lr\n \n lr = tf.keras.callbacks.LearningRateScheduler(lrfn, verbose = True)\n\n plt.figure(figsize=(20,5))\n rng = [i for i in range(50)]\n y = [lrfn(x) for x in rng]\n plt.plot(rng, np.log10(y),'-o'); \n plt.xlabel('epoch',size=14); plt.ylabel('log10 learning rate',size=14)\n plt.title('Training Schedule',size=16); plt.show()\n\n[1]: https://www.kaggle.com/mgornergoogle/custom-training-loop-with-100-flowers-on-tpu", "1303847": "Thanks for sharing the details and congratulations on the position (even though you suffered some shakeup). I like the different illustrations, they make the process clear. I also envy the 4 x V100 GPUs. :D\n\nIn the Ensemble section, you mention that you don't use folds. I guess the reason is time limitations?", "1304057": "cdeotte we used a similar strategy for 'smart stochastic' sampling or 'positive sampling' (mask postive). \n\nHowever I see different concerns with such an approach:\n- what if your model never learns what to do on full white or full black areas during training?\n- what if there exists similar patterns that you do not want to segment (like FC gloms for example). The model never see examples to differentiate them.\n\nOur way to deal with this was to have a percentage of completely random tiles shown during training (10 %to 1%).\nI think next step would be to do 'positive' and 'negative' sampling (for example showing hard false positives tiles more often).\n\nHave you tried any other sampling strategy?", "1304306": "Those are great points and will most likely improve the data loader.\n\nIn my case, i didn't try to improve my model. I spent my entire time hand labeling, pseudo labeling, and hill climbing ensembling to maximize public LB score and acquire the best public LB labels I could. When building ensembles, I downloaded every public notebook `submission.csv` and mixed those in using genetic algorithms to maximize public LB. I also randomly added and removed labeled gloms and found many annotator pattens and errors that were not discussed in the forums.\n\nAs a result, i had very accurate public LB labels that contained many patterns that were not present in the train data. I was hoping that private LB contained the same new patterns that public LB had. If that were the case then only a model trained with overfitted public LB labels would find them.\n\nAs it turns out, the private LB consisted of only easy images similar to the easiest train images. I'm surprised that my untuned model did so well. One submission of my model without training on public test labels actually achieved 8th place Gold private LB 9487. In retrospect, i should have just improved my model CV and ignored acquiring public LB labels.", "1304336": "To further clarify, if the private test data contained one image like public test d488c759a it **would not matter** if our model had a good or bad CV. The winning team would be the team with the best labeled d488c759a image and used it to train with.", "1304349": "This is our best-performing sub that we didn't select. But I still can't seem to understand why this performed so well in the private dataset, I simply used **5 folds** cross-validation w/o any **post-processing** ☹️.\n",
"1304406": "Did you use image size reduction?",
"1304416": "yes, **2x** & **4x**",
"1304619": "Wow, private score 0.9499, great model",
"1304740": "This stochastic dataloader is a great idea, but I wonder if you used rasterio to do this and how much performance penalty it incurs during training (in terms of training speed)? I also thought about doing something like this, but in my experience separating preprocessing and training as much as possible always resulted in much faster training and thus experimentation",
"1304764": "awsaf49 It's really strange that many people have high private score on the model that has very low on public lb. My best private model was my second submission. Just a single fold on reduce 4 (256x256). Really strange indeed.",
"1304811": "Training was fast because i stored all the complete train images in memory, I did not read from disk during training which is slow. \n\nAt full resolution, the 15 train images combined is **41.6GB**. At 2x scale reduction, all 15 images combined is **10.4GB**. At 3x scale reduction, all 15 images are **4.6GB**. At 4x reduction, all 15 train images are **2.6GB**.",
"1304848": "I see. That makes a lot of sense! I guess I overestimated how much memory this type of method would require",
"1307890": "> In the Ensemble section, you mention that you don't use folds.\n\nI joined the competition late and my plan was to find the best masks for the infamous d488c759a image. So once i got my model running and it outperformed public notebooks, i didn't spend time computing CV nor optimizing my model anymore.\n\nInstead, i iIteratively trained my model on hand label and pseudo labels. Each time it predicted better masks for d488c759a. I also used genetic algorithms to ensemble my model with public `submission.csv` files and made random changes to the mask. Eventually, i got my public LB up to 0.944.\n\nFor my final submission, i added the 5 test images with their 0.944 public test masks to my `balanced stochastic data loader`. My final submission had public LB 0.9431 and private LB 0.9476. If private test contained a strange image like d488c759a, then my private test score would still be high and other teams would be lower.",
"1326444": "Thank you for always posting good solutions in every competition! I'm curious about creating a data loader randomly with Stochastic Sampling, can I see how you wrote the code?\n\nCongratulations on 23rd place!!",
"1326556": "Thank you. First i read all the train images and masks into memory\n\n train_images = []\n train_masks = []\n train_sizes = []\n\n for k in range(15):\n \n name = train.iloc[k,0]\n print(name,', ',end='')\n \n img = np.squeeze( tiff.imread('/raid/Kaggle/hubmap/train/'+name+'.tiff') )\n if img.shape[0]==3: img = np.transpose(img,[1,2,0])\n train_images.append(img)\n train_sizes.append(img.shape[:2])\n \n rle = train.iloc[k,1]\n mask = rle2mask(rle, shape=(img.shape[1],img.shape[0]))\n train_masks.append(mask)`",
"1326561": "Next, here is my TF Keras dataloader\n\n IMG_SIZE = 768\n IMG_SIZE2 = 1024\n SHRINK_SIZE = 3\n\n class DataGenerator(tf.keras.utils.Sequence):\n 'Generates data for Keras'\n def __init__(self, imgs, masks, batch_size=24, shuffle=False, augment=False,\n crops=128, size=IMG_SIZE, size2=IMG_SIZE2, shrink=SHRINK_SIZE): \n\n self.imgs = imgs\n self.masks = masks\n self.batch_size = batch_size\n self.crops = crops\n self.size = size\n self.size2 = size2\n self.shrink = shrink\n self.shuffle = shuffle\n self.augment = augment\n self.on_epoch_end()\n \n def __len__(self):\n 'Denotes the number of batches per epoch'\n ct = int( np.ceil(self.crops * len( self.imgs ) / self.batch_size ) )\n return ct\n\n def __getitem__(self, index):\n 'Generate one batch of data'\n indexes = self.indexes[index*self.batch_size:(index+1)*self.batch_size]\n X, y = self.__data_generation(indexes)\n if self.augment: \n X2 = np.zeros((len(indexes),self.size,self.size,3),dtype='float32')\n y2 = np.zeros((len(indexes),self.size,self.size,1),dtype='float32')\n X2, y2 = self.__augment_batch(X, y)\n else:\n X2 = X\n y2 = y\n \n return X2, y2\n\n def on_epoch_end(self):\n 'Updates indexes after each epoch'\n self.indexes = np.arange( self.crops * len( self.imgs ) )\n if self.shuffle: np.random.shuffle(self.indexes)\n \n def __data_generation(self, indexes):\n 'Generates data containing batch_size samples' \n \n X = np.zeros((len(indexes),self.size2,self.size2,3),dtype='float32')\n y = np.zeros((len(indexes),self.size2,self.size2,1),dtype='float32')\n \n for k in range(len(indexes)):\n i = np.random.randint(0,len(self.imgs))\n img = self.imgs[i]\n mask = self.masks[i]\n \n sm = 0; ct = 0\n while (sm==0)&(ct<25):\n a = np.random.randint(0,img.shape[0]-self.size2*self.shrink)\n b = np.random.randint(0,img.shape[1]-self.size2*self.shrink)\n sm = np.sum(mask[a:a+self.size*self.shrink,b:b+self.size*self.shrink])\n ct += 1\n \n X[k,] = img[a:a+self.size2*self.shrink,b:b+self.size2*self.shrink,][::self.shrink,::self.shrink,]/255.\n y[k,:,:,0] = mask[a:a+self.size2*self.shrink,b:b+self.size2*self.shrink][::self.shrink,::self.shrink]\n \n return X,y\n \n def __random_transform(self, img, msk):\n composition = albu.Compose([\n albu.HorizontalFlip(p=0.5),\n albu.VerticalFlip(p=0.5),\n albu.ShiftScaleRotate(rotate_limit=25, scale_limit=0.15, shift_limit=0, p=0.75),\n albu.CoarseDropout(max_holes=16, max_height=64 ,max_width=64 ,p=0.5),\n albu.ColorJitter(brightness=0.25, contrast=0.25, saturation=0.25, hue=0.25, p=0.75),\n albu.GridDistortion(num_steps=5, distort_limit=0.3, interpolation=1, p=0.5),\n albu.RandomCrop(height=self.size, width=self.size, p=1.0)\n ])\n tmp = composition(image=img, mask=msk)\n return tmp['image'], tmp['mask']\n \n def __augment_batch(self, img_batch, mask_batch):\n img_batch2 = np.zeros((len(img_batch),self.size,self.size,3))\n mask_batch2 = np.zeros((len(mask_batch),self.size,self.size,1))\n for i in range(img_batch.shape[0]):\n img_batch2[i, ],mask_batch2[i, ] = self.__random_transform(img_batch[i, ],mask_batch[i, ])\n return img_batch2,mask_batch2",
"1326591": "Thank you for your kind reply always!!"
},
"source": "meta"
}