![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1014468%2F5b0da37195853fc9dd5822eba9603843%2FScreen%20Shot%202023-06-17%20at%2012.09.12.png?generation=1687018176238566&alt=media\")
This was our first competition as a team: @kristoferschobert, @awgross, @mjulianimsj75, and myself. We're very happy with the 49th place finish. We even managed to shake up a little from the 65th position on public.
\nTL;DR: Large models and validated ensembles built upon the popular 2.5d baseline
\nWe spent a significant amount of time building our own flexible fastai pipeline from scratch, but we weren't able to achieve the same scores as with our variations on @tanakar's public notebook that we were experimenting with on the side. We used fastai's unet_learner, extended to use timm image models as backbone. We had custom callbacks and a streamlined trained & inference structure already prepared for ensembling. We used a custom dataloader that held the relevant layers of the full fragments in memory and was indexing into them dynamically at training & inference time. Not sure whether it's more efficient than the 2.5d baseline dataloader, but it felt elegant ;-)
What those fastai experiments gave us was a scan through all the (relevant) layers/slices of each fragment; showing where most of the signal was located. We used 3 slices per experiment, and stepped through successive starting minimum slices. E.g. the experiment for min slice = 28 contained the slices 28, 29, 30. The results are below, and suggested that the top and bottom slices were lacking in detectable ink, which was primarily found in the center slices. There was a gradual increase and decline in signal, and the locations of these ingress & egress were slightly different for each fragment. Using 3 slices per experiment gave us slightly reduced resolution in the z-axis, but made those validation scores somewhat more robust:
With an appropriate 2.5 weeks left to go, we decided to pivot fully to using the 2.5d public baseline as the core of our experimentation setup. Here are some things we did slightly differently from other solution write-ups:
\nWe settled on using 8 slices/channels starting from slice 28, partly based on the above depth scan.
We focused on large models, most prominently eca_nfnet_l2 and maxvit_large models from timm with image sizes 224, 384, and 512 (@awgross).
In addition to the segmentation-models-pytorch Unet architecture, we found Linknet to perform even slightly better (@mjulianimsj75).
We validated on fragments 1 & 3, which gave us reliable correlations with the public LB (@kristoferschobert).
We validated our ensemble selections using the out-of-fold scores for fragments 1 & 3; tuning percentile thresholds simultaneously. Our best thresholds were usually 94%. Our local ensemble scores were also pretty well correlated with the public & private LB.
It was our best ensemble of 6 maxvit models (without the contribution of eca_nfnet or other backbones), which scored consistently highest on local validation, public LB, and private LB; and allowed us to pick our best submission to sneak into the top 50.
This was our first competition as a team: @kristoferschobert, @awgross, @mjulianimsj75, and myself. We're very happy with the 49th place finish. We even managed to shake up a little from the 65th position on public.
\nTL;DR: Large models and validated ensembles built upon the popular 2.5d baseline
\nWe spent a significant amount of time building our own flexible fastai pipeline from scratch, but we weren't able to achieve the same scores as with our variations on @tanakar's public notebook that we were experimenting with on the side. We used fastai's unet_learner, extended to use timm image models as backbone. We had custom callbacks and a streamlined trained & inference structure already prepared for ensembling. We used a custom dataloader that held the relevant layers of the full fragments in memory and was indexing into them dynamically at training & inference time. Not sure whether it's more efficient than the 2.5d baseline dataloader, but it felt elegant ;-)
What those fastai experiments gave us was a scan through all the (relevant) layers/slices of each fragment; showing where most of the signal was located. We used 3 slices per experiment, and stepped through successive starting minimum slices. E.g. the experiment for min slice = 28 contained the slices 28, 29, 30. The results are below, and suggested that the top and bottom slices were lacking in detectable ink, which was primarily found in the center slices. There was a gradual increase and decline in signal, and the locations of these ingress & egress were slightly different for each fragment. Using 3 slices per experiment gave us slightly reduced resolution in the z-axis, but made those validation scores somewhat more robust:
With an appropriate 2.5 weeks left to go, we decided to pivot fully to using the 2.5d public baseline as the core of our experimentation setup. Here are some things we did slightly differently from other solution write-ups:
\nWe settled on using 8 slices/channels starting from slice 28, partly based on the above depth scan.
We focused on large models, most prominently eca_nfnet_l2 and maxvit_large models from timm with image sizes 224, 384, and 512 (@awgross).
In addition to the segmentation-models-pytorch Unet architecture, we found Linknet to perform even slightly better (@mjulianimsj75).
We validated on fragments 1 & 3, which gave us reliable correlations with the public LB (@kristoferschobert).
We validated our ensemble selections using the out-of-fold scores for fragments 1 & 3; tuning percentile thresholds simultaneously. Our best thresholds were usually 94%. Our local ensemble scores were also pretty well correlated with the public & private LB.
It was our best ensemble of 6 maxvit models (without the contribution of eca_nfnet or other backbones), which scored consistently highest on local validation, public LB, and private LB; and allowed us to pick our best submission to sneak into the top 50.
Congratulations on your silver medal, and thank you for taking the time to write up what worked and what didn't.
", "rawMarkdown": "Congratulations on your silver medal, and thank you for taking the time to write up what worked and what didn't.", "votes": null }, { "id": "2310876", "postDate": "06/20/2023 17:03:34", "content": "Congratulations! 🥳 And thank you for your interesting write-up! Initially I wanted to set something up with fastai as well but then switched to @tanakar's public notebook, just as you did.
\nSince you really worked something out with fastai, it would be great, if you would share some code. I would be super interested in exploring what you did!
\nKeep it up!
", "rawMarkdown": "Congratulations! 🥳 And thank you for your interesting write-up! Initially I wanted to set something up with fastai as well but then switched to @tanakar's public notebook, just as you did.\n\nSince you really worked something out with fastai, it would be great, if you would share some code. I would be super interested in exploring what you did!\n\nKeep it up!", "votes": null }, { "id": "2312491", "postDate": "06/22/2023 00:34:45", "content": "Thanks! I like to use fastai as a starter baseline because it's usually, well, fast to set up. It's kinda reassuring that other people were struggling with it as well in this competition. I still plan to do more investigating into what went wrong.
\nThe approach that gave the training scores from the plot above wasn't that complex, after all. Let me try to summarize:
\nThe trainer is a simple unet_learner, with some changes to enable timm models; following from this issue. The timm tweaks seemed to work well, but we didn't really get to utilizing it because the learner didn't produce good LB models either with or without the tweaks.
The dataloaders are build around an approach that loads the complete fragments (with the corresponding layers) into RAM as a dictionary; e.g. fragments_all['1'] is fragment 1. Then it indexes into them like this:
def get_img(example, img_size):\n fragment = example[]\n x = example[] \n y = example[]\n return fragments_all[fragment][x:x+img_size, y:y+img_size, ...]\nAnd the same for get_labels to be used in this datablock:
dblock = DataBlock(\n (, mask_block),\n get_x = partial(, img_size = img_size),\n get_y = partial(, img_size = img_size),\n n_inp=1,\n splitter = ColSplitter(),\n item_tfms = aug_tfms\n)\nWhere img_block and mask_block are essentially just TransformBlocks that take and return tensors. Everything is a tensor; no PILImage types involved after creating the fragments_all dictionary entries. It's possible that this is the source of some of the trouble, although when inspecting random batches they always looked good to me. The aug_tfms use an Albumentation wrapper like here and were also in charge of normalization.
As the datablock suggests, it ingests a dataframe df to create the dataloaders. This dataframe holds the x and y coordinates for each image of img_size x img_size within the fragment. So that the dataloaders are dynamically indexing into the fragment at runtime. Essentially you put a grid like this over your fragment, then index into it:
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1014468%2Fba1db55ea919b4057b90169f41c0074a%2FScreen%20Shot%202023-06-21%20at%2020.24.36.png?generation=1687393519791527&alt=media\")
That's pretty much it. We tried different losses, including smp.losses.SoftBCEWithLogitsLoss like in the public baseline, Adam and AdamW optimizers, different learning rates with and without lr_find, dynamic thresholds for final scores, and probably some other things that I'm forgetting right now.
Hope that helps. Let me know if you have any feedback or question on any of this.
", "rawMarkdown": "Thanks! I like to use fastai as a starter baseline because it's usually, well, fast to set up. It's kinda reassuring that other people were struggling with it as well in this competition. I still plan to do more investigating into what went wrong.\n\nThe approach that gave the training scores from the plot above wasn't that complex, after all. Let me try to summarize:\n\nThe trainer is a simple `unet_learner`, with some changes to enable `timm` models; following from [this issue](https://github.com/fastai/fastai/pull/3717). The `timm` tweaks seemed to work well, but we didn't really get to utilizing it because the learner didn't produce good LB models either with or without the tweaks.\n\nThe dataloaders are build around an approach that loads the complete fragments (with the corresponding layers) into RAM as a dictionary; e.g. `fragments_all['1']` is fragment 1. Then it indexes into them like this:\n\n```\ndef get_img(example, img_size):\n fragment = example['fragment']\n x = example['x'] \n y = example['y']\n return fragments_all[fragment][x:x+img_size, y:y+img_size, ...]\n```\n\nAnd the same for `get_labels` to be used in this datablock:\n\n```\ndblock = DataBlock(\n blocks=(img_block, mask_block),\n get_x = partial(get_img, img_size = img_size),\n get_y = partial(get_labels, img_size = img_size),\n n_inp=1,\n splitter = ColSplitter(),\n item_tfms = aug_tfms\n)\n```\n\nWhere `img_block` and `mask_block` are essentially just `TransformBlock`s that take and return tensors. Everything is a tensor; no `PILImage` types involved after creating the `fragments_all` dictionary entries. It's possible that this is the source of some of the trouble, although when inspecting random batches they always looked good to me. The `aug_tfms` use an [Albumentation wrapper like here](https://docs.fast.ai/tutorial.albumentations.html) and were also in charge of normalization.\n\nAs the datablock suggests, it ingests a dataframe `df` to create the dataloaders. This dataframe holds the `x` and `y` coordinates for each image of img_size x img_size within the fragment. So that the dataloaders are dynamically indexing into the fragment at runtime. Essentially you put a grid like this over your fragment, then index into it:\n\n\n\nThat's pretty much it. We tried different losses, including `smp.losses.SoftBCEWithLogitsLoss` like in the public baseline, Adam and AdamW optimizers, different learning rates with and without `lr_find`, dynamic thresholds for final scores, and probably some other things that I'm forgetting right now.\n\nHope that helps. Let me know if you have any feedback or question on any of this.", "votes": null }, { "id": "2313935", "postDate": "06/23/2023 04:05:24", "content": "It helps a lot! Thank you :) I am trying fastai again on https://www.kaggle.com/competitions/google-research-identify-contrails-reduce-global-warming and will share as soon as I've got something running.
", "rawMarkdown": "It helps a lot! Thank you :) I am trying fastai again on https://www.kaggle.com/competitions/google-research-identify-contrails-reduce-global-warming and will share as soon as I've got something running.", "votes": null }, { "id": "2342801", "postDate": "07/13/2023 06:59:10", "content": "https://www.kaggle.com/crustacean/train-fastai-baseline
\nhttps://www.kaggle.com/crustacean/inference-fastai-baseline
Congratulations on your silver medal, and thank you for taking the time to write up what worked and what didn't.
", "votes": null, "replies": [] }, { "id": 2310876, "author_name": "crustacean", "author_url": "", "post_date": "06/20/2023 17:03:34", "content": "Congratulations! 🥳 And thank you for your interesting write-up! Initially I wanted to set something up with fastai as well but then switched to @tanakar's public notebook, just as you did.
\nSince you really worked something out with fastai, it would be great, if you would share some code. I would be super interested in exploring what you did!
\nKeep it up!
", "votes": null, "replies": [ { "id": 2312491, "author_name": "headsortails", "author_url": "", "post_date": "06/22/2023 00:34:45", "content": "Thanks! I like to use fastai as a starter baseline because it's usually, well, fast to set up. It's kinda reassuring that other people were struggling with it as well in this competition. I still plan to do more investigating into what went wrong.
\nThe approach that gave the training scores from the plot above wasn't that complex, after all. Let me try to summarize:
\nThe trainer is a simple unet_learner, with some changes to enable timm models; following from this issue. The timm tweaks seemed to work well, but we didn't really get to utilizing it because the learner didn't produce good LB models either with or without the tweaks.
The dataloaders are build around an approach that loads the complete fragments (with the corresponding layers) into RAM as a dictionary; e.g. fragments_all['1'] is fragment 1. Then it indexes into them like this:
def get_img(example, img_size):\n fragment = example[]\n x = example[] \n y = example[]\n return fragments_all[fragment][x:x+img_size, y:y+img_size, ...]\nAnd the same for get_labels to be used in this datablock:
dblock = DataBlock(\n (, mask_block),\n get_x = partial(, img_size = img_size),\n get_y = partial(, img_size = img_size),\n n_inp=1,\n splitter = ColSplitter(),\n item_tfms = aug_tfms\n)\nWhere img_block and mask_block are essentially just TransformBlocks that take and return tensors. Everything is a tensor; no PILImage types involved after creating the fragments_all dictionary entries. It's possible that this is the source of some of the trouble, although when inspecting random batches they always looked good to me. The aug_tfms use an Albumentation wrapper like here and were also in charge of normalization.
As the datablock suggests, it ingests a dataframe df to create the dataloaders. This dataframe holds the x and y coordinates for each image of img_size x img_size within the fragment. So that the dataloaders are dynamically indexing into the fragment at runtime. Essentially you put a grid like this over your fragment, then index into it:
That's pretty much it. We tried different losses, including smp.losses.SoftBCEWithLogitsLoss like in the public baseline, Adam and AdamW optimizers, different learning rates with and without lr_find, dynamic thresholds for final scores, and probably some other things that I'm forgetting right now.
Hope that helps. Let me know if you have any feedback or question on any of this.
", "votes": null, "replies": [ { "id": 2313935, "author_name": "crustacean", "author_url": "", "post_date": "06/23/2023 04:05:24", "content": "It helps a lot! Thank you :) I am trying fastai again on https://www.kaggle.com/competitions/google-research-identify-contrails-reduce-global-warming and will share as soon as I've got something running.
", "votes": null, "replies": [ { "id": 2342801, "author_name": "crustacean", "author_url": "", "post_date": "07/13/2023 06:59:10", "content": "https://www.kaggle.com/crustacean/train-fastai-baseline
\nhttps://www.kaggle.com/crustacean/inference-fastai-baseline