First of all, we want to thank the organizers and the Kaggle team for hosting the competition. We believe that someday AI will open a big road to exploring the Universe.
\nBesides that, we want to point out the following projects:
\nClassification ensemble with MSDA and focal loss. 2 models types: nf-regnetb1, HRNet18. Spatial cadence with guide layer for keeping ABACAD information
\nMost of our experiments were on original resolution (img/sec reasons). Within the last couple of weeks, we scaled up to current image sizes.
\nWe use spatial vstack, 6 frames side by side, as most of you did. Samples upscaled with cubic interpolation x3 by frequency axis, scaled down to 6x160 by time axis. The resulting sample size is 960x768
We use 'guide layer' as the second input layer, mask 960x768 of -1/1 structure: 1 for 'A' frames,-1 for 'B', 'C', 'D' frames
NN input is ~70% random crop from 960x768x2
Augmentation (on batches, hand-coded and bit of kornia):
\nCV split 15k / 45k, cv is correlated with LB
\nWe feel like MSDA is one of the key parts of this solution.
\nMixUp Used on all batches, alpha=2. But mixing any label within an anomaly will result in an anomaly label. Labels and permuted labels are the same. It was done not to penalize NN with loss from the mixed anomaly. So one can say it is not MixUp at all.
\nWe tried other techniques like AugMix, but it's hard to scale up image processing speed with a larger image size. Therefore, we end up using only batch-wise augmentations.
Used models: timm nf-regnetb1 - our light one, timm HRNet18 - larger one. Such pick is random, but we feel like regnet ideas are correct, and HRnet looks like a good choice for this competition.
\nTraining time for nf-regnetb1 is ~7hrs, for HRNet ~24hrs (4xV100)
\nTTA: flips, multiplier, sometimes additional Gaussian noise(mean=0, std=[0,.1,.2]).
\nEnsemble, merge is simple average:
\nCV score for ensemble parts: .901, .901, .908, .908
\nLB: .794, .794, .797, .797
\nUpdate. Best single model HRNet18 488x896 PB 0.80095; LB 0.79766
\nChannel-wise cadence, consistency loss, AUC proxy loss, frequency normalizing, whitening, dynamic sampling, selective sampling, autoencoders, VAE, VQVAE, MEMAE, more of XXXAE and unsupervised methods, PSEUDO LABELING, best was non-harming one, KNN, DBSCAN clustering of train/ test with adversarial models, frames decomposition, transformers, logits ensembles, shuffling cadences, creating additional data from test B, C, D frames, space2depth and other stems.
\nProbably a personal record.
\nThere was a belief in finding some data insights that could give that jump to the final score. However, spending nearly all of our time and efforts were not worth it.
\nAs we fail to use almost all of our data-related discoveries (such as difference between train and test) we left that writing to other teams.
\nFirst of all, we want to thank the organizers and the Kaggle team for hosting the competition. We believe that someday AI will open a big road to exploring the Universe.
\nBesides that, we want to point out the following projects:
\nClassification ensemble with MSDA and focal loss. 2 models types: nf-regnetb1, HRNet18. Spatial cadence with guide layer for keeping ABACAD information
\nMost of our experiments were on original resolution (img/sec reasons). Within the last couple of weeks, we scaled up to current image sizes.
\nWe use spatial vstack, 6 frames side by side, as most of you did. Samples upscaled with cubic interpolation x3 by frequency axis, scaled down to 6x160 by time axis. The resulting sample size is 960x768
We use 'guide layer' as the second input layer, mask 960x768 of -1/1 structure: 1 for 'A' frames,-1 for 'B', 'C', 'D' frames
NN input is ~70% random crop from 960x768x2
Augmentation (on batches, hand-coded and bit of kornia):
\nCV split 15k / 45k, cv is correlated with LB
\nWe feel like MSDA is one of the key parts of this solution.
\nMixUp Used on all batches, alpha=2. But mixing any label within an anomaly will result in an anomaly label. Labels and permuted labels are the same. It was done not to penalize NN with loss from the mixed anomaly. So one can say it is not MixUp at all.
\nWe tried other techniques like AugMix, but it's hard to scale up image processing speed with a larger image size. Therefore, we end up using only batch-wise augmentations.
Used models: timm nf-regnetb1 - our light one, timm HRNet18 - larger one. Such pick is random, but we feel like regnet ideas are correct, and HRnet looks like a good choice for this competition.
\nTraining time for nf-regnetb1 is ~7hrs, for HRNet ~24hrs (4xV100)
\nTTA: flips, multiplier, sometimes additional Gaussian noise(mean=0, std=[0,.1,.2]).
\nEnsemble, merge is simple average:
\nCV score for ensemble parts: .901, .901, .908, .908
\nLB: .794, .794, .797, .797
\nUpdate. Best single model HRNet18 488x896 PB 0.80095; LB 0.79766
\nChannel-wise cadence, consistency loss, AUC proxy loss, frequency normalizing, whitening, dynamic sampling, selective sampling, autoencoders, VAE, VQVAE, MEMAE, more of XXXAE and unsupervised methods, PSEUDO LABELING, best was non-harming one, KNN, DBSCAN clustering of train/ test with adversarial models, frames decomposition, transformers, logits ensembles, shuffling cadences, creating additional data from test B, C, D frames, space2depth and other stems.
\nProbably a personal record.
\nThere was a belief in finding some data insights that could give that jump to the final score. However, spending nearly all of our time and efforts were not worth it.
\nAs we fail to use almost all of our data-related discoveries (such as difference between train and test) we left that writing to other teams.
\nWow! It's really technical solution.
\nGreat work!
Congrats on 4th Place
\nVery interesting solution @bakeryproducts especially the upscaling and preprocessing part , thanks for sharing , can you share the code for your solution as well, it will be great for us to learn
Thanks, we didnt plan to, but maybe in future
", "rawMarkdown": "Thanks, we didnt plan to, but maybe in future" } ] }, { "id": 1480472, "postDate": "2021-08-19T03:27:02.647Z", "content": "Very cool. Soloing the comp especially the last 3 weeks where discussions and kernels were all but dead (except for that one funny ensemble rickroll kernel), sometimes it felt like there's nothing interesting to learn. However reading through your solution has put a few good things on my radar. Thank you for sharing.
", "rawMarkdown": "Very cool. Soloing the comp especially the last 3 weeks where discussions and kernels were all but dead (except for that one funny ensemble rickroll kernel), sometimes it felt like there's nothing interesting to learn. However reading through your solution has put a few good things on my radar. Thank you for sharing.", "replies": [ { "id": 1481662, "postDate": "2021-08-19T16:09:11.950Z", "content": "And this is what i dont like about platform. We cant really start discussion until its over and people are tired after a month or more of intensive research. Im happy you get something from my write-up.
", "rawMarkdown": "And this is what i dont like about platform. We cant really start discussion until its over and people are tired after a month or more of intensive research. Im happy you get something from my write-up." } ] } ], "comments": [ { "id": 1481078, "author_name": "WOOSUNG YOON", "author_url": "", "post_date": "2021-08-19T09:49:43.693000", "content": "Wow! It's really technical solution.
\nGreat work!
Congrats on 4th Place
\nVery interesting solution @bakeryproducts especially the upscaling and preprocessing part , thanks for sharing , can you share the code for your solution as well, it will be great for us to learn
Thanks, we didnt plan to, but maybe in future
", "votes": 0, "replies": [] } ] }, { "id": 1480472, "author_name": "عثمان", "author_url": "", "post_date": "2021-08-19T03:27:02.647000", "content": "Very cool. Soloing the comp especially the last 3 weeks where discussions and kernels were all but dead (except for that one funny ensemble rickroll kernel), sometimes it felt like there's nothing interesting to learn. However reading through your solution has put a few good things on my radar. Thank you for sharing.
", "votes": 0, "replies": [ { "id": 1481662, "author_name": "Gleb", "author_url": "", "post_date": "2021-08-19T16:09:11.950000", "content": "And this is what i dont like about platform. We cant really start discussion until its over and people are tired after a month or more of intensive research. Im happy you get something from my write-up.
", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "1480361": "# Intro\n\nFirst of all, we want to thank the organizers and the Kaggle team for hosting the competition. We believe that someday AI will open a big road to exploring the Universe.\n\nBesides that, we want to point out the following projects:\n- [timm, rwightman](https://pypi.org/project/timm/)\n- [pytorch](https://pytorch.org/)\n- [qubvel, TTA, cls models, seg models, etc. Great stuff](https://github.com/qubvel/)\n- feel free to mention any project/author we forget about. We are happy to update this post with references\n\n# TLDR\n\nClassification ensemble with MSDA and focal loss. 2 models types: nf-regnetb1, HRNet18. Spatial cadence with guide layer for keeping ABACAD information\n\n\n# Input, preprocessing\n\nMost of our experiments were on original resolution (img/sec reasons). Within the last couple of weeks, we scaled up to current image sizes.\n\n - We use spatial vstack, 6 frames side by side, as most of you did. Samples upscaled with cubic interpolation x3 by frequency axis, scaled down to 6x160 by time axis. The resulting sample size is 960x768\n\n - We use 'guide layer' as the second input layer, mask 960x768 of -1/1 structure: 1 for 'A' frames,-1 for 'B', 'C', 'D' frames\n\n - NN input is ~70% random crop from 960x768x2\n\nAugmentation (on batches, hand-coded and bit of kornia):\n- Random Crop\n- HFlip, VFlip\n- Special tuned MSDA\n\nCV split 15k / 45k, cv is correlated with LB\n\n### MSDA\n\nWe feel like MSDA is one of the key parts of this solution.\n\n - [MixUp](https://arxiv.org/abs/1710.09412) Used on all batches, alpha=2. **But** mixing any label within an anomaly will result in an anomaly label. Labels and permuted labels are the same. It was done not to penalize NN with loss from the mixed anomaly. So one can say it is not MixUp at all.\n\n- [CutMix](https://arxiv.org/abs/1905.04899) Used on all batches, with parameters from official implementation. **But** mixing happening in two separate sub-batches: the first is only anomaly batch, and the second is rest. It was done due to the inability to distinguish the position of the signal inside of the frame. So one can say it is not CutMix at all.\n \n We tried other techniques like AugMix, but it's hard to scale up image processing speed with a larger image size. Therefore, we end up using only batch-wise augmentations.\n\n# Train\nUsed models: timm nf-regnetb1 - our light one, timm HRNet18 - larger one. Such pick is random, but we feel like regnet ideas are correct, and HRnet looks like a good choice for this competition.\n\nTraining time for nf-regnetb1 is ~7hrs, for HRNet ~24hrs (4xV100)\n\n### Train Features\n\n - [FocalLoss](https://arxiv.org/abs/1708.02002), gamma=2, alpha=.7\n - [DDP](https://github.com/pytorch/examples/blob/master/distributed/ddp/README.md) multiple GPUs\n - [AMP](https://pytorch.org/docs/stable/notes/amp_examples.html) to reduces memory requirements for training models, enabling larger models or larger batches\n - [EMA](https://fastai.github.io/timmdocs/training_modelEMA) (timm)\n - usual LR stuff, warmup, cosine decay \n - early in competition SAM, but with AMP and DDP it requires much careful tuning, fails with numerical instability (and x2 training time)\n - all data in the tensor dataset, sharded between GPUs (which is not a good idea accuracy-wise, but it is fast)\n\n\n# Inference, postprocess\n\nTTA: flips, multiplier, sometimes additional Gaussian noise(mean=0, std=[0,.1,.2]).\n\nEnsemble, merge is simple average:\n\n - [nf-regnetb1](https://arxiv.org/pdf/2101.08692.pdf), 384, 768, split1 \n - [nf-regnetb1](https://arxiv.org/pdf/2101.08692.pdf), 384, 768, split2, minor MSDA changes\n - [HRNet18](https://github.com/HRNet/HRNet-Image-Classification), 448, 896, split1\n - [HRNet18](https://github.com/HRNet/HRNet-Image-Classification), 448, 896, split1, minor MSDA changes, Focal with lower gamma and alpha\n\nCV score for ensemble parts: .901, .901, .908, .908\n\nLB: .794, .794, .797, .797\n\n# LB basic roadmap\n\n- .780 simple cnn, but with MSDA\n- .783 +guide layer\n- .788 +focal \n- .794 +upscale resolution\n- .797 +heavy model\n- .800 +ensembles\n\nUpdate. Best single model HRNet18 488x896 PB 0.80095; LB 0.79766\n\n# Shortlist of things that did not work\n\nChannel-wise cadence, consistency loss, AUC proxy loss, frequency normalizing, whitening, dynamic sampling, selective sampling, autoencoders, VAE, VQVAE, MEMAE, more of XXXAE and unsupervised methods, *PSEUDO LABELING, best was non-harming one*, KNN, DBSCAN clustering of train/ test with adversarial models, frames decomposition, transformers, logits ensembles, shuffling cadences, creating additional data from test B, C, D frames, space2depth and other stems.\n\nProbably a personal record.\n\n_______________\n\n### P.S.\nThere was a belief in finding some data insights that could give that jump to the final score. However, spending nearly all of our time and efforts were not worth it. \n\nAs we fail to use almost all of our data-related discoveries (such as difference between train and test) we left that writing to other teams.\n\n### Upd after reading 1-st place, congrats btw\n\n1. We used only new data, old data did give a little boost to us, so little that we dropped that.\n2. We also find sin-shaped signal and also tried to add it with setigen/handcode, but with no improvements to score\n3. We also added near-1 probabilities as pseudo labels, but score immediately drop to .75. So we did DBSCAN clustering on test images and select certain cluster, that 'base', (not a cluster). Adding it as pseudo labels dropped score only slightly, and basically we could not go further then that.\n4. Cleaning images for us ended at trimmed normalization by frequency and time, which visually helped, but not scoring-wise \n", "1481078": "Wow! It's really technical solution.\nGreat work!", "1480728": "Congrats on 4th Place\nVery interesting solution @bakeryproducts especially the upscaling and preprocessing part , thanks for sharing , can you share the code for your solution as well, it will be great for us to learn", "1480472": "Very cool. Soloing the comp especially the last 3 weeks where discussions and kernels were all but dead (except for that one funny ensemble rickroll kernel), sometimes it felt like there's nothing interesting to learn. However reading through your solution has put a few good things on my radar. Thank you for sharing." } }