Thanks to the all hosts & participants.
\nEspecially for the @ttahara 's baseline[1]. I reused the notebook throughout my experiments.
Also, Google Colabo Pro & Kaggle Notebook.
\nStacking of 5 models (Table 1).
\nTable 1: Models and Score of Final Submission
\n| Model | \nPrivate LB | \nPublic LB | \n
|---|---|---|
| EfficientNet-B2, 512x512, 5-fold | \n0.763 | \n0.764 | \n
| EfficientNet-B0, 512x512, 5-fold | \n0.757 | \n0.759 | \n
| EfficientNet-B0, 512x512, 3-fold, controlling sampling rate, logical-or on mix-upped target | \n0.730 | \n0.733 | \n
| ResNet18d, 512x512, 5-fold, controlling sampling rate | \n0.745 | \n0.744 | \n
| ResNet18d, 320x320, 5-fold | \n0.719 | \n0.720 | \n
| Stacking 5 models (1-hidden-layer MLP) | \n0.764 | \n0.765 | \n
In the early trials, I was wondering about class imbalance problem. Since AUC-ROC equally evaluates the minority and majority class, I thought simple BCE loss over-penalize the positive set, and make it hard to detect them (which I finally found that's NOT TRUE).
\nSo I balanced class by controlling sampling rate of training set (i.e. oversampling positive set or/and downsampling negative set). I also added the train/test's B-observations (1, 3, 5 channels) as the negative sets to help models learning the background information. A typical settings are shown in table2.
The trial boosted the training speed, but at the same time it suppress the score: feeding the whole dataset improves score about +0.03. I think it is because \"knowing background\" is an important information for the competition.
\nTable 2: Number of Samples for Train Dataset
\n| target | \ntrain-pos-A | \ntrain-pos-B | \ntrain-neg-A | \ntotal | \nratio | \nafter mix-up | \n
|---|---|---|---|---|---|---|
| 0 | \n0 | \n4,000 | \n4,000 | \n8,000 | \n0.67 | \n0.45 | \n
| 1 | \n4,000 | \n0 | \n0 | \n4,000 | \n0.33 | \n0.55 | \n
| total | \n4,000 | \n4,000 | \n4,000 | \n12,000 | \n1.0 | \n1.0 | \n
[1] https://www.kaggle.com/ttahara/rerun-seti-e-t-resnet18d-baseline
\n[2] https://www.kaggle.com/c/seti-breakthrough-listen/discussion/265973
\n[3] https://www.kaggle.com/tuckerarrants/seti-learned-image-resizing
Thanks to the all hosts & participants.
\nEspecially for the @ttahara 's baseline[1]. I reused the notebook throughout my experiments.
Also, Google Colabo Pro & Kaggle Notebook.
\nStacking of 5 models (Table 1).
\nTable 1: Models and Score of Final Submission
\n| Model | \nPrivate LB | \nPublic LB | \n
|---|---|---|
| EfficientNet-B2, 512x512, 5-fold | \n0.763 | \n0.764 | \n
| EfficientNet-B0, 512x512, 5-fold | \n0.757 | \n0.759 | \n
| EfficientNet-B0, 512x512, 3-fold, controlling sampling rate, logical-or on mix-upped target | \n0.730 | \n0.733 | \n
| ResNet18d, 512x512, 5-fold, controlling sampling rate | \n0.745 | \n0.744 | \n
| ResNet18d, 320x320, 5-fold | \n0.719 | \n0.720 | \n
| Stacking 5 models (1-hidden-layer MLP) | \n0.764 | \n0.765 | \n
In the early trials, I was wondering about class imbalance problem. Since AUC-ROC equally evaluates the minority and majority class, I thought simple BCE loss over-penalize the positive set, and make it hard to detect them (which I finally found that's NOT TRUE).
\nSo I balanced class by controlling sampling rate of training set (i.e. oversampling positive set or/and downsampling negative set). I also added the train/test's B-observations (1, 3, 5 channels) as the negative sets to help models learning the background information. A typical settings are shown in table2.
The trial boosted the training speed, but at the same time it suppress the score: feeding the whole dataset improves score about +0.03. I think it is because \"knowing background\" is an important information for the competition.
\nTable 2: Number of Samples for Train Dataset
\n| target | \ntrain-pos-A | \ntrain-pos-B | \ntrain-neg-A | \ntotal | \nratio | \nafter mix-up | \n
|---|---|---|---|---|---|---|
| 0 | \n0 | \n4,000 | \n4,000 | \n8,000 | \n0.67 | \n0.45 | \n
| 1 | \n4,000 | \n0 | \n0 | \n4,000 | \n0.33 | \n0.55 | \n
| total | \n4,000 | \n4,000 | \n4,000 | \n12,000 | \n1.0 | \n1.0 | \n
[1] https://www.kaggle.com/ttahara/rerun-seti-e-t-resnet18d-baseline
\n[2] https://www.kaggle.com/c/seti-breakthrough-listen/discussion/265973
\n[3] https://www.kaggle.com/tuckerarrants/seti-learned-image-resizing
As for top ranker's solutions, it seems larger model performs better.
\nI am curious about anyone who achieves high score with relatively small models.
\nIf anyone who can achieve the high score with small model, please share the approach.
nf-regnet b1 and efficientnet b2 are about same capacity
", "rawMarkdown": "nf-regnet b1 and efficientnet b2 are about same capacity", "votes": 1 }, { "id": 1483128, "postDate": "2021-08-20T13:07:39.687Z", "content": "@bakeryproducts Thank you for sharing.
\nDo you think the score difference is due to the model choice or other trials (or both)?
\nAccording to your solution, you tried other trials like larger image size, logical-OR on mix-upped target, freezing backbone etc. Which trial do you think is the most effective to the score?
", "rawMarkdown": "@bakeryproducts Thank you for sharing.\n\nDo you think the score difference is due to the model choice or other trials (or both)?\n\nAccording to your solution, you tried other trials like larger image size, logical-OR on mix-upped target, freezing backbone etc. Which trial do you think is the most effective to the score?" }, { "id": 1483156, "postDate": "2021-08-20T13:29:04.110Z", "content": "Hi bilzard. Nice job, you did well.
\nI am similarly confused. I used big models such as EfficientNetB6, B4 and B3. And I used large image sizes like 384, 512, and 768. None-the-less, i barely got Bronze medal. My public LB only reached LB 779.
\nI think mixup, augmentation, regularization are the key factors to prevent models from overfitting train data and generalize to test data. I did not use much of these, so I'm thinking this was my shortcoming. I'm currently running experiments now to see if this is indeed the missing factor.
", "rawMarkdown": "Hi bilzard. Nice job, you did well.\n\nI am similarly confused. I used big models such as EfficientNetB6, B4 and B3. And I used large image sizes like 384, 512, and 768. None-the-less, i barely got Bronze medal. My public LB only reached LB 779.\n\nI think mixup, augmentation, regularization are the key factors to prevent models from overfitting train data and generalize to test data. I did not use much of these, so I'm thinking this was my shortcoming. I'm currently running experiments now to see if this is indeed the missing factor.", "votes": 1 }, { "id": 1483164, "postDate": "2021-08-20T13:32:44.373Z", "content": "Image size, epochs, are crucial, tricks like MSDA( mixup, etc), freezing bb, are boosting good model even further. 20 epochs is nothing, my submit models never early exited until epoch ~300.
\n(for now) You will get much more from doing right things and monitoring everything, then from tricks. And you gotta be careful with augmentations, they are good, but sometimes break logic of sample. For example you cant vertical flip image from car dashcam.
Update. to be fair number of epochs to converge depends on a lot of things and you probably should just monitor your validation.
", "rawMarkdown": "Image size, epochs, are crucial, tricks like MSDA( mixup, etc), freezing bb, are boosting good model even further. 20 epochs is nothing, my submit models never early exited until epoch ~300.\n(for now) You will get much more from doing right things and monitoring everything, then from tricks. And you gotta be careful with augmentations, they are good, but sometimes break logic of sample. For example you cant vertical flip image from car dashcam.\n\nUpdate. to be fair number of epochs to converge depends on a lot of things and you probably should just monitor your validation.", "votes": 1 }, { "id": 1483180, "postDate": "2021-08-20T13:44:58.577Z", "content": "Our models converged after 20 epochs (also without cleaning and early 0.800 score), but we used all data (also old).
", "rawMarkdown": "Our models converged after 20 epochs (also without cleaning and early 0.800 score), but we used all data (also old).", "votes": 2 }, { "id": 1486594, "postDate": "2021-08-23T05:09:26.153Z", "content": "@cdeotte Thank you for sharing your model and score. Larger model, larger image size was the key to the score LB 0.779. In my situation, I don't have my own workstation, I have to train within the limit of cloud service like Google Colab and Kaggle notebook. So I didn't dare to train larger model than EfficientNet-B2. I think sufficient computing resource is de facto standard for today's kaggle competition.
\n@bakeryproducts I didn't come up with training over hundreds of epochs. It's like training the model from scratch. Thank you for the advice.
\n@philippsinger Variation of data was also crucial to this competition.
\nPersonally, one more important thing is EDA. I did some EDA and error analysis, but I couldn't find out useful information from them, or, if I find something (S-signal, CV-LB difference), I didn't dig further or doing something by utilizing the fact.
", "rawMarkdown": "@cdeotte Thank you for sharing your model and score. Larger model, larger image size was the key to the score LB 0.779. In my situation, I don't have my own workstation, I have to train within the limit of cloud service like Google Colab and Kaggle notebook. So I didn't dare to train larger model than EfficientNet-B2. I think sufficient computing resource is de facto standard for today's kaggle competition.\n\n@bakeryproducts I didn't come up with training over hundreds of epochs. It's like training the model from scratch. Thank you for the advice.\n\n@philippsinger Variation of data was also crucial to this competition. \n\nPersonally, one more important thing is EDA. I did some EDA and error analysis, but I couldn't find out useful information from them, or, if I find something (S-signal, CV-LB difference), I didn't dig further or doing something by utilizing the fact. " } ] } ], "comments": [ { "id": 1481235, "author_name": "Bilzard", "author_url": "", "post_date": "2021-08-19T11:34:05.370000", "content": "As for top ranker's solutions, it seems larger model performs better.
\nI am curious about anyone who achieves high score with relatively small models.
\nIf anyone who can achieve the high score with small model, please share the approach.
nf-regnet b1 and efficientnet b2 are about same capacity
", "votes": 1, "replies": [] }, { "id": 1483128, "author_name": "Bilzard", "author_url": "", "post_date": "2021-08-20T13:07:39.687000", "content": "@bakeryproducts Thank you for sharing.
\nDo you think the score difference is due to the model choice or other trials (or both)?
\nAccording to your solution, you tried other trials like larger image size, logical-OR on mix-upped target, freezing backbone etc. Which trial do you think is the most effective to the score?
", "votes": 0, "replies": [] }, { "id": 1483156, "author_name": "Chris Deotte", "author_url": "", "post_date": "2021-08-20T13:29:04.110000", "content": "Hi bilzard. Nice job, you did well.
\nI am similarly confused. I used big models such as EfficientNetB6, B4 and B3. And I used large image sizes like 384, 512, and 768. None-the-less, i barely got Bronze medal. My public LB only reached LB 779.
\nI think mixup, augmentation, regularization are the key factors to prevent models from overfitting train data and generalize to test data. I did not use much of these, so I'm thinking this was my shortcoming. I'm currently running experiments now to see if this is indeed the missing factor.
", "votes": 1, "replies": [] }, { "id": 1483164, "author_name": "Gleb", "author_url": "", "post_date": "2021-08-20T13:32:44.373000", "content": "Image size, epochs, are crucial, tricks like MSDA( mixup, etc), freezing bb, are boosting good model even further. 20 epochs is nothing, my submit models never early exited until epoch ~300.
\n(for now) You will get much more from doing right things and monitoring everything, then from tricks. And you gotta be careful with augmentations, they are good, but sometimes break logic of sample. For example you cant vertical flip image from car dashcam.
Update. to be fair number of epochs to converge depends on a lot of things and you probably should just monitor your validation.
", "votes": 1, "replies": [] }, { "id": 1483180, "author_name": "Psi", "author_url": "", "post_date": "2021-08-20T13:44:58.577000", "content": "Our models converged after 20 epochs (also without cleaning and early 0.800 score), but we used all data (also old).
", "votes": 2, "replies": [] }, { "id": 1486594, "author_name": "Bilzard", "author_url": "", "post_date": "2021-08-23T05:09:26.153000", "content": "@cdeotte Thank you for sharing your model and score. Larger model, larger image size was the key to the score LB 0.779. In my situation, I don't have my own workstation, I have to train within the limit of cloud service like Google Colab and Kaggle notebook. So I didn't dare to train larger model than EfficientNet-B2. I think sufficient computing resource is de facto standard for today's kaggle competition.
\n@bakeryproducts I didn't come up with training over hundreds of epochs. It's like training the model from scratch. Thank you for the advice.
\n@philippsinger Variation of data was also crucial to this competition.
\nPersonally, one more important thing is EDA. I did some EDA and error analysis, but I couldn't find out useful information from them, or, if I find something (S-signal, CV-LB difference), I didn't dig further or doing something by utilizing the fact.
", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "1481156": "Thanks to the all hosts & participants.\nEspecially for the @ttahara 's baseline[1]. I reused the notebook throughout my experiments.\n\nAlso, Google Colabo Pro & Kaggle Notebook.\n\n## Final Model\n\nStacking of 5 models (Table 1).\n\n