Hello kagglers,
\nAlongside EfficientNetB0 and ResNet34d, I have (successfully) trained EfficientNetB1, using the same batch size of 16. Since most participants use the first two models, I have been experimenting for the previous days with different models and optimizers: EfficientNetB1 (AdamW optimizer), EfficientNetB3 (RMSProp optimizer) and ResNet50 (SGD optimizer, with momentum). From the first epoch, I concluded that larger models, namely EfficientNetB3 and ResNet50 perform worse compared to lighter models, scoring around 1.25 on the Kullback Liebler divergence loss.
\nAs for EfficientNetB1, despite doing better on the CV than EfficientNetB0 and ResNet34d, having a best score of 0.53 compared to 0.55 for the other two, it did slightly worse on the public LB: 0.46 compared to 0.42 (for EfficientNetB0) and 0.45 (for ResNet34d).
\nAt this juncture, my approach is to create a weighted ensembler from the three models (ResNet34d, EffNetB0 and EffNetB1), hoping to get a better score than 0.42 on the public LB.
\nIf you want to experiment with the weights of EfficientNetB1, you can visit this dataset.
", "messages": [ { "id": "2635260", "postDate": "02/04/2024 09:00:09", "content": "Hello kagglers,
\nAlongside EfficientNetB0 and ResNet34d, I have (successfully) trained EfficientNetB1, using the same batch size of 16. Since most participants use the first two models, I have been experimenting for the previous days with different models and optimizers: EfficientNetB1 (AdamW optimizer), EfficientNetB3 (RMSProp optimizer) and ResNet50 (SGD optimizer, with momentum). From the first epoch, I concluded that larger models, namely EfficientNetB3 and ResNet50 perform worse compared to lighter models, scoring around 1.25 on the Kullback Liebler divergence loss.
\nAs for EfficientNetB1, despite doing better on the CV than EfficientNetB0 and ResNet34d, having a best score of 0.53 compared to 0.55 for the other two, it did slightly worse on the public LB: 0.46 compared to 0.42 (for EfficientNetB0) and 0.45 (for ResNet34d).
\nAt this juncture, my approach is to create a weighted ensembler from the three models (ResNet34d, EffNetB0 and EffNetB1), hoping to get a better score than 0.42 on the public LB.
\nIf you want to experiment with the weights of EfficientNetB1, you can visit this dataset.
", "rawMarkdown": "Hello kagglers,\n\nAlongside EfficientNetB0 and ResNet34d, I have (successfully) trained [EfficientNetB1](https://www.kaggle.com/code/andreasbis/hms-train-efficientnetb1), using the same batch size of 16. Since most participants use the first two models, I have been experimenting for the previous days with different models and optimizers: EfficientNetB1 (AdamW optimizer), EfficientNetB3 (RMSProp optimizer) and ResNet50 (SGD optimizer, with momentum). From the first epoch, I concluded that larger models, namely EfficientNetB3 and ResNet50 perform worse compared to lighter models, scoring around 1.25 on the Kullback Liebler divergence loss. \n\nAs for EfficientNetB1, despite doing better on the CV than EfficientNetB0 and ResNet34d, having a best score of 0.53 compared to 0.55 for the other two, it did slightly worse on the public LB: 0.46 compared to 0.42 (for EfficientNetB0) and 0.45 (for ResNet34d).\n\nAt this juncture, my approach is to create a weighted ensembler from the three models (ResNet34d, EffNetB0 and EffNetB1), hoping to get a better score than 0.42 on the public LB. \n\nIf you want to experiment with the weights of EfficientNetB1, you can visit [this dataset](https://www.kaggle.com/datasets/andreasbis/efficientnetb1-for-eeg-classification-weights).", "votes": null }, { "id": "2635421", "postDate": "02/04/2024 11:45:56", "content": "I think this is the effect of ensembling, you are using only one model for your CV and the average of 5 for your LB.
\nThere is a variation ('underfitting') for a smaller architecture which can give a nice boost when ensembled, a bigger architecture tends to overfit very quickly
\nIt might be better to work with effnetb0 for now and optimize your hyperparameters or the way you process the data
Indeed, a weighted ensembler of the three models (ResNet34d, EffNetB0, EffNetB1) did better than the inidividual models, scoring 0.41 on the public LB.
\nNonetheless, considering your place in the competition, how do you approach data preprocessing?
", "rawMarkdown": "Indeed, a weighted ensembler of the three models (ResNet34d, EffNetB0, EffNetB1) did better than the inidividual models, scoring 0.41 on the public LB. \n\nNonetheless, considering your place in the competition, how do you approach data preprocessing?", "votes": null }, { "id": "2635725", "postDate": "02/04/2024 15:07:42", "content": "There are a lot of things that can be done, for example try to create new spectrograms and see how that might affect your cv.
\nIf you are stuck, you can read the top solutions from the
\nprevious image classification competitions, that will give you some inspiration and new ideas to try.
My experience is that the larger the model, the faster it overfit, I tried a few things to try to reduce the overfit, the only one that had a good effect was reducing the batch, but when doing this the training became slower.
", "rawMarkdown": "My experience is that the larger the model, the faster it overfit, I tried a few things to try to reduce the overfit, the only one that had a good effect was reducing the batch, but when doing this the training became slower.", "votes": null }, { "id": "2635987", "postDate": "02/04/2024 18:18:20", "content": "What batch sizes are you using?
", "rawMarkdown": "What batch sizes are you using?", "votes": null }, { "id": "2638671", "postDate": "02/06/2024 12:30:50", "content": "The best result in pythorch model was 16
", "rawMarkdown": "The best result in pythorch model was 16", "votes": null }, { "id": "2638905", "postDate": "02/06/2024 14:51:31", "content": "Depends also on the optimizer, e.g. Adam is known to be more batch-size over-fit sensitive
", "rawMarkdown": "Depends also on the optimizer, e.g. Adam is known to be more batch-size over-fit sensitive", "votes": null } ], "comments": [ { "id": 2635421, "author_name": "ahmedelfazouan", "author_url": "", "post_date": "02/04/2024 11:45:56", "content": "I think this is the effect of ensembling, you are using only one model for your CV and the average of 5 for your LB.
\nThere is a variation ('underfitting') for a smaller architecture which can give a nice boost when ensembled, a bigger architecture tends to overfit very quickly
\nIt might be better to work with effnetb0 for now and optimize your hyperparameters or the way you process the data
Indeed, a weighted ensembler of the three models (ResNet34d, EffNetB0, EffNetB1) did better than the inidividual models, scoring 0.41 on the public LB.
\nNonetheless, considering your place in the competition, how do you approach data preprocessing?
", "votes": null, "replies": [ { "id": 2635725, "author_name": "ahmedelfazouan", "author_url": "", "post_date": "02/04/2024 15:07:42", "content": "There are a lot of things that can be done, for example try to create new spectrograms and see how that might affect your cv.
\nIf you are stuck, you can read the top solutions from the
\nprevious image classification competitions, that will give you some inspiration and new ideas to try.
My experience is that the larger the model, the faster it overfit, I tried a few things to try to reduce the overfit, the only one that had a good effect was reducing the batch, but when doing this the training became slower.
", "votes": null, "replies": [ { "id": 2635987, "author_name": "andreasbis", "author_url": "", "post_date": "02/04/2024 18:18:20", "content": "What batch sizes are you using?
", "votes": null, "replies": [ { "id": 2638671, "author_name": "rafaelzimmermann1", "author_url": "", "post_date": "02/06/2024 12:30:50", "content": "The best result in pythorch model was 16
", "votes": null, "replies": [] } ] }, { "id": 2638905, "author_name": "danielwicz", "author_url": "", "post_date": "02/06/2024 14:51:31", "content": "Depends also on the optimizer, e.g. Adam is known to be more batch-size over-fit sensitive
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2635260": "Hello kagglers,\n\nAlongside EfficientNetB0 and ResNet34d, I have (successfully) trained [EfficientNetB1](https://www.kaggle.com/code/andreasbis/hms-train-efficientnetb1), using the same batch size of 16. Since most participants use the first two models, I have been experimenting for the previous days with different models and optimizers: EfficientNetB1 (AdamW optimizer), EfficientNetB3 (RMSProp optimizer) and ResNet50 (SGD optimizer, with momentum). From the first epoch, I concluded that larger models, namely EfficientNetB3 and ResNet50 perform worse compared to lighter models, scoring around 1.25 on the Kullback Liebler divergence loss. \n\nAs for EfficientNetB1, despite doing better on the CV than EfficientNetB0 and ResNet34d, having a best score of 0.53 compared to 0.55 for the other two, it did slightly worse on the public LB: 0.46 compared to 0.42 (for EfficientNetB0) and 0.45 (for ResNet34d).\n\nAt this juncture, my approach is to create a weighted ensembler from the three models (ResNet34d, EffNetB0 and EffNetB1), hoping to get a better score than 0.42 on the public LB. \n\nIf you want to experiment with the weights of EfficientNetB1, you can visit [this dataset](https://www.kaggle.com/datasets/andreasbis/efficientnetb1-for-eeg-classification-weights).", "2635421": "I think this is the effect of ensembling, you are using only one model for your CV and the average of 5 for your LB.\nThere is a variation ('underfitting') for a smaller architecture which can give a nice boost when ensembled, a bigger architecture tends to overfit very quickly\nIt might be better to work with effnetb0 for now and optimize your hyperparameters or the way you process the data", "2635644": "Indeed, a weighted ensembler of the three models (ResNet34d, EffNetB0, EffNetB1) did better than the inidividual models, scoring 0.41 on the public LB. \n\nNonetheless, considering your place in the competition, how do you approach data preprocessing?", "2635725": "There are a lot of things that can be done, for example try to create new spectrograms and see how that might affect your cv.\nIf you are stuck, you can read the top solutions from the \nprevious image classification competitions, that will give you some inspiration and new ideas to try.", "2635909": "My experience is that the larger the model, the faster it overfit, I tried a few things to try to reduce the overfit, the only one that had a good effect was reducing the batch, but when doing this the training became slower.", "2635987": "What batch sizes are you using?", "2638671": "The best result in pythorch model was 16", "2638905": "Depends also on the optimizer, e.g. Adam is known to be more batch-size over-fit sensitive" }, "source": "meta" }