{ "id": 245841, "title": "(EfficientNet vs ResNet)πŸ”₯ Baseline Architecture Study πŸ₯Ό (Weights Provided)", "url": "/competitions/siim-covid19-detection/discussion/245841", "author_name": "", "post_date": "2021-06-12T14:55:24.085822800Z", "votes": 36, "comment_count": 6, "views": 0, "content": "

\"\"

\n

Link to the Weights and Biases Dashboard.

\n

The Model Weights for EfficientNet and ResNet trained on 512 x 512 images is available in the SIIM-COVID19 Baseline Model Weights Dataset

\n

Results πŸ“‹

\n

The following table contains the metrics averaged across 5 (GroupKFold) runs trained for 20 epochs.

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Namebest_val_lossval_lossloss
EfficientNetB4-512-Baseline1.22782418727874761.4012400627136230.28386542797088626
EfficientNetB6-512-Baseline1.22518880367279051.442957568168640.3071795642375946
EfficientNetB7-512-Baseline1.23322021961212161.43624005317687970.3244903475046158
EfficientNetB5-512-Baseline1.22872149944305421.41365578174591060.33519356995821
ResNet50V2-512-Baseline0.93187031745910640.97405095100402840.6614007711410522
ResNet101V2-512-Baseline0.9240637660026550.93258529901504520.7936959743499756
ResNet152V2-512-Baseline0.93843325376510610.9562308907508850.8386225461959839
", "messages": [ { "id": "1346706", "postDate": "06/12/2021 14:55:24", "content": "

\"\"

\n

Link to the Weights and Biases Dashboard.

\n

The Model Weights for EfficientNet and ResNet trained on 512 x 512 images is available in the SIIM-COVID19 Baseline Model Weights Dataset

\n

Results πŸ“‹

\n

The following table contains the metrics averaged across 5 (GroupKFold) runs trained for 20 epochs.

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Namebest_val_lossval_lossloss
EfficientNetB4-512-Baseline1.22782418727874761.4012400627136230.28386542797088626
EfficientNetB6-512-Baseline1.22518880367279051.442957568168640.3071795642375946
EfficientNetB7-512-Baseline1.23322021961212161.43624005317687970.3244903475046158
EfficientNetB5-512-Baseline1.22872149944305421.41365578174591060.33519356995821
ResNet50V2-512-Baseline0.93187031745910640.97405095100402840.6614007711410522
ResNet101V2-512-Baseline0.9240637660026550.93258529901504520.7936959743499756
ResNet152V2-512-Baseline0.93843325376510610.9562308907508850.8386225461959839
", "rawMarkdown": "![](https://github.com/SauravMaheshkar/siim-covid19/blob/main/assets/SIIM-COVID19%20Github%20Banner.png?raw=true)\n\nLink to the [**Weights and Biases Dashboard**](https://wandb.ai/sauravmaheshkar/siim-covid19).\n\nThe Model Weights for EfficientNet and ResNet trained on [512 x 512 images](https://www.kaggle.com/h053473666/siimcovid19-512-img-png-600-study-png) is available in the [**SIIM-COVID19 Baseline Model Weights**](https://www.kaggle.com/sauravmaheshkar/siimcovid19model-weights) Dataset\n\n# Results πŸ“‹\n\nThe following table contains the metrics averaged across **5 (GroupKFold)** runs trained for **20 epochs**. \n\n|Name |best_val_loss |val_loss |loss |\n|---------------------------|------------------|------------------|-------------------|\n|EfficientNetB4-512-Baseline|1.2278241872787476|1.401240062713623 |**0.28386542797088626**|\n|EfficientNetB6-512-Baseline|1.2251888036727905|1.44295756816864 |0.3071795642375946 |\n|EfficientNetB7-512-Baseline|1.2332202196121216|1.4362400531768797|0.3244903475046158 |\n|EfficientNetB5-512-Baseline|1.2287214994430542|1.4136557817459106|0.33519356995821 |\n|ResNet50V2-512-Baseline |0.9318703174591064|0.9740509510040284|0.6614007711410522 |\n|ResNet101V2-512-Baseline |0.924063766002655 |**0.9325852990150452**|0.7936959743499756 |\n|ResNet152V2-512-Baseline |0.9384332537651061|0.956230890750885 |0.8386225461959839 |", "votes": null }, { "id": "1347173", "postDate": "06/13/2021 03:55:33", "content": "

Thanks. Resnet seems to perform consistently better on validation set

", "rawMarkdown": "Thanks. Resnet seems to perform consistently better on validation set", "votes": null }, { "id": "1354304", "postDate": "06/17/2021 14:12:53", "content": "

So my guess is that Resnet actually generalizes more than EfficientNet ?
\nAlso, the spread of the losses of the EfficientNet results worries me… Could this demonstrate that EfficientNet as a tendency to overfit?

\n

Finally, from what I understand, the loss term is the training loss, right? In this case, are we able to gather the result of efficientNet at Epoch 6-7 where the spread is similar to the one of Resnet to see if EfficientNet is just overfitting due to too many epochs?

", "rawMarkdown": "So my guess is that Resnet actually generalizes more than EfficientNet ?\nAlso, the spread of the losses of the EfficientNet results worries me... Could this demonstrate that EfficientNet as a tendency to overfit?\n\nFinally, from what I understand, the loss term is the training loss, right? In this case, are we able to gather the result of efficientNet at Epoch 6-7 where the spread is similar to the one of Resnet to see if EfficientNet is just overfitting due to too many epochs?", "votes": null }, { "id": "1356374", "postDate": "06/19/2021 01:18:31", "content": "

Thanks for sharing your experiments. In my setup and also best public kernel effnets seem to perform better. But it is also known that resnets perform well on this type of medical data. It might be better to share results on these results on the local validation mAP metric. Although, CE correlates with mAP to some extend its not a perfect proxy. What is the loss here in your case?

", "rawMarkdown": "Thanks for sharing your experiments. In my setup and also best public kernel effnets seem to perform better. But it is also known that resnets perform well on this type of medical data. It might be better to share results on these results on the local validation mAP metric. Although, CE correlates with mAP to some extend its not a perfect proxy. What is the loss here in your case?", "votes": null }, { "id": "1366782", "postDate": "06/27/2021 07:32:12", "content": "

Thanks for your feedback. All experiments were performed using Adam, categorical cross entropy and multi-label AUC.

", "rawMarkdown": "Thanks for your feedback. All experiments were performed using Adam, categorical cross entropy and multi-label AUC.", "votes": null }, { "id": "1370999", "postDate": "06/30/2021 14:51:52", "content": "

Thank you for sharing the experiment! Resent seems better.

", "rawMarkdown": "Thank you for sharing the experiment! Resent seems better.", "votes": null }, { "id": "1384831", "postDate": "07/12/2021 08:48:52", "content": "

yup, Resnet seems to perform better on the validation set

", "rawMarkdown": "yup, Resnet seems to perform better on the validation set", "votes": null } ], "comments": [ { "id": 1347173, "author_name": "artnotintelligence", "author_url": "", "post_date": "06/13/2021 03:55:33", "content": "

Thanks. Resnet seems to perform consistently better on validation set

", "votes": null, "replies": [] }, { "id": 1354304, "author_name": "peterbonnesoeur", "author_url": "", "post_date": "06/17/2021 14:12:53", "content": "

So my guess is that Resnet actually generalizes more than EfficientNet ?
\nAlso, the spread of the losses of the EfficientNet results worries me… Could this demonstrate that EfficientNet as a tendency to overfit?

\n

Finally, from what I understand, the loss term is the training loss, right? In this case, are we able to gather the result of efficientNet at Epoch 6-7 where the spread is similar to the one of Resnet to see if EfficientNet is just overfitting due to too many epochs?

", "votes": null, "replies": [] }, { "id": 1356374, "author_name": "keremt", "author_url": "", "post_date": "06/19/2021 01:18:31", "content": "

Thanks for sharing your experiments. In my setup and also best public kernel effnets seem to perform better. But it is also known that resnets perform well on this type of medical data. It might be better to share results on these results on the local validation mAP metric. Although, CE correlates with mAP to some extend its not a perfect proxy. What is the loss here in your case?

", "votes": null, "replies": [ { "id": 1366782, "author_name": "sauravmaheshkar", "author_url": "", "post_date": "06/27/2021 07:32:12", "content": "

Thanks for your feedback. All experiments were performed using Adam, categorical cross entropy and multi-label AUC.

", "votes": null, "replies": [] } ] }, { "id": 1370999, "author_name": "liushuzhi", "author_url": "", "post_date": "06/30/2021 14:51:52", "content": "

Thank you for sharing the experiment! Resent seems better.

", "votes": null, "replies": [ { "id": 1384831, "author_name": "sauravmaheshkar", "author_url": "", "post_date": "07/12/2021 08:48:52", "content": "

yup, Resnet seems to perform better on the validation set

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1346706": "![](https://github.com/SauravMaheshkar/siim-covid19/blob/main/assets/SIIM-COVID19%20Github%20Banner.png?raw=true)\n\nLink to the [**Weights and Biases Dashboard**](https://wandb.ai/sauravmaheshkar/siim-covid19).\n\nThe Model Weights for EfficientNet and ResNet trained on [512 x 512 images](https://www.kaggle.com/h053473666/siimcovid19-512-img-png-600-study-png) is available in the [**SIIM-COVID19 Baseline Model Weights**](https://www.kaggle.com/sauravmaheshkar/siimcovid19model-weights) Dataset\n\n# Results πŸ“‹\n\nThe following table contains the metrics averaged across **5 (GroupKFold)** runs trained for **20 epochs**. \n\n|Name |best_val_loss |val_loss |loss |\n|---------------------------|------------------|------------------|-------------------|\n|EfficientNetB4-512-Baseline|1.2278241872787476|1.401240062713623 |**0.28386542797088626**|\n|EfficientNetB6-512-Baseline|1.2251888036727905|1.44295756816864 |0.3071795642375946 |\n|EfficientNetB7-512-Baseline|1.2332202196121216|1.4362400531768797|0.3244903475046158 |\n|EfficientNetB5-512-Baseline|1.2287214994430542|1.4136557817459106|0.33519356995821 |\n|ResNet50V2-512-Baseline |0.9318703174591064|0.9740509510040284|0.6614007711410522 |\n|ResNet101V2-512-Baseline |0.924063766002655 |**0.9325852990150452**|0.7936959743499756 |\n|ResNet152V2-512-Baseline |0.9384332537651061|0.956230890750885 |0.8386225461959839 |", "1347173": "Thanks. Resnet seems to perform consistently better on validation set", "1354304": "So my guess is that Resnet actually generalizes more than EfficientNet ?\nAlso, the spread of the losses of the EfficientNet results worries me... Could this demonstrate that EfficientNet as a tendency to overfit?\n\nFinally, from what I understand, the loss term is the training loss, right? In this case, are we able to gather the result of efficientNet at Epoch 6-7 where the spread is similar to the one of Resnet to see if EfficientNet is just overfitting due to too many epochs?", "1356374": "Thanks for sharing your experiments. In my setup and also best public kernel effnets seem to perform better. But it is also known that resnets perform well on this type of medical data. It might be better to share results on these results on the local validation mAP metric. Although, CE correlates with mAP to some extend its not a perfect proxy. What is the loss here in your case?", "1366782": "Thanks for your feedback. All experiments were performed using Adam, categorical cross entropy and multi-label AUC.", "1370999": "Thank you for sharing the experiment! Resent seems better.", "1384831": "yup, Resnet seems to perform better on the validation set" }, "source": "meta" }