{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"markdown","source":"# Cassava Leaf Disease Classification\n\nChallenge is [HERE](https://www.kaggle.com/c/cassava-leaf-disease-classification)\n\n## Data\n\n* Train Image Resolution : (500 ~ 800)x(500 ~ 800)\n* Test Image Resolution : 800x600\n* Total 5 classes (4 for the diseases, 1 for healthy)\n* There are lots of noisy labels (both train & test)\n\n### Label Distribution\n\n| class | label | cleaned label |\n| :---: | :---: | :---: |\n| 0 | 1492 | 1381 |\n| 1 | 3476 | 3389 |\n| 2 | 3017 | 2836 |\n| 3 | 15462 | 15905 |\n| 4 | 2890 | 2826 |\n\n* cleaned 19 + 20 datasets : [here](https://www.kaggle.com/kozistr/leaf-disease-cleaned)\n\n% cleaned label : pseudo label (got from my best lb models, 0.905)\n\n## To-Do\n\n1. clean the whole train dataset\n\n## Works\n\n1. extra data (using 2019 + 2020 data)\n2. TTA (n_iter = 4 is best)\n3. smooth cross entropy loss (maybe...?)\n4. heavy augmentations (brightness, contrastive, flip, etc...) (?)\n5. `CutMix` + `FMix` (?)\n\n## Not Works\n\n1. `SnapMix` Augmentations\n2. too small or big backbone (ResNeXt50~, EffNet-B5~ )\n3. bi-tempered loss\n4. taylor category cross entropy loss\n5. some augmentations (e.g. ChannelDropout, GridDistortion, RGDShift)\n6. tuning on 2020 validation dataset\n\n## Local Performance\n\n### ResNeSt50\n\n| fold | res | fmix | cutmix | snapmix | loss | epochs | lr | lr scheduler | optimizer | dataset | cv |\n| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| 0/5 | 512 | x | x | o | focal cosine | 20 | 1e-4 | cosine | AdamW | 20 | 0.88598 |\n| 1/5 | 512 | x | x | o | focal cosine | 20 | 1e-4 | cosine | AdamW | 20 | 0.88037 |\n| 2/5 | 512 | x | x | o | focal cosine | 20 | 1e-4 | cosine | AdamW | 20 | 0.88268 |\n| 3/5 | 512 | x | x | o | focal cosine | 20 | 1e-4 | cosine | AdamW | 20 | 0.89297 |\n| 4/5 | 512 | x | x | o | focal cosine | 20 | 1e-4 | cosine | AdamW | 20 | 0.89133 |\n| | | | | | | | | | | | | |\n\n### ResNeSt50-fast-4s2x40d\n\n| fold | res | fmix | cutmix | snapmix | loss | epochs | lr | lr scheduler | optimizer | dataset | cv |\n| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| 0/5 | 512 | o | o | x | scce | 20 | 1e-4 | cosine | AdamW | 20 | 0.88318 |\n| 1/5 | 512 | o | o | x | scce | 20 | 1e-4 | cosine | AdamW | 20 | 0.89136 |\n| 2/5 | 512 | o | o | x | scce | 20 | 1e-4 | cosine | AdamW | 20 | 0.88666 |\n| 3/5 | 512 | o | o | x | scce | 20 | 1e-4 | cosine | AdamW | 20 | 0.89437 |\n| 4/5 | 512 | o | o | x | scce | 20 | 1e-4 | cosine | AdamW | 20 | ??? |\n| | | | | | | | | | | | |\n| 0/5 | 512 | x | x | x | bi-tempered | 20 | 1e-4 | cosine | AdamW | 19 + 20 | 0.88800 |\n| 1/5 | 512 | x | x | x | bi-tempered | 20 | 1e-4 | cosine | AdamW | 19 + 20 | 0.88591 |\n| 2/5 | 512 | x | x | x | bi-tempered | 20 | 1e-4 | cosine | AdamW | 19 + 20 | 0.89140 |\n| 3/5 | 512 | x | x | x | bi-tempered | 20 | 1e-4 | cosine | AdamW | 19 + 20 | 0.89026 |\n| 4/5 | 512 | x | x | x | bi-tempered | 20 | 1e-4 | cosine | AdamW | 19 + 20 | 0.88874 |\n| | | | | | | | | | | | |\n| 0/5 | 512 | x | x | x | focal cosine | 20 | 1e-4 | cosine | RAdam | 19 + 20 | 0.88990 |\n| 1/5 | 512 | x | x | x | focal cosine | 20 | 1e-4 | cosine | RAdam | 19 + 20 | 0.88610 |\n| 2/5 | 512 | x | x | x | focal cosine | 20 | 1e-4 | cosine | RAdam | 19 + 20 | 0.89254 |\n| 3/5 | 512 | x | x | x | focal cosine | 20 | 1e-4 | cosine | RAdam | 19 + 20 | 0.88950 |\n| 4/5 | 512 | x | x | x | focal cosine | 20 | 1e-4 | cosine | RAdam | 19 + 20 | 0.89216 |\n| | | | | | | | | | | | |\n| 0/5 | 512 | o | o | x | scce .2 | 15 | 1e-4 | cosine | AdamP | 19 + 20 | 0.89863 |\n| 1/5 | 512 | o | o | x | scce .2 | 15 | 1e-4 | cosine | AdamP | 19 + 20 | 0.89617 |\n| 2/5 | 512 | o | o | x | scce .2 | 15 | 1e-4 | cosine | AdamP | 19 + 20 | 0.89596 |\n| 3/5 | 512 | o | o | x | scce .2 | 15 | 1e-4 | cosine | AdamP | 19 + 20 | 0.90184 |\n| 4/5 | 512 | o | o | x | scce .2 | 15 | 1e-4 | cosine | AdamP | 19 + 20 | 0.89956 |\n| | | | | | | | | | | | |\n\n### EfficientNet-B3\n\n| w | fold | res | fmix | cutmix | snapmix | loss | epochs | lr | lr scheduler | optimizer | dataset | cv |\n| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| ns | 0/5 | 512 | x | o | x | scce .2 | 20 | 1e-4 | cosine | RAdam | 19 + 20 | 0.89066 |\n| ns | 1/5 | 512 | x | o | x | scce .2 | 20 | 1e-4 | cosine | RAdam | 19 + 20 | 0.89560 |\n| ns | 2/5 | 512 | x | o | x | scce .2 | 20 | 1e-4 | cosine | RAdam | 19 + 20 | 0.89539 |\n| ns | 3/5 | 512 | x | o | x | scce .2 | 20 | 1e-4 | cosine | RAdam | 19 + 20 | 0.89368 |\n| ns | 4/5 | 512 | x | o | x | scce .2 | 20 | 1e-4 | cosine | RAdam | 19 + 20 | 0.89349 |\n| | | | | | | | | | | | | |\n\n### EfficientNet-B4\n\n| w | fold | res | fmix | cutmix | snapmix | loss | epochs | lr | lr scheduler | optimizer | dataset | cv |\n| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| ns | 0/5 | 512 | x | x | x | focal cosine | 25 | 1e-4 | cosine | AdamW | 20 | 0.88879 |\n| | | | | | | | | | | | | |\n| ns | 0/5 | 512 | x | x | x | focal cosine | 20 | 1e-4 | cosine | RAdam | 19 + 20 | 0.88876 |\n| ns | 1/5 | 512 | x | x | x | focal cosine | 20 | 1e-4 | cosine | RAdam | 19 + 20 | 0.89123 |\n| ns | 2/5 | 512 | x | x | x | focal cosine | 20 | 1e-4 | cosine | RAdam | 19 + 20 | 0.89330 |\n| ns | 3/5 | 512 | x | x | x | focal cosine | 20 | 1e-4 | cosine | RAdam | 19 + 20 | 0.89216 |\n| ns | 4/5 | 512 | x | x | x | focal cosine | 20 | 1e-4 | cosine | RAdam | 19 + 20 | 0.88798 |\n| | | | | | | | | | | | | |\n| ns | 0/5 | 512 | o | o | x | scce .2 | 10 | 1e-4 | cosine | AdamP | 19 + 20 | 0.88573 |\n| ns | 1/5 | 512 | o | o | x | scce .2 | 10 | 1e-4 | cosine | AdamP | 19 + 20 | 0.88838 |\n| ns | 2/5 | 512 | o | o | x | scce .2 | 10 | 1e-4 | cosine | AdamP | 19 + 20 | 0.89235 |\n| ns | 3/5 | 512 | o | o | x | scce .2 | 10 | 1e-4 | cosine | AdamP | 19 + 20 | 0.89444 |\n| ns | 4/5 | 512 | o | o | x | scce .2 | 10 | 1e-4 | cosine | AdamP | 19 + 20 | 0.89216 |\n| | | | | | | | | | | | | |\n| ns | 0/5 | 512 | o | o | x | scce .1 | 15 | 1e-4 | cosine | AdamP | pseudo | 0.95539 |\n| ns | 1/5 | 512 | o | o | x | scce .1 | 15 | 1e-4 | cosine | AdamP | pseudo | 0.96071 |\n| ns | 2/5 | 512 | o | o | x | scce .1 | 15 | 1e-4 | cosine | AdamP | pseudo | 0.95671 |\n| ns | 3/5 | 512 | o | o | x | scce .1 | 15 | 1e-4 | cosine | AdamP | pseudo | 0.95842 |\n| ns | 4/5 | 512 | o | o | x | scce .1 | 15 | 1e-4 | cosine | AdamP | pseudo | 0.95633 |\n| | | | | | | | | | | | | |\n\n### ViT-L\n\n| p | fold | res | fmix | cutmix | snapmix | loss | epochs | lr | lr scheduler | optimizer | dataset | cv |\n| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| 16 | 0/5 | 384 | x | x | x | bi-tempered | 7 | 1e-4 | cosine anl | AdamW | 19 + 20 | 0.88453 |\n| 16 | 1/5 | 384 | x | x | x | bi-tempered | 7 | 1e-4 | cosine anl | AdamW | 19 + 20 | 0.89024 |\n| 16 | 2/5 | 384 | x | x | x | bi-tempered | 7 | 1e-4 | cosine anl | AdamW | 19 + 20 | 0.89024 |\n| 16 | 3/5 | 384 | x | x | x | bi-tempered | 7 | 1e-4 | cosine anl | AdamW | 19 + 20 | 0.89005 |\n| 16 | 4/5 | 384 | x | x | x | bi-tempered | 7 | 1e-4 | cosine anl | AdamW | 19 + 20 | 0.89043 |\n| | | | | | | | | | | | | |\n\n### DeiT-B\n\n| p | fold | res | fmix | cutmix | snapmix | loss | epochs | lr | lr scheduler | optimizer | dataset | cv |\n| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| 16 | 0/5 | 384 | x | x | x | scce .2 | 10 | 1e-4 | cosine | AdamP | 19 + 20 | 0.88724 |\n| 16 | 1/5 | 384 | x | x | x | scce .2 | 10 | 1e-4 | cosine | AdamP | 19 + 20 | 0.88990 |\n| 16 | 2/5 | 384 | x | x | x | scce .2 | 10 | 1e-4 | cosine | AdamP | 19 + 20 | 0.88893 |\n| 16 | 3/5 | 384 | x | x | x | scce .2 | 10 | 1e-4 | cosine | AdamP | 19 + 20 | 0.88969 |\n| 16 | 4/5 | 384 | x | x | x | scce .2 | 10 | 1e-4 | cosine | AdamP | 19 + 20 | 0.89235 |\n| | | | | | | | | | | | | |\n| 16 | 0/5 | 384 | o | o | x | scce .2 | 10 | 1e-4 | cosine | AdamP | 19 + 20 | 0.88838 |\n| 16 | 1/5 | 384 | o | o | x | scce .2 | 10 | 1e-4 | cosine | AdamP | 19 + 20 | 0.88762 |\n| 16 | 2/5 | 384 | o | o | x | scce .2 | 10 | 1e-4 | cosine | AdamP | 19 + 20 | 0.88570 |\n| 16 | 3/5 | 384 | o | o | x | scce .2 | 10 | 1e-4 | cosine | AdamP | 19 + 20 | 0.89026 |\n| 16 | 4/5 | 384 | o | o | x | scce .2 | 10 | 1e-4 | cosine | AdamP | 19 + 20 | x |\n| | | | | | | | | | | | | |\n\n## LB Performance\n\n1. ResNeSt50\n2. **ResNeSt50-fast-4s2x40d**\n3. **EfficientNet-B4**\n4. ResNeXt50-32x4d (public)\n5. **EfficientNet-B3**\n6. ViT-L/16\n7. DeiT-B/16\n\n### Single Models\n\n| no | arch | folds | res | data | n_tta | lb |\n| :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| 0  | ResNeSt50 fc                              | 5 | 512 | 20      | 3 | 0.892 |\n| 1  | ResNeSt50-fast-4s2x40d cutmix + fmix scce | 5 | 512 | 20      | 3 | 0.898 |\n| 2  | 1-(1, 3, 4 folds)                         | 3 | 512 | 20      | 3 | 0.894 |\n| 3  | EffNet-B4 fc                              | 5 | 512 | 19 + 20 | 5 | 0.900 |\n| 4  | ResNeSt50-fast-4s2x40d bi-tempered        | 5 | 512 | 19 + 20 | 5 | 0.895 |\n| 5  | ResNeSt50-fast-4s2x40d fc                 | 5 | 512 | 19 + 20 | 5 | 0.900 |\n| 6  | ResNeXt50-32x4d cce                       | 5 | 512 | 19 + 20 | 4 | 0.894 |\n| 7  | EffNet-B3 cutmix + scce                   | 5 | 512 | 19 + 20 | 4 | 0.896 |\n| 8  | ViT-L/16 bi-tempered                      | 5 | 384 | 19 + 20 | 4 | 0.882 |\n| 9  | DeiT-B/16 scce                            | 5 | 384 | 19 + 20 | 4 | 0.894 |\n| 10 | DeiT-B/16 cutmix + fmix scce              | 5 | 384 | 19 + 20 | 4 |   x   |\n| 11 | EffNet-B4 cutmix + fmix scce              | 5 | 512 | 19 + 20 | 4 | 0.900 |\n| 12 | ResNeSt50-fast-4s2x40d cutmix + fmix scce | 5 | 512 | 19 + 20 | 4 | 0.898 |\n| 13 | EffNet-B4 cutmix + fmix scce pseudo       | 5 | 512 | 19 + 20 | 4 | 0.898 |\n\n### Ensembles\n\n| no | arch | folds | res | data | n_tta | lb |\n| :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| 0  | No 0 + No 1               | 5 + 5         | 512      | 20      | 3    | 0.896 |\n| 1  | No 1 + No 3               | 5 + 5         | 512      | 19 + 20 | 3    | 0.903 |\n| 2  | No 1 + No 3               | 5 + 5         | 512      | 19 + 20 | 5    | 0.905 |\n| 3  | No 1 + No 3               | 5 + 5         | 512      | 19 + 20 | 7    | 0.903 |\n| 4  | No 3 + No 4               | 5 + 5         | 512      | 19 + 20 | 5    | 0.901 |\n| 5  | No 1 + No 3 + No 4        | 5 + 5 + 5     | 512      | 19 + 20 | 5    | 0.902 |\n| 6  | No 3 + No 5               | 5 + 5         | 512      | 19 + 20 | 5    | 0.904 |\n| 7  | No 3 + No 5               | 5 + 5         | 512      | 19 + 20 | 7    | 0.904 |\n| 8  | No 1 + No 3 + No 5        | 5 + 5 + 5     | 512      | 19 + 20 | 5    | 0.905 |\n| 9  | No 1 + No 3 + No 5        | 5 + 5 + 5     | 512      | 19 + 20 | 4    | 0.906 |\n| 10 | No 3 + No 5 + No 6        | 5 + 5 + 5     | 512      | 19 + 20 | 4    | 0.902 |\n| 11 | No 1 + No 3 + No 5 + No 6 | 5 + 5 + 5 + 5 | 512      | 19 + 20 | 4    | 0.905 |\n| 12 | No 3 + No 5 + No 7        | 5 + 5 + 5     | 512      | 19 + 20 | 4    | 0.902 |\n| 13 | No 1 + No 3 + No 5 + No 7 | 5 + 5 + 5 + 5 | 512      | 19 + 20 | 4    | 0.903 |\n| 14 | No 1 + No 3 + No 5 + No 7 | 5 + 5 + 5 + 5 | 512      | 19 + 20 | 5    | 0.904 |\n| 15 | No 1 + No 3 + No 5 + No 8 | 5 + 5 + 5 + 5 | 512, 384 | 19 + 20 | 4, 2 | 0.905 |\n| 16 | No 1 + No 3 + No 5 + No 9 | 5 + 5 + 5 + 5 | 512, 384 | 19 + 20 | 4    | 0.903 |\n| 17 | No 3 + No 5 + No 9        | 5 + 5 + 5     | 512, 384 | 19 + 20 | 4    | 0.901 |\n| 18 | No 3 + No 5 + No 9 + No 7 | 5 + 5 + 5 + 5 | 512, 384 | 19 + 20 | 4    | 0.900 |\n| 19 | No 1 + No 3 + No 5 + No 9 | 5 + 5 + 5 + 4 | 512, 384 | 19 + 20 | 4    | 0.904 |\n\n| no | arch | folds | res | data | n_tta | lb |\n| :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| 20 | No 1 + No 5 + No 11                               | 5 + 5 + 5                 | 512 | 19 + 20 | 4 | 0.906 |\n| 21 | No 1 + No 3 + No 5 + No 11                        | 5 + 5 + 5 + 5             | 512 | 19 + 20 | 4 | **0.907** |\n| 22 | No 1 + No 3 + No 5 + No 7 + No 11                 | 5 + 5 + 5 + 5 + 5         | 512 | 19 + 20 | 4 | 0.905 |\n| 23 | No 3 + No 5 + No 7 + No 11                        | 5 + 5 + 5 + 5             | 512 | 19 + 20 | 4 | 0.903 |\n| 24 | No 3 + No 5 + No 11 + No 12                       | 5 + 5 + 5 + 5             | 512 | 19 + 20 | 4 | 0.905 |\n| 25 | No 1 + No 3 + No 5 + No 11 + No 12                | 5 + 5 + 5 + 5 + 5         | 512 | 19 + 20 | 4 | 0.907 |\n| 26 | No 3 + No 5 + No 7 + No 11 + No 12                | 5 + 5 + 5 + 5 + 5         | 512 | 19 + 20 | 4 | 0.904 |\n| 27 | No 1 + No 3 + No 5 + No 7 + No 11 + No 12         | 5 + 5 + 5 + 5 + 5 + 5     | 512 | 19 + 20 | 4 | 0.905 |\n| 28 | No 1 + No 3 + No 11 + No 12                       | 5 + 5 + 5 + 5             | 512 | 19 + 20 | 4 | 0.905 |\n| 29 | No 3 + No 5 + No 12 + No 13                       | 5 + 5 + 5 + 5             | 512 | 19 + 20 | 4 | 0.903 |\n| 30 | No 1 + No 3 + No 5 + No 7 + No 11 + No 13         | 5 + 5 + 5 + 5 + 5 + 5     | 512 | 19 + 20 | 4 | 0.905 |\n| 31 | No 3 + No 5 + No 7 + No 11 + No 12 + No 13        | 5 + 5 + 5 + 5 + 5 + 5     | 512 | 19 + 20 | 4 | 0.904 |\n| 32 | No 1 + No 3 + No 5 + No 11 + No 12 + No 13        | 5 + 5 + 5 + 5 + 5 + 5     | 512 | 19 + 20 | 4 | 0.905 |\n| 33 | No 1 + No 3 + No 5 + No 7 + No 11 + No 12 + No 13 | 5 + 5 + 5 + 5 + 5 + 5 + 5 | 512 | 19 + 20 | 4 | 0.905 |\n| 34 | No 1 + No 3 + No 5 + No 13                        | 5 + 5 + 5 + 5             | 512 | 19 + 20 | 4 | 0.902 |\n| 35 | No 1 + No 3 + No 5 + No 11 + No 13                | 5 + 5 + 5 + 5 + 5         | 512 | 19 + 20 | 4 | 0.905 |\n\n### Fold Selection\n\n* based on best LB scores *Ensembles 9*\n\n* model power : `RST50 cutmix + fmix scce > Effnet-B4 > RST50 fc`\n* However, several experiments show that the following model power doesn't proportionate with the ensemble weights.\n\n| folds | n_tta | lb |\n| :---: | :---: | :---: |\n| 0 + 1 + 2 | 4 | 0.905 |\n| 0 + 2 + 3 | 4 | 0.903 |\n| 0 + 2 + 4 | 4 | 0.903 |\n| 0 + 1 + 2 + 3 | 4 | 0.904 |\n| | | |\n| 4 + 5 + 5 | 4 | 0.903 |\n| 5 + 4 + 5 | 4 | 0.905 |\n| 5 + 5 + 4 | 4 | 0.904 |\n| | | |\n\n## CV Performance\n\n| arch | model | class 0 acc | class 1 acc | class 2 acc | class 3 acc | class 4 acc | macro f1 | weighted f1 | acc | \n| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| RST50 cf v1         | No 1  | 62.2654 | 80.1496 | 82.1346 | 95.8932 | 80.7266 | 80.5728 | 88.6871 | 88.6699 | \n| EffNet-b4 fc        | No 3  | 61.7962 | 82.5086 | 78.9857 | 97.5553 | 75.6401 | 80.5967 | 88.8362 | 89.0117 |\n| RST50 fc            | No 5  | 60.7239 | 83.4292 | 80.5767 | 97.2255 | 76.4014 | 80.7895 | 88.9890 | 89.1445 |\n| EffNet-b3 c         | No 7  | 68.1635 | 83.1991 | 81.1734 | 97.5100 | 74.0484 | 81.7483 | 89.3790 | 89.5129 |\n| ViT-L16 bi          | No 8  | 65.2815 | 82.7100 | 79.0852 | 97.2384 | 74.5675 | 80.7256 | 88.8202 | 88.9433 |\n| EffNet-b4 cf        | No 11 | 60.8579 | 85.4430 | 82.9632 | 96.8439 | 73.2180 | 80.7463 | 88.9672 | 89.1180 |\n| RST50 cf v2         | No 12 | 62.8016 | 85.8170 | 82.3003 | 97.4648 | 76.4706 | 82.0774 | 89.7713 | 89.9229 |\n| EffNet-b4 cf pseudo | No 13 | 63.5389 | 84.5800 | 82.3334 | 97.2707 | 77.4740 | 82.1038 | 89.6712 | 89.8014 |\n|  |  |  |  |  |  |  |  |  |  |\n\n| model | class 0 acc | class 1 acc | class 2 acc | class 3 acc | class 4 acc | macro f1 | weighted f1 | acc | \n| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| Ensembles  9       | 64.8794 | 84.1197 | 81.7700 | 97.8140 | 78.8235 | 82.6708 | 90.0947 | 90.2191 | \n| Ensembles  9 tuned | 64.5442 | 84.3498 | 81.7700 | 97.8399 | 78.7889 | 82.6716 | 90.1123 | 90.2419 |\n| Ensembles 11       | 66.2869 | 84.7526 | 81.7368 | 97.9110 | 77.9239 | 82.9196 | 90.2023 | 90.3368 |\n| Ensembles 11 tuned | 66.5550 | 84.7814 | 82.2672 | 97.8787 | 78.5121 | 83.1136 | 90.3413 | 90.4621 |\n| Ensembles 15 tuned | 65.4826 | 84.2060 | 82.0351 | 97.8011 | 79.2734 | 82.8680 | 90.2281 | 90.3368 |\n\n* Ensembles  9 weights : `[0.33547759, 0.30181755, 0.28914882]`\n* Ensembles 11 weights : `[0.32264375, 0.19517635, 0.10858799, 0.33353971]`\n* Ensembles 15 weights : `[0.15093364, 0.15979473, 0.38676311, 0.26914147]`\n* Ensembles 20 weights : `[0.31301767, 0.29456512, 0.31728454]`\n\n\n| model | class 0 acc | class 1 acc | class 2 acc | class 3 acc | class 4 acc | macro f1 | weighted f1 | acc | \n| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| Ensembles 20       | 63.6729 | 85.1266 | 82.6318 | 97.5553 | 78.5813 | 82.6335 | 90.0737 | 90.2039 |\n| Ensembles 20 tuned | 63.6059 | 85.1841 | 82.6318 | 97.5553 | 78.7543 | 82.6726 | 90.0974 | 90.2267 |\n| Ensembles 21       | 64.3432 | 85.2417 | 82.1014 | 97.8011 | 78.1315 | 82.7324 | 90.1517 | 90.2912 |\n| Ensembles 21 tuned | 64.8794 | 85.1266 | 81.9357 | 97.8463 | 78.6851 | 82.9210 | 90.2426 | 90.3748 |\n| Ensembles 22       | 65.8177 | 85.3280 | 82.3003 | 97.8593 | 77.8547 | 83.0168 | 90.2767 | 90.4127 |\n| Ensembles 22 tuned | 66.9571 | 85.2417 | 82.2340 | 97.8851 | 78.3737 | 83.3007 | 90.4056 | 90.5304 | \n| Ensembles 23       | 66.2869 | 84.9252 | 81.7368 | 97.8399 | 76.0900 | 82.5127 | 89.9713 | 90.1166 | \n| Ensembles 23 tuned | 66.8231 | 84.4937 | 81.8694 | 97.8916 | 76.5052 | 82.6741 | 90.0376 | 90.1811 | \n| Ensembles 24       | 64.0751 | 85.3855 | 81.9357 | 97.8399 | 77.0242 | 82.4735 | 90.0215 | 90.1773 | \n| Ensembles 24 tuned | 64.4102 | 85.5869 | 82.2009 | 97.8657 | 77.6471 | 82.7948 | 90.1814 | 90.3368 | \n| Ensembles 25       | 64.5442 | 85.7307 | 82.1677 | 97.8269 | 78.5121 | 82.9857 | 90.2875 | 90.4317 | \n| Ensembles 25 tuned | 64.8794 | 85.9609 | 83.0958 | 97.7429 | 79.4118 | 83.3772 | 90.5082 | 90.6367 | \n| Ensembles 27       | 65.4155 | 85.4718 | 82.1677 | 97.8787 | 78.4775 | 83.0997 | 90.3339 | 90.4735 | \n| Ensembles 27 tuned | 66.3539 | 85.7595 | 82.8969 | 97.7687 | 79.2042 | 83.5287 | 90.5402 | **90.6633** | \n| Ensembles 28       | 64.5442 | 86.0184 | 82.3334 | 97.8011 | 78.7543 | 83.0754 | 90.3645 | 90.5001 |\n| Ensembles 28 tuned | 64.9464 | 85.9609 | 82.9632 | 97.7623 | 79.0311 | 83.3157 | 90.4593 | 90.5950 |\n|  |  |  |  |  |  |  |  |  |\n\n* **Ensembles 21 weights v1** : `[0.21145703, 0.18271946, 0.26915887, 0.29597817]`\n* Ensembles 21 weights v2     : `[0.30978996, 0.22192819, 0.17375666, 0.29452519]`  \n* Ensembles 21 weights v3     : `[0.10884351, 0.2009535, 0.18187667, 0.20405396]`\n* Ensembles 21 weights v4     : `[0.21411924, 0.18756664, 0.26507698, 0.31649887]`\n* Ensembles 22 weights v1     : `[0.28008428, 0.08930099, 0.19287446, 0.13415098, 0.2855688]`\n* Ensembles 22 weights v2     : `[0.22772560, 0.10498674, 0.19491591, 0.18363636, 0.2887354]`\n* Ensembles 23 weights        : `[0.33566536, 0.10897427, 0.19606722, 0.31792425]`\n* Ensembles 24 weights v1     : `[0.22155271, 0.1881944, 0.38943474, 0.1644162]`\n* Ensembles 24 weights v2     : `[0.29872177, 0.41167376, 0.92104135, 0.51346469]`\n* Ensembles 25 weights v1     : `[0.16956903, 0.11774465, 0.3500565, 0.06345556, 0.27596693]`\n* Ensembles 25 weights v2     : `[0.40763181, 0.23739473, 0.89755062, 0.15559366, 0.78777895]`\n* Ensembles 25 weights v3     : `[0.213048, 0.1057116, 0.37792647, 0.10013445, 0.33992517]`\n* Ensembles 27 weights v1     : `[0.17215925, 0.06356686, 0.09478203, 0.29867107, 0.12876395, 0.23019573]`\n* Ensembles 27 weights v2     : `[0.34618164, 0.19092364, 0.38515934, 0.91232422, 0.00026995, 0.70023081]`\n* Ensembles 28 weights v1     : `[0.17911332, 0.10146231, 0.40657403, 0.30869513]`\n* Ensembles 28 weights v2     : `[0.13729324, 0.19570104, 0.81195183, 0.38014906]`\n\n### Corrected No. 1 CV\n\n| model | class 0 acc | class 1 acc | class 2 acc | class 3 acc | class 4 acc | macro f1 | weighted f1 | acc | \n| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| Ensembles 21    | 62.8016 | 84.7238 | 81.5711 | 97.6782 | 77.3702 | 82.0227 | 89.7710 | 89.9191 |\n| Ensembles 21 v1 | 63.4048 | 85.2417 | 81.7037 | 97.7364 | 77.0242 | 82.2381 | 89.8789 | 90.0330 |\n| Ensembles 21 v2 | 63.6729 | 85.2417 | 81.9357 | 97.7299 | 77.0588 | 82.3207 | 89.9247 | 90.0748 |\n| Ensembles 25    | 63.6059 | 85.2129 | 81.9357 | 97.7558 | 77.7509 | 82.4575 | 90.0111 | 90.1583 |\n| Ensembles 25 v1 | 64.0751 | 85.8458 | 82.4992 | 97.7817 | 77.4740 | 82.7302 | 90.1680 | 90.3178 |\n| Ensembles 25 v2 | 64.4772 | 85.5869 | 82.4992 | 97.7687 | 77.7509 | 82.7749 | 90.1875 | 90.3292 |\n| Ensembles 27    | 65.2815 | 85.1841 | 81.9688 | 97.8140 | 77.9239 | 82.8437 | 90.1635 | 90.3064 |\n| Ensembles 27 v2 | 66.4879 | 85.5293 | 82.3666 | 97.8528 | 78.0277 | 83.2241 | 90.3626 | 90.5001 |\n| Ensembles 29    | 65.2145 | 85.2992 | 81.9688 | 97.7881 | 77.8547 | 82.8115 | 90.1542 | 90.2950 |\n| Ensembles 29 v1 | 64.8794 | 85.4430 | 82.2672 | 97.8075 | 78.0623 | 82.9062 | 90.2181 | 90.3634 |\n| Ensembles 29 v2 | 65.1475 | 85.4718 | 82.5323 | 97.7558 | 78.0623 | 82.9704 | 90.2405 | 90.3824 |\n| Ensembles 30    | 65.0134 | 85.0115 | 82.2009 | 97.8011 | 78.1661 | 82.8279 | 90.1728 | 90.3140 |\n| Ensembles 30 v2 | 66.3539 | 85.3855 | 82.6318 | 97.8011 | 78.3391 | 83.2327 | 90.3764 | **90.5077** |\n| Ensembles 31    | 65.5496 | 85.0690 | 82.2009 | 97.7881 | 77.1626 | 82.7226 | 90.0923 | 90.2343 |\n| Ensembles 31 v1 | 65.8177 | 85.5869 | 82.4992 | 97.8011 | 77.7163 | 83.0625 | 90.2783 | 90.4203 |\n| Ensembles 31 v2 | 66.0188 | 85.7307 | 82.4992 | 97.8140 | 77.8547 | 83.1540 | 90.3332 | 90.4735 |\n| Ensembles 32    | 64.1421 | 85.2417 | 82.1677 | 97.7429 | 78.0969 | 82.6899 | 90.1047 | 90.2495 |\n| Ensembles 32 v1 | 64.6783 | 85.4143 | 82.3997 | 97.8011 | 78.0969 | 82.8833 | 90.2178 | 90.3634 |\n| Ensembles 32 v2 | 65.1475 | 85.3280 | 82.3003 | 97.8399 | 78.5467 | 83.0375 | 90.2989 | 90.4393 |\n| Ensembles 33    |  |  |  |  |  |  |  |  |\n| Ensembles 33 v2 | 66.4209 | 85.5581 | 82.5323 | 97.8205 | 78.1315 | 83.2456 | 90.3761 | **90.5114** |\n| Ensembles 34    | 63.8070 | 84.4649 | 81.8363 | 97.7429 | 78.1315 | 82.3955 | 89.9517 | 90.0938 |\n| Ensembles 34 v2 | 64.3432 | 84.7238 | 82.1346 | 97.7623 | 77.9931 | 82.5808 | 90.0474 | 90.1887 |\n| Ensembles 35    | 63.8070 | 85.0403 | 82.0683 | 97.6717 | 77.8547 | 82.4988 | 89.9779 | 90.1242 |\n| Ensembles 35 v2 | 64.2761 | 84.7814 | 82.1677 | 97.7558 | 77.9585 | 82.5788 | 90.0460 | 90.1887 |\n|  |  |  |  |  |  |  |  |  |\n\n* Ensembles 21 weighted v1 : `[0.31792425, 0.33566536, 0.19606722, 0.10897427]`\n* Ensembles 21 weighted v2 : `[0.90971052, 0.82730546, 0.38318065, 0.26230337]`\n* Ensembles 25 weighted v1 : `[0.16892105, 0.13211221, 0.54735528, 0.06826233, 0.06961585]`\n* Ensembles 25 weighted v2 : `[0.3306592, 0.25242486, 0.8238158, 0.11380859, 0.22893606]`\n* Ensembles 27 weighted v1 : x\n* Ensembles 27 weighted v2 : `[0.64438387, 0.06787352, 0.21374317, 0.92894338, 0.30073056, 0.25681572]`\n* Ensembles 29 weighted v1 : `[0.16127426, 0.14639634, 0.40610428, 0.25388546]`\n* Ensembles 29 weighted v2 : `[0.14392024, 0.26999754, 0.66471911, 0.10123768]`\n* Ensembles 30 weighted v2 : `[0.62811276, 0.23633465, 0.25867451, 0.95243224, 0.10789748, 0.5444512]`\n* Ensembles 31 weighted v1 : `[0.15856572, 0.03564624, 0.09446308, 0.06490467, 0.51857468, 0.12564358]`\n* Ensembles 31 weighted v2 : `[0.37460579, 0.06460235, 0.06938154, 0.22316517, 0.83760474, 0.26799389]`\n* Ensembles 32 weighted v1 : `[0.03564624, 0.15856572, 0.12564358, 0.51857468, 0.06490467, 0.09446308]`\n* Ensembles 32 weighted v2 : `[2.47e-05, 0.58436138, 0.5120863, 0.87538525, 0.61622132, 0.31402191]`\n* Ensembles 33 weighted v2 : `[0.50718439, 0.16996559, 0.25776149, 0.13448321, 0.92798265, 0.2179295, 0.22300932]`\n* Ensembles 34 weighted v2 : `[0.72592935, 0.90095763, 0.65667935, 0.53306918]`\n* Ensembles 35 weighted v2 : `[0.29556101, 0.95351582, 0.96590418, 0.82782731, 0.73989029]`\n* Ensembles 35 weighted v3 : `[0.24627387, 0.12512951, 0.18096359, 0.07093961, 0.37141577]`\n\n% Usually, `v1` is a version of *Simplex* method, `v2`+ are versions of *Optuna*.\n\n### 2020 validation\n\n| model | class 0 acc | class 1 acc | class 2 acc | class 3 acc | class 4 acc | macro f1 | weighted f1 | acc | \n| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| Ensembles 21 tuned | 67.7093 | 83.3714 | 81.5172 | 97.7504 | 77.3768 | 82.4344 | 90.4021 | 90.4893 | \n| Ensembles 22 tuned | 69.2732 | 82.9603 | 81.3495 | 97.8416 | 77.4932 | 82.6708 | 90.4903 | **90.5781** |\n\n* Ensembles 21 weights v1 (20) : `[0.23779558, 0.24279284, 0.10746264, 0.41194895]`\n* Ensembles 22 weights v1 (20) : `[0.22084675, 0.08012831, 0.24442862, 0.07493153, 0.37966478]`\n\n### Orders\n\n1. `effnetb3-cutmix`\n2. `effnetb4-cutmix-fmix`\n3. `effnetb4-fcl`\n4. `resnest50_fast_4s2x40d-cutmix-fmix`\n5. `resnest50_fast_4s2x40d-fcl`\n6. `resnest50_fast_4s2x40d-fmix-cutmix`\n7. `vit_l16`\n\n## Validation\n\n* validation kernel : [here](https://www.kaggle.com/kozistr/valid-rns50-effnet-b4-vit-l-16)\n* leaf_disease_validation : [here](https://www.kaggle.com/kozistr/leaf-disease-validation)\n\n## Trained Models\n\n* leaf_disease_resnest50 : [here](https://www.kaggle.com/kozistr/leaf-disease-resnest50)\n* leaf_disease_effnet : [here](https://www.kaggle.com/kozistr/leaf-disease-effnet)\n* leaf_disease_vit : [here](https://www.kaggle.com/kozistr/leaf-disease-vit)\n* leaf_disease_deit : [here](https://www.kaggle.com/kozistr/leaf-disease-deit)\n\n## Inference\n\n* inference kernel : [here](https://www.kaggle.com/kozistr/inference-resnest50-effnet-b4-ensembles)\n\n| try | num of models | description | n_tta | time |\n| :---: | :---: | :---: | :---: | :---: |\n| 0 | 5 | ResNeSt50-fast-4s2x40d | 5 | about 2 hours |\n| 1 | 10 (5 + 5) | EffNet-B4 + ResNeSt50-fast-4s2x40d | 3 | about 2 ~ 3 hours |\n| 2 | 10 (5 + 5) | EffNet-B4 + ResNeSt50-fast-4s2x40d | 5 | about 4 hours |\n| 3 | 15 (5 + 5 + 5) | EffNet-B4 + ResNeSt50-fast-4s2x40d x 2 | 5 | about 6 hours |\n| 4 | 5 | ViT-L/16 | 4 | about 6 hours |\n| 5 | 5 | DeiT-B/16 | 4 | about 2 hours |\n\n## Term\n\n* `cce` : category cross entropy loss\n* `scce` : smooth category cross entropy loss\n* `fc` : focal cosine loss\n* `taylor` : taylor category cross entropy loss\n* `bi-tempered` : bi-tempered loss\n* `anl` : annealing\n* `cf` : cutmix + fmix\n"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}