{"cells":[{"metadata":{},"cell_type":"markdown","source":"## Our highest scoring private solution\n\nThe competition is over. And I am sure that most of the teams have not choosen their best private solutions. Unfortunately .... \n\nBut we would like to share our best private scoring solution.\n\nTraining setup:\n- model - EfficentNet 6 pretrained on Noisy student\n- resolution - 384x384\n- Augmentations :\n```python\nalbu.Compose([\n    albu.HorizontalFlip(p=0.5),\n    albu.VerticalFlip(p=0.5),\n\n    AdvancedHairAugmentation(p=0.5, hairs_folder='/ssd_data/melanoma_classification/melanoma_hairs/'),\n\n    albu.ShiftScaleRotate(scale_limit=0.3, rotate_limit=0, shift_limit=0.3, p=0.6, border_mode=0),\n\n    albu.OneOf(\n        [\n            albu.RandomContrast(p=1),\n            albu.HueSaturationValue(p=1),\n        ],\n        p=0.9,\n    ),\n    albu.RandomRotate90(),\n    albu.Cutout(\n        max_h_size=16,\n        max_w_size=16\n    ),\n    Microscope(p=0.5),\n])\n```\n- used external data from ISIC 2018 comp\n- batch size 16 with 4 steps of gradient accumulation\n- using MultiScaleDropout for classifier (https://arxiv.org/pdf/1905.09788.pdf)\n- different learning rates (backbone - 0.0001 and classifier - 0.005)\n- scheduler:\n```python\ntorch.optim.lr_scheduler.ReduceLROnPlateau(\n    optimizer, \n    patience=2,\n    factor=0.25, \n    min_lr=1e-6,\n    mode='max'\n)\n```\n- Adam optimizer\n- Simple BCE loss\n\nInference setup:\n- While training 3 best checkpoints by validation roc_auc for each fold were saved\n- Then for each fold SWA of these 3 checkpoints were used\n- TTAs (12 TTAs): \n```python \naug_combinations = [\n    lambda : None,\n    lambda : albu.HorizontalFlip(always_apply=True),\n    lambda : albu.VerticalFlip(always_apply=True),\n    lambda : Microscope(always_apply=True),\n    lambda : albu.Compose([\n        albu.VerticalFlip(always_apply=True),\n        albu.HorizontalFlip(always_apply=True),\n    ]),\n    lambda : albu.RandomRotate90(always_apply=True)\n] + [\n    lambda : albu.Compose([\n                            albu.HorizontalFlip(p=0.5),\n                            albu.VerticalFlip(p=0.5),\n        \n                            AdvancedHairAugmentation(p=0.5, hairs_folder='/ssd_data/melanoma_classification/hairs/'),\n    \n                            albu.JpegCompression(),\n                            albu.RGBShift(),\n                            albu.CLAHE(),\n                            albu.Blur(),\n\n                            albu.ShiftScaleRotate(scale_limit=0.3, rotate_limit=0, shift_limit=0.3, p=0.75, border_mode=0),\n\n                            albu.OneOf(\n                                [\n                                    albu.RandomContrast(p=1),\n                                    albu.HueSaturationValue(p=1),\n                                ],\n                                p=0.9,\n                            ),\n                            albu.RandomRotate90(),\n                            albu.Cutout(\n                                num_holes=3,\n                                max_h_size=16,\n                                max_w_size=16\n                            ),\n                            Microscope(p=0.5),\n                        ])] * 6\n```\n- And then simple average results from 5 folds \n\nValidation results:\n- Mean CV score : 0.9308377465318983\n- Std CV score : 0.004282178007858335\n- OOF score : 0.92830","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import pandas as pd\n\nsub = pd.read_csv('../input/best-private-sub/effnet6_image_chrisgroupkfold_fastplatoschedule_cutoutshiftsrotates_customaugs_denseelu_mdropout_difflrs_distributed_bigbatch_nodupls_384res_extrenaldata1817_batchacum4_SWA_6ttamicroscope6randomttas_meanfolds.csv')\nsub.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Many thanks to Kaggle team, all participants and especially my teammates:\n- Rohit Agarwal (https://www.kaggle.com/rohitagarwal)\n- Ashish Gupta (https://www.kaggle.com/roydatascience)\n- Mukharbek Organokov (https://www.kaggle.com/muhakabartay)\n\nGood Kaggling!","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}