{"cells":[{"metadata":{},"cell_type":"markdown","source":"It's the first time I use fastai in Kaggle competition, I use fastai with 7fold cross validation, and inceptionresnetv2 + resnet50 ensemble. Pretty simple edition, and may seems to be stupid to some experienced Kaggler.\nThis kernel can get both 0.626 in private & public LB, but I didn't select is for my another kernel get 0.627 in public LB and 0.623 in Private LB, which makes me get 71st place.  \nWell, fastai is easy to use, but it's difficult to get an outstanding result.\nBTW, thanks for [ods.ai] Alex Lekov for his kernel https://www.kaggle.com/itslek/fastai-resnet50-imet-v4-2 and thanks for Miguel Pinto for his kernel https://www.kaggle.com/mnpinto/imet-fastai-starter"},{"metadata":{},"cell_type":"markdown","source":"The following part is similar to the kernels mentioned above, you can ignore it"},{"metadata":{"trusted":true},"cell_type":"code","source":"\n#======================================From here to end just a demo to train the resnet50 models===========================================\n#I also trained inceptionresnetv2, with the internel open and use !pip install pretrainedmodels to directly use inceptionresnetv2. \n#I only need the imet pretrained resnet50 models, internel open but cannot submit result is ok for me\nimport fastai\nfrom fastai.vision import *\nfastai.__version__\n\n#Here just for example, use BATCH  = 72 and SIZE   = 320. It's appropriate for pre-trained at first SIZE= 256 and then 288, at last, 320.\n\nBATCH  = 72\nSIZE   = 320\npath = Path('../input/imet-2019-fgvc6/') # iMet data path\n\nfrom torch.utils import model_zoo\nPath('models').mkdir(exist_ok=True)\n!cp '../input/resnet50/resnet50.pth' 'models/'\ndef load_url(*args, **kwargs):\n    model_dir = Path('models')\n    filename  = 'resnet50.pth'\n    if not (model_dir/filename).is_file(): raise FileNotFoundError\n    return torch.load(model_dir/filename)\nmodel_zoo.load_url = load_url\n\ntrain_df = pd.read_csv(path/'train.csv')\ntrain_df.head()\n\nlabels_df = pd.read_csv(path/'labels.csv')\nlabels_df.head()\n\ntest_df = pd.read_csv(path/'sample_submission.csv')\ntest_df.head()\n\n\ntfms = get_transforms(do_flip=True, flip_vert=False, max_rotate=0.10, max_zoom=1.5, max_warp=0.2, max_lighting=0.2,\n                     xtra_tfms=[(symmetric_warp(magnitude=(-0,0), p=0)),])\n\ntrain, test = [ImageList.from_df(df, path=path, cols='id', folder=folder, suffix='.png') \n               for df, folder in zip([train_df, test_df], ['train', 'test'])]\n\n\nclass FocalLoss(nn.Module):\n    def __init__(self, gamma=2):\n        super().__init__()\n        self.gamma = gamma\n\n    def forward(self, logit, target):\n        target = target.float()\n        max_val = (-logit).clamp(min=0)\n        loss = logit - logit * target + max_val + \\\n               ((-max_val).exp() + (-logit - max_val).exp()).log()\n\n        invprobs = F.logsigmoid(-logit * (target * 2.0 - 1.0))\n        loss = (invprobs * self.gamma).exp() * loss\n        if len(loss.size())==2:\n            loss = loss.sum(dim=1)\n        return loss.mean()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here I use 7 fold cross-validation. However, fastai **automatically determined the number of class** according to training dataset, which will lead to 1102 or 1101, 1100 classes rather than 1103 classes. Because the categories are imbalanced, some of categories only appears once: if it is divided into validation dataset, it will not be recoreded in training dataset. \nSo I use n_splits=15, and find out the divisions that number of classes are 1103 in training dataset, they are: set0, set1, set2, set3, set7, set8 and set10."},{"metadata":{"trusted":true},"cell_type":"code","source":"\nfrom sklearn.model_selection import KFold\nimport numpy as np\nkf = KFold(n_splits=15, random_state=43, shuffle=True)\n\nkf\n\nn = 0\nfor train_index, test_index in kf.split(train):\n    print(\"TRAIN:\", train_index, \"TEST:\", test_index)\n# Change if n == 0 to n == 1,2,3,7,8,10 to use other divisions\n    if n == 0:\n        break\n    n+=1\n\nX_train, X_test = train[train_index], train[test_index]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here, for example, train resnet50 for 1 time. In fact, you can train here at least 15 times with resnet50 in 32400 seconds, save the model again, next, load it again and again by learn.load function."},{"metadata":{"trusted":true},"cell_type":"code","source":"data = (train.split_by_list(X_train,X_test)\n        .label_from_df(cols='attribute_ids', label_delim=' ')\n        .add_test(test)\n        .transform(tfms, size=SIZE, resize_method=ResizeMethod.PAD, padding_mode='border',)\n        .databunch(path=Path('.'), bs=BATCH).normalize(imagenet_stats)\n\n       )\nprint('Data loaded')\nlearn = cnn_learner(data, base_arch=models.resnet50, loss_func=FocalLoss(), metrics=fbeta,pretrained=True)\n#use learn.load function to use your previous trained model, move the model to correct file position by using !cp function\n#learn.load('resnet50-0-v4-35')\nlearn.unfreeze()\nlearn.fit_one_cycle(1, slice(1e-5,1e-2))\nlearn.freeze()\nlearn.save('resnet50-0-v4-1', return_path=True)\n#=================================This is the end of pre-trained resnet50 demo====================================================","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The following code is actually what I used in the last submission result file. It's time saving by only loading models and do predictions. In public datasets, I only used less than 1500 seconds to do prediction in 7-fold inceptionresnetv2 and resnet50."},{"metadata":{"trusted":true},"cell_type":"code","source":"Incepres_BATCH  = 36\nIncepres_SIZE   = 320\npath = Path('../input/imet-2019-fgvc6/') # iMet data path","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"Res_BATCH  = 36\nRes_SIZE   = 320","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train, test = [ImageList.from_df(df, path=path, cols='id', folder=folder, suffix='.png') \n               for df, folder in zip([train_df, test_df], ['train', 'test'])]\nIncepres_data = (train.split_by_rand_pct(0.1, seed=42)\n        .label_from_df(cols='attribute_ids', label_delim=' ')\n        .add_test(test)\n        .transform(tfms, size=Incepres_SIZE, resize_method=ResizeMethod.PAD, padding_mode='border',)\n        .databunch(path=Path('.'), bs=Incepres_BATCH).normalize(imagenet_stats))\nRes_data = (train.split_by_rand_pct(0.1, seed=42)\n        .label_from_df(cols='attribute_ids', label_delim=' ')\n        .add_test(test)\n        .transform(tfms, size=Res_SIZE, resize_method=ResizeMethod.PAD, padding_mode='border',)\n        .databunch(path=Path('.'), bs=Res_BATCH).normalize(imagenet_stats))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here you should copy the key source codes in github repositories in pytorch pretrained-models cadene https://github.com/Cadene/pretrained-models.pytorch and fastai https://github.com/fastai/fastai to avoid using internet"},{"metadata":{"trusted":true},"cell_type":"code","source":"from fastai.vision import learner\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Copied from https://github.com/Cadene/pretrained-models.pytorch, and just use for inceptionresnetv2\nfrom __future__ import print_function, division, absolute_import\n\nimport torch\n\nimport torch.nn as nn\n\nimport torch.utils.model_zoo as model_zoo\n\nimport os\n\nimport sys\n\n\n\n__all__ = ['InceptionResNetV2', 'inceptionresnetv2']\n\n\n\npretrained_settings = {\n\n    'inceptionresnetv2': {\n\n        'imagenet': {\n\n            'url': 'http://data.lip6.fr/cadene/pretrainedmodels/inceptionresnetv2-520b38e4.pth',\n\n            'input_space': 'RGB',\n\n            'input_size': [3, 299, 299],\n\n            'input_range': [0, 1],\n\n            'mean': [0.5, 0.5, 0.5],\n\n            'std': [0.5, 0.5, 0.5],\n\n            'num_classes': 1000\n\n        },\n\n        'imagenet+background': {\n\n            'url': 'http://data.lip6.fr/cadene/pretrainedmodels/inceptionresnetv2-520b38e4.pth',\n\n            'input_space': 'RGB',\n\n            'input_size': [3, 299, 299],\n\n            'input_range': [0, 1],\n\n            'mean': [0.5, 0.5, 0.5],\n\n            'std': [0.5, 0.5, 0.5],\n\n            'num_classes': 1001\n\n        }\n\n    }\n\n}\n\n\n\n\n\nclass BasicConv2d(nn.Module):\n\n\n\n    def __init__(self, in_planes, out_planes, kernel_size, stride, padding=0):\n\n        super(BasicConv2d, self).__init__()\n\n        self.conv = nn.Conv2d(in_planes, out_planes,\n\n                              kernel_size=kernel_size, stride=stride,\n\n                              padding=padding, bias=False) # verify bias false\n\n        self.bn = nn.BatchNorm2d(out_planes,\n\n                                 eps=0.001, # value found in tensorflow\n\n                                 momentum=0.1, # default pytorch value\n\n                                 affine=True)\n\n        self.relu = nn.ReLU(inplace=False)\n\n\n\n    def forward(self, x):\n\n        x = self.conv(x)\n\n        x = self.bn(x)\n\n        x = self.relu(x)\n\n        return x\n\n\n\n\n\nclass Mixed_5b(nn.Module):\n\n\n\n    def __init__(self):\n\n        super(Mixed_5b, self).__init__()\n\n\n\n        self.branch0 = BasicConv2d(192, 96, kernel_size=1, stride=1)\n\n\n\n        self.branch1 = nn.Sequential(\n\n            BasicConv2d(192, 48, kernel_size=1, stride=1),\n\n            BasicConv2d(48, 64, kernel_size=5, stride=1, padding=2)\n\n        )\n\n\n\n        self.branch2 = nn.Sequential(\n\n            BasicConv2d(192, 64, kernel_size=1, stride=1),\n\n            BasicConv2d(64, 96, kernel_size=3, stride=1, padding=1),\n\n            BasicConv2d(96, 96, kernel_size=3, stride=1, padding=1)\n\n        )\n\n\n\n        self.branch3 = nn.Sequential(\n\n            nn.AvgPool2d(3, stride=1, padding=1, count_include_pad=False),\n\n            BasicConv2d(192, 64, kernel_size=1, stride=1)\n\n        )\n\n\n\n    def forward(self, x):\n\n        x0 = self.branch0(x)\n\n        x1 = self.branch1(x)\n\n        x2 = self.branch2(x)\n\n        x3 = self.branch3(x)\n\n        out = torch.cat((x0, x1, x2, x3), 1)\n\n        return out\n\n\n\n\n\nclass Block35(nn.Module):\n\n\n\n    def __init__(self, scale=1.0):\n\n        super(Block35, self).__init__()\n\n\n\n        self.scale = scale\n\n\n\n        self.branch0 = BasicConv2d(320, 32, kernel_size=1, stride=1)\n\n\n\n        self.branch1 = nn.Sequential(\n\n            BasicConv2d(320, 32, kernel_size=1, stride=1),\n\n            BasicConv2d(32, 32, kernel_size=3, stride=1, padding=1)\n\n        )\n\n\n\n        self.branch2 = nn.Sequential(\n\n            BasicConv2d(320, 32, kernel_size=1, stride=1),\n\n            BasicConv2d(32, 48, kernel_size=3, stride=1, padding=1),\n\n            BasicConv2d(48, 64, kernel_size=3, stride=1, padding=1)\n\n        )\n\n\n\n        self.conv2d = nn.Conv2d(128, 320, kernel_size=1, stride=1)\n\n        self.relu = nn.ReLU(inplace=False)\n\n\n\n    def forward(self, x):\n\n        x0 = self.branch0(x)\n\n        x1 = self.branch1(x)\n\n        x2 = self.branch2(x)\n\n        out = torch.cat((x0, x1, x2), 1)\n\n        out = self.conv2d(out)\n\n        out = out * self.scale + x\n\n        out = self.relu(out)\n\n        return out\n\n\n\n\n\nclass Mixed_6a(nn.Module):\n\n\n\n    def __init__(self):\n\n        super(Mixed_6a, self).__init__()\n\n\n\n        self.branch0 = BasicConv2d(320, 384, kernel_size=3, stride=2)\n\n\n\n        self.branch1 = nn.Sequential(\n\n            BasicConv2d(320, 256, kernel_size=1, stride=1),\n\n            BasicConv2d(256, 256, kernel_size=3, stride=1, padding=1),\n\n            BasicConv2d(256, 384, kernel_size=3, stride=2)\n\n        )\n\n\n\n        self.branch2 = nn.MaxPool2d(3, stride=2)\n\n\n\n    def forward(self, x):\n\n        x0 = self.branch0(x)\n\n        x1 = self.branch1(x)\n\n        x2 = self.branch2(x)\n\n        out = torch.cat((x0, x1, x2), 1)\n\n        return out\n\n\n\n\n\nclass Block17(nn.Module):\n\n\n\n    def __init__(self, scale=1.0):\n\n        super(Block17, self).__init__()\n\n\n\n        self.scale = scale\n\n\n\n        self.branch0 = BasicConv2d(1088, 192, kernel_size=1, stride=1)\n\n\n\n        self.branch1 = nn.Sequential(\n\n            BasicConv2d(1088, 128, kernel_size=1, stride=1),\n\n            BasicConv2d(128, 160, kernel_size=(1,7), stride=1, padding=(0,3)),\n\n            BasicConv2d(160, 192, kernel_size=(7,1), stride=1, padding=(3,0))\n\n        )\n\n\n\n        self.conv2d = nn.Conv2d(384, 1088, kernel_size=1, stride=1)\n\n        self.relu = nn.ReLU(inplace=False)\n\n\n\n    def forward(self, x):\n\n        x0 = self.branch0(x)\n\n        x1 = self.branch1(x)\n\n        out = torch.cat((x0, x1), 1)\n\n        out = self.conv2d(out)\n\n        out = out * self.scale + x\n\n        out = self.relu(out)\n\n        return out\n\n\n\n\n\nclass Mixed_7a(nn.Module):\n\n\n\n    def __init__(self):\n\n        super(Mixed_7a, self).__init__()\n\n\n\n        self.branch0 = nn.Sequential(\n\n            BasicConv2d(1088, 256, kernel_size=1, stride=1),\n\n            BasicConv2d(256, 384, kernel_size=3, stride=2)\n\n        )\n\n\n\n        self.branch1 = nn.Sequential(\n\n            BasicConv2d(1088, 256, kernel_size=1, stride=1),\n\n            BasicConv2d(256, 288, kernel_size=3, stride=2)\n\n        )\n\n\n\n        self.branch2 = nn.Sequential(\n\n            BasicConv2d(1088, 256, kernel_size=1, stride=1),\n\n            BasicConv2d(256, 288, kernel_size=3, stride=1, padding=1),\n\n            BasicConv2d(288, 320, kernel_size=3, stride=2)\n\n        )\n\n\n\n        self.branch3 = nn.MaxPool2d(3, stride=2)\n\n\n\n    def forward(self, x):\n\n        x0 = self.branch0(x)\n\n        x1 = self.branch1(x)\n\n        x2 = self.branch2(x)\n\n        x3 = self.branch3(x)\n\n        out = torch.cat((x0, x1, x2, x3), 1)\n\n        return out\n\n\n\n\n\nclass Block8(nn.Module):\n\n\n\n    def __init__(self, scale=1.0, noReLU=False):\n\n        super(Block8, self).__init__()\n\n\n\n        self.scale = scale\n\n        self.noReLU = noReLU\n\n\n\n        self.branch0 = BasicConv2d(2080, 192, kernel_size=1, stride=1)\n\n\n\n        self.branch1 = nn.Sequential(\n\n            BasicConv2d(2080, 192, kernel_size=1, stride=1),\n\n            BasicConv2d(192, 224, kernel_size=(1,3), stride=1, padding=(0,1)),\n\n            BasicConv2d(224, 256, kernel_size=(3,1), stride=1, padding=(1,0))\n\n        )\n\n\n\n        self.conv2d = nn.Conv2d(448, 2080, kernel_size=1, stride=1)\n\n        if not self.noReLU:\n\n            self.relu = nn.ReLU(inplace=False)\n\n\n\n    def forward(self, x):\n\n        x0 = self.branch0(x)\n\n        x1 = self.branch1(x)\n\n        out = torch.cat((x0, x1), 1)\n\n        out = self.conv2d(out)\n\n        out = out * self.scale + x\n\n        if not self.noReLU:\n\n            out = self.relu(out)\n\n        return out\n\n\n\n\n\nclass InceptionResNetV2(nn.Module):\n\n\n\n    def __init__(self, num_classes=1001):\n\n        super(InceptionResNetV2, self).__init__()\n\n        # Special attributs\n\n        self.input_space = None\n\n        self.input_size = (299, 299, 3)\n\n        self.mean = None\n\n        self.std = None\n\n        # Modules\n\n        self.conv2d_1a = BasicConv2d(3, 32, kernel_size=3, stride=2)\n\n        self.conv2d_2a = BasicConv2d(32, 32, kernel_size=3, stride=1)\n\n        self.conv2d_2b = BasicConv2d(32, 64, kernel_size=3, stride=1, padding=1)\n\n        self.maxpool_3a = nn.MaxPool2d(3, stride=2)\n\n        self.conv2d_3b = BasicConv2d(64, 80, kernel_size=1, stride=1)\n\n        self.conv2d_4a = BasicConv2d(80, 192, kernel_size=3, stride=1)\n\n        self.maxpool_5a = nn.MaxPool2d(3, stride=2)\n\n        self.mixed_5b = Mixed_5b()\n\n        self.repeat = nn.Sequential(\n\n            Block35(scale=0.17),\n\n            Block35(scale=0.17),\n\n            Block35(scale=0.17),\n\n            Block35(scale=0.17),\n\n            Block35(scale=0.17),\n\n            Block35(scale=0.17),\n\n            Block35(scale=0.17),\n\n            Block35(scale=0.17),\n\n            Block35(scale=0.17),\n\n            Block35(scale=0.17)\n\n        )\n\n        self.mixed_6a = Mixed_6a()\n\n        self.repeat_1 = nn.Sequential(\n\n            Block17(scale=0.10),\n\n            Block17(scale=0.10),\n\n            Block17(scale=0.10),\n\n            Block17(scale=0.10),\n\n            Block17(scale=0.10),\n\n            Block17(scale=0.10),\n\n            Block17(scale=0.10),\n\n            Block17(scale=0.10),\n\n            Block17(scale=0.10),\n\n            Block17(scale=0.10),\n\n            Block17(scale=0.10),\n\n            Block17(scale=0.10),\n\n            Block17(scale=0.10),\n\n            Block17(scale=0.10),\n\n            Block17(scale=0.10),\n\n            Block17(scale=0.10),\n\n            Block17(scale=0.10),\n\n            Block17(scale=0.10),\n\n            Block17(scale=0.10),\n\n            Block17(scale=0.10)\n\n        )\n\n        self.mixed_7a = Mixed_7a()\n\n        self.repeat_2 = nn.Sequential(\n\n            Block8(scale=0.20),\n\n            Block8(scale=0.20),\n\n            Block8(scale=0.20),\n\n            Block8(scale=0.20),\n\n            Block8(scale=0.20),\n\n            Block8(scale=0.20),\n\n            Block8(scale=0.20),\n\n            Block8(scale=0.20),\n\n            Block8(scale=0.20)\n\n        )\n\n        self.block8 = Block8(noReLU=True)\n\n        self.conv2d_7b = BasicConv2d(2080, 1536, kernel_size=1, stride=1)\n\n        self.avgpool_1a = nn.AvgPool2d(8, count_include_pad=False)\n\n        self.last_linear = nn.Linear(1536, num_classes)\n\n\n\n    def features(self, input):\n\n        x = self.conv2d_1a(input)\n\n        x = self.conv2d_2a(x)\n\n        x = self.conv2d_2b(x)\n\n        x = self.maxpool_3a(x)\n\n        x = self.conv2d_3b(x)\n\n        x = self.conv2d_4a(x)\n\n        x = self.maxpool_5a(x)\n\n        x = self.mixed_5b(x)\n\n        x = self.repeat(x)\n\n        x = self.mixed_6a(x)\n\n        x = self.repeat_1(x)\n\n        x = self.mixed_7a(x)\n\n        x = self.repeat_2(x)\n\n        x = self.block8(x)\n\n        x = self.conv2d_7b(x)\n\n        return x\n\n\n\n    def logits(self, features):\n\n        x = self.avgpool_1a(features)\n\n        x = x.view(x.size(0), -1)\n\n        x = self.last_linear(x)\n\n        return x\n\n\n\n    def forward(self, input):\n\n        x = self.features(input)\n\n        x = self.logits(x)\n\n        return x\n\n\n\ndef inceptionresnetv2(num_classes=1000, pretrained='imagenet'):\n\n    r\"\"\"InceptionResNetV2 model architecture from the\n\n    `\"InceptionV4, Inception-ResNet...\" <https://arxiv.org/abs/1602.07261>`_ paper.\n\n    \"\"\"\n\n    if pretrained:\n\n        settings = pretrained_settings['inceptionresnetv2'][pretrained]\n\n        assert num_classes == settings['num_classes'],\"num_classes should be {}, but is {}\".format(settings['num_classes'], num_classes)\n\n\n\n        # both 'imagenet'&'imagenet+background' are loaded from same parameters\n\n        model = InceptionResNetV2(num_classes=1001)\n\n        model.load_state_dict(model_zoo.load_url(settings['url']))\n\n\n\n        if pretrained == 'imagenet':\n\n            new_last_linear = nn.Linear(1536, 1000)\n\n            new_last_linear.weight.data = model.last_linear.weight.data[1:]\n\n            new_last_linear.bias.data = model.last_linear.bias.data[1:]\n\n            model.last_linear = new_last_linear\n\n\n\n        model.input_space = settings['input_space']\n\n        model.input_size = settings['input_size']\n\n        model.input_range = settings['input_range']\n\n\n\n        model.mean = settings['mean']\n\n        model.std = settings['std']\n\n    else:\n\n        model = InceptionResNetV2(num_classes=num_classes)\n\n    return model\n\n\n\n'''\n\nTEST\n\nRun this code with:\n\n```\n\ncd $HOME/pretrained-models.pytorch\n\npython -m pretrainedmodels.inceptionresnetv2\n\n```\n\n'''\n# Comment these code, we will not use them\n\n#if __name__ == '__main__':\n\n\n\n #   assert inceptionresnetv2(num_classes=10, pretrained=None)\n\n  #  print('success')\n\n   # assert inceptionresnetv2(num_classes=1000, pretrained='imagenet')\n\n    #print('success')\n\n   # assert inceptionresnetv2(num_classes=1001, pretrained='imagenet+background')\n\n    #print('success')\n\n\n\n    # fail\n\n   # assert inceptionresnetv2(num_classes=1001, pretrained='imagenet')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# copied from https://github.com/fastai/fastai/blob/master/fastai/vision/models/cadene_models.py\ndef get_incepres_model(model_name:str, pretrained:bool, seq:bool=False, pname:str='imagenet', **kwargs):\n\n    pretrained = pname if pretrained else None\n\n    model = inceptionresnetv2(pretrained=pretrained, **kwargs)\n\n    return nn.Sequential(*model.children()) if seq else model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def myinceptionresnetv2(pretrained:bool=False):  \n    return get_incepres_model('inceptionresnetv2', pretrained, seq=True)\n\n\nlearner.model_meta[myinceptionresnetv2] = {'cut': -2, 'split': lambda m: (m[0][9],     m[1])}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#!ls ../input/15foldincepres/15foldincepres.pth","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\n\n!ls ../input/incepres15fold320 \n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The following code just show the prediction matrix, in fastai, don't use learn.TTA for it required 8 times of running time!"},{"metadata":{"trusted":true},"cell_type":"code","source":"#cv0.593\n!cp ../input/incepres15fold320/inceptionres-0-v4-52.pth ./models/\nIncepres_learn0 = cnn_learner(Incepres_data, base_arch=myinceptionresnetv2, loss_func=FocalLoss(), metrics=fbeta,pretrained=False)\nIncepres_learn0.load('inceptionres-0-v4-52')\n#test_preds = learn0.TTA(ds_type=DatasetType.Test)\nIncepres_test_preds0 = Incepres_learn0.get_preds(ds_type=DatasetType.Test)\nIncepres_test_preds0","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#cv0.597\n!cp ../input/incepres15fold320/inceptionres-1-v4-54.pth ./models/\nIncepres_learn1 = cnn_learner(Incepres_data, base_arch=myinceptionresnetv2, loss_func=FocalLoss(), metrics=fbeta,pretrained=False)\nIncepres_learn1.load('inceptionres-1-v4-54')\n#test_preds = learn0.TTA(ds_type=DatasetType.Test)\nIncepres_test_preds1 = Incepres_learn1.get_preds(ds_type=DatasetType.Test)\nIncepres_test_preds1","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#cv0.588\n!cp ../input/incepres15fold320/inceptionres-2-v4-52.pth ./models/\nIncepres_learn2 = cnn_learner(Incepres_data, base_arch=myinceptionresnetv2, loss_func=FocalLoss(), metrics=fbeta,pretrained=False)\nIncepres_learn2.load('inceptionres-2-v4-52')\n#test_preds = learn0.TTA(ds_type=DatasetType.Test)\nIncepres_test_preds2 = Incepres_learn2.get_preds(ds_type=DatasetType.Test)\nIncepres_test_preds2","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#cv0.597\n!cp ../input/incepres15fold320/inceptionres-8-v4-60.pth ./models/\nIncepres_learn8 = cnn_learner(Incepres_data, base_arch=myinceptionresnetv2, loss_func=FocalLoss(), metrics=fbeta,pretrained=False)\nIncepres_learn8.load('inceptionres-8-v4-60')\n#test_preds = learn0.TTA(ds_type=DatasetType.Test)\nIncepres_test_preds8 = Incepres_learn8.get_preds(ds_type=DatasetType.Test)\nIncepres_test_preds8","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#cv0.588\n!cp ../input/incepres15fold320/inceptionres-3-v4-49-n.pth ./models/\nIncepres_learn3 = cnn_learner(Incepres_data, base_arch=myinceptionresnetv2, loss_func=FocalLoss(), metrics=fbeta,pretrained=False)\nIncepres_learn3.load('inceptionres-3-v4-49-n')\n#test_preds = learn0.TTA(ds_type=DatasetType.Test)\nIncepres_test_preds3 = Incepres_learn3.get_preds(ds_type=DatasetType.Test)\nIncepres_test_preds3","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#cv0.588\n!cp ../input/incepres15fold320/inceptionres-7-v4-60-n.pth ./models/\nIncepres_learn7 = cnn_learner(Incepres_data, base_arch=myinceptionresnetv2, loss_func=FocalLoss(), metrics=fbeta,pretrained=False)\nIncepres_learn7.load('inceptionres-7-v4-60-n')\n#test_preds = learn0.TTA(ds_type=DatasetType.Test)\nIncepres_test_preds7 = Incepres_learn7.get_preds(ds_type=DatasetType.Test)\nIncepres_test_preds7","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Res_test_preds0","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#cv0.594\n!cp ../input/incepres15fold320/inceptionres-10-v4-48.pth ./models/\nIncepres_learn10 = cnn_learner(Incepres_data, base_arch=myinceptionresnetv2, loss_func=FocalLoss(), metrics=fbeta,pretrained=False)\nIncepres_learn10.load('inceptionres-10-v4-48')\n#test_preds = learn0.TTA(ds_type=DatasetType.Test)\nIncepres_test_preds10 = Incepres_learn10.get_preds(ds_type=DatasetType.Test)\nIncepres_test_preds10","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!ls ../input/resnet50imet15fold320","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#!ls ../input/resnet50-imet-15fold/resnet50-0-v4-35.pth\n!cp ../input/resnet50imet15fold320/resnet50-0-v4-597.pth ./models/\nRes_learn0 = cnn_learner(Res_data, base_arch=models.resnet50, loss_func=FocalLoss(), metrics=fbeta,pretrained=False)\nRes_learn0.load('resnet50-0-v4-597')\n\nRes_test_preds0 = Res_learn0.get_preds(ds_type=DatasetType.Test)\nRes_test_preds0","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!cp ../input/resnet50imet15fold320/resnet50-1-v4-603.pth ./models/\nRes_learn1 = cnn_learner(Res_data, base_arch=models.resnet50, loss_func=FocalLoss(), metrics=fbeta,pretrained=False)\nRes_learn1.load('resnet50-1-v4-603')\n\nRes_test_preds1 = Res_learn1.get_preds(ds_type=DatasetType.Test)\nRes_test_preds1","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!cp ../input/resnet50imet15fold320/resnet50-2-v4-597.pth ./models/\nRes_learn2 = cnn_learner(Res_data, base_arch=models.resnet50, loss_func=FocalLoss(), metrics=fbeta,pretrained=False)\nRes_learn2.load('resnet50-2-v4-597')\n\nRes_test_preds2 = Res_learn2.get_preds(ds_type=DatasetType.Test)\nRes_test_preds2","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!cp ../input/resnet50imet15fold320/resnet50-3-v4-597.pth ./models/\nRes_learn3 = cnn_learner(Res_data, base_arch=models.resnet50, loss_func=FocalLoss(), metrics=fbeta,pretrained=False)\nRes_learn3.load('resnet50-3-v4-597')\n\nRes_test_preds3 = Res_learn3.get_preds(ds_type=DatasetType.Test)\nRes_test_preds3","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!cp ../input/resnet50imet15fold320/resnet50-7-v4-597.pth ./models/\nRes_learn7 = cnn_learner(Res_data, base_arch=models.resnet50, loss_func=FocalLoss(), metrics=fbeta,pretrained=False)\nRes_learn7.load('resnet50-7-v4-597')\n\nRes_test_preds7 = Res_learn7.get_preds(ds_type=DatasetType.Test)\nRes_test_preds7","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!cp ../input/resnet50imet15fold320/resnet50-8-v4-603.pth ./models/\nRes_learn8 = cnn_learner(Res_data, base_arch=models.resnet50, loss_func=FocalLoss(), metrics=fbeta,pretrained=False)\nRes_learn8.load('resnet50-8-v4-603')\n\nRes_test_preds8 = Res_learn8.get_preds(ds_type=DatasetType.Test)\nRes_test_preds8","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!cp ../input/resnet50imet15fold320/resnet50-10-v4-599.pth ./models/\nRes_learn10 = cnn_learner(Res_data, base_arch=models.resnet50, loss_func=FocalLoss(), metrics=fbeta,pretrained=False)\nRes_learn10.load('resnet50-10-v4-599')\n\nRes_test_preds10 = Res_learn10.get_preds(ds_type=DatasetType.Test)\nRes_test_preds10","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#import numpy as np\n#np.shape(np.array(Res_test_preds0[0]))\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Here use take average of everything, or you can appoint the weight, it's simple\ntest_preds = ((Incepres_test_preds0[0]*1+Incepres_test_preds1[0]*1.1\n               +Incepres_test_preds2[0]*0.9+Incepres_test_preds3[0]*0.9\n               +Incepres_test_preds7[0]*0.9+Incepres_test_preds8[0]*1.1\n               +Incepres_test_preds10[0]*1.1)/14+(Res_test_preds0[0]*1+Res_test_preds1[0]*1\n               +Res_test_preds2[0]*1+Res_test_preds3[0]*1\n               +Res_test_preds7[0]*1+Res_test_preds8[0]*1\n               +Res_test_preds10[0]*1)/14,Incepres_test_preds1[1])\n#test_preds = ((test_preds0[0]+test_preds1[0]+test_preds2[0]+test_preds3[0]+test_preds7[0]+test_preds8[0]+test_preds10[0])/7,test_preds1[1])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"i2c = np.array([[i, c] for c, i in Incepres_learn0.data.train_ds.y.c2i.items()]).astype(int)\ndef join_preds(preds, thr):\n    return [' '.join(i2c[np.where(t==1)[0],1].astype(str)) for t in (preds[0].sigmoid()>thr).long()]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Most of models get good result around 0.260, 0.270 is also an applicable threshold\ntest_df.attribute_ids = join_preds(test_preds, 0.260)\ntest_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df.to_csv('submission.csv', index=False) ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}