{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Power Rangers Solution"},{"metadata":{},"cell_type":"markdown","source":"# Starting points"},{"metadata":{},"cell_type":"markdown","source":"We have images with different cloth parts and we have to identify cloth id and attributes"},{"metadata":{},"cell_type":"markdown","source":"We will use classical deep learning approach. We will use this [awesome framework](https://github.com/qubvel/segmentation_models.pytorch)"},{"metadata":{},"cell_type":"markdown","source":"# pip installs"},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install segmentation-models-pytorch","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Imports"},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\nimport numpy as np\nimport pandas as pd\nimport torch\nimport cv2\nimport collections\nimport segmentation_models_pytorch as smp\nimport albumentations as albu\nimport torch\nimport seaborn as sns\n\nfrom torch.utils.data import DataLoader\nfrom sklearn.model_selection import train_test_split\nfrom glob import glob\nfrom os import path\nfrom PIL import Image\nfrom tqdm import tqdm\nfrom matplotlib import pyplot as plt\n\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Constants"},{"metadata":{"trusted":true},"cell_type":"code","source":"ROOT_PATH_TRAIN = '/kaggle/input/imaterialist-fashion-2019-FGVC6/train'\nDF_PATH_TRAIN = '/kaggle/input/imaterialist-fashion-2019-FGVC6/train.csv'\nPATH_TO_MODEL_WEIGHTS = '/kaggle/input/za-cho-takoe-testovoe/best_model.pth'\n\nIMAGE_SIZE = (512, 512)\nIAMGE_PREDICTION_SIZE = (256, 256)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Utils fucntions"},{"metadata":{"trusted":true},"cell_type":"code","source":"def rle_decode(mask_rle, shape):\n    '''\n    mask_rle: run-length as string formated: [start0] [length0] [start1] [length1]... in 1d array\n    shape: (height,width) of array to return\n    Returns numpy array according to the shape, 1 - mask, 0 - background\n    '''\n    shape = (shape[1], shape[0])\n    s = mask_rle.split()\n    # gets starts & lengths 1d arrays\n    starts, lengths = [np.asarray(x, dtype=int) for x in (s[0::2], s[1::2])]\n    starts -= 1\n    # gets ends 1d array\n    ends = starts + lengths\n    # creates blank mask image 1d array\n    img = np.zeros(shape[0] * shape[1], dtype=np.uint8)\n    # sets mark pixles\n    for lo, hi in zip(starts, ends):\n        img[lo:hi] = 1\n    # reshape as a 2d mask image\n    return img.reshape(shape).T","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def rle_to_string(runs):\n    return ' '.join(str(x) for x in runs)\n\ndef rle_encode(mask):\n    pixels = mask.T.flatten()\n    # We need to allow for cases where there is a '1' at either end of the sequence.\n    # We do this by padding with a zero at each end when needed.\n    use_padding = False\n    if pixels[0] or pixels[-1]:\n        use_padding = True\n        pixel_padded = np.zeros([len(pixels) + 2], dtype=pixels.dtype)\n        pixel_padded[1:-1] = pixels\n        pixels = pixel_padded\n    rle = np.where(pixels[1:] != pixels[:-1])[0] + 2\n    if use_padding:\n        rle = rle - 1\n    rle[1::2] = rle[1::2] - rle[:-1:2]\n    return rle_to_string(rle)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def visualise_masks(image_name, real_df, predicted_df=None, r_p=ROOT_PATH_TRAIN, im_class=None):\n    # get image\n    img_path = os.path.join(r_p, image_name)\n    img = cv2.imread(img_path)\n    img = cv2.cvtColor(img,cv2.COLOR_BGR2RGB)\n    img = cv2.resize(img, IMAGE_SIZE)\n    \n    # get original mask\n    if im_class is not None:\n        original_raws = real_df[(real_df['ImageId'] == image_name) & (real_df['ClassId'] == im_class)]\n    else:\n        original_raws = real_df[real_df['ImageId'] == image_name]\n                \n    o_h, o_w = int(original_raws['Height'].mean()), int(original_raws['Width'].mean())\n    \n    o_mask = np.zeros(IMAGE_SIZE)        \n        \n    for annotation in original_raws['EncodedPixels']:\n        o_mask += cv2.resize(rle_decode(annotation, (o_h, o_w)), IMAGE_SIZE)\n        \n    o_mask = (o_mask > 0.5).astype(np.uint8)\n    \n    if predicted_df is not None:\n        # get predicted mask\n        if im_class is not None:\n            predicted_raws = predicted_df[(predicted_df['ImageId'] == image_name) & (predicted_df['ClassId'] == im_class)]\n        else:\n            predicted_raws = predicted_df[predicted_df['ImageId'] == image_name]\n                \n        p_mask = np.zeros(IMAGE_SIZE)\n        \n        for annotation in predicted_raws['EncodedPixels']:\n            p_mask += rle_decode(annotation, IMAGE_SIZE)\n        \n        p_mask = (p_mask > 0.5).astype(np.uint8)\n    \n    fig=plt.figure(figsize=(20, 20))\n    fig.add_subplot(1, 3, 1)\n    plt.title('image')\n    plt.imshow(img)\n    fig.add_subplot(1, 3, 2)\n    plt.title('original_mask')\n    plt.imshow(o_mask)\n    \n    if predicted_df is not None:\n        fig.add_subplot(1, 3, 3)\n        plt.title('predicted_mask')\n        plt.imshow(p_mask)\n        plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Data exploration"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df = pd.read_csv(DF_PATH_TRAIN)\ntrain_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In our Classid we have Category id and it's attributes. Predicting attributes is a really hard stuff and competion best practises showed that in most cases it will only harm the perfomence of your Neural Net. So we will lets get Category id"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df['CategoryId'] = train_df.ClassId.apply(lambda x: str(x).split(\"_\")[0])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(20, 7))\nsns.countplot(train_df['CategoryId']);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Moreover in our dataframe we have to have unique pair (ImageId, ClassId) but in our initial dataframe we have each pair for each mask segment.\nIn the following example we have duplicates with ClassId - 31, 32"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df[train_df['ImageId'] == '00000663ed1ff0c4e0132b9b9ac53f6e.jpg']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"visualise_masks('00000663ed1ff0c4e0132b9b9ac53f6e.jpg', train_df, im_class='31')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So lets fix it"},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_unique_class_id_df(inital_df):\n    temp_df = inital_df.groupby(['ImageId','ClassId'])['EncodedPixels'].agg(lambda x: ' '.join(list(x))).reset_index()\n    size_df = inital_df.groupby(['ImageId','ClassId'])['Height', 'Width'].mean().reset_index()\n    temp_df = temp_df.merge(size_df, on=['ImageId','ClassId'], how='left')\n    \n    return temp_df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df = get_unique_class_id_df(train_df)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df[train_df['ImageId'] == '00000663ed1ff0c4e0132b9b9ac53f6e.jpg']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"visualise_masks('00000663ed1ff0c4e0132b9b9ac53f6e.jpg', train_df, im_class='31')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Train/Test split"},{"metadata":{},"cell_type":"markdown","source":"Of course, it is better to have train/validation/test datasets. But now we will stop on train/test. Why?\n* We do not have Early Stopping and other stuff, that will overfit us\n* Orgs did not clarify this point properly "},{"metadata":{},"cell_type":"markdown","source":"We will make train/test split on Classid combinations "},{"metadata":{"trusted":true},"cell_type":"code","source":"def create_one_represent_class(df_param):\n    v_c_df = df_param['CategoryId'].value_counts().reset_index()\n    one_represent = v_c_df.loc[v_c_df['CategoryId'] == 1, 'index'].tolist()\n    df_param.loc[df_param['CategoryId'].isin(one_represent), 'CategoryId'] = 'one_represent'\n    return df_param\n\ndef custom_train_test_split(df_param):\n    \n    df_param['CategoryId'] = df_param.ClassId.apply(lambda x: str(x).split(\"_\")[0])\n    \n    img_categ = train_df.groupby('ImageId')['CategoryId'].apply(list).reset_index()\n    img_categ['CategoryId'] = img_categ['CategoryId'].apply(lambda x: ' '.join(sorted(x)))\n    \n    img_categ = create_one_represent_class(img_categ)\n    \n    img_train, img_val  = train_test_split(img_categ, test_size=0.2, random_state=42, stratify=img_categ['CategoryId'])\n    \n    df_param = df_param.drop(columns='CategoryId')\n    \n    df_train = df_param[df_param['ImageId'].isin(img_train['ImageId'])].reset_index(drop=True)\n    df_val = df_param[df_param['ImageId'].isin(img_val['ImageId'])].reset_index(drop=True)\n    \n    return df_train, df_val","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df = pd.read_csv(DF_PATH_TRAIN)\ntrain_df, val_df = custom_train_test_split(train_df)\n\ntrain_df = get_unique_class_id_df(train_df)\nval_df = get_unique_class_id_df(val_df)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Lets take a look on our categoryid distribution"},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(20, 7))\nplt.title('Train')\nsns.countplot(train_df['ClassId'].apply(lambda x: str(x).split(\"_\")[0]));","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(20, 7))\nplt.title('Validation')\nsns.countplot(val_df['ClassId'].apply(lambda x: str(x).split(\"_\")[0]));","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Awesome matching !!!"},{"metadata":{},"cell_type":"markdown","source":"So now we are ready for DEEEEEEEEEEEEEEEEEEEEEEEEEEEEEep Learning"},{"metadata":{},"cell_type":"markdown","source":"![](https://pbs.twimg.com/media/CnN1pRvUAAA0lWq.jpg)"},{"metadata":{},"cell_type":"markdown","source":"# DL part"},{"metadata":{},"cell_type":"markdown","source":"We will use classical Unet architecture with mobilenet backbone, pretrained on ImageNet. Why ?\n* Unet is classic\n* Backbone is light enough to train it in Kaggle Kernel"},{"metadata":{},"cell_type":"markdown","source":"![Unet](https://i.stack.imgur.com/DjXVU.png)"},{"metadata":{},"cell_type":"markdown","source":"![](http://static.issue.life/Content/img/17-03-2019/636884100747932656.png)"},{"metadata":{},"cell_type":"markdown","source":"We have trained our Net only for 2 epochs (Kaggle Kernel limitation on time)\n* With Dice Loss\n* With Horizontal and Vertical flip augmentations\n* With Adam optimizer\n* Data was normalized by ImageNet stats"},{"metadata":{},"cell_type":"markdown","source":"## Dataset"},{"metadata":{"trusted":true},"cell_type":"code","source":"class UnetDataset(torch.utils.data.Dataset):\n    def __init__(self, image_dir, df, height, width, augmentation=None, preprocessing=None):\n        \n        self.preprocessing = preprocessing\n        self.augmentation = augmentation\n        \n        self.image_dir = image_dir\n        self.df = df\n        \n        self.height = height\n        self.width = width\n        \n        self.image_info = collections.defaultdict(dict)\n        \n        self.df['CategoryId'] = self.df.ClassId.apply(lambda x: str(x).split(\"_\")[0])\n        self.num_classes = self.df['CategoryId'].nunique()\n        \n        temp_df = self.df.groupby('ImageId')['EncodedPixels', 'CategoryId'].agg(lambda x: list(x)).reset_index()\n        size_df = self.df.groupby('ImageId')['Height', 'Width'].mean().reset_index()\n        temp_df = temp_df.merge(size_df, on='ImageId', how='left')\n        \n        for index, row in tqdm(temp_df.iterrows(), total=len(temp_df)):\n            image_id = row['ImageId']\n            image_path = os.path.join(self.image_dir, image_id)\n            self.image_info[index][\"image_id\"] = image_id\n            self.image_info[index][\"image_path\"] = image_path\n            self.image_info[index][\"width\"] = self.width\n            self.image_info[index][\"height\"] = self.height\n            self.image_info[index][\"labels\"] = row[\"CategoryId\"]\n            self.image_info[index][\"orig_height\"] = row[\"Height\"]\n            self.image_info[index][\"orig_width\"] = row[\"Width\"]\n            self.image_info[index][\"annotations\"] = row[\"EncodedPixels\"]\n\n    def __getitem__(self, idx):\n        \n        img_path = self.image_info[idx][\"image_path\"]\n        \n        img = cv2.imread(img_path)\n        img = cv2.cvtColor(img,cv2.COLOR_BGR2RGB)\n        img = cv2.resize(img, (self.width, self.height))\n        \n        # apply preprocessing\n        if self.preprocessing is not None:\n            img = self.preprocessing(image=img)['image']\n            \n        return img, os.path.basename(img_path)\n\n    def __len__(self):\n        return len(self.image_info)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Preprocessing"},{"metadata":{"trusted":true},"cell_type":"code","source":"def to_tensor(x, **kwargs):\n    return x.transpose(2, 0, 1).astype('float32')\n\ndef get_preprocessing(preprocessing_fn):\n    \"\"\"Construct preprocessing transform\n    \n    Args:\n        preprocessing_fn (callbale): data normalization function \n            (can be specific for each pretrained neural network)\n    Return:\n        transform: albumentations.Compose\n    \n    \"\"\"\n    \n    _transform = [\n        albu.Lambda(image=preprocessing_fn),\n        albu.Lambda(image=to_tensor, mask=to_tensor),\n    ]\n    return albu.Compose(_transform)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Model"},{"metadata":{"trusted":true},"cell_type":"code","source":"ENCODER = 'mobilenet_v2'\nENCODER_WEIGHTS = 'imagenet'\nDEVICE = 'cuda'\n\nACTIVATION = 'sigmoid'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model = smp.Unet(\n    encoder_name=ENCODER, \n    encoder_weights=ENCODER_WEIGHTS, \n    classes=46, \n    activation=ACTIVATION,\n)\n\npreprocessing_fn = smp.encoders.get_preprocessing_fn(ENCODER, ENCODER_WEIGHTS)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model.load_state_dict(torch.load(PATH_TO_MODEL_WEIGHTS, \n                                 map_location=torch.device(DEVICE)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model = model.to(DEVICE)\nmodel.eval();","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Dataloader"},{"metadata":{"trusted":true},"cell_type":"code","source":"test_dataset = UnetDataset(\n    ROOT_PATH_TRAIN,\n    val_df,\n    IAMGE_PREDICTION_SIZE[0],\n    IAMGE_PREDICTION_SIZE[0], \n    preprocessing=get_preprocessing(preprocessing_fn),\n)\n\ntest_loader = DataLoader(test_dataset, batch_size=32, shuffle=False, num_workers=2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Inference"},{"metadata":{"trusted":true},"cell_type":"code","source":"def create_final_df(result_dict):\n    final_df = {'ImageId':[], 'EncodedPixels':[], 'ClassId':[]}\n    for im_name, im_dict in result_dict.items():\n        final_df['ImageId'] += [im_name]*len(im_dict)\n        for cls_id, enc_p in im_dict.items():\n            final_df['EncodedPixels'].append(enc_p)\n            final_df['ClassId'].append(cls_id)\n            \n    return pd.DataFrame(final_df)\n\nclass InfernceModel(object):\n    def __init__(self, nn_model, device, output_size=(512,512), threshold=0.5, size_min_mask=250):\n        self.nn_model = nn_model\n        self.output_size = output_size\n        self.threshold = threshold\n        self.size_min_mask = size_min_mask\n        self.device = device\n        \n    def __call__(self, inf_dataloader):\n        result = {}\n        with torch.no_grad():\n            for batch in tqdm(inf_dataloader, \n                              total=len(inf_dataloader.dataset) // inf_dataloader.batch_size):\n                names = batch[1]\n                batch = model(batch[0].to(self.device)).detach().cpu().numpy()\n                for idx in range(batch.shape[0]):\n                    cur_result = self.post_process(batch[idx])\n                    if len(cur_result) > 0:\n                        result[names[idx]] = cur_result\n                        \n        final_df = create_final_df(result) \n        final_df['Height'] = self.output_size[0]\n        final_df['Width'] = self.output_size[1]\n        final_df['ClassId'] = final_df['ClassId'].astype(str)\n                    \n        return final_df\n        \n    def post_process(self, mask):\n        rle_mask = {}\n        for idx in range(mask.shape[0]):\n            item_mask = mask[idx]\n            \n            item_mask = (item_mask > self.threshold).astype(np.uint8)\n            item_mask = cv2.resize(item_mask, self.output_size)\n            \n            if item_mask.sum() < self.size_min_mask:\n                continue\n            else:\n                rle_mask[str(idx)] = rle_encode(item_mask)\n            \n        return rle_mask","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"inf_model = InfernceModel(nn_model=model, device=DEVICE)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"predicted_df = inf_model(test_loader)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"predicted_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Evaluate results"},{"metadata":{},"cell_type":"markdown","source":"### Visualize"},{"metadata":{},"cell_type":"markdown","source":"Firstly, on all classes together"},{"metadata":{"trusted":true},"cell_type":"code","source":"visualise_masks('000b3a87508b0fa185fbd53ecbe2e4c6.jpg', val_df, predicted_df)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"visualise_masks('fffc631acce2e28e1628de685d40c980.jpg', val_df, predicted_df)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"visualise_masks('ffec8295f37df6ea12eecbb60d2c23d4.jpg', val_df, predicted_df)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So we get some main cloth parts"},{"metadata":{},"cell_type":"markdown","source":"Now lets look at different CategoryID"},{"metadata":{},"cell_type":"markdown","source":"Firstly, on some frequent"},{"metadata":{"trusted":true},"cell_type":"code","source":"visualise_masks('001039acb67251508b1b32fd37a49f43.jpg', val_df, predicted_df, im_class='31')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"visualise_masks('005ccdb239e2d6cfe62506dd6eb5693e.jpg', val_df, predicted_df, im_class='10')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Got it. Also these masks are huge enough for our week net"},{"metadata":{},"cell_type":"markdown","source":"But as for smaller masks and less frequent"},{"metadata":{"trusted":true},"cell_type":"code","source":"visualise_masks('000cd2e13d1bdd28f480304d7bb9e1ca.jpg', val_df, predicted_df, im_class='23')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"visualise_masks('05e6ef19957d43524d972de6d2f41b57.jpg', val_df, predicted_df, im_class='45')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Not very good results for these masks"},{"metadata":{},"cell_type":"markdown","source":"### Metric"},{"metadata":{"trusted":true},"cell_type":"code","source":"# mainly from here https://www.kaggle.com/kyazuki/calculate-evaluation-score\n\ndef IoU(A,B):\n    AorB = np.logical_or(A,B).astype('int')\n    AandB = np.logical_and(A,B).astype('int')\n    IoU = AandB.sum() / AorB.sum()\n    return IoU\n\ndef IoU_threshold(data):\n    # Note: This rle_to_mask should be called before loop below for speed-up! We currently implement here to reduse memory usage.\n    mask_gt = rle_decode(data['EncodedPixels_true'], (int(data['Height_true']), int(data['Width_true'])))\n    mask_pred = rle_decode(data['EncodedPixels_pred'], (int(data['Height_pred']), int(data['Width_pred'])))\\\n    \n    if (int(data['Height_true']), int(data['Width_true'])) != IMAGE_SIZE:\n        mask_gt = cv2.resize(mask_gt, IMAGE_SIZE)\n    if (int(data['Height_pred']), int(data['Width_pred'])) != IMAGE_SIZE:\n        mask_pred = cv2.resize(mask_pred, IMAGE_SIZE)\n    \n    mask_gt = mask_gt > 0.5\n    mask_pred = mask_pred > 0.5\n    \n    return IoU(mask_gt, mask_pred)\n\ndef best_metric(true_df, pred_df):\n    eval_df = pd.merge(true_df, pred_df, how='outer', on=['ImageId', 'ClassId'], suffixes=['_true', '_pred'])\n\n    # IoU for True Positive\n    idx_ = eval_df['EncodedPixels_true'].notnull() & eval_df['EncodedPixels_pred'].notnull()\n    IoU = eval_df[idx_].apply(IoU_threshold, axis=1)\n\n    # False Positive\n    fp = (eval_df['EncodedPixels_true'].isnull() & eval_df['EncodedPixels_pred'].notnull()).sum()\n\n    # False Negative\n    fn = (eval_df['EncodedPixels_true'].notnull() & eval_df['EncodedPixels_pred'].isnull()).sum()\n\n    threshold_IoU = np.arange(0.5, 1.0, 0.05)\n    scores = []\n    for th in threshold_IoU:\n        # True Positive\n        tp = (IoU > th).sum()\n        iou_fp = (IoU <= th).sum()\n\n        # False Positive (not Ground Truth) + False Positive (under IoU threshold)\n        fp = fp + iou_fp\n\n        # Calculate evaluation score\n        score = tp / (tp + fp + fn)\n        scores.append(score)\n\n    mean_score = sum(scores) / len(threshold_IoU)\n    return mean_score","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Lets test it"},{"metadata":{"trusted":true},"cell_type":"code","source":"images_to_count_metric = val_df['ImageId'].unique()[:100]\n\nbest_metric(val_df[val_df['ImageId'].isin(images_to_count_metric)], val_df[val_df['ImageId'].isin(images_to_count_metric)])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"And now evaluate our prediction"},{"metadata":{"trusted":true},"cell_type":"code","source":"best_metric(val_df, predicted_df)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## What can be improved"},{"metadata":{},"cell_type":"markdown","source":"EVERYTHING"},{"metadata":{},"cell_type":"markdown","source":"* Use MaskRCNN or other appropriate architecture for biag amount of classes\n* Use bigger backbone\n* Train until Net converged\n* Use bigger augmentations\n* Tune net hyperparams\n* Use TwoHead Net\n* Use Test Time Augmentations\n* Use Blending\n\nAnd so on ...."},{"metadata":{},"cell_type":"markdown","source":"Why we did not do all this stuff\n\nBecause\n\n* 30 hours per week of railway\n* one GPU per account\n* execution - up to 6 hours"},{"metadata":{},"cell_type":"markdown","source":"# Final thoughts or kozaki pishut pismo BESTu"},{"metadata":{},"cell_type":"markdown","source":"![](https://zaxid.net/resources/photos/news/640x360_DIR/201511/1371769.jpg?201805281755)"},{"metadata":{},"cell_type":"markdown","source":"We think that such format of test task is one of the worst variants. Why ?\n* This task requires Deep Learning background, which is overkill for contest, where most participants are students \n* Even if participant has Deep Learning background, he/she will need a lot of computational resources (GPUs), which cost money or use his/her GPU Quota on Kaggle\n* Metric and Train/Test were not provided. So all participants will have hardly biased results. It is important, because evaluating DL/ML/DS tasks by 'beautiful code' or 'interesting approach' or something like this is not the best practice. Of course, these criteria should be taken into account, but main criteria is target metric !"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}