{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"Thanks to the great work finished by **REMEK KINAS** !\n\nhttps://www.kaggle.com/code/remekkinas/remove-background-salient-object-detection/notebook\n\nThis notbook is ***based on*** the notebook above. Change a little parameters and **refactor** to get  **effecient** detection result.\n\n**Changes**\n\n+ **CLAHE** : use CLAHE in opencv to make image brighter and a higer constrast\n\n+ **THRESHOLD** : change from 0.3 to 0.9\n\n+ **BoundingRec** : import from cv2, which used to find the bounding of whale(after background remove) and resize all the image to the same size.\n\n+ **Other changes** : fix latent exception\n\n+ **Something New** : there are about 5% images which has a bad result after u-net background remove. I use the a area threshold to sift the valid images. Those invalid remain the original images. \n","metadata":{}},{"cell_type":"markdown","source":"## 1. SETUP NOTEBOOK (MODULES)\nLet's clone U-2-Net repository","metadata":{}},{"cell_type":"code","source":"!git clone https://github.com/shreyas-bk/U-2-Net\n    \nimport sys\nsys.path.append('./U-2-Net')","metadata":{"execution":{"iopub.status.busy":"2022-04-04T00:55:28.698308Z","iopub.execute_input":"2022-04-04T00:55:28.698918Z","iopub.status.idle":"2022-04-04T00:55:30.310123Z","shell.execute_reply.started":"2022-04-04T00:55:28.698825Z","shell.execute_reply":"2022-04-04T00:55:30.309317Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import cv2 as cv\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport pandas as pd\nimport torch\n\nfrom data_loader import RescaleT\nfrom data_loader import ToTensor\nfrom data_loader import ToTensorLab\nfrom data_loader import SalObjDataset\nfrom model import U2NET \nfrom model import U2NETP \n\nfrom IPython.display import display\nfrom PIL import Image as Img\n\nfrom torchvision import transforms\nfrom torch.utils.data import Dataset, DataLoader\nfrom torch.autograd import Variable","metadata":{"execution":{"iopub.status.busy":"2022-04-04T00:55:44.814805Z","iopub.execute_input":"2022-04-04T00:55:44.81549Z","iopub.status.idle":"2022-04-04T00:55:44.821721Z","shell.execute_reply.started":"2022-04-04T00:55:44.815453Z","shell.execute_reply":"2022-04-04T00:55:44.821046Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2. TAKE CANDIDATES FOR BACKGROUND REMOVAL","metadata":{}},{"cell_type":"markdown","source":"For this demo I use cropped images from dataset provided by @phalanx [cropped&resized(512x512) dataset using detic](https://www.kaggle.com/c/happy-whale-and-dolphin/discussion/305503). Thank you for contributing in this competition.","metadata":{}},{"cell_type":"code","source":"train_df = pd.read_csv(\"../input/whale2-cropped-dataset/train2.csv\")\n\ninput_path = \"../input/whale2-cropped-dataset/cropped_train_images/cropped_train_images\"\n\nroot_in = '../input/whale2-cropped-dataset'\nroot_out = './'# root_out give the folder direciton of after-processed images, csv..","metadata":{"execution":{"iopub.status.busy":"2022-04-04T00:57:45.94532Z","iopub.execute_input":"2022-04-04T00:57:45.945584Z","iopub.status.idle":"2022-04-04T00:57:46.02318Z","shell.execute_reply.started":"2022-04-04T00:57:45.945555Z","shell.execute_reply":"2022-04-04T00:57:46.022423Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import os \nos.makedirs('./subtraction_train')\nos.makedirs('./subtraction_test')","metadata":{"execution":{"iopub.status.busy":"2022-04-04T00:56:29.582689Z","iopub.execute_input":"2022-04-04T00:56:29.583579Z","iopub.status.idle":"2022-04-04T00:56:29.588009Z","shell.execute_reply.started":"2022-04-04T00:56:29.583541Z","shell.execute_reply":"2022-04-04T00:56:29.587265Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's take one of TOP10 individual. You can experiment with other individuals. Quality of prediction depends on photo but I will work on improving prediction (I will probably train this model on custom data).","metadata":{}},{"cell_type":"code","source":"train_df.individual_id.value_counts().head(10)","metadata":{"execution":{"iopub.status.busy":"2022-04-04T00:56:35.811164Z","iopub.execute_input":"2022-04-04T00:56:35.811693Z","iopub.status.idle":"2022-04-04T00:56:35.842105Z","shell.execute_reply.started":"2022-04-04T00:56:35.811651Z","shell.execute_reply":"2022-04-04T00:56:35.841438Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"For experiments we use only one individual ID=281504409737... just to check solution performance and quality of mask.","metadata":{}},{"cell_type":"code","source":"paths = [file for file in train_df.image]\nimg_to_draw = [input_path + '/' + file for file in train_df.image]","metadata":{"execution":{"iopub.status.busy":"2022-04-04T00:57:49.088802Z","iopub.execute_input":"2022-04-04T00:57:49.089522Z","iopub.status.idle":"2022-04-04T00:57:49.124505Z","shell.execute_reply.started":"2022-04-04T00:57:49.089486Z","shell.execute_reply":"2022-04-04T00:57:49.123565Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, axes = plt.subplots(5, 5, figsize=(20,20))\n\nfor idx, img in enumerate(img_to_draw[0:25]):\n    i = idx % 5 \n    j = idx // 5\n    image = Img.open(img)\n    iar_shp = np.array(image).shape\n    axes[i, j].axis('off')\n    axes[i, j].imshow(image)\n    \nplt.subplots_adjust(wspace=0.05, hspace=0.05)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-04-04T00:57:50.20864Z","iopub.execute_input":"2022-04-04T00:57:50.209279Z","iopub.status.idle":"2022-04-04T00:57:52.989297Z","shell.execute_reply.started":"2022-04-04T00:57:50.209238Z","shell.execute_reply":"2022-04-04T00:57:52.987722Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 3. Use CLAHE","metadata":{}},{"cell_type":"markdown","source":"**Here we use GPU to process the images**","metadata":{}},{"cell_type":"code","source":"THRESHOLD = 0.9\nBATCH_SIZE = 64\nDEVICE = \"cuda\" if torch.cuda.is_available() else \"cpu\"","metadata":{"execution":{"iopub.status.busy":"2022-04-04T00:58:01.357282Z","iopub.execute_input":"2022-04-04T00:58:01.357821Z","iopub.status.idle":"2022-04-04T00:58:01.426649Z","shell.execute_reply.started":"2022-04-04T00:58:01.357765Z","shell.execute_reply":"2022-04-04T00:58:01.424002Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def CLAHE_Convert(origin_input):\n    imidx = origin_input['imidx']\n    label = origin_input['label']\n    clahe = cv.createCLAHE(clipLimit=3, tileGridSize=(24,32))\n    img = np.asarray(origin_input['image'])\n    img = cv.cvtColor(img, cv.COLOR_BGR2HSV)\n    img[:,:,-1] = clahe.apply(img[:,:,-1])\n    img = cv.cvtColor(img, cv.COLOR_HSV2BGR)\n    \n    return {'imidx':imidx, 'image':img,'label':label}\n\ndef BilateralFilter_Convert(origin_input):\n    imidx = origin_input['imidx']\n    label = origin_input['label']\n    img = np.asarray(origin_input['image'])\n    img = cv.bilateralFilter(img,5,75,75)\n    \n    return {'imidx':imidx, 'image':img,'label':label}\n\ndef EqualizeHist_Convert(origin_input):\n    imidx = origin_input['imidx']\n    label = origin_input['label']\n    img = np.asarray(origin_input['image'])\n    img = cv.cvtColor(img, cv.COLOR_BGR2HSV)\n    img[:,:,-1] = cv.equalizeHist(img[:,:,-1])\n    img = cv.cvtColor(img, cv.COLOR_HSV2BGR)\n    \n    return {'imidx':imidx, 'image':img,'label':label}","metadata":{"execution":{"iopub.status.busy":"2022-04-04T01:14:43.903505Z","iopub.execute_input":"2022-04-04T01:14:43.904314Z","iopub.status.idle":"2022-04-04T01:14:43.914017Z","shell.execute_reply.started":"2022-04-04T01:14:43.904278Z","shell.execute_reply":"2022-04-04T01:14:43.913238Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def normPRED(d):\n    ma = torch.max(d)\n    mi = torch.min(d)\n    dn = (d-mi)/(ma-mi)\n    return dn","metadata":{"execution":{"iopub.status.busy":"2022-04-04T00:58:12.015879Z","iopub.execute_input":"2022-04-04T00:58:12.016667Z","iopub.status.idle":"2022-04-04T00:58:12.021604Z","shell.execute_reply.started":"2022-04-04T00:58:12.016615Z","shell.execute_reply":"2022-04-04T00:58:12.020855Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def pred_unet(model, imgs):\n    '''The old version of pred_unet!\n        Used just to show the difference'''\n    test_salobj_dataset = SalObjDataset(img_name_list = imgs, lbl_name_list = [], transform = transforms.Compose([RescaleT(320),ToTensorLab(flag=0)]))\n    test_salobj_dataloader = DataLoader(test_salobj_dataset, batch_size=1, shuffle=False, num_workers = 0)\n    \n    for i_test, data_test in enumerate(test_salobj_dataloader):\n        \n        inputs_test = data_test['image']\n        inputs_test = inputs_test.type(torch.FloatTensor)\n\n        if torch.cuda.is_available():\n            inputs_test = Variable(inputs_test.cuda())\n        else:\n            inputs_test = Variable(inputs_test)\n\n        d1, d2, d3, d4, d5, d6, d7 = net(inputs_test)\n\n        predict = d5[:,0,:,:]\n        predict = normPRED(predict)\n        \n        del d1, d2, d3, d4, d5, d6, d7\n\n        predict = predict.squeeze()\n        predict_np = predict.cpu().data.numpy()\n\n        # Masked image - using threshold you can soften/sharpen mask boundaries\n        predict_np[predict_np > THRESHOLD] = 1\n        predict_np[predict_np <= THRESHOLD] = 0\n        mask = Img.fromarray(predict_np*255).convert('RGB')\n        image = Img.open(imgs[0])\n        imask = mask.resize((image.width, image.height), resample=Img.BILINEAR)\n        back = Img.new(\"RGB\", (image.width, image.height), (255, 255, 255))\n        mask = imask.convert('L')\n        im_out = Img.composite(image, back, mask)\n        \n        # Sailient mask \n        salient_mask = np.array(image)\n        mask_layer = np.array(imask)        \n        mask_layer[mask_layer == 255] = 50 # offest on RED channel\n        salient_mask[:,:,0] += mask_layer[:,:, 0]\n        salient_mask = np.clip(salient_mask, 0, 255) \n    \n    return np.array(im_out), np.array(image), np.array(salient_mask), np.array(mask)","metadata":{"execution":{"iopub.status.busy":"2022-04-04T01:22:00.13502Z","iopub.execute_input":"2022-04-04T01:22:00.135312Z","iopub.status.idle":"2022-04-04T01:22:00.149592Z","shell.execute_reply.started":"2022-04-04T01:22:00.135278Z","shell.execute_reply":"2022-04-04T01:22:00.147352Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def pred_unet_CLAHE_show(model, imgs):\n    '''Used just to show the difference,\n        Do not use it to write image'''\n    test_salobj_dataset = SalObjDataset(img_name_list = imgs, lbl_name_list = [], transform = transforms.Compose([CLAHE_Convert, RescaleT(320),ToTensorLab(flag=0)]))\n    test_salobj_dataloader = DataLoader(test_salobj_dataset, batch_size=1, shuffle=False, num_workers = 0)\n    \n    for i_test, data_test in enumerate(test_salobj_dataloader):\n        \n        inputs_test = data_test['image']\n        inputs_test = inputs_test.type(torch.FloatTensor)\n\n        if torch.cuda.is_available():\n            inputs_test = Variable(inputs_test.cuda())\n        else:\n            inputs_test = Variable(inputs_test)\n\n        d1, d2, d3, d4, d5, d6, d7 = net(inputs_test)\n\n        predict = d5[:,0,:,:]\n        predict = normPRED(predict)\n        \n        del d1, d2, d3, d4, d5, d6, d7\n\n        predict = predict.squeeze()\n        predict_np = predict.cpu().data.numpy()\n\n        # Masked image - using threshold you can soften/sharpen mask boundaries\n        predict_np[predict_np > THRESHOLD] = 1\n        predict_np[predict_np <= THRESHOLD] = 0\n        mask = Img.fromarray(predict_np*255).convert('RGB')\n        image = Img.open(imgs[0])\n        imask = mask.resize((image.width, image.height), resample=Img.BILINEAR)\n        back = Img.new(\"RGB\", (image.width, image.height), (255, 255, 255))\n        mask = imask.convert('L')\n        im_out = Img.composite(image, back, mask)\n        \n        # Sailient mask \n        salient_mask = np.array(image)\n        mask_layer = np.array(imask)        \n        mask_layer[mask_layer == 255] = 50 # offest on RED channel\n        salient_mask[:,:,0] += mask_layer[:,:, 0]\n        salient_mask = np.clip(salient_mask, 0, 255) \n    \n    return np.array(im_out), np.array(image), np.array(salient_mask), np.array(mask)","metadata":{"execution":{"iopub.status.busy":"2022-04-04T01:22:01.78386Z","iopub.execute_input":"2022-04-04T01:22:01.784322Z","iopub.status.idle":"2022-04-04T01:22:01.796418Z","shell.execute_reply.started":"2022-04-04T01:22:01.784285Z","shell.execute_reply":"2022-04-04T01:22:01.795575Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"VALID_THERESHOLD = 0.2 # use the a area threshold to sift the valid images","metadata":{"execution":{"iopub.status.busy":"2022-04-04T00:58:12.620641Z","iopub.execute_input":"2022-04-04T00:58:12.621369Z","iopub.status.idle":"2022-04-04T00:58:12.62485Z","shell.execute_reply.started":"2022-04-04T00:58:12.621314Z","shell.execute_reply":"2022-04-04T00:58:12.624089Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def pred_unet_CLAHE(model, imgs, paths, train=False):\n    salobj_dataset = SalObjDataset(img_name_list = imgs, lbl_name_list = [], transform = transforms.Compose([CLAHE_Convert, RescaleT(320), ToTensorLab(flag=0)]))\n    salobj_dataloader = DataLoader(salobj_dataset, batch_size=BATCH_SIZE, shuffle=False, num_workers = 1)\n    \n    model.to(DEVICE)\n    \n    error_log = []\n    for i, batch in enumerate(salobj_dataloader):\n        model.eval()\n        img_batch = batch['image'] # (batch size, channel, row, columns)\n        images = img_batch.type(torch.float32) # change torch.double -> torch.float\n        images = images.to(DEVICE) \n        with torch.no_grad():\n            d1, d2, d3, d4, d5, d6, d7 = model(images)\n        \n        predict = d5[:,0,:,:]# (batch size, 1, row, columns)\n        predict = normPRED(predict)\n\n        del d1, d2, d3, d4, d5, d6, d7\n\n        predict = predict.squeeze()# (batch size, row, columns)\n        predict_np = predict.cpu().numpy()\n\n        # Masked image - using threshold you can soften/sharpen mask boundaries\n        predict_np[predict_np > THRESHOLD] = 1\n        predict_np[predict_np <= THRESHOLD] = 0\n\n        for j in range(len(predict_np)):\n            file = paths[i * BATCH_SIZE + j]\n            mask_np = predict_np[j]\n            mask = Img.fromarray(mask_np*255).convert('RGB')\n            image = Img.open(imgs[i * BATCH_SIZE + j])\n            mask = mask.resize((image.width, image.height), resample=Img.BILINEAR)\n            back = Img.new(\"RGB\", (image.width, image.height), (255, 255, 255)) # WHITE Backgroud\n            mask = mask.convert('L')\n            im_out = Img.composite(image, back, mask)\n            \n            mask_rs = np.array(mask)\n            \n            x,y,w,h = cv.boundingRect (mask_rs)\n            ymin = y\n            ymax = y + h\n            xmin = x\n            xmax = x + w\n            \n            #used to sift invalid image by the \"1\"(white) area / rectangle area(white and black)\n            if (x,y,w,h) == (0,0,0,0) or (((mask_rs != 0).sum()) / (w * h)) < a:\n                crop_img = np.array(image)\n                error_log.append(file)\n                print('Failed:\\t', file)\n            else:\n                im_out_np = np.array(im_out)\n                crop_img = im_out_np[ymin:ymax, xmin:xmax]\n                crop_img = cv.resize(crop_img, (im_out_np.shape[0], im_out_np.shape[1]), interpolation = cv.INTER_AREA)\n            if train:\n                cv.imwrite(os.path.join(root_out, 'subtraction_train', file), crop_img)\n            else:\n                cv.imwrite(os.path.join(root_out, 'subtraction_test', file), crop_img)\n    if train:\n        pd.DataFrame({'image':error_log}).to_csv(os.path.join(root_out, 'Ignorance_Train_Img.csv'))\n    else:\n        pd.DataFrame({'image':error_log}).to_csv(os.path.join(root_out, 'Ignorance_Test_Img.csv'))","metadata":{"execution":{"iopub.status.busy":"2022-04-04T01:32:40.383846Z","iopub.execute_input":"2022-04-04T01:32:40.384159Z","iopub.status.idle":"2022-04-04T01:32:40.402566Z","shell.execute_reply.started":"2022-04-04T01:32:40.384125Z","shell.execute_reply":"2022-04-04T01:32:40.401842Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"UNET2_SMALL = False","metadata":{"execution":{"iopub.status.busy":"2022-04-04T01:01:17.161396Z","iopub.execute_input":"2022-04-04T01:01:17.162098Z","iopub.status.idle":"2022-04-04T01:01:17.167497Z","shell.execute_reply.started":"2022-04-04T01:01:17.162061Z","shell.execute_reply":"2022-04-04T01:01:17.165103Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%capture\n\nif UNET2_SMALL:\n    model_dir = \"./U-2-Net/u2netp.pth\"  # Faster ... a lot (!) but less accurate\n    net = U2NETP(3,1) \nelse:\n    model_dir = \"../input/u-square-net-model/u2net.pth\"\n    net = U2NET(3,1) \n\n\nif torch.cuda.is_available():\n    net.load_state_dict(torch.load(model_dir))\n    net.cuda()\nelse:        \n    net.load_state_dict(torch.load(model_dir, map_location=torch.device('cpu')))\n\nnet.eval()","metadata":{"execution":{"iopub.status.busy":"2022-04-04T01:01:17.805227Z","iopub.execute_input":"2022-04-04T01:01:17.805769Z","iopub.status.idle":"2022-04-04T01:01:23.322943Z","shell.execute_reply.started":"2022-04-04T01:01:17.80573Z","shell.execute_reply":"2022-04-04T01:01:23.322107Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 4. U2-Net RESULT VISUALIZATION","metadata":{}},{"cell_type":"code","source":"fig, axes = plt.subplots(2, 1, figsize=(30,20))\ni_m_t = img_to_draw[0:1]\nfor idx, img in enumerate(i_m_t):\n    #CLAHE\n    image, im_oryg, sal_map, mask = pred_unet(net, [i_m_t[idx]]) \n    result = np.concatenate((im_oryg, sal_map, image), axis=1)\n    result_img = Img.fromarray(result)\n    axes[idx].axis('off')\n    axes[idx].set_title('Original', fontsize=32)\n    axes[idx].imshow(result_img)\n    \n    \n    image, im_oryg, sal_map, mask = pred_unet_CLAHE_show(net, [i_m_t[idx]]) \n    result = np.concatenate((im_oryg, sal_map, image), axis=1)\n    result_img = Img.fromarray(result)\n    axes[idx+1].axis('off')\n    axes[idx+1].set_title('After CLAHE', fontsize=32)\n    axes[idx+1].imshow(result_img)\n    \n\n\nplt.subplots_adjust(wspace=0.05, hspace=0.1)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-04-04T01:27:18.853507Z","iopub.execute_input":"2022-04-04T01:27:18.853828Z","iopub.status.idle":"2022-04-04T01:27:20.295263Z","shell.execute_reply.started":"2022-04-04T01:27:18.853788Z","shell.execute_reply":"2022-04-04T01:27:20.294642Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 5. TRANSFORM & WRITE IMAGES","metadata":{}},{"cell_type":"markdown","source":"Let's use parameter ***train*** to indicate that we are select and process the images from train or test images","metadata":{}},{"cell_type":"code","source":"# Transform and write Train images\n'''train_df = pd.read_csv(\"../input/whale2-cropped-dataset/train2.csv\")\n\ninput_path = \"../input/whale2-cropped-dataset/cropped_train_images/cropped_train_images\"\npaths = [file for file in train_df.image]\nimg_to_draw = [input_path + '/' + file for file in train_df.image]\npred_unet_CLAHE(net, img_to_draw, paths, train=True)'''","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Transform and write Test images\n'''test_df = pd.read_csv(os.path.join(root_in, 'test2.csv'))\n\ninput_path = os.path.join(root_in, \"cropped_test_images/cropped_test_images\")\npaths = [file for file in test_df.image]\nimg_to_draw = [input_path + '/' + file for file in test_df.image]\npred_unet_CLAHE(net, img_to_draw, paths, train=False)'''\n","metadata":{"execution":{"iopub.status.busy":"2022-04-04T01:35:28.596303Z","iopub.execute_input":"2022-04-04T01:35:28.596561Z","iopub.status.idle":"2022-04-04T01:35:28.650695Z","shell.execute_reply.started":"2022-04-04T01:35:28.596532Z","shell.execute_reply":"2022-04-04T01:35:28.649982Z"},"trusted":true},"execution_count":null,"outputs":[]}]}