Hello everyone
\nI followed the steps below to implement pseudo label in submit notebook：
①Inference on the test dataset, generate classification confidences and test labels.
\n②Select the test set with confidence higher than 0.95 (pseudo label dataset) and add it to the training set.
\n③Retrain the training dataset and pseudo label dataset
\n④Final test and generate submission.csv
It always prompts Submission CSV Not Found error. According to my analysis, after submitting the results, the error will be reported when the first or second step is completed. The code do not start to retrain. But putting multiple pictures in the test set folder on your own computer can run normally and generate submission.csv files. It can be run in the kaggle notebook, too, and a test set image is added to the training set and retrained.
\nI still don't understand what is wrong.
Hello everyone
\nI followed the steps below to implement pseudo label in submit notebook：
①Inference on the test dataset, generate classification confidences and test labels.
\n②Select the test set with confidence higher than 0.95 (pseudo label dataset) and add it to the training set.
\n③Retrain the training dataset and pseudo label dataset
\n④Final test and generate submission.csv
It always prompts Submission CSV Not Found error. According to my analysis, after submitting the results, the error will be reported when the first or second step is completed. The code do not start to retrain. But putting multiple pictures in the test set folder on your own computer can run normally and generate submission.csv files. It can be run in the kaggle notebook, too, and a test set image is added to the training set and retrained.
\nI still don't understand what is wrong.
And here is my code
\npackage_path = '../input/pytorch-image-model' #'../input/efficientnet-pytorch-07/efficientnet_pytorch-0.7.0'\nimport sys; sys.path.append(package_path)\n# Since the length of the comment cannot exceed 20000, so I delete the import *\n\nCFG = {\n 'fold_num': 5,\n 'seed': 719,\n 'model_arch': 'tf_efficientnet_b4_ns',\n 'img_size': 512, #512\n 'epochs': 1, #10\n 'train_bs': 8,\n 'valid_bs': 8,\n 'T_0': 1,\n 'lr': 1e-4,\n 'min_lr': 1e-6,\n 'weight_decay':1e-6,\n 'num_workers': 4,\n 'accum_iter': 2, # suppoprt to do batch accumulation for backprop with effectively larger batch size\n 'verbose_step': 1,\n 'device': 'cuda:0',\n 'tta': 1,\n 'used_epochs': [0],\n 'pseudo_used_epochs': [0],\n 'weights': [1],\n 'pseudo_weights':[1]\n}\n\ntrain = pd.read_csv('../input/cassava-leaf-disease-classification/train.csv')\nsubmission = pd.read_csv('../input/cassava-leaf-disease-classification/sample_submission.csv')\n\ndef seed_everything(seed):\n random.seed(seed)\n os.environ['PYTHONHASHSEED'] = str(seed)\n np.random.seed(seed)\n torch.manual_seed(seed)\n torch.cuda.manual_seed(seed)\n torch.backends.cudnn.deterministic = True\n torch.backends.cudnn.benchmark = True\n\ndef get_img(path):\n im_bgr = cv2.imread(path)\n im_rgb = im_bgr[:, :, ::-1]\n #print(im_rgb)\n return im_rgb\n\nclass CassavaDataset(Dataset):\n def __init__(\n self, df, data_root, transforms=None, output_label=True\n ):\n\n super().__init__()\n self.df = df.reset_index(drop=True).copy()\n self.transforms = transforms\n self.data_root = data_root\n self.output_label = output_label\n\n def __len__(self):\n return self.df.shape[0]\n\n def __getitem__(self, index: int):\n\n # get labels\n if self.output_label:\n target = self.df.iloc[index]['label']\n\n path = \"{}/{}\".format(self.data_root, self.df.iloc[index]['image_id'])\n\n img = get_img(path)\n\n if self.transforms:\n img = self.transforms(image=img)['image']\n\n # do label smoothing\n if self.output_label == True:\n return img, target\n else:\n return img\n\n\nclass pseudo_CassavaDataset(Dataset):\n def __init__(\n self, df, data_root, pseudo_data_root, transforms=None, output_label=True\n ):\n\n super().__init__()\n self.df = df.reset_index(drop=True).copy()\n self.transforms = transforms\n self.data_root = data_root\n self.pseudo_data_root = pseudo_data_root\n self.output_label = output_label\n\n\n def __len__(self):\n return self.df.shape[0]\n\n def __getitem__(self, index: int):\n\n # get labels\n if self.output_label:\n target = self.df.iloc[index]['label']\n\n path = \"{}/{}\".format(self.data_root, self.df.iloc[index]['image_id'])\n pseudo_path = \"{}/{}\".format(self.pseudo_data_root, self.df.iloc[index]['image_id'])\n\n\n if os.path.exists(path):\n img = get_img(path)\n else:\n img = get_img(pseudo_path)\n\n\n if self.transforms:\n img = self.transforms(image=img)['image']\n\n # do label smoothing\n if self.output_label == True:\n return img, target\n else:\n return img\n\n\n\nfrom albumentations import (\n HorizontalFlip, VerticalFlip, IAAPerspective, ShiftScaleRotate, CLAHE, RandomRotate90,\n Transpose, ShiftScaleRotate, Blur, OpticalDistortion, GridDistortion, HueSaturationValue,\n IAAAdditiveGaussianNoise, GaussNoise, MotionBlur, MedianBlur, IAAPiecewiseAffine, RandomResizedCrop,\n IAASharpen, IAAEmboss, RandomBrightnessContrast, Flip, OneOf, Compose, Normalize, Cutout, CoarseDropout, ShiftScaleRotate, CenterCrop, Resize\n)\n\nfrom albumentations.pytorch import ToTensorV2\n\n\ndef get_train_transforms():\n return Compose([\n RandomResizedCrop(CFG['img_size'], CFG['img_size']),\n Transpose(p=0.5),\n HorizontalFlip(p=0.5),\n VerticalFlip(p=0.5),\n ShiftScaleRotate(p=0.5),\n HueSaturationValue(hue_shift_limit=0.2, sat_shift_limit=0.2, val_shift_limit=0.2, p=0.5),\n RandomBrightnessContrast(brightness_limit=(-0.1,0.1), contrast_limit=(-0.1, 0.1), p=0.5),\n Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225], max_pixel_value=255.0, p=1.0),\n CoarseDropout(p=0.5),\n Cutout(p=0.5),\n ToTensorV2(p=1.0),\n ], p=1.)\n\n\ndef get_valid_transforms():\n return Compose([\n CenterCrop(CFG['img_size'], CFG['img_size'], p=1.),\n Resize(CFG['img_size'], CFG['img_size']),\n Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225], max_pixel_value=255.0, p=1.0),\n ToTensorV2(p=1.0),\n ], p=1.)\n\ndef get_inference_transforms():\n return Compose([\n RandomResizedCrop(CFG['img_size'], CFG['img_size']),\n Transpose(p=0.5),\n HorizontalFlip(p=0.5),\n VerticalFlip(p=0.5),\n HueSaturationValue(hue_shift_limit=0.2, sat_shift_limit=0.2, val_shift_limit=0.2, p=0.5),\n RandomBrightnessContrast(brightness_limit=(-0.1,0.1), contrast_limit=(-0.1, 0.1), p=0.5),\n Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225], max_pixel_value=255.0, p=1.0),\n ToTensorV2(p=1.0),\n ], p=1.)\n\n\nclass CassvaImgClassifier(nn.Module):\n def __init__(self, model_arch, n_class, pretrained=False):\n super().__init__()\n self.model = timm.create_model(model_arch, pretrained=pretrained)\n n_features = self.model.classifier.in_features\n self.model.classifier = nn.Linear(n_features, n_class)\n\n def forward(self, x):\n x = self.model(x)\n return x\n\n\n\ndef prepare_dataloader(df, trn_idx, val_idx, data_root='../input/cassava-leaf-disease-classification/train_images/'):\n\n from catalyst.data.sampler import BalanceClassSampler\n\n train_ = df.loc[trn_idx,:].reset_index(drop=True)\n valid_ = df.loc[val_idx,:].reset_index(drop=True)\n\n train_ds = CassavaDataset(train_, data_root, transforms=get_train_transforms(), output_label=True, one_hot_label=False, do_fmix=False, do_cutmix=False)\n valid_ds = CassavaDataset(valid_, data_root, transforms=get_valid_transforms(), output_label=True)\n\n train_loader = torch.utils.data.DataLoader(\n train_ds,\n batch_size=CFG['train_bs'],\n pin_memory=False,\n drop_last=False,\n shuffle=True, \n num_workers=CFG['num_workers'],\n #sampler=BalanceClassSampler(labels=train_['label'].values, mode=\"downsampling\")\n )\n val_loader = torch.utils.data.DataLoader(\n valid_ds, \n batch_size=CFG['valid_bs'],\n num_workers=CFG['num_workers'],\n shuffle=False,\n pin_memory=False,\n )\n return train_loader, val_loader\n\n\ndef prepare_pseudo_dataloader(df, trn_idx, val_idx, \n data_root='../input/cassava-leaf-disease-classification/train_images/',\n pseudo_data_root='../input/cassava-leaf-disease-classification/test_images/'\n ):\n\n from catalyst.data.sampler import BalanceClassSampler\n\n train_ = df.loc[trn_idx,:].reset_index(drop=True)\n valid_ = df.loc[val_idx,:].reset_index(drop=True)\n\n train_ds = pseudo_CassavaDataset(train_, data_root,pseudo_data_root, transforms=get_train_transforms(), output_label=True)\n valid_ds = pseudo_CassavaDataset(valid_, data_root,pseudo_data_root, transforms=get_valid_transforms(), output_label=True)\n\n train_loader = torch.utils.data.DataLoader(\n train_ds,\n batch_size=CFG['train_bs'],\n pin_memory=False,\n drop_last=False,\n shuffle=True, \n num_workers=CFG['num_workers'],\n #sampler=BalanceClassSampler(labels=train_['label'].values, mode=\"downsampling\")\n )\n val_loader = torch.utils.data.DataLoader(\n valid_ds, \n batch_size=CFG['valid_bs'],\n num_workers=CFG['num_workers'],\n shuffle=False,\n pin_memory=False,\n )\n return train_loader, val_loader\n\n\n\ndef train_one_epoch(epoch, model, loss_fn, optimizer, train_loader, device, scheduler=None, schd_batch_update=False):\n model.train()\n\n t = time.time()\n running_loss = None\n\n pbar = tqdm(enumerate(train_loader), total=len(train_loader))\n for step, (imgs, image_labels) in pbar:\n imgs = imgs.to(device).float()\n image_labels = image_labels.to(device).long()\n\n #print(image_labels.shape, exam_label.shape)\n with autocast():\n image_preds = model(imgs) #output = model(input)\n #print(image_preds.shape, exam_pred.shape)\n\n loss = loss_fn(image_preds, image_labels)\n\n scaler.scale(loss).backward()\n\n if running_loss is None:\n running_loss = loss.item()\n else:\n running_loss = running_loss * .99 + loss.item() * .01\n\n if ((step + 1) % CFG['accum_iter'] == 0) or ((step + 1) == len(train_loader)):\n # may unscale_ here if desired (e.g., to allow clipping unscaled gradients)\n\n scaler.step(optimizer)\n scaler.update()\n optimizer.zero_grad() \n\n if scheduler is not None and schd_batch_update:\n scheduler.step()\n\n if ((step + 1) % CFG['verbose_step'] == 0) or ((step + 1) == len(train_loader)):\n description = f'epoch {epoch} loss: {running_loss:.4f}'\n\n pbar.set_description(description)\n\n if scheduler is not None and not schd_batch_update:\n scheduler.step()\n\ndef valid_one_epoch(epoch, model, loss_fn, val_loader, device, scheduler=None, schd_loss_update=False):\n model.eval()\n\n t = time.time()\n loss_sum = 0\n sample_num = 0\n image_preds_all = []\n image_targets_all = []\n\n pbar = tqdm(enumerate(val_loader), total=len(val_loader))\n for step, (imgs, image_labels) in pbar:\n imgs = imgs.to(device).float()\n image_labels = image_labels.to(device).long()\n\n image_preds = model(imgs) #output = model(input)\n #print(image_preds.shape, exam_pred.shape)\n image_preds_all += [torch.argmax(image_preds, 1).detach().cpu().numpy()]\n image_targets_all += [image_labels.detach().cpu().numpy()]\n\n loss = loss_fn(image_preds, image_labels)\n\n loss_sum += loss.item()*image_labels.shape[0]\n sample_num += image_labels.shape[0] \n\n if ((step + 1) % CFG['verbose_step'] == 0) or ((step + 1) == len(val_loader)):\n description = f'epoch {epoch} loss: {loss_sum/sample_num:.4f}'\n pbar.set_description(description)\n\n image_preds_all = np.concatenate(image_preds_all)\n image_targets_all = np.concatenate(image_targets_all)\n print('validation multi-class accuracy = {:.4f}'.format((image_preds_all==image_targets_all).mean()))\n\n if scheduler is not None:\n if schd_loss_update:\n scheduler.step(loss_sum/sample_num)\n else:\n scheduler.step()\n\ndef inference_one_epoch(model, data_loader, device):\n model.eval()\n\n image_preds_all = []\n\n pbar = tqdm(enumerate(data_loader), total=len(data_loader))\n for step, (imgs) in pbar:\n imgs = imgs.to(device).float()\n\n image_preds = model(imgs) \n image_preds_all += [torch.softmax(image_preds, 1).detach().cpu().numpy()]\n\n\n\n image_preds_all = np.concatenate(image_preds_all, axis=0)\n return image_preds_all\n\nkwargs={'map_location':lambda storage, loc: storage.cuda(0)}\ndef load_GPUS(model,model_path,kwargs):\n state_dict = torch.load(model_path,**kwargs)\n # create new OrderedDict that does not contain `module.`\n from collections import OrderedDict\n new_state_dict = OrderedDict()\n for k, v in state_dict.items():\n name = k[7:] # remove `module.\n new_state_dict[name] = v\n # load params\n model.load_state_dict(new_state_dict)\n return model\n\n\n###### test and select conf_threshold_final>* to reform the novel training dataset\nif not os.path.exists('result'):\n os.mkdir('result')\nfor iiiiii in range(1):\n\n conf_threshold = 0.001\n conf_threshold_final = 0.95\n seed_everything(CFG['seed'])\n\n folds_test = StratifiedKFold(n_splits=CFG['fold_num']).split(np.arange(train.shape[0]), train.label.values)\n\n for fold, (trn_idx, val_idx) in enumerate(folds_test):\n # we'll train fold 0 first\n if fold > 0:\n break \n\n print('Inference fold {} started'.format(fold))\n\n valid_ = train.loc[val_idx,:].reset_index(drop=True)\n valid_ds = CassavaDataset(valid_, '../input/cassava-leaf-disease-classification/train_images/', \n transforms=get_inference_transforms(), output_label=False)\n\n test_df_pseudo = pd.DataFrame()\n test_df_pseudo['image_id'] = list(os.listdir('../input/cassava-leaf-disease-classification/test_images/'))\n test_ds = CassavaDataset(test_df_pseudo, '../input/cassava-leaf-disease-classification/test_images/', \n transforms=get_inference_transforms(), output_label=False)\n\n val_loader = torch.utils.data.DataLoader(\n valid_ds, \n batch_size=CFG['valid_bs'],\n num_workers=CFG['num_workers'],\n shuffle=False,\n pin_memory=False,\n )\n\n tst_loader = torch.utils.data.DataLoader(\n test_ds, \n batch_size=CFG['valid_bs'],\n num_workers=CFG['num_workers'],\n shuffle=False,\n pin_memory=False,\n )\n\n device = torch.device(CFG['device'])\n model = CassvaImgClassifier(CFG['model_arch'], train.label.nunique()).to(device)\n\n tst_preds = []\n testdf_psuedo = []\n\n\n #for epoch in range(CFG['epochs']-3):\n for i, epoch in enumerate(CFG['used_epochs']): \n load_GPUS(model,('../input/e-b4-e20-5fold-extradata-grid/{}_fold_{}_{}'.format(CFG['model_arch'], fold, epoch)), kwargs)\n\n with torch.no_grad():\n for _ in range(CFG['tta']):\n tst_preds += [CFG['weights'][i]/sum(CFG['weights'])/CFG['tta']*inference_one_epoch(model, tst_loader, device)]\n\n tst_preds = np.mean(tst_preds, axis=0) *CFG['tta']*len(CFG['used_epochs'])\n\n\n del model\n torch.cuda.empty_cache()\n #print(len(tst_preds))\n pseudo_pred = []\n for ii in range(len(tst_preds)):\n if (tst_preds[ii][np.where(tst_preds == np.max(tst_preds))[1][0]]) > conf_threshold:\n testdf_psuedo.append(tst_preds[ii].tolist())\n np.array(testdf_psuedo)\n\n\n test_df_pseudo['label0'] = np.max(testdf_psuedo, axis=1)\n test_df_pseudo['label'] = np.argmax(testdf_psuedo, axis=1)\n\n test_df_pseudo = test_df_pseudo[test_df_pseudo['label0']>conf_threshold_final]\n test_df_pseudo.drop('label0', axis = 1, inplace = True)\n\n frames = [train,test_df_pseudo]\n test_df_pseudo = pd.concat(frames,axis=0,ignore_index=True)\n print(test_df_pseudo.tail())\n\n\n\n\n ##############retrain\n test_imgs = os.listdir('../input/cassava-leaf-disease-classification/test_images/')\n\n folds_retrain = StratifiedKFold(n_splits=CFG['fold_num'], shuffle=True, \n random_state=CFG['seed']).split(np.arange(test_df_pseudo.shape[0]),test_df_pseudo.label.values)\n for fold, (trn_idx, val_idx) in enumerate(folds_retrain):\n if fold > 0:\n break \n\n print('Training with {} started'.format(fold))\n\n print(len(trn_idx), len(val_idx))\n train_loader, val_loader = prepare_pseudo_dataloader(test_df_pseudo, trn_idx, val_idx, \n data_root='../input/cassava-leaf-disease-classification/train_images/',\n pseudo_data_root='../input/cassava-leaf-disease-classification/test_images/'\n )\n\n\n device = torch.device(CFG['device'])\n\n model = CassvaImgClassifier(CFG['model_arch'], test_df_pseudo.label.nunique(), pretrained=False).to(device)\n model = load_GPUS(model,('../input/e-b4-e20-5fold-extradata-grid/tf_efficientnet_b4_ns_fold_0_2'), kwargs)\n\n scaler = GradScaler() \n optimizer = torch.optim.Adam(model.parameters(), lr=CFG['lr'], weight_decay=CFG['weight_decay'])\n scheduler = torch.optim.lr_scheduler.CosineAnnealingWarmRestarts(optimizer, T_0=CFG['T_0'], T_mult=1, eta_min=CFG['min_lr'], last_epoch=-1)\n\n loss_tr = nn.CrossEntropyLoss().to(device) \n loss_fn = nn.CrossEntropyLoss().to(device)\n\n for epoch in range(CFG['epochs']):\n train_one_epoch(epoch, model, loss_tr, optimizer, train_loader, device, scheduler=scheduler, schd_batch_update=False)\n\n with torch.no_grad():\n valid_one_epoch(epoch, model, loss_fn, val_loader, device, scheduler=None, schd_loss_update=False)\n\n torch.save(model.state_dict(),'./result/{}_fold_{}_{}'.format(CFG['model_arch'], fold, epoch))\n\n #torch.save(model.cnn_model.state_dict(),'{}/cnn_model_fold_{}_{}'.format(CFG['model_path'], fold, CFG['tag']))\n del model, optimizer, train_loader, val_loader, scaler, scheduler\n torch.cuda.empty_cache()\n\n\n ################final test \n seed_everything(CFG['seed'])\n\n folds = StratifiedKFold(n_splits=CFG['fold_num']).split(np.arange(train.shape[0]), train.label.values)\n\n for fold, (trn_idx, val_idx) in enumerate(folds):\n if fold > 0:\n break \n print('Inference fold {} started'.format(fold))\n\n valid_ = train.loc[val_idx,:].reset_index(drop=True)\n valid_ds = CassavaDataset(valid_, '../input/cassava-leaf-disease-classification/train_images/', transforms=get_inference_transforms(), output_label=False)\n\n test_final = pd.DataFrame()\n test_final['image_id'] = list(os.listdir('../input/cassava-leaf-disease-classification/test_images/'))\n test_ds = CassavaDataset(test_final, '../input/cassava-leaf-disease-classification/test_images/', transforms=get_inference_transforms(), output_label=False)\n\n val_loader = torch.utils.data.DataLoader(\n valid_ds, \n batch_size=CFG['valid_bs'],\n num_workers=CFG['num_workers'],\n shuffle=False,\n pin_memory=False,\n )\n\n tst_loader = torch.utils.data.DataLoader(\n test_ds, \n batch_size=CFG['valid_bs'],\n num_workers=CFG['num_workers'],\n shuffle=False,\n pin_memory=False,\n )\n\n device = torch.device(CFG['device'])\n model = CassvaImgClassifier(CFG['model_arch'], train.label.nunique()).to(device)\n tst_preds_final = []\n\n for i, epoch in enumerate(CFG['pseudo_used_epochs']): \n model.load_state_dict(torch.load('./result/{}_fold_{}_{}'.format(CFG['model_arch'], fold, epoch)))\n with torch.no_grad():\n for _ in range(CFG['tta']):\n tst_preds_final += [CFG['weights'][i]/sum(CFG['weights'])/CFG['tta']*inference_one_epoch(model, tst_loader, device)]\n tst_preds_final = np.mean(tst_preds_final, axis=0) \n del model\n torch.cuda.empty_cache() \n\n test_final['label'] = np.argmax(tst_preds_final, axis=1)\n print(test_final.head())\n test_final.to_csv('submission.csv', index=False)\n@juliaelliott
\nHello, can we use pseudo labeling approach like this. I am looking forward to your reply.
maybe it crashed OOM then no csv generated, or time limit exceeded.
", "rawMarkdown": "maybe it crashed OOM then no csv generated, or time limit exceeded.", "votes": null }, { "id": "1121572", "postDate": "12/21/2020 18:36:25", "content": "@wilyzh Pseudolabeling is permitted as long as it is fully automated. For specificity, hand-labeling of the test set is not permitted (and should largely be mitigated by the hidden test set), so as long as you are using a truly automated approach to pseudolabeling, this is fine.
", "rawMarkdown": "wilyzh Pseudolabeling is permitted as long as it is fully automated. For specificity, hand-labeling of the test set is not permitted (and should largely be mitigated by the hidden test set), so as long as you are using a truly automated approach to pseudolabeling, this is fine.", "votes": null }, { "id": "1121607", "postDate": "12/21/2020 19:07:47", "content": "Try just 1 fold, not 5 folds.
\np.s. Personally, I don't think it's a step to try.
\n@wilyzh
Thank you for your reply!😄
", "rawMarkdown": "Thank you for your reply!😄", "votes": null }, { "id": "1121914", "postDate": "12/22/2020 02:46:34", "content": "Thank you for providing suggestion. I've reduced the batch size to 8, but this problem still occurs. Maybe it is caused by other reasons.
", "rawMarkdown": "Thank you for providing suggestion. I've reduced the batch size to 8, but this problem still occurs. Maybe it is caused by other reasons.", "votes": null }, { "id": "1121916", "postDate": "12/22/2020 02:48:30", "content": "@piantic. Thank you for providing suggestion, I will try it later😁
", "rawMarkdown": "piantic. Thank you for providing suggestion, I will try it later😁", "votes": null }, { "id": "1124738", "postDate": "12/24/2020 07:09:02", "content": "Perhaps you should select the submission.csv file and click the submit button.
\nps. pseudo label did not improve my LB
And here is my code
\npackage_path = '../input/pytorch-image-model' #'../input/efficientnet-pytorch-07/efficientnet_pytorch-0.7.0'\nimport sys; sys.path.append(package_path)\n# Since the length of the comment cannot exceed 20000, so I delete the import *\n\nCFG = {\n 'fold_num': 5,\n 'seed': 719,\n 'model_arch': 'tf_efficientnet_b4_ns',\n 'img_size': 512, #512\n 'epochs': 1, #10\n 'train_bs': 8,\n 'valid_bs': 8,\n 'T_0': 1,\n 'lr': 1e-4,\n 'min_lr': 1e-6,\n 'weight_decay':1e-6,\n 'num_workers': 4,\n 'accum_iter': 2, # suppoprt to do batch accumulation for backprop with effectively larger batch size\n 'verbose_step': 1,\n 'device': 'cuda:0',\n 'tta': 1,\n 'used_epochs': [0],\n 'pseudo_used_epochs': [0],\n 'weights': [1],\n 'pseudo_weights':[1]\n}\n\ntrain = pd.read_csv('../input/cassava-leaf-disease-classification/train.csv')\nsubmission = pd.read_csv('../input/cassava-leaf-disease-classification/sample_submission.csv')\n\ndef seed_everything(seed):\n random.seed(seed)\n os.environ['PYTHONHASHSEED'] = str(seed)\n np.random.seed(seed)\n torch.manual_seed(seed)\n torch.cuda.manual_seed(seed)\n torch.backends.cudnn.deterministic = True\n torch.backends.cudnn.benchmark = True\n\ndef get_img(path):\n im_bgr = cv2.imread(path)\n im_rgb = im_bgr[:, :, ::-1]\n #print(im_rgb)\n return im_rgb\n\nclass CassavaDataset(Dataset):\n def __init__(\n self, df, data_root, transforms=None, output_label=True\n ):\n\n super().__init__()\n self.df = df.reset_index(drop=True).copy()\n self.transforms = transforms\n self.data_root = data_root\n self.output_label = output_label\n\n def __len__(self):\n return self.df.shape[0]\n\n def __getitem__(self, index: int):\n\n # get labels\n if self.output_label:\n target = self.df.iloc[index]['label']\n\n path = \"{}/{}\".format(self.data_root, self.df.iloc[index]['image_id'])\n\n img = get_img(path)\n\n if self.transforms:\n img = self.transforms(image=img)['image']\n\n # do label smoothing\n if self.output_label == True:\n return img, target\n else:\n return img\n\n\nclass pseudo_CassavaDataset(Dataset):\n def __init__(\n self, df, data_root, pseudo_data_root, transforms=None, output_label=True\n ):\n\n super().__init__()\n self.df = df.reset_index(drop=True).copy()\n self.transforms = transforms\n self.data_root = data_root\n self.pseudo_data_root = pseudo_data_root\n self.output_label = output_label\n\n\n def __len__(self):\n return self.df.shape[0]\n\n def __getitem__(self, index: int):\n\n # get labels\n if self.output_label:\n target = self.df.iloc[index]['label']\n\n path = \"{}/{}\".format(self.data_root, self.df.iloc[index]['image_id'])\n pseudo_path = \"{}/{}\".format(self.pseudo_data_root, self.df.iloc[index]['image_id'])\n\n\n if os.path.exists(path):\n img = get_img(path)\n else:\n img = get_img(pseudo_path)\n\n\n if self.transforms:\n img = self.transforms(image=img)['image']\n\n # do label smoothing\n if self.output_label == True:\n return img, target\n else:\n return img\n\n\n\nfrom albumentations import (\n HorizontalFlip, VerticalFlip, IAAPerspective, ShiftScaleRotate, CLAHE, RandomRotate90,\n Transpose, ShiftScaleRotate, Blur, OpticalDistortion, GridDistortion, HueSaturationValue,\n IAAAdditiveGaussianNoise, GaussNoise, MotionBlur, MedianBlur, IAAPiecewiseAffine, RandomResizedCrop,\n IAASharpen, IAAEmboss, RandomBrightnessContrast, Flip, OneOf, Compose, Normalize, Cutout, CoarseDropout, ShiftScaleRotate, CenterCrop, Resize\n)\n\nfrom albumentations.pytorch import ToTensorV2\n\n\ndef get_train_transforms():\n return Compose([\n RandomResizedCrop(CFG['img_size'], CFG['img_size']),\n Transpose(p=0.5),\n HorizontalFlip(p=0.5),\n VerticalFlip(p=0.5),\n ShiftScaleRotate(p=0.5),\n HueSaturationValue(hue_shift_limit=0.2, sat_shift_limit=0.2, val_shift_limit=0.2, p=0.5),\n RandomBrightnessContrast(brightness_limit=(-0.1,0.1), contrast_limit=(-0.1, 0.1), p=0.5),\n Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225], max_pixel_value=255.0, p=1.0),\n CoarseDropout(p=0.5),\n Cutout(p=0.5),\n ToTensorV2(p=1.0),\n ], p=1.)\n\n\ndef get_valid_transforms():\n return Compose([\n CenterCrop(CFG['img_size'], CFG['img_size'], p=1.),\n Resize(CFG['img_size'], CFG['img_size']),\n Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225], max_pixel_value=255.0, p=1.0),\n ToTensorV2(p=1.0),\n ], p=1.)\n\ndef get_inference_transforms():\n return Compose([\n RandomResizedCrop(CFG['img_size'], CFG['img_size']),\n Transpose(p=0.5),\n HorizontalFlip(p=0.5),\n VerticalFlip(p=0.5),\n HueSaturationValue(hue_shift_limit=0.2, sat_shift_limit=0.2, val_shift_limit=0.2, p=0.5),\n RandomBrightnessContrast(brightness_limit=(-0.1,0.1), contrast_limit=(-0.1, 0.1), p=0.5),\n Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225], max_pixel_value=255.0, p=1.0),\n ToTensorV2(p=1.0),\n ], p=1.)\n\n\nclass CassvaImgClassifier(nn.Module):\n def __init__(self, model_arch, n_class, pretrained=False):\n super().__init__()\n self.model = timm.create_model(model_arch, pretrained=pretrained)\n n_features = self.model.classifier.in_features\n self.model.classifier = nn.Linear(n_features, n_class)\n\n def forward(self, x):\n x = self.model(x)\n return x\n\n\n\ndef prepare_dataloader(df, trn_idx, val_idx, data_root='../input/cassava-leaf-disease-classification/train_images/'):\n\n from catalyst.data.sampler import BalanceClassSampler\n\n train_ = df.loc[trn_idx,:].reset_index(drop=True)\n valid_ = df.loc[val_idx,:].reset_index(drop=True)\n\n train_ds = CassavaDataset(train_, data_root, transforms=get_train_transforms(), output_label=True, one_hot_label=False, do_fmix=False, do_cutmix=False)\n valid_ds = CassavaDataset(valid_, data_root, transforms=get_valid_transforms(), output_label=True)\n\n train_loader = torch.utils.data.DataLoader(\n train_ds,\n batch_size=CFG['train_bs'],\n pin_memory=False,\n drop_last=False,\n shuffle=True, \n num_workers=CFG['num_workers'],\n #sampler=BalanceClassSampler(labels=train_['label'].values, mode=\"downsampling\")\n )\n val_loader = torch.utils.data.DataLoader(\n valid_ds, \n batch_size=CFG['valid_bs'],\n num_workers=CFG['num_workers'],\n shuffle=False,\n pin_memory=False,\n )\n return train_loader, val_loader\n\n\ndef prepare_pseudo_dataloader(df, trn_idx, val_idx, \n data_root='../input/cassava-leaf-disease-classification/train_images/',\n pseudo_data_root='../input/cassava-leaf-disease-classification/test_images/'\n ):\n\n from catalyst.data.sampler import BalanceClassSampler\n\n train_ = df.loc[trn_idx,:].reset_index(drop=True)\n valid_ = df.loc[val_idx,:].reset_index(drop=True)\n\n train_ds = pseudo_CassavaDataset(train_, data_root,pseudo_data_root, transforms=get_train_transforms(), output_label=True)\n valid_ds = pseudo_CassavaDataset(valid_, data_root,pseudo_data_root, transforms=get_valid_transforms(), output_label=True)\n\n train_loader = torch.utils.data.DataLoader(\n train_ds,\n batch_size=CFG['train_bs'],\n pin_memory=False,\n drop_last=False,\n shuffle=True, \n num_workers=CFG['num_workers'],\n #sampler=BalanceClassSampler(labels=train_['label'].values, mode=\"downsampling\")\n )\n val_loader = torch.utils.data.DataLoader(\n valid_ds, \n batch_size=CFG['valid_bs'],\n num_workers=CFG['num_workers'],\n shuffle=False,\n pin_memory=False,\n )\n return train_loader, val_loader\n\n\n\ndef train_one_epoch(epoch, model, loss_fn, optimizer, train_loader, device, scheduler=None, schd_batch_update=False):\n model.train()\n\n t = time.time()\n running_loss = None\n\n pbar = tqdm(enumerate(train_loader), total=len(train_loader))\n for step, (imgs, image_labels) in pbar:\n imgs = imgs.to(device).float()\n image_labels = image_labels.to(device).long()\n\n #print(image_labels.shape, exam_label.shape)\n with autocast():\n image_preds = model(imgs) #output = model(input)\n #print(image_preds.shape, exam_pred.shape)\n\n loss = loss_fn(image_preds, image_labels)\n\n scaler.scale(loss).backward()\n\n if running_loss is None:\n running_loss = loss.item()\n else:\n running_loss = running_loss * .99 + loss.item() * .01\n\n if ((step + 1) % CFG['accum_iter'] == 0) or ((step + 1) == len(train_loader)):\n # may unscale_ here if desired (e.g., to allow clipping unscaled gradients)\n\n scaler.step(optimizer)\n scaler.update()\n optimizer.zero_grad() \n\n if scheduler is not None and schd_batch_update:\n scheduler.step()\n\n if ((step + 1) % CFG['verbose_step'] == 0) or ((step + 1) == len(train_loader)):\n description = f'epoch {epoch} loss: {running_loss:.4f}'\n\n pbar.set_description(description)\n\n if scheduler is not None and not schd_batch_update:\n scheduler.step()\n\ndef valid_one_epoch(epoch, model, loss_fn, val_loader, device, scheduler=None, schd_loss_update=False):\n model.eval()\n\n t = time.time()\n loss_sum = 0\n sample_num = 0\n image_preds_all = []\n image_targets_all = []\n\n pbar = tqdm(enumerate(val_loader), total=len(val_loader))\n for step, (imgs, image_labels) in pbar:\n imgs = imgs.to(device).float()\n image_labels = image_labels.to(device).long()\n\n image_preds = model(imgs) #output = model(input)\n #print(image_preds.shape, exam_pred.shape)\n image_preds_all += [torch.argmax(image_preds, 1).detach().cpu().numpy()]\n image_targets_all += [image_labels.detach().cpu().numpy()]\n\n loss = loss_fn(image_preds, image_labels)\n\n loss_sum += loss.item()*image_labels.shape[0]\n sample_num += image_labels.shape[0] \n\n if ((step + 1) % CFG['verbose_step'] == 0) or ((step + 1) == len(val_loader)):\n description = f'epoch {epoch} loss: {loss_sum/sample_num:.4f}'\n pbar.set_description(description)\n\n image_preds_all = np.concatenate(image_preds_all)\n image_targets_all = np.concatenate(image_targets_all)\n print('validation multi-class accuracy = {:.4f}'.format((image_preds_all==image_targets_all).mean()))\n\n if scheduler is not None:\n if schd_loss_update:\n scheduler.step(loss_sum/sample_num)\n else:\n scheduler.step()\n\ndef inference_one_epoch(model, data_loader, device):\n model.eval()\n\n image_preds_all = []\n\n pbar = tqdm(enumerate(data_loader), total=len(data_loader))\n for step, (imgs) in pbar:\n imgs = imgs.to(device).float()\n\n image_preds = model(imgs) \n image_preds_all += [torch.softmax(image_preds, 1).detach().cpu().numpy()]\n\n\n\n image_preds_all = np.concatenate(image_preds_all, axis=0)\n return image_preds_all\n\nkwargs={'map_location':lambda storage, loc: storage.cuda(0)}\ndef load_GPUS(model,model_path,kwargs):\n state_dict = torch.load(model_path,**kwargs)\n # create new OrderedDict that does not contain `module.`\n from collections import OrderedDict\n new_state_dict = OrderedDict()\n for k, v in state_dict.items():\n name = k[7:] # remove `module.\n new_state_dict[name] = v\n # load params\n model.load_state_dict(new_state_dict)\n return model\n\n\n###### test and select conf_threshold_final>* to reform the novel training dataset\nif not os.path.exists('result'):\n os.mkdir('result')\nfor iiiiii in range(1):\n\n conf_threshold = 0.001\n conf_threshold_final = 0.95\n seed_everything(CFG['seed'])\n\n folds_test = StratifiedKFold(n_splits=CFG['fold_num']).split(np.arange(train.shape[0]), train.label.values)\n\n for fold, (trn_idx, val_idx) in enumerate(folds_test):\n # we'll train fold 0 first\n if fold > 0:\n break \n\n print('Inference fold {} started'.format(fold))\n\n valid_ = train.loc[val_idx,:].reset_index(drop=True)\n valid_ds = CassavaDataset(valid_, '../input/cassava-leaf-disease-classification/train_images/', \n transforms=get_inference_transforms(), output_label=False)\n\n test_df_pseudo = pd.DataFrame()\n test_df_pseudo['image_id'] = list(os.listdir('../input/cassava-leaf-disease-classification/test_images/'))\n test_ds = CassavaDataset(test_df_pseudo, '../input/cassava-leaf-disease-classification/test_images/', \n transforms=get_inference_transforms(), output_label=False)\n\n val_loader = torch.utils.data.DataLoader(\n valid_ds, \n batch_size=CFG['valid_bs'],\n num_workers=CFG['num_workers'],\n shuffle=False,\n pin_memory=False,\n )\n\n tst_loader = torch.utils.data.DataLoader(\n test_ds, \n batch_size=CFG['valid_bs'],\n num_workers=CFG['num_workers'],\n shuffle=False,\n pin_memory=False,\n )\n\n device = torch.device(CFG['device'])\n model = CassvaImgClassifier(CFG['model_arch'], train.label.nunique()).to(device)\n\n tst_preds = []\n testdf_psuedo = []\n\n\n #for epoch in range(CFG['epochs']-3):\n for i, epoch in enumerate(CFG['used_epochs']): \n load_GPUS(model,('../input/e-b4-e20-5fold-extradata-grid/{}_fold_{}_{}'.format(CFG['model_arch'], fold, epoch)), kwargs)\n\n with torch.no_grad():\n for _ in range(CFG['tta']):\n tst_preds += [CFG['weights'][i]/sum(CFG['weights'])/CFG['tta']*inference_one_epoch(model, tst_loader, device)]\n\n tst_preds = np.mean(tst_preds, axis=0) *CFG['tta']*len(CFG['used_epochs'])\n\n\n del model\n torch.cuda.empty_cache()\n #print(len(tst_preds))\n pseudo_pred = []\n for ii in range(len(tst_preds)):\n if (tst_preds[ii][np.where(tst_preds == np.max(tst_preds))[1][0]]) > conf_threshold:\n testdf_psuedo.append(tst_preds[ii].tolist())\n np.array(testdf_psuedo)\n\n\n test_df_pseudo['label0'] = np.max(testdf_psuedo, axis=1)\n test_df_pseudo['label'] = np.argmax(testdf_psuedo, axis=1)\n\n test_df_pseudo = test_df_pseudo[test_df_pseudo['label0']>conf_threshold_final]\n test_df_pseudo.drop('label0', axis = 1, inplace = True)\n\n frames = [train,test_df_pseudo]\n test_df_pseudo = pd.concat(frames,axis=0,ignore_index=True)\n print(test_df_pseudo.tail())\n\n\n\n\n ##############retrain\n test_imgs = os.listdir('../input/cassava-leaf-disease-classification/test_images/')\n\n folds_retrain = StratifiedKFold(n_splits=CFG['fold_num'], shuffle=True, \n random_state=CFG['seed']).split(np.arange(test_df_pseudo.shape[0]),test_df_pseudo.label.values)\n for fold, (trn_idx, val_idx) in enumerate(folds_retrain):\n if fold > 0:\n break \n\n print('Training with {} started'.format(fold))\n\n print(len(trn_idx), len(val_idx))\n train_loader, val_loader = prepare_pseudo_dataloader(test_df_pseudo, trn_idx, val_idx, \n data_root='../input/cassava-leaf-disease-classification/train_images/',\n pseudo_data_root='../input/cassava-leaf-disease-classification/test_images/'\n )\n\n\n device = torch.device(CFG['device'])\n\n model = CassvaImgClassifier(CFG['model_arch'], test_df_pseudo.label.nunique(), pretrained=False).to(device)\n model = load_GPUS(model,('../input/e-b4-e20-5fold-extradata-grid/tf_efficientnet_b4_ns_fold_0_2'), kwargs)\n\n scaler = GradScaler() \n optimizer = torch.optim.Adam(model.parameters(), lr=CFG['lr'], weight_decay=CFG['weight_decay'])\n scheduler = torch.optim.lr_scheduler.CosineAnnealingWarmRestarts(optimizer, T_0=CFG['T_0'], T_mult=1, eta_min=CFG['min_lr'], last_epoch=-1)\n\n loss_tr = nn.CrossEntropyLoss().to(device) \n loss_fn = nn.CrossEntropyLoss().to(device)\n\n for epoch in range(CFG['epochs']):\n train_one_epoch(epoch, model, loss_tr, optimizer, train_loader, device, scheduler=scheduler, schd_batch_update=False)\n\n with torch.no_grad():\n valid_one_epoch(epoch, model, loss_fn, val_loader, device, scheduler=None, schd_loss_update=False)\n\n torch.save(model.state_dict(),'./result/{}_fold_{}_{}'.format(CFG['model_arch'], fold, epoch))\n\n #torch.save(model.cnn_model.state_dict(),'{}/cnn_model_fold_{}_{}'.format(CFG['model_path'], fold, CFG['tag']))\n del model, optimizer, train_loader, val_loader, scaler, scheduler\n torch.cuda.empty_cache()\n\n\n ################final test \n seed_everything(CFG['seed'])\n\n folds = StratifiedKFold(n_splits=CFG['fold_num']).split(np.arange(train.shape[0]), train.label.values)\n\n for fold, (trn_idx, val_idx) in enumerate(folds):\n if fold > 0:\n break \n print('Inference fold {} started'.format(fold))\n\n valid_ = train.loc[val_idx,:].reset_index(drop=True)\n valid_ds = CassavaDataset(valid_, '../input/cassava-leaf-disease-classification/train_images/', transforms=get_inference_transforms(), output_label=False)\n\n test_final = pd.DataFrame()\n test_final['image_id'] = list(os.listdir('../input/cassava-leaf-disease-classification/test_images/'))\n test_ds = CassavaDataset(test_final, '../input/cassava-leaf-disease-classification/test_images/', transforms=get_inference_transforms(), output_label=False)\n\n val_loader = torch.utils.data.DataLoader(\n valid_ds, \n batch_size=CFG['valid_bs'],\n num_workers=CFG['num_workers'],\n shuffle=False,\n pin_memory=False,\n )\n\n tst_loader = torch.utils.data.DataLoader(\n test_ds, \n batch_size=CFG['valid_bs'],\n num_workers=CFG['num_workers'],\n shuffle=False,\n pin_memory=False,\n )\n\n device = torch.device(CFG['device'])\n model = CassvaImgClassifier(CFG['model_arch'], train.label.nunique()).to(device)\n tst_preds_final = []\n\n for i, epoch in enumerate(CFG['pseudo_used_epochs']): \n model.load_state_dict(torch.load('./result/{}_fold_{}_{}'.format(CFG['model_arch'], fold, epoch)))\n with torch.no_grad():\n for _ in range(CFG['tta']):\n tst_preds_final += [CFG['weights'][i]/sum(CFG['weights'])/CFG['tta']*inference_one_epoch(model, tst_loader, device)]\n tst_preds_final = np.mean(tst_preds_final, axis=0) \n del model\n torch.cuda.empty_cache() \n\n test_final['label'] = np.argmax(tst_preds_final, axis=1)\n print(test_final.head())\n test_final.to_csv('submission.csv', index=False)\nmaybe it crashed OOM then no csv generated, or time limit exceeded.
", "votes": null, "replies": [] }, { "id": 1121607, "author_name": "piantic", "author_url": "", "post_date": "12/21/2020 19:07:47", "content": "Try just 1 fold, not 5 folds.
\np.s. Personally, I don't think it's a step to try.
\n@wilyzh
Thank you for providing suggestion. I've reduced the batch size to 8, but this problem still occurs. Maybe it is caused by other reasons.
", "votes": null, "replies": [] }, { "id": 1121916, "author_name": "wilyzh", "author_url": "", "post_date": "12/22/2020 02:48:30", "content": "@piantic. Thank you for providing suggestion, I will try it later😁
", "votes": null, "replies": [] }, { "id": 1124738, "author_name": "miraclebcool", "author_url": "", "post_date": "12/24/2020 07:09:02", "content": "Perhaps you should select the submission.csv file and click the submit button.
\nps. pseudo label did not improve my LB
@juliaelliott
\nHello, can we use pseudo labeling approach like this. I am looking forward to your reply.
@wilyzh Pseudolabeling is permitted as long as it is fully automated. For specificity, hand-labeling of the test set is not permitted (and should largely be mitigated by the hidden test set), so as long as you are using a truly automated approach to pseudolabeling, this is fine.
", "votes": null, "replies": [] }, { "id": 1121910, "author_name": "wilyzh", "author_url": "", "post_date": "12/22/2020 02:41:07", "content": "Thank you for your reply!😄
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1120627": "Hello everyone\nI followed the steps below to implement pseudo label in submit notebook：\n\n①Inference on the test dataset, generate classification confidences and test labels.\n②Select the test set with confidence higher than 0.95 (pseudo label dataset) and add it to the training set.\n③Retrain the training dataset and pseudo label dataset\n④Final test and generate submission.csv\n\nIt always prompts *Submission CSV Not Found* error. According to my analysis, after submitting the results, the error will be reported when the first or second step is completed. The code do not start to retrain. But putting multiple pictures in the test set folder on your own computer can run normally and generate *submission.csv* files. It can be run in the kaggle notebook, too, and a test set image is added to the training set and retrained.\nI still don't understand what is wrong.", "1120634": "And here is my code\n```\npackage_path = '../input/pytorch-image-model' #'../input/efficientnet-pytorch-07/efficientnet_pytorch-0.7.0'\nimport sys; sys.path.append(package_path)\n# Since the length of the comment cannot exceed 20000, so I delete the import *\n\nCFG = {\n 'fold_num': 5,\n 'seed': 719,\n 'model_arch': 'tf_efficientnet_b4_ns',\n 'img_size': 512, #512\n 'epochs': 1, #10\n 'train_bs': 8,\n 'valid_bs': 8,\n 'T_0': 1,\n 'lr': 1e-4,\n 'min_lr': 1e-6,\n 'weight_decay':1e-6,\n 'num_workers': 4,\n 'accum_iter': 2, # suppoprt to do batch accumulation for backprop with effectively larger batch size\n 'verbose_step': 1,\n 'device': 'cuda:0',\n 'tta': 1,\n 'used_epochs': [0],\n 'pseudo_used_epochs': [0],\n 'weights': [1],\n 'pseudo_weights':[1]\n}\n\ntrain = pd.read_csv('../input/cassava-leaf-disease-classification/train.csv')\nsubmission = pd.read_csv('../input/cassava-leaf-disease-classification/sample_submission.csv')\n\ndef seed_everything(seed):\n random.seed(seed)\n os.environ['PYTHONHASHSEED'] = str(seed)\n np.random.seed(seed)\n torch.manual_seed(seed)\n torch.cuda.manual_seed(seed)\n torch.backends.cudnn.deterministic = True\n torch.backends.cudnn.benchmark = True\n \ndef get_img(path):\n im_bgr = cv2.imread(path)\n im_rgb = im_bgr[:, :, ::-1]\n #print(im_rgb)\n return im_rgb\n\nclass CassavaDataset(Dataset):\n def __init__(\n self, df, data_root, transforms=None, output_label=True\n ):\n \n super().__init__()\n self.df = df.reset_index(drop=True).copy()\n self.transforms = transforms\n self.data_root = data_root\n self.output_label = output_label\n \n def __len__(self):\n return self.df.shape[0]\n \n def __getitem__(self, index: int):\n \n # get labels\n if self.output_label:\n target = self.df.iloc[index]['label']\n \n path = \"{}/{}\".format(self.data_root, self.df.iloc[index]['image_id'])\n \n img = get_img(path)\n \n if self.transforms:\n img = self.transforms(image=img)['image']\n \n # do label smoothing\n if self.output_label == True:\n return img, target\n else:\n return img\n\n \nclass pseudo_CassavaDataset(Dataset):\n def __init__(\n self, df, data_root, pseudo_data_root, transforms=None, output_label=True\n ):\n \n super().__init__()\n self.df = df.reset_index(drop=True).copy()\n self.transforms = transforms\n self.data_root = data_root\n self.pseudo_data_root = pseudo_data_root\n self.output_label = output_label\n\n \n def __len__(self):\n return self.df.shape[0]\n \n def __getitem__(self, index: int):\n \n # get labels\n if self.output_label:\n target = self.df.iloc[index]['label']\n \n path = \"{}/{}\".format(self.data_root, self.df.iloc[index]['image_id'])\n pseudo_path = \"{}/{}\".format(self.pseudo_data_root, self.df.iloc[index]['image_id'])\n\n \n if os.path.exists(path):\n img = get_img(path)\n else:\n img = get_img(pseudo_path)\n\n \n if self.transforms:\n img = self.transforms(image=img)['image']\n \n # do label smoothing\n if self.output_label == True:\n return img, target\n else:\n return img\n \n\n \nfrom albumentations import (\n HorizontalFlip, VerticalFlip, IAAPerspective, ShiftScaleRotate, CLAHE, RandomRotate90,\n Transpose, ShiftScaleRotate, Blur, OpticalDistortion, GridDistortion, HueSaturationValue,\n IAAAdditiveGaussianNoise, GaussNoise, MotionBlur, MedianBlur, IAAPiecewiseAffine, RandomResizedCrop,\n IAASharpen, IAAEmboss, RandomBrightnessContrast, Flip, OneOf, Compose, Normalize, Cutout, CoarseDropout, ShiftScaleRotate, CenterCrop, Resize\n)\n\nfrom albumentations.pytorch import ToTensorV2\n\n\ndef get_train_transforms():\n return Compose([\n RandomResizedCrop(CFG['img_size'], CFG['img_size']),\n Transpose(p=0.5),\n HorizontalFlip(p=0.5),\n VerticalFlip(p=0.5),\n ShiftScaleRotate(p=0.5),\n HueSaturationValue(hue_shift_limit=0.2, sat_shift_limit=0.2, val_shift_limit=0.2, p=0.5),\n RandomBrightnessContrast(brightness_limit=(-0.1,0.1), contrast_limit=(-0.1, 0.1), p=0.5),\n Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225], max_pixel_value=255.0, p=1.0),\n CoarseDropout(p=0.5),\n Cutout(p=0.5),\n ToTensorV2(p=1.0),\n ], p=1.)\n \n \ndef get_valid_transforms():\n return Compose([\n CenterCrop(CFG['img_size'], CFG['img_size'], p=1.),\n Resize(CFG['img_size'], CFG['img_size']),\n Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225], max_pixel_value=255.0, p=1.0),\n ToTensorV2(p=1.0),\n ], p=1.)\n\ndef get_inference_transforms():\n return Compose([\n RandomResizedCrop(CFG['img_size'], CFG['img_size']),\n Transpose(p=0.5),\n HorizontalFlip(p=0.5),\n VerticalFlip(p=0.5),\n HueSaturationValue(hue_shift_limit=0.2, sat_shift_limit=0.2, val_shift_limit=0.2, p=0.5),\n RandomBrightnessContrast(brightness_limit=(-0.1,0.1), contrast_limit=(-0.1, 0.1), p=0.5),\n Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225], max_pixel_value=255.0, p=1.0),\n ToTensorV2(p=1.0),\n ], p=1.)\n\n\nclass CassvaImgClassifier(nn.Module):\n def __init__(self, model_arch, n_class, pretrained=False):\n super().__init__()\n self.model = timm.create_model(model_arch, pretrained=pretrained)\n n_features = self.model.classifier.in_features\n self.model.classifier = nn.Linear(n_features, n_class)\n \n def forward(self, x):\n x = self.model(x)\n return x\n \n\n \ndef prepare_dataloader(df, trn_idx, val_idx, data_root='../input/cassava-leaf-disease-classification/train_images/'):\n \n from catalyst.data.sampler import BalanceClassSampler\n \n train_ = df.loc[trn_idx,:].reset_index(drop=True)\n valid_ = df.loc[val_idx,:].reset_index(drop=True)\n \n train_ds = CassavaDataset(train_, data_root, transforms=get_train_transforms(), output_label=True, one_hot_label=False, do_fmix=False, do_cutmix=False)\n valid_ds = CassavaDataset(valid_, data_root, transforms=get_valid_transforms(), output_label=True)\n \n train_loader = torch.utils.data.DataLoader(\n train_ds,\n batch_size=CFG['train_bs'],\n pin_memory=False,\n drop_last=False,\n shuffle=True, \n num_workers=CFG['num_workers'],\n #sampler=BalanceClassSampler(labels=train_['label'].values, mode=\"downsampling\")\n )\n val_loader = torch.utils.data.DataLoader(\n valid_ds, \n batch_size=CFG['valid_bs'],\n num_workers=CFG['num_workers'],\n shuffle=False,\n pin_memory=False,\n )\n return train_loader, val_loader\n\n\ndef prepare_pseudo_dataloader(df, trn_idx, val_idx, \n data_root='../input/cassava-leaf-disease-classification/train_images/',\n pseudo_data_root='../input/cassava-leaf-disease-classification/test_images/'\n ):\n \n from catalyst.data.sampler import BalanceClassSampler\n \n train_ = df.loc[trn_idx,:].reset_index(drop=True)\n valid_ = df.loc[val_idx,:].reset_index(drop=True)\n \n train_ds = pseudo_CassavaDataset(train_, data_root,pseudo_data_root, transforms=get_train_transforms(), output_label=True)\n valid_ds = pseudo_CassavaDataset(valid_, data_root,pseudo_data_root, transforms=get_valid_transforms(), output_label=True)\n \n train_loader = torch.utils.data.DataLoader(\n train_ds,\n batch_size=CFG['train_bs'],\n pin_memory=False,\n drop_last=False,\n shuffle=True, \n num_workers=CFG['num_workers'],\n #sampler=BalanceClassSampler(labels=train_['label'].values, mode=\"downsampling\")\n )\n val_loader = torch.utils.data.DataLoader(\n valid_ds, \n batch_size=CFG['valid_bs'],\n num_workers=CFG['num_workers'],\n shuffle=False,\n pin_memory=False,\n )\n return train_loader, val_loader\n\n\n\ndef train_one_epoch(epoch, model, loss_fn, optimizer, train_loader, device, scheduler=None, schd_batch_update=False):\n model.train()\n\n t = time.time()\n running_loss = None\n\n pbar = tqdm(enumerate(train_loader), total=len(train_loader))\n for step, (imgs, image_labels) in pbar:\n imgs = imgs.to(device).float()\n image_labels = image_labels.to(device).long()\n\n #print(image_labels.shape, exam_label.shape)\n with autocast():\n image_preds = model(imgs) #output = model(input)\n #print(image_preds.shape, exam_pred.shape)\n\n loss = loss_fn(image_preds, image_labels)\n \n scaler.scale(loss).backward()\n\n if running_loss is None:\n running_loss = loss.item()\n else:\n running_loss = running_loss * .99 + loss.item() * .01\n\n if ((step + 1) % CFG['accum_iter'] == 0) or ((step + 1) == len(train_loader)):\n # may unscale_ here if desired (e.g., to allow clipping unscaled gradients)\n\n scaler.step(optimizer)\n scaler.update()\n optimizer.zero_grad() \n \n if scheduler is not None and schd_batch_update:\n scheduler.step()\n\n if ((step + 1) % CFG['verbose_step'] == 0) or ((step + 1) == len(train_loader)):\n description = f'epoch {epoch} loss: {running_loss:.4f}'\n \n pbar.set_description(description)\n \n if scheduler is not None and not schd_batch_update:\n scheduler.step()\n \ndef valid_one_epoch(epoch, model, loss_fn, val_loader, device, scheduler=None, schd_loss_update=False):\n model.eval()\n\n t = time.time()\n loss_sum = 0\n sample_num = 0\n image_preds_all = []\n image_targets_all = []\n \n pbar = tqdm(enumerate(val_loader), total=len(val_loader))\n for step, (imgs, image_labels) in pbar:\n imgs = imgs.to(device).float()\n image_labels = image_labels.to(device).long()\n \n image_preds = model(imgs) #output = model(input)\n #print(image_preds.shape, exam_pred.shape)\n image_preds_all += [torch.argmax(image_preds, 1).detach().cpu().numpy()]\n image_targets_all += [image_labels.detach().cpu().numpy()]\n \n loss = loss_fn(image_preds, image_labels)\n \n loss_sum += loss.item()*image_labels.shape[0]\n sample_num += image_labels.shape[0] \n\n if ((step + 1) % CFG['verbose_step'] == 0) or ((step + 1) == len(val_loader)):\n description = f'epoch {epoch} loss: {loss_sum/sample_num:.4f}'\n pbar.set_description(description)\n \n image_preds_all = np.concatenate(image_preds_all)\n image_targets_all = np.concatenate(image_targets_all)\n print('validation multi-class accuracy = {:.4f}'.format((image_preds_all==image_targets_all).mean()))\n \n if scheduler is not None:\n if schd_loss_update:\n scheduler.step(loss_sum/sample_num)\n else:\n scheduler.step()\n \ndef inference_one_epoch(model, data_loader, device):\n model.eval()\n\n image_preds_all = []\n \n pbar = tqdm(enumerate(data_loader), total=len(data_loader))\n for step, (imgs) in pbar:\n imgs = imgs.to(device).float()\n \n image_preds = model(imgs) \n image_preds_all += [torch.softmax(image_preds, 1).detach().cpu().numpy()]\n\n \n \n image_preds_all = np.concatenate(image_preds_all, axis=0)\n return image_preds_all\n\nkwargs={'map_location':lambda storage, loc: storage.cuda(0)}\ndef load_GPUS(model,model_path,kwargs):\n state_dict = torch.load(model_path,**kwargs)\n # create new OrderedDict that does not contain `module.`\n from collections import OrderedDict\n new_state_dict = OrderedDict()\n for k, v in state_dict.items():\n name = k[7:] # remove `module.\n new_state_dict[name] = v\n # load params\n model.load_state_dict(new_state_dict)\n return model\n\n\n###### test and select conf_threshold_final>* to reform the novel training dataset\nif not os.path.exists('result'):\n os.mkdir('result')\nfor iiiiii in range(1):\n\n conf_threshold = 0.001\n conf_threshold_final = 0.95\n seed_everything(CFG['seed'])\n\n folds_test = StratifiedKFold(n_splits=CFG['fold_num']).split(np.arange(train.shape[0]), train.label.values)\n\n for fold, (trn_idx, val_idx) in enumerate(folds_test):\n # we'll train fold 0 first\n if fold > 0:\n break \n\n print('Inference fold {} started'.format(fold))\n\n valid_ = train.loc[val_idx,:].reset_index(drop=True)\n valid_ds = CassavaDataset(valid_, '../input/cassava-leaf-disease-classification/train_images/', \n transforms=get_inference_transforms(), output_label=False)\n\n test_df_pseudo = pd.DataFrame()\n test_df_pseudo['image_id'] = list(os.listdir('../input/cassava-leaf-disease-classification/test_images/'))\n test_ds = CassavaDataset(test_df_pseudo, '../input/cassava-leaf-disease-classification/test_images/', \n transforms=get_inference_transforms(), output_label=False)\n\n val_loader = torch.utils.data.DataLoader(\n valid_ds, \n batch_size=CFG['valid_bs'],\n num_workers=CFG['num_workers'],\n shuffle=False,\n pin_memory=False,\n )\n\n tst_loader = torch.utils.data.DataLoader(\n test_ds, \n batch_size=CFG['valid_bs'],\n num_workers=CFG['num_workers'],\n shuffle=False,\n pin_memory=False,\n )\n\n device = torch.device(CFG['device'])\n model = CassvaImgClassifier(CFG['model_arch'], train.label.nunique()).to(device)\n\n tst_preds = []\n testdf_psuedo = []\n\n\n #for epoch in range(CFG['epochs']-3):\n for i, epoch in enumerate(CFG['used_epochs']): \n load_GPUS(model,('../input/e-b4-e20-5fold-extradata-grid/{}_fold_{}_{}'.format(CFG['model_arch'], fold, epoch)), kwargs)\n\n with torch.no_grad():\n for _ in range(CFG['tta']):\n tst_preds += [CFG['weights'][i]/sum(CFG['weights'])/CFG['tta']*inference_one_epoch(model, tst_loader, device)]\n\n tst_preds = np.mean(tst_preds, axis=0) *CFG['tta']*len(CFG['used_epochs'])\n\n\n del model\n torch.cuda.empty_cache()\n #print(len(tst_preds))\n pseudo_pred = []\n for ii in range(len(tst_preds)):\n if (tst_preds[ii][np.where(tst_preds == np.max(tst_preds))[1][0]]) > conf_threshold:\n testdf_psuedo.append(tst_preds[ii].tolist())\n np.array(testdf_psuedo)\n\n\n test_df_pseudo['label0'] = np.max(testdf_psuedo, axis=1)\n test_df_pseudo['label'] = np.argmax(testdf_psuedo, axis=1)\n\n test_df_pseudo = test_df_pseudo[test_df_pseudo['label0']>conf_threshold_final]\n test_df_pseudo.drop('label0', axis = 1, inplace = True)\n\n frames = [train,test_df_pseudo]\n test_df_pseudo = pd.concat(frames,axis=0,ignore_index=True)\n print(test_df_pseudo.tail())\n\n \n \n \n ##############retrain\n test_imgs = os.listdir('../input/cassava-leaf-disease-classification/test_images/')\n\n folds_retrain = StratifiedKFold(n_splits=CFG['fold_num'], shuffle=True, \n random_state=CFG['seed']).split(np.arange(test_df_pseudo.shape[0]),test_df_pseudo.label.values)\n for fold, (trn_idx, val_idx) in enumerate(folds_retrain):\n if fold > 0:\n break \n\n print('Training with {} started'.format(fold))\n\n print(len(trn_idx), len(val_idx))\n train_loader, val_loader = prepare_pseudo_dataloader(test_df_pseudo, trn_idx, val_idx, \n data_root='../input/cassava-leaf-disease-classification/train_images/',\n pseudo_data_root='../input/cassava-leaf-disease-classification/test_images/'\n )\n\n\n device = torch.device(CFG['device'])\n\n model = CassvaImgClassifier(CFG['model_arch'], test_df_pseudo.label.nunique(), pretrained=False).to(device)\n model = load_GPUS(model,('../input/e-b4-e20-5fold-extradata-grid/tf_efficientnet_b4_ns_fold_0_2'), kwargs)\n\n scaler = GradScaler() \n optimizer = torch.optim.Adam(model.parameters(), lr=CFG['lr'], weight_decay=CFG['weight_decay'])\n scheduler = torch.optim.lr_scheduler.CosineAnnealingWarmRestarts(optimizer, T_0=CFG['T_0'], T_mult=1, eta_min=CFG['min_lr'], last_epoch=-1)\n\n loss_tr = nn.CrossEntropyLoss().to(device) \n loss_fn = nn.CrossEntropyLoss().to(device)\n\n for epoch in range(CFG['epochs']):\n train_one_epoch(epoch, model, loss_tr, optimizer, train_loader, device, scheduler=scheduler, schd_batch_update=False)\n\n with torch.no_grad():\n valid_one_epoch(epoch, model, loss_fn, val_loader, device, scheduler=None, schd_loss_update=False)\n\n torch.save(model.state_dict(),'./result/{}_fold_{}_{}'.format(CFG['model_arch'], fold, epoch))\n\n #torch.save(model.cnn_model.state_dict(),'{}/cnn_model_fold_{}_{}'.format(CFG['model_path'], fold, CFG['tag']))\n del model, optimizer, train_loader, val_loader, scaler, scheduler\n torch.cuda.empty_cache()\n \n\n ################final test \n seed_everything(CFG['seed'])\n\n folds = StratifiedKFold(n_splits=CFG['fold_num']).split(np.arange(train.shape[0]), train.label.values)\n\n for fold, (trn_idx, val_idx) in enumerate(folds):\n if fold > 0:\n break \n print('Inference fold {} started'.format(fold))\n\n valid_ = train.loc[val_idx,:].reset_index(drop=True)\n valid_ds = CassavaDataset(valid_, '../input/cassava-leaf-disease-classification/train_images/', transforms=get_inference_transforms(), output_label=False)\n\n test_final = pd.DataFrame()\n test_final['image_id'] = list(os.listdir('../input/cassava-leaf-disease-classification/test_images/'))\n test_ds = CassavaDataset(test_final, '../input/cassava-leaf-disease-classification/test_images/', transforms=get_inference_transforms(), output_label=False)\n\n val_loader = torch.utils.data.DataLoader(\n valid_ds, \n batch_size=CFG['valid_bs'],\n num_workers=CFG['num_workers'],\n shuffle=False,\n pin_memory=False,\n )\n\n tst_loader = torch.utils.data.DataLoader(\n test_ds, \n batch_size=CFG['valid_bs'],\n num_workers=CFG['num_workers'],\n shuffle=False,\n pin_memory=False,\n )\n\n device = torch.device(CFG['device'])\n model = CassvaImgClassifier(CFG['model_arch'], train.label.nunique()).to(device)\n tst_preds_final = []\n\n for i, epoch in enumerate(CFG['pseudo_used_epochs']): \n model.load_state_dict(torch.load('./result/{}_fold_{}_{}'.format(CFG['model_arch'], fold, epoch)))\n with torch.no_grad():\n for _ in range(CFG['tta']):\n tst_preds_final += [CFG['weights'][i]/sum(CFG['weights'])/CFG['tta']*inference_one_epoch(model, tst_loader, device)]\n tst_preds_final = np.mean(tst_preds_final, axis=0) \n del model\n torch.cuda.empty_cache() \n\n test_final['label'] = np.argmax(tst_preds_final, axis=1)\n print(test_final.head())\n test_final.to_csv('submission.csv', index=False)\n\n```", "1120735": "juliaelliott \nHello, can we use pseudo labeling approach like this. I am looking forward to your reply.", "1120779": "maybe it crashed OOM then no csv generated, or time limit exceeded.", "1121572": "wilyzh Pseudolabeling is permitted as long as it is fully automated. For specificity, hand-labeling of the test set is not permitted (and should largely be mitigated by the hidden test set), so as long as you are using a truly automated approach to pseudolabeling, this is fine.", "1121607": "Try just 1 fold, not 5 folds.\n\np.s. Personally, I don't think it's a step to try.\n@wilyzh", "1121910": "Thank you for your reply!😄", "1121914": "Thank you for providing suggestion. I've reduced the batch size to 8, but this problem still occurs. Maybe it is caused by other reasons.", "1121916": "piantic. Thank you for providing suggestion, I will try it later😁", "1124738": "Perhaps you should select the submission.csv file and click the submit button. \nps. pseudo label did not improve my LB" }, "source": "meta" }