{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.14","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"nvidiaTeslaT4","dataSources":[{"sourceId":71750,"databundleVersionId":7863244,"sourceType":"competition"},{"sourceId":9383865,"sourceType":"datasetVersion","datasetId":5693145},{"sourceId":9387303,"sourceType":"datasetVersion","datasetId":5695857},{"sourceId":101191,"sourceType":"modelInstanceVersion","modelInstanceId":84842,"modelId":109072},{"sourceId":108600,"sourceType":"modelInstanceVersion","modelInstanceId":90956,"modelId":115189},{"sourceId":112964,"sourceType":"modelInstanceVersion","modelInstanceId":94745,"modelId":118956}],"dockerImageVersionId":30761,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# HOW TO: beat top-3 in Super Resolution in Video Games competition\n## Score: ≈33.5","metadata":{"_uuid":"a8ab65a1-ba23-4119-91f2-5aa5bf593c85","_cell_guid":"d4f84a8c-08a0-4712-92e5-9743f90f2884","trusted":true}},{"cell_type":"markdown","source":"# <a id='0'>Content</a>\n\n- <a href='#1'>Task description</a>  \n- <a href='#2'>Ideas summary</a>  \n- <a href='#3'>Inspect images: Choose a threshold to filter out the images</a>  \n- <a href='#4'>Model</a>   \n - <a href='#5'>Prepare training files</a>   \n - <a href='#6'>Clone repositories, install requirements</a>   \n - <a href='#7'>Add PSNR filter to train file</a>   \n - <a href='#8'>Change yaml: add paths, num iterations</a>  \n - <a href='#9'>Use prepared train.py</a>   \n- <a href='#10'>Inference</a>   \n- <a href='#11'>Encode and submit</a>   ","metadata":{"_uuid":"8e522061-e99b-4e23-8c1c-fd0005040390","_cell_guid":"e8760a59-4237-4aed-8e49-a2e4e86397d3","trusted":true}},{"cell_type":"markdown","source":"# <a id='1'>Task description</a>  \n\nThere are screenshots from games in poor resolution, we need to:  \n- **Increase the resolution** by 4x  \n- Make the picture **more “clear”**, including remove the blurring of borders  \n- **Restore details**, the outlines of which are poorly recognizable (but at the same time do not add unnecessary things)  \n\nIn train dataset we have **low resolution and high resolution images**. Low resolution images are 270x480 pixels, high resolution images are 1080x1920 pixels.\nIn test dataset we have **low resolution images which are cropped to 64x64 pixels**. High resolution test images are not publicly available and are cropped in the same way to 256x256 pixels. The type of cropping is random paired cropping  \n\nThe key metric is **Peak Signal-to-Noise Ratio (PSNR)** between the predicted high resolution images and the observed high resolution images","metadata":{"_uuid":"a08df816-b8db-4fd2-a3c8-39c0b2d307b3","_cell_guid":"930887a0-8e4c-4ae2-9372-0db16ab5ff5b","trusted":true}},{"cell_type":"markdown","source":"### Just using the HAT-S-x4 model already allows you to beat the benchmark base. Fine-tuning allows you to beat the third place, and fine-tuning with data cleaning - the second place\n\n# <a id='2'>Ideas summary</a>  \n\n**Disclamer! Most of the actions in this notebook are crutches needed because of working with Kaggle**\n\n* **check the train data**: there are pairs of pictures that don't have the same image content (e.g. hr has lighted lamps and lr doesn't). there are also many duplicates (same pairs of pictures). delete what you think is trash  \n\n*I have deleted pairs that have PSNR (from ignit.metrics) less than 18*  \n\n* **use augmentations in dataloader** (in addition to classical rotations we need pair crop, because we have cropped images on the test).  \n\n*I have horizontal + vertical rotations, paired random crop*\n\n* **take some pre-trained models** (pay attention to what it was trained on) and compare the quality specifically in this competition.  \n\n*I checked it visually + through trial submissions (compared EDSR, ESRGAN, HAT, HAT small, Real HAT GAN).*\n\n* **you can use [Chainner](https://chainner.app/)** to speed up inference and to test pre-trained models without fine-tuning.  \n\n* **fine-tune the model on your own data.** for some models you need to take ready-made train.py, an example of how to work with them (for HAT) in kaggle below.  \n* **the process can be paralleled** as in the examples of the model developers, but in kaggle it does not seem to be possible to do so\n\n*I have [HAT-S_SRx4](https://github.com/XPixelGroup/HAT), prepared train.py according to instructions from model developers + ≈60,000 iterations (but part is skipped because of mismatched pictures in a pair)*\n\n* It would be useful to also **split the data into training and validation**.  \n\n*I didn't do that*\n\n* In this contest the order of images in submission was initially important, adapting to this, the code for encode was taken using susmission example","metadata":{"_uuid":"3092b703-8b82-472f-bef2-36b496fbbb7c","_cell_guid":"502b287c-d974-4c1e-bd26-5ce1e6250c2a","trusted":true}},{"cell_type":"code","source":"!pip install torch torchvision torchmetrics opencv-python imagehash pyyaml","metadata":{"_uuid":"8110f195-47e8-45a0-9a20-c5257d0a5ea9","_cell_guid":"7a4c7c58-a926-4ba0-ab82-6dba05c99c73","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-09-14T22:54:08.970783Z","iopub.execute_input":"2024-09-14T22:54:08.971334Z","iopub.status.idle":"2024-09-14T22:54:23.284737Z","shell.execute_reply.started":"2024-09-14T22:54:08.971300Z","shell.execute_reply":"2024-09-14T22:54:23.283717Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!pip install git+https://github.com/eugenesiow/super-image.git","metadata":{"_uuid":"152263b9-8faa-4dc9-a254-a127943f1f74","_cell_guid":"abc53919-7ccd-4a23-86ba-4c6d57475f5f","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-09-14T22:54:23.287025Z","iopub.execute_input":"2024-09-14T22:54:23.287739Z","iopub.status.idle":"2024-09-14T22:54:50.423945Z","shell.execute_reply.started":"2024-09-14T22:54:23.287688Z","shell.execute_reply":"2024-09-14T22:54:50.422902Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import os\nimport numpy as np\nimport torch\nimport torchvision\nfrom torch.utils.data import Dataset, DataLoader\nfrom PIL import Image\nimport albumentations as A\nfrom albumentations.pytorch import ToTensorV2\nfrom ignite.metrics import PSNR\nimport matplotlib.pyplot as plt\nfrom tqdm import tqdm\nfrom torchvision import transforms\nfrom super_image import EdsrModel\nimport imagehash","metadata":{"_uuid":"32f25fe3-ddcb-4e09-a699-7e20019afc8e","_cell_guid":"d3400b7c-7fa7-4422-b91e-2b2756381bc3","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-09-14T22:54:50.425329Z","iopub.execute_input":"2024-09-14T22:54:50.425663Z","iopub.status.idle":"2024-09-14T22:54:58.128363Z","shell.execute_reply.started":"2024-09-14T22:54:50.425628Z","shell.execute_reply":"2024-09-14T22:54:58.127599Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# <a id='3'>Inspect images</a>  \nChoose a threshold to filter out the images","metadata":{"_uuid":"96487399-b953-4aeb-bbae-c3077f95be0a","_cell_guid":"2a8be608-c329-447b-9a3f-57b103ae7006","trusted":true}},{"cell_type":"code","source":"with open(\"/kaggle/input/suspects/suspects_to_choose_threshold.txt\", \"r\", encoding=\"utf-8\") as file:\n    suspects = file.readlines()\n\n# Удаляем символ переноса строки из каждой строки (если он есть)\nsuspects = [suspect.strip() for suspect in suspects]","metadata":{"_uuid":"95434a56-c33c-45a1-924d-023d2ed76a96","_cell_guid":"61c0ebba-c82f-48ea-9d66-1d50075abc48","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-09-14T22:54:58.130324Z","iopub.execute_input":"2024-09-14T22:54:58.130717Z","iopub.status.idle":"2024-09-14T22:54:58.151114Z","shell.execute_reply.started":"2024-09-14T22:54:58.130684Z","shell.execute_reply":"2024-09-14T22:54:58.150265Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def inspect_psnr_pairs(high_res_dir, low_res_dir, \n                       mismatched_pairs, num_samples=10):\n    counter = 0\n    for pair_name in mismatched_pairs:\n        if counter == num_samples:\n            break\n        hr_path = os.path.join(high_res_dir, pair_name)\n        lr_path = os.path.join(low_res_dir, pair_name)\n\n        hr_img = Image.open(hr_path).convert(\"RGB\")\n        lr_img = Image.open(lr_path).convert(\"RGB\")\n        \n        # Resize images to 256x256 for PSNR check\n        high_res_image_resized = hr_img.resize((256, 256), Image.BICUBIC)\n        low_res_image_resized = lr_img.resize((256, 256), Image.BICUBIC)\n\n        # Convert to numpy array\n        high_res_image_resized = np.array(high_res_image_resized)\n        low_res_image_resized = np.array(low_res_image_resized)\n\n        # Calculate PSNR\n        psnr_metric = PSNR(data_range=255)\n        psnr_metric.update((torch.tensor(high_res_image_resized).unsqueeze(0), torch.tensor(low_res_image_resized).unsqueeze(0)))\n        psnr_value = psnr_metric.compute()\n        psnr_metric.reset()\n        \n        if psnr_value <= 18:\n            counter += 1\n            fig, axs = plt.subplots(1, 2, figsize=(10, 5))\n            axs[0].imshow(hr_img)\n            axs[0].set_title('High Res')\n            axs[1].imshow(lr_img)\n            axs[1].set_title('Low Res')\n            fig.suptitle(f'PSNR between High Res and Low Res Images: {psnr_value:.2f}', fontsize=16)\n            psnr_value\n            plt.show()\n\n# Пример использования\ninspect_psnr_pairs('/kaggle/input/super-resolution-in-video-games/train/hr', \n                   '/kaggle/input/super-resolution-in-video-games/train/lr', \n                   suspects)","metadata":{"_uuid":"87b7c245-a6c7-421c-b7d8-c9a458fa396c","_cell_guid":"51e3ffe1-f58f-48d3-bef3-a0aa17e59461","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-09-14T22:54:58.152205Z","iopub.execute_input":"2024-09-14T22:54:58.152504Z","iopub.status.idle":"2024-09-14T22:55:31.763675Z","shell.execute_reply.started":"2024-09-14T22:54:58.152454Z","shell.execute_reply":"2024-09-14T22:55:31.762808Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# <a id='4'>Model</a>","metadata":{"_uuid":"9267dc62-efa6-437a-a193-8e8fbfa84f08","_cell_guid":"4c0fcdc3-a06c-4752-a9e5-070d61a9a374","trusted":true}},{"cell_type":"markdown","source":"## <a id='5'>Prepare training files</a>","metadata":{"_uuid":"03067957-0ab8-4605-8515-19cf4eff626c","_cell_guid":"1ce93d67-79a7-4775-aaec-03a99fd93b37","trusted":true}},{"cell_type":"markdown","source":"### <a id='6'>Clone repositories, install requirements</a>","metadata":{"_uuid":"2c1f572d-951f-467b-b0a0-1e9ff90c173d","_cell_guid":"7a37baba-f637-43ae-919c-deace1fda8e9","trusted":true}},{"cell_type":"code","source":"!git clone https://github.com/XPixelGroup/HAT.git","metadata":{"_uuid":"4d18f07c-f314-46d6-9199-203712e1c2f2","_cell_guid":"56554735-4e3d-48bf-bc27-37e64e6ed758","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-09-14T12:07:18.945908Z","iopub.execute_input":"2024-09-14T12:07:18.946874Z","iopub.status.idle":"2024-09-14T12:07:21.169112Z","shell.execute_reply.started":"2024-09-14T12:07:18.946822Z","shell.execute_reply":"2024-09-14T12:07:21.167999Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%cd HAT","metadata":{"_uuid":"23ac5c18-7ac5-4baa-af9a-db692ccdfa34","_cell_guid":"276cc5f2-e349-4e85-803f-e5c262618c4a","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-09-14T12:07:22.609315Z","iopub.execute_input":"2024-09-14T12:07:22.610219Z","iopub.status.idle":"2024-09-14T12:07:22.616245Z","shell.execute_reply.started":"2024-09-14T12:07:22.610175Z","shell.execute_reply":"2024-09-14T12:07:22.615365Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!pip install -r requirements.txt\n!python setup.py develop","metadata":{"_uuid":"f8570ee6-e775-450e-96a5-8db9e0f43612","_cell_guid":"93e109bc-02c8-4829-9960-4e4f3c65a26b","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-09-14T12:07:24.024951Z","iopub.execute_input":"2024-09-14T12:07:24.025317Z","iopub.status.idle":"2024-09-14T12:08:01.367681Z","shell.execute_reply.started":"2024-09-14T12:07:24.025283Z","shell.execute_reply":"2024-09-14T12:08:01.366552Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### <a id='7'>Add PSNR filter to train file</a>","metadata":{"_uuid":"09b37f0a-5fb3-4905-91fc-f80c1acc4c59","_cell_guid":"b0f31aea-da9a-45a1-b739-a5795084d5dd","trusted":true}},{"cell_type":"markdown","source":"We **cannot remove bad pairs from folders in kaggle input** because it is an unchangeable object. Also, it is **not possible to save cleaned folders in kaggle working** because there is not enough memory. Therefore, let's make a crutch and **skip the learning iteration** if PSNR is lower than 18","metadata":{"_uuid":"f25128a8-987b-403c-8697-adddf029086e","_cell_guid":"ef8736e0-9ecd-4771-89bf-397c8b2d9b9e","trusted":true}},{"cell_type":"code","source":"%%bash\necho \"\"\"\nimport datetime\nimport logging\nimport math\nimport time\nimport torch\nimport torch.nn.functional as F\n\nfrom os import path as osp\n\nfrom basicsr.data import build_dataloader, build_dataset\nfrom basicsr.data.data_sampler import EnlargedSampler\nfrom basicsr.data.prefetch_dataloader import CPUPrefetcher, CUDAPrefetcher\nfrom basicsr.models import build_model\nfrom basicsr.utils import (AvgTimer, MessageLogger, check_resume, get_env_info, get_root_logger, get_time_str,\n                           init_tb_logger, init_wandb_logger, make_exp_dirs, mkdir_and_rename, scandir)\nfrom basicsr.utils.options import copy_opt_file, dict2str, parse_options\n\nfrom ignite.metrics import PSNR\n\n\ndef init_tb_loggers(opt):\n    # initialize wandb logger before tensorboard logger to allow proper sync\n    if (opt['logger'].get('wandb') is not None) and (opt['logger']['wandb'].get('project')\n                                                     is not None) and ('debug' not in opt['name']):\n        assert opt['logger'].get('use_tb_logger') is True, ('should turn on tensorboard when using wandb')\n        init_wandb_logger(opt)\n    tb_logger = None\n    if opt['logger'].get('use_tb_logger') and 'debug' not in opt['name']:\n        tb_logger = init_tb_logger(log_dir=osp.join(opt['root_path'], 'tb_logger', opt['name']))\n    return tb_logger\n\n\ndef create_train_val_dataloader(opt, logger):\n    # create train and val dataloaders\n    train_loader, val_loaders = None, []\n    for phase, dataset_opt in opt['datasets'].items():\n        if phase == 'train':\n            dataset_enlarge_ratio = dataset_opt.get('dataset_enlarge_ratio', 1)\n            train_set = build_dataset(dataset_opt)\n            train_sampler = EnlargedSampler(train_set, opt['world_size'], opt['rank'], dataset_enlarge_ratio)\n            train_loader = build_dataloader(\n                train_set,\n                dataset_opt,\n                num_gpu=opt['num_gpu'],\n                dist=opt['dist'],\n                sampler=train_sampler,\n                seed=opt['manual_seed'])\n\n            num_iter_per_epoch = math.ceil(\n                len(train_set) * dataset_enlarge_ratio / (dataset_opt['batch_size_per_gpu'] * opt['world_size']))\n            total_iters = int(opt['train']['total_iter'])\n            total_epochs = math.ceil(total_iters / (num_iter_per_epoch))\n            logger.info('Training statistics:' +\n                        f'\\n\\tNumber of train images: {len(train_set)}' +\n                        f'\\n\\tDataset enlarge ratio: {dataset_enlarge_ratio}' +\n                        '\\n\\tBatch size per gpu: {}'.format(dataset_opt['batch_size_per_gpu']) +\n                        '\\n\\tWorld size (gpu number): {}'.format(opt['world_size']) +\n                        f'\\n\\tRequire iter number per epoch: {num_iter_per_epoch}' +\n                        f'\\n\\tTotal epochs: {total_epochs}; iters: {total_iters}.')\n        elif phase.split('_')[0] == 'val':\n            val_set = build_dataset(dataset_opt)\n            val_loader = build_dataloader(\n                val_set, dataset_opt, num_gpu=opt['num_gpu'], dist=opt['dist'], sampler=None, seed=opt['manual_seed'])\n            logger.info('Number of val images/folders in {}: {}'.format(dataset_opt['name'], len(val_set)))\n            val_loaders.append(val_loader)\n        else:\n            raise ValueError(f'Dataset phase {phase} is not recognized.')\n\n    return train_loader, train_sampler, val_loaders, total_epochs, total_iters\n\n\ndef load_resume_state(opt):\n    resume_state_path = None\n    if opt['auto_resume']:\n        state_path = osp.join('experiments', opt['name'], 'training_states')\n        if osp.isdir(state_path):\n            states = list(scandir(state_path, suffix='state', recursive=False, full_path=False))\n            if len(states) != 0:\n                states = [float(v.split('.state')[0]) for v in states]\n                resume_state_path = osp.join(state_path, f'{max(states):.0f}.state')\n                opt['path']['resume_state'] = resume_state_path\n    else:\n        if opt['path'].get('resume_state'):\n            resume_state_path = opt['path']['resume_state']\n\n    if resume_state_path is None:\n        resume_state = None\n    else:\n        device_id = torch.cuda.current_device()\n        resume_state = torch.load(resume_state_path, map_location=lambda storage, loc: storage.cuda(device_id))\n        check_resume(opt, resume_state['iter'])\n    return resume_state\n\n\ndef train_pipeline(root_path):\n    # parse options, set distributed setting, set ramdom seed\n    opt, args = parse_options(root_path, is_train=True)\n    opt['root_path'] = root_path\n\n    torch.backends.cudnn.benchmark = True\n    # torch.backends.cudnn.deterministic = True\n\n    # load resume states if necessary\n    resume_state = load_resume_state(opt)\n    # mkdir for experiments and logger\n    if resume_state is None:\n        make_exp_dirs(opt)\n        if opt['logger'].get('use_tb_logger') and 'debug' not in opt['name'] and opt['rank'] == 0:\n            mkdir_and_rename(osp.join(opt['root_path'], 'tb_logger', opt['name']))\n\n    # copy the yml file to the experiment root\n    copy_opt_file(args.opt, opt['path']['experiments_root'])\n\n    # WARNING: should not use get_root_logger in the above codes, including the called functions\n    # Otherwise the logger will not be properly initialized\n    log_file = osp.join(opt['path']['log'], 'train_{}_{}.log'.format(opt['name'], get_time_str()))\n    logger = get_root_logger(logger_name='basicsr', log_level=logging.INFO, log_file=log_file)\n    logger.info(get_env_info())\n    logger.info(dict2str(opt))\n    # initialize wandb and tb loggers\n    tb_logger = init_tb_loggers(opt)\n\n    # create train and validation dataloaders\n    result = create_train_val_dataloader(opt, logger)\n    train_loader, train_sampler, val_loaders, total_epochs, total_iters = result\n\n    # create model\n    model = build_model(opt)\n    if resume_state:  # resume training\n        model.resume_training(resume_state)  # handle optimizers and schedulers\n        logger.info('Resuming training from epoch: {}, '.format(resume_state['epoch']) + 'iter: {}.'.format(resume_state['iter']))\n        start_epoch = resume_state['epoch']\n        current_iter = resume_state['iter']\n    else:\n        start_epoch = 0\n        current_iter = 0\n\n    # create message logger (formatted outputs)\n    msg_logger = MessageLogger(opt, current_iter, tb_logger)\n\n    # dataloader prefetcher\n    prefetch_mode = opt['datasets']['train'].get('prefetch_mode')\n    if prefetch_mode is None or prefetch_mode == 'cpu':\n        prefetcher = CPUPrefetcher(train_loader)\n    elif prefetch_mode == 'cuda':\n        prefetcher = CUDAPrefetcher(train_loader, opt)\n        logger.info(f'Use {prefetch_mode} prefetch dataloader')\n        if opt['datasets']['train'].get('pin_memory') is not True:\n            raise ValueError('Please set pin_memory=True for CUDAPrefetcher.')\n    else:\n        raise ValueError(f'Wrong prefetch_mode {prefetch_mode}.' 'Supported ones are: None, cuda, cpu.')\n\n    # training\n    logger.info(f'Start training from epoch: {start_epoch}, iter: {current_iter}')\n    data_timer, iter_timer = AvgTimer(), AvgTimer()\n    start_time = time.time()\n\n    for epoch in range(start_epoch, total_epochs + 1):\n        train_sampler.set_epoch(epoch)\n        prefetcher.reset()\n        train_data = prefetcher.next()\n        while train_data is not None:\n            data_timer.record()\n\n            current_iter += 1\n            if current_iter > total_iters:\n                break\n            # Calculate PSNR\n            device = train_data['gt'].device\n            psnr_metric = PSNR(data_range=255) #.to(device)\n            \n            psnr_metric.update((F.interpolate(train_data['gt'], size=(256, 256), mode='bilinear', align_corners=False).unsqueeze(0), F.interpolate(train_data['lq'], size=(256, 256), mode='bilinear', align_corners=False).unsqueeze(0)))\n            psnr_value = psnr_metric.compute()\n            psnr_metric.reset()\n            \n            if psnr_value >= 18:\n                # update learning rate\n                model.update_learning_rate(current_iter, warmup_iter=opt['train'].get('warmup_iter', -1))\n            \n                # training\n                model.feed_data(train_data)\n                model.optimize_parameters(current_iter)\n\n                iter_timer.record()\n                if current_iter == 1:\n                    # reset start time in msg_logger for more accurate eta_time\n                    # not work in resume mode\n                    msg_logger.reset_start_time()\n                # log\n                if current_iter % opt['logger']['print_freq'] == 0:\n                    log_vars = {'epoch': epoch, 'iter': current_iter}\n                    log_vars.update({'lrs': model.get_current_learning_rate()})\n                    log_vars.update({'time': iter_timer.get_avg_time(), 'data_time': data_timer.get_avg_time()})\n                    log_vars.update(model.get_current_log())\n                    msg_logger(log_vars)\n\n                # save models and training states\n                if current_iter % opt['logger']['save_checkpoint_freq'] == 0:\n                    logger.info('Saving models and training states.')\n                    model.save(epoch, current_iter)\n\n                # validation\n                if opt.get('val') is not None and (current_iter % opt['val']['val_freq'] == 0):\n                    if len(val_loaders) > 1:\n                        logger.warning('Multiple validation datasets are *only* supported by SRModel.')\n                    for val_loader in val_loaders:\n                        model.validation(val_loader, current_iter, tb_logger, opt['val']['save_img'])\n\n                data_timer.start()\n                iter_timer.start()\n                train_data = prefetcher.next()\n            else:\n                if current_iter == 1:\n                    msg_logger.reset_start_time()\n                iter_timer.record()\n                log_vars = {'epoch': epoch, 'iter': current_iter}\n                if current_iter % opt['logger']['print_freq'] == 0:\n                    msg_logger(log_vars)\n            # end of iter\n\n    # end of epoch\n\n    consumed_time = str(datetime.timedelta(seconds=int(time.time() - start_time)))\n    logger.info(f'End of training. Time consumed: {consumed_time}')\n    logger.info('Save the latest model.')\n    model.save(epoch=-1, current_iter=-1)  # -1 stands for the latest\n    if opt.get('val') is not None:\n        for val_loader in val_loaders:\n            model.validation(val_loader, current_iter, tb_logger, opt['val']['save_img'])\n    if tb_logger:\n        tb_logger.close()\n\n\nif __name__ == '__main__':\n    root_path = osp.abspath(osp.join(__file__, osp.pardir, osp.pardir))\n    train_pipeline(root_path)\n\"\"\" > /opt/conda/lib/python3.10/site-packages/basicsr/train.py","metadata":{"_uuid":"df565f35-45d2-4e40-bdd8-d8a7b2305342","_cell_guid":"b1ae0be8-dd97-426e-aebc-8bf189f9bf5a","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-09-14T12:08:01.370321Z","iopub.execute_input":"2024-09-14T12:08:01.370992Z","iopub.status.idle":"2024-09-14T12:08:01.409675Z","shell.execute_reply.started":"2024-09-14T12:08:01.370942Z","shell.execute_reply":"2024-09-14T12:08:01.408709Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### <a id='8'>Change yaml: add paths, num iterations</a>","metadata":{"_uuid":"6bcb2108-900f-4bed-b60b-4eef11c6e493","_cell_guid":"4aa61f03-5e52-4987-b64b-76f7015ea905","trusted":true}},{"cell_type":"markdown","source":"### Lifehack how to do fine tuning of a pre-trained model in kaggle:\n\n1) start fine-tuning a pre-trained loaded model **so that at the moment of finishing you can save the model to your computer** (fine-tuned model will appear in the folder /kaggle/working/HAT/experiments -> name of your experiment specified in the config. *mine is train_HAT-S_SRx4_finetune_from_SRx2)*\n\n*for me 3000 iterations took about an hour*\n\n2) **upload the saved model back to the notebook and train it again** (by changing the path to the model in the config).","metadata":{"_uuid":"2c0f19d6-d129-4d72-8b80-9fdd5e75e018","_cell_guid":"d92f64c4-5682-4511-9db8-d247ac2dc869","trusted":true}},{"cell_type":"code","source":"NUM_ITERATIONS = 24000\nMODEL_PATH = '/kaggle/input/hat-s_srx4-clean-6/pytorch/default/1/HAT-S_SRx4-clean-6.pth'","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import yaml \n\nfile_path = '/kaggle/working/HAT/options/train/train_HAT-S_SRx4_finetune_from_SRx2.yml'\n\nwith open(file_path, 'r') as file:\n    data = yaml.safe_load(file)\n\n# Меняем данные\ndata['datasets']['train'] = {\n    'name': 'SR in Video Games',\n    'type': 'PairedImageDataset',\n    'dataroot_gt': '/kaggle/input/super-resolution-in-video-games/train/hr',\n    'dataroot_lq': '/kaggle/input/super-resolution-in-video-games/train/lr',\n    'io_backend': {\n        'type': 'disk'\n        },\n    'gt_size': 256,\n    'use_hflip': True,\n    'use_rot': True,\n\n    # data loader\n    'use_shuffle': True,\n    'num_worker_per_gpu': 2,\n    'batch_size_per_gpu': 4,\n    'dataset_enlarge_ratio': 1,\n    'prefetch_mode': 'cuda',\n    'pin_memory': True\n}\n\ndata['train']['total_iter'] = NUM_ITERATIONS\ndata['num_gpu'] = 1\ndata['pbar'] = True\ndata['path']['pretrain_network_g'] = MODEL_PATH\n\n# Удаление валидационной части\nfor val in ['val', 'val_2', 'val_3']:\n    if val in data['datasets']:\n        del data['datasets'][val]\n\n# Запись измененных данных обратно в файл\nwith open(file_path, 'w') as file:\n    yaml.safe_dump(data, file)","metadata":{"_uuid":"5ae3a087-bff5-4bad-a994-e6da6cac6fcc","_cell_guid":"9a6b2b76-cc03-4169-a5d4-0f9a583c369b","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-09-14T12:08:01.410898Z","iopub.execute_input":"2024-09-14T12:08:01.411227Z","iopub.status.idle":"2024-09-14T12:08:01.475914Z","shell.execute_reply.started":"2024-09-14T12:08:01.411195Z","shell.execute_reply":"2024-09-14T12:08:01.475036Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Fix bugs**","metadata":{"_uuid":"64d84f4e-6f37-4993-9e00-b45d149a1954","_cell_guid":"d6eea98b-7135-4ef1-b16f-13667c030a9a","trusted":true}},{"cell_type":"code","source":"!sed -i 's/from torchvision.transforms.functional_tensor import rgb_to_grayscale/from torchvision.transforms.functional import rgb_to_grayscale/' /opt/conda/lib/python3.10/site-packages/basicsr/data/degradations.py","metadata":{"_uuid":"2acaf458-8b0e-4f7f-b50c-c1fc2f6ba1d5","_cell_guid":"6328a7a3-6b13-4c68-bdba-7223e95f5959","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-09-14T12:08:01.478154Z","iopub.execute_input":"2024-09-14T12:08:01.478852Z","iopub.status.idle":"2024-09-14T12:08:02.467491Z","shell.execute_reply.started":"2024-09-14T12:08:01.478804Z","shell.execute_reply":"2024-09-14T12:08:02.466142Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## <a id='9'>Use prepared train.py</a>","metadata":{"_uuid":"667d1685-cf3d-40a5-8c36-4516bb015c23","_cell_guid":"6f97b767-682b-4532-a1c0-17e67dbe8ba5","trusted":true}},{"cell_type":"code","source":"!python hat/train.py -opt options/train/train_HAT-S_SRx4_finetune_from_SRx2.yml","metadata":{"_uuid":"2f7a3c1c-65c3-4bad-9084-8f811e101329","_cell_guid":"9afe0da5-0cb8-47ad-8f7a-3526563e9734","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2024-09-14T12:08:02.469308Z","iopub.execute_input":"2024-09-14T12:08:02.469759Z","iopub.status.idle":"2024-09-14T20:41:04.527667Z","shell.execute_reply.started":"2024-09-14T12:08:02.469711Z","shell.execute_reply":"2024-09-14T20:41:04.523235Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# <a id='10'>Inference</a> \nUse chainner (screen below) or this code may be helpful","metadata":{"_uuid":"ebd4e4db-a51d-4525-84a0-e83b32483bec","_cell_guid":"f1104d5e-3a2e-47cb-bdee-15b714c749c5","trusted":true}},{"cell_type":"code","source":"from IPython.display import Image\nimage_path = \"/kaggle/input/chainner-example/IMG_0573.JPEG\"\nImage(filename=image_path)","metadata":{"execution":{"iopub.status.busy":"2024-09-15T16:23:29.762066Z","iopub.execute_input":"2024-09-15T16:23:29.762377Z","iopub.status.idle":"2024-09-15T16:23:29.785692Z","shell.execute_reply.started":"2024-09-15T16:23:29.762333Z","shell.execute_reply":"2024-09-15T16:23:29.784862Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class TestDataset(Dataset):\n    def __init__(self, test_folder, transform=None):\n        self.test_folder = test_folder\n        self.transform = transform\n        self.filenames = os.listdir(test_folder)\n\n    def __len__(self):\n        return len(self.filenames)\n\n    def __getitem__(self, idx):\n        img_path = os.path.join(self.test_folder, self.filenames[idx])\n        img = Image.open(img_path).convert('RGB')\n        if self.transform:\n            img = self.transform(img)\n        return img, self.filenames[idx]\n\nmodel.eval()\n\n# Путь к папке с тестовыми изображениями\ntest_folder = '/kaggle/input/super-resolution-in-video-games/test/lr'\n\n# Преобразования для данных\ntransform = transforms.Compose([\n    transforms.ToTensor()\n])\n\n# Создание Dataset и DataLoader\ntest_dataset = TestDataset(test_folder, transform=transform)\ntest_loader = DataLoader(test_dataset, batch_size=4, shuffle=False)\n\npredictions = []\nfilenames = []\nids = []\n\n# Проход по всем батчам в DataLoader с использованием tqdm для отображения прогресса\nfor batch_idx, (data, filename) in enumerate(tqdm(test_loader, desc=\"Predicting\")):\n    data = data.to(device)\n\n    # Предикт с использованием модели\n    with torch.no_grad():\n        prediction = model(data)\n\n    # Преобразование тензора в массив numpy\n    prediction = prediction.permute(0, 2, 3, 1).cpu().numpy()\n    prediction = (prediction * 255).astype('uint8')\n\n    # Кодирование изображений\n    for i in range(prediction.shape[0]):\n        encoded_prediction = encode(prediction[i])\n        predictions.append(encoded_prediction)\n        filenames.append(filename[i])\n        ids.append(batch_idx * test_loader.batch_size + i)\n\n# Создание DataFrame для сабмита\nsubmission = pd.DataFrame({\n    'id': ids,\n    'filename': filenames,\n    'rle': predictions\n})\n\n# Сохранение результатов в CSV файл\nsubmission.to_csv('submission.csv', index=False)","metadata":{"_uuid":"ccc4c178-fdb5-47e3-8416-611d1b27a9e3","_cell_guid":"48513afb-d38d-47e8-b011-54e74fe2c114","collapsed":false,"jupyter":{"outputs_hidden":false},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# <a id='11'>Encode and submit (if using Chainner)</a>","metadata":{"_uuid":"033f065b-2b9e-4ca5-88a2-2df099bfa3b8","_cell_guid":"0df1502b-90d7-4cc7-ba72-283612db1e5b","trusted":true}},{"cell_type":"code","source":"import base64\nimport zlib\n\nimport cv2\nimport numpy as np\n\nimport os\nfrom PIL import Image\n\nimport pandas as pd\n\nfrom tqdm import tqdm\n\ndef encode(img: np.ndarray) -> bytes:\n    \"\"\"\n    Lossless encoding of images for submission on kaggle platform.\n\n    Parameters\n    ----------\n    img : np.ndarray\n        cv2.imread(f) - BGR image in (h, w, c) format, c = 3 (even for png format).\n        \n    Returns\n    -------\n    bytes\n        Encoded image as bytes.\n    \"\"\"\n    img_to_encode = img.astype(np.uint8)\n    img_to_encode = img_to_encode.flatten()\n    img_to_encode = np.append(img_to_encode, -1)\n\n    cnt, rle = 1, []\n    for i in range(1, img_to_encode.shape[0]):\n        if img_to_encode[i] == img_to_encode[i - 1]:\n            cnt += 1\n            if cnt > 255:\n                rle += [img_to_encode[i - 1], 255]\n                cnt = 1\n        else:\n            rle += [img_to_encode[i - 1], cnt]\n                        cnt = 1\n\n    compressed = zlib.compress(bytes(rle), zlib.Z_BEST_COMPRESSION)\n    base64_bytes = base64.b64encode(compressed)\n    return base64_bytes","metadata":{"_uuid":"da67778f-d863-48b9-915a-bf8f4b43b1d0","_cell_guid":"d83a6257-a61e-457c-867e-bd26a74dadc8","collapsed":false,"jupyter":{"outputs_hidden":false},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def prediction(\n    sample_submission: str,\n    lr_folder: str,\n    output_file: str\n) -> None:\n    \"\"\"\n    Get prediction for sample submission using huggingface model.\n\n    Parameters\n    ----------\n    sample_submission : str\n        Path to sample submission file.\n    lr_folder : str\n        Path to test dataset folder with LR images.\n    output_file : str\n        Path to output file with predictions.\n    model : str\n        Name (repository) of huggingface model.\n    device : str\n        Device on which the model will be run.\n    simple_resize : str\n            If specified then the upscaling will be done using deterministic interpolation.\n\n    Returns\n    -------\n    None\n    \"\"\"\n    submission_df = pd.read_csv(sample_submission)\n\n    filenames = submission_df[\"filename\"].values\n    for i in tqdm(range(len(filenames))):\n        filename = filenames[i]\n        init_img = cv2.imread(os.path.join(lr_folder, filename))\n        submission_df.loc[i, \"rle\"] = encode(init_img)\n    submission_df.to_csv(output_file, index=False)","metadata":{"_uuid":"95ad912c-9601-440d-a612-ed2fe54546a6","_cell_guid":"eafb2f03-6981-453c-9567-cbdf752881a3","collapsed":false,"jupyter":{"outputs_hidden":false},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"prediction(\n    sample_submission= 'your_path_to/sample_submission.csv',\n    lr_folder= 'your_path_to/hr',\n    output_file= f'your_path_to/{NUM_ITERATIONS}it_submission.csv'\n)","metadata":{"_uuid":"cd2e2c14-0dad-4a02-830b-dc11fa1e1b41","_cell_guid":"b82b8952-ca77-424a-8abf-55aac7780164","collapsed":false,"jupyter":{"outputs_hidden":false},"trusted":true},"execution_count":null,"outputs":[]}]}