{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Fast&Automated clean PyTorch pipeline [Train]"},{"metadata":{},"cell_type":"markdown","source":"**VERSION 5:**   \nCheckpoints for Apex have been added.  \n[Inference](https://www.kaggle.com/vadimtimakin/fast-automated-clean-pytorch-pipeline-inference?scriptVersionId=49096143) has been also updated.\n"},{"metadata":{},"cell_type":"markdown","source":"**VERSION 4 features:**\n* Now it's even faster with persistent workers and turned off Pin Memory!\n* Added support for custom loss functions, optimizer, schedulers via config\n* Getting last layer bug fix\n* Testloader bug in test function fix\n* Added code for creating a working repository (but it's commented)\n* Warnings filter turned on\n* Now early stopping depends on f1-score, not validation loss.\n* Checkpoint system improvements and bug fixes\n* Description formatting"},{"metadata":{},"cell_type":"markdown","source":"**VERSION 3:** stopflag bug fixed"},{"metadata":{},"cell_type":"markdown","source":"### Hello!  \n\nI had two goals in mind when I was writing this notebook:  \n1) Сreate the fastest possible pipeline with which you can quickly test your ideas.  \n2) Make it fully automated so that ideas can be checked by changing only the config.\n\nSo what are the features of this pipeline?  \n1) Using Apex to speed up your training.   \n2) Using multiprocessing to speed up your training as well.  \n3) The presence of very convenient config that allows you to change everything from hyperparams to augmentations and even add custom ones.    \n4) Variability - you can try whatever PyTorch model just by changing its name in the config.  \n5) Simplicity - almost no extra packages, just Python, Pytorch and Albumentations.   \n6) Using f1 score instead of accuracy because of huge class imbalance.\n\n### Inference is [here](https://www.kaggle.com/vadimtimakin/fast-automated-clean-pytorch-pipeline-inference?scriptVersionId=49096143).\n"},{"metadata":{},"cell_type":"markdown","source":"It's possible to achieve 0.75+ accuracy with training model (resnet18) for only 3 minutes and 0.85+ accuracy with training model (resnext101_32x8d) for only 15 minutes, just with only flip augmentation.\nYou should keep in mind that the best way to check all of your ideas is to do it with small model like resnet18 (its set as default here), then you'll just be able to move it to bigger models.\n"},{"metadata":{},"cell_type":"markdown","source":"At the same you should understand that the key to the victory in this competition is data processing. You can easily add new augmentations, loss functions, schedulers and optimizers in config to use them during the training. Moreover you can create custom ones by starting its name in the config from \"/custom/\" and creating fuction or class with the same name which returns an augmentation or just representing loss fucntion, scheduler or optimizer and takes parameters if needed (example below)."},{"metadata":{},"cell_type":"markdown","source":"## Set up"},{"metadata":{},"cell_type":"markdown","source":"### Config"},{"metadata":{"trusted":true},"cell_type":"code","source":"class cfg:\n    \"\"\"Main config.\"\"\"\n    \n    seed = 42  # random seed\n    \n    experiment_name = \"Default\"  # Name of the current approach\n    debug = False  # Debug flag. If set to True, train data will be decreased for faster experiments.\n    pathtoimgs = \"../input/cassava-leaf-disease-classification/train_images\"  # Path to folder with train images\n    pathtocsv = \"../input/cassava-leaf-disease-classification/train.csv\"  # Path to csv-file with targets\n    path = \"\"  # Working directory, the place where the logs and the weigths will be saved\n    log = \"log.txt\"  # If exists, all the logs will be saved in this txt-file as well\n    chk = \"\"  # Path to model checkpoint (weights).\n              # If exists, the model will be uploaded from this checkpoint.\n    device = \"cuda\"  # Device\n    \n    # Model's config\n    modelname = \"resnet18\"  # PyTorch model\n    pretrained = True               # Pretrained flag\n    \n    # Training config\n    numepochs = 5       # Number of epochs\n    earlystopping = 7     # Interrupts training after a certain number of epochs if the valloss stops decreasing,\n                           # set the value to \"-1\" if you want to turn off this option.\n    trainbatchsize = 128    # Train Batch Size\n    trainshuffle = True    # Shuffle during training\n    valbatchsize = 128      # Validation Batch Size\n    valshuffle = False     # Shuffle during validation\n    testbatchsize = 1      # Test Batch Size\n    testshuffle = False    # Shuffle during testing\n    verbose = True         # If set to True draws plots of loss changes and test metric changes.\n    savestep = 10          # Number of epochs in loop before saving model. \n                           # 10 means that weights will be saved each 10 epochs.\n    numworkers = 4         # Number of workers\n    apex = True            # Using Apex for training flag\n    apexoptlvl = \"O2\"      # Apex optimization level. O3 breaks the training.\n    trainsize, valsize, testsize = 0.9, 0.1, 0.0  # Sizes for split. You can set 0.0 value for testsize,\n                                                  # in this case test won't be used. \n\n    # Transforms' config\n    pretransforms = [     # Pre-transforms \n        dict(\n            name=\"Resize\",\n            params=dict(\n                height=256,\n                width=256,\n                p=1.0,\n            )\n        ),\n        dict(\n            name=\"Normalize\",\n            params=dict(\n                mean=[0.485, 0.456, 0.406],\n                std=[0.229, 0.224, 0.225],\n                max_pixel_value=255.0,\n                p=1.0\n            )\n        ),\n    ]\n    \n    augmentations = [     # Training augmentations\n        dict(\n            name=\"HorizontalFlip\",\n            params=dict(\n                always_apply=False,\n                p=0.5,\n            )\n        ),\n    ]\n    \n    posttransforms = [    # Post-transforms\n        dict(\n            name=\"/custom/totensor\",\n            params=dict(\n            )\n        ),\n    ]\n    \n    # Optimizer's config\n    optimizer = \"AdamW\"  # PyTorch optimizer\n    optimizerparams = dict(\n        lr=0.001,           # Learning rate\n        weight_decay=0.01,  # Weight decay\n    )\n    \n    # Scheduler's confing\n    scheduler = \"ReduceLROnPlateau\"  # PyTorch scheduler\n    schedulerparams = dict(\n        mode=\"min\",    # Mode\n        patience=5,    # Number of epochs before decreasing learning rate\n        factor=0.1,    # Factor of changing learning rate\n        verbose=True,  # The stdout message about reducing learning rate\n    )\n    \n    # Loss function's config\n    lossfn = \"CrossEntropyLoss\"  # PyTorch loss fucntion\n    lossfnparams = {}\n    \n    # Don't change\n    NUMCLASSES = 5  # CONST\n    # Can be changed only with uploading the model from the checkpoint\n    stopflag = 0    \n    schedulerstate = None \n    optimdict = None","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Imports"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np\nimport cv2\nimport gc\nimport random\nimport torch\nimport os\nimport pandas as pd\nimport torchvision.models as models\nimport torch.optim as optim\nimport torch.nn as nn\nimport albumentations as A\nfrom albumentations.pytorch import ToTensor\nimport time\nfrom tqdm.notebook import tqdm\nfrom matplotlib import pyplot as plt\nfrom sklearn.metrics import f1_score\nfrom sklearn.model_selection import train_test_split\nfrom torch.nn.modules.module import ModuleAttributeError\nimport warnings\nwarnings.filterwarnings(\"ignore\")\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Reproducibility"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"def fullseed(seed=42):\n    \"\"\"Sets the random seeds.\"\"\"\n    random.seed(seed)\n    np.random.seed(seed)\n    torch.manual_seed(seed)\n    torch.cuda.manual_seed(seed)\n    torch.backends.cudnn.deterministic = True\n    torch.backends.cudnn.benchmark = True\n    os.environ['PYTHONHASHSEED'] = str(seed)\n\n    \nfullseed(cfg.seed)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Initializing device"},{"metadata":{"trusted":true},"cell_type":"code","source":"assert torch.cuda.is_available() or not cfg.device == \"cuda\", \"cuda isn't available\"\ndevice = torch.device(cfg.device)\nprint(device)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Installing apex "},{"metadata":{"trusted":true},"cell_type":"code","source":"if cfg.apex:    \n    !git clone https://github.com/NVIDIA/apex && cd apex && pip install -v --no-cache-dir --global-option=\"--cpp_ext\" --global-option=\"--cuda_ext\" . --user && cd .. && rm -rf apex\n    from apex import amp","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Custom fucntions and classes"},{"metadata":{"trusted":true},"cell_type":"code","source":"def totensor():\n    \"\"\"An example of custom transform function.\"\"\"\n    return A.pytorch.ToTensor()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Initializing functions"},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_model(cfg):\n    \"\"\"Get PyTorch model.\"\"\"\n    if cfg.chk:   # Loading model from the checkpoint\n        model = getattr(models, cfg.modelname)(pretrained=False)\n        # Changing the last layer according the number of classes\n        lastlayer = list(model._modules)[-1]\n        try:\n            setattr(model, lastlayer, nn.Linear(in_features=getattr(model, lastlayer).in_features,\n                                                out_features=cfg.NUMCLASSES, bias=True))\n        except ModuleAttributeError:\n            setattr(model, lastlayer, nn.Linear(in_features=getattr(model, lastlayer)[1].in_features,\n                                    out_features=cfg.NUMCLASSES, bias=True))\n        cp = torch.load(cfg.chk)\n        if 'model' in cp:\n            model.load_state_dict(cp['model'])\n        else:\n            model.load_state_dict(cp)\n        if 'epoch' in cp:\n            epoch = int(cp['epoch'])\n        if 'trainloss' in cp:\n            trainloss = cp['trainloss']\n        if 'valloss' in cp:\n            valloss = cp['valloss']\n        if 'metric' in cp:\n            metric = cp['metric']\n        if 'optimizer' in cp:\n            cfg.optimdict = cp['optimizer']\n            lr = cp['optimizer'][\"param_groups\"][0]['lr']\n        if 'stopflag' in cp:\n            stopflag = cp['stopflag']\n            cfg.stopflag = stopflag\n        if 'scheduler' in cp:\n            cfg.schedulerstate = cp['scheduler']\n        print(\"Uploading model from the checkpoint...\",\n              \"\\nEpoch:\", epoch,\n              \"\\nTrain Loss:\", trainloss,\n              \"\\nVal Loss:\", valloss,\n              \"\\nMetrics:\", metric,\n              \"\\nlr:\", lr,\n              \"\\nstopflag:\", stopflag)\n    else:    # Creating a new model\n        model = getattr(models, cfg.modelname)(pretrained=cfg.pretrained)\n        # Changing the last layer according the number of classes\n        lastlayer = list(model._modules)[-1]\n        try:\n            setattr(model, lastlayer, nn.Linear(in_features=getattr(model, lastlayer).in_features,\n                                                out_features=cfg.NUMCLASSES, bias=True))\n        except ModuleAttributeError:\n            setattr(model, lastlayer, nn.Linear(in_features=getattr(model, lastlayer)[1].in_features,\n                                                out_features=cfg.NUMCLASSES, bias=True))\n    return model.to(cfg.device)\n\n\ndef get_optimizer(model, cfg):\n    \"Get PyTorch optimizer.\"\n    optimizer =  globals()[cfg.optimizer[8:]](model.parameters(), **cfg.optimizerparams) \\\n    if cfg.optimizer.startswith(\"/custom/\") \\\n    else getattr(optim, cfg.optimizer)(model.parameters(), **cfg.optimizerparams)\n    if cfg.optimdict:\n        optimizer.load_state_dict(cfg.optimdict)\n    return optimizer\n\n\ndef get_scheduler(optimizer, cfg):\n    \"\"\"Get PyTorch scheduler.\"\"\"\n    if cfg.schedulerstate:\n        return cfg.schedulerstate\n    return  globals()[cfg.scheduler[8:]](optimizer, **cfg.schedulerparams) \\\n    if cfg.scheduler.startswith(\"/custom/\") \\\n    else getattr(optim.lr_scheduler, cfg.scheduler)(optimizer, **cfg.schedulerparams)\n    \n\ndef get_lossfn(cfg):\n    \"\"\"Get PyTorch loss function.\"\"\"\n    return  globals()[cfg.lossfn[8:]](**cfg.lossfnparams) \\\n    if cfg.lossfn.startswith(\"/custom/\") \\\n    else getattr(nn, cfg.lossfn)(**cfg.lossfnparams)\n    \n\ndef get_transforms(cfg):\n    \"\"\"Get train and test augmentations.\"\"\"\n    pretransforms = [globals()[item[\"name\"][8:]](**item[\"params\"]) if item[\"name\"].startswith(\"/custom/\") \n                     else getattr(A, item[\"name\"])(**item[\"params\"]) for item in cfg.pretransforms]\n    augmentations = [globals()[item[\"name\"][8:]](**item[\"params\"]) if item[\"name\"].startswith(\"/custom/\") \n                     else getattr(A, item[\"name\"])(**item[\"params\"]) for item in cfg.augmentations]\n    posttransforms = [globals()[item[\"name\"][8:]](**item[\"params\"]) if item[\"name\"].startswith(\"/custom/\") \n                     else getattr(A, item[\"name\"])(**item[\"params\"]) for item in cfg.posttransforms]\n    train = A.Compose(pretransforms + augmentations + posttransforms)\n    test = A.Compose(pretransforms + posttransforms)\n    return train, test","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def datagenerator(cfg):\n    \"\"\"Generates data (images and targets) for train and test.\"\"\"\n    \n    print(\"Getting the data...\")\n    assert cfg.trainsize + cfg.valsize + cfg.testsize == 1, \"the sum of the split size must be equal to 1.\"\n    data = pd.read_csv(cfg.pathtocsv)\n    if cfg.debug:\n        data = data.sample(n=1000, random_state=cfg.seed).reset_index(drop=True)\n    targets = list(data[\"label\"])\n    files = list(data[\"image_id\"])\n    \n    # If test size is equal zero, we split the data only into train and validation parts, \n    # otherwise we split it into train, validation and test parts.\n    trainimgs, testimgs, traintargets, testtargets = train_test_split(files, targets, train_size=cfg.trainsize,\n                                                                      test_size=cfg.valsize+cfg.testsize,\n                                                                      random_state=cfg.seed, stratify=targets)\n    if cfg.testsize == 0:\n        return trainimgs, traintargets, testimgs, testtargets\n    \n    valimgs,testimgs, valtargets, testtargets = train_test_split(testimgs, testtargets,\n                                                                  train_size=cfg.valsize,\n                                                                  test_size=cfg.testsize,\n                                                                  random_state=cfg.seed, \n                                                                  stratify=testtargets)\n    return trainimgs, traintargets, valimgs, valtargets, testimgs, testtargets ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class CassavaDataset(torch.utils.data.Dataset):\n    \"\"\"Cassava Dataset for uploading images and targets.\"\"\"\n    \n    def __init__(self, cfg, images, targets, transforms):\n        self.images = images           # List with images\n        self.targets = targets         # List with targets\n        self.transforms = transforms   # Transforms\n        self.cfg = cfg                 # Config\n        \n    def __getitem__(self, idx):\n        img = cv2.imread(os.path.join(self.cfg.pathtoimgs, self.images[idx]))\n        img = torch.FloatTensor(self.transforms(image=img)[\"image\"])\n        target = torch.LongTensor([int(self.targets[idx])])\n        return img, target\n\n    def __len__(self):\n        return len(self.targets)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_loaders(cfg):\n    \"\"\"Getting dataloaders for train, validation (and test, if needed).\"\"\"\n    trainforms, testforms = get_transforms(cfg)\n    \n    # If test size is equal zero, we create the loaders only for train and validation parts, \n    # otherwise we create the loaders for train, validation and test parts.\n    if cfg.testsize != 0.0:\n        trainimgs, traintargets, valimgs, valtargets, testimgs, testtargets = datagenerator(cfg)\n        traindataset = CassavaDataset(cfg, trainimgs, traintargets, trainforms)\n        valdataset = CassavaDataset(cfg, valimgs, valtargets, testforms)\n        testdataset = CassavaDataset(cfg, testimgs, testtargets, testforms)\n        trainloader = torch.utils.data.DataLoader(traindataset,\n                                                  shuffle=cfg.trainshuffle,\n                                                  batch_size=cfg.trainbatchsize,\n                                                  pin_memory=False,\n                                                  num_workers=cfg.numworkers,\n                                                  persistent_workers=True)\n        valloader = torch.utils.data.DataLoader(valdataset,\n                                                shuffle=cfg.valshuffle,\n                                                batch_size=cfg.valbatchsize,\n                                                pin_memory=False,\n                                                num_workers=cfg.numworkers,\n                                                persistent_workers=True)\n        testloader = torch.utils.data.DataLoader(testdataset,\n                                                 shuffle=cfg.testshuffle,\n                                                 batch_size=cfg.testbatchsize,\n                                                 pin_memory=False,\n                                                 num_workers=cfg.numworkers,\n                                                 persistent_workers=True)\n        return trainloader, valloader, testloader\n    \n    else:\n        trainimgs, traintargets, valimgs, valtargets = datagenerator(cfg)\n        traindataset = CassavaDataset(cfg, trainimgs, traintargets, trainforms)\n        valdataset = CassavaDataset(cfg, valimgs, valtargets, testforms)\n        trainloader = torch.utils.data.DataLoader(traindataset,\n                                                  shuffle=cfg.trainshuffle,\n                                                  batch_size=cfg.trainbatchsize,\n                                                  pin_memory=False,\n                                                  num_workers=cfg.numworkers,\n                                                  persistent_workers=True)\n        valloader = torch.utils.data.DataLoader(valdataset,\n                                                shuffle=cfg.valshuffle,\n                                                batch_size=cfg.valbatchsize,\n                                                pin_memory=False,\n                                                num_workers=cfg.numworkers,\n                                                persistent_workers=True)\n        return trainloader, valloader","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def savemodel(model, epoch, trainloss, valloss, metric, optimizer, stopflag, name, scheduler):\n    \"\"\"Saves PyTorch model.\"\"\"\n    chk = {\n        'model': model.state_dict(),\n        'epoch': epoch,\n        'trainloss': trainloss,\n        'valloss': valloss,\n        'metric': metric,\n        'optimizer': optimizer.state_dict(),\n        'stopflag': stopflag,\n        'scheduler': scheduler,\n    }\n    if cfg.apex:\n        chk[\"amp\"] = amp.state_dict()\n    torch.save(chk, os.path.join(cfg.path, name))\n\n\ndef drawplot(trainlosses, vallosses, metrics):\n    \"\"\"Draws plots of loss changes and test metric changes.\"\"\"\n    # Validation and train loss changes\n    plt.plot(range(len(trainlosses)), trainlosses, label='Train Loss')\n    plt.plot(range(len(vallosses)), vallosses, label='Val Loss')\n    plt.legend()\n    plt.title(\"Validation and train loss changes\")\n    plt.show()\n    # Test metrics changes\n    plt.plot(range(len(metrics)), metrics, label='Metrics')\n    plt.legend()\n    plt.title(\"Test metrics changes\")\n    plt.show()\n    \n    \ndef printreport(t, trainloss, valloss, metric, record):\n    \"\"\"Prints epoch's report.\"\"\"\n    print(f'Time: {t} s')\n    print(f'Train Loss: {trainloss:.4f}')\n    print(f'Val Loss: {valloss:.4f}')\n    print(f'Metrics: {metric:.4f}')\n    print(f'Current best Val Loss: {record:.4f}')\n\n    \ndef savelog(path, epoch, trainloss, valloss, metric):\n    \"\"\"Saves the epoch's log.\"\"\"\n    with open(path, \"a\") as file:\n        file.write(\"epoch: \" + str(epoch) + \" trainloss: \" + str(\n            trainloss) + \" valloss: \" + str(valloss) + \" metrics: \" + str(metric) + \"\\n\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def train(model, trainloader, optimizer, lossfn):\n    \"\"\"Train loop.\"\"\"\n    print(\"Training\")\n    model.train()\n    totalloss = 0.0\n    \n    for batch in tqdm(trainloader):\n        inputs, labels = batch\n        labels = labels.squeeze(1).to(cfg.device)\n        optimizer.zero_grad()\n        outputs = model(inputs.to(cfg.device))\n        loss = lossfn(outputs, labels)\n        loss.backward()\n        optimizer.step()\n        totalloss += loss.item()\n        \n    return totalloss / len(trainloader)\n\n\ndef validation(model, valloader, lossfn):\n    \"\"\"Validation loop.\"\"\"\n    print(\"Validating\")\n    model.eval()\n    totalloss = 0.0\n    preds, targets = [], []\n    \n    with torch.no_grad():\n        for batch in tqdm(valloader):\n            inputs, labels = batch\n            labels = labels.squeeze(1).to(cfg.device)\n            outputs = model(inputs.to(cfg.device))\n            for idx in np.argmax(outputs.cpu(), axis=1):\n                preds.append([1 if idx == i else 0 for i in range(5)])\n            for j in labels:\n                targets.append([1 if i == j else 0 for i in range(5)])\n            loss = lossfn(outputs, labels)\n            totalloss += loss.item()\n    \n    score = f1_score(targets, preds, average='weighted')\n    return totalloss / len(valloader), score\n\n\ndef test(model, testloader, lossfn):\n    \"\"\"Testing loop.\"\"\"\n    print(\"Testing\")\n    model.eval()\n    totalloss = 0.0\n    preds, targets = [], []\n    \n    with torch.no_grad():\n        for batch in tqdm(testloader):\n            inputs, labels = batch\n            labels = labels.squeeze(1).to(cfg.device)\n            outputs = model(inputs.to(cfg.device))\n            for idx in np.argmax(outputs.cpu(), axis=1):\n                preds.append([1 if idx == i else 0 for i in range(5)])\n            for j in labels:\n                targets.append([1 if i == j else 0 for i in range(5)])\n            loss = lossfn(outputs, labels)\n            totalloss += loss.item()\n\n    score = f1_score(targets, preds, average='weighted')\n    print(\"Test Loss:\", totalloss / len(testloader),\n          \"\\nTest metrics:\", score)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Training"},{"metadata":{"trusted":true},"cell_type":"code","source":"def run(cfg):\n    \"\"\"Main function.\"\"\"\n    \n    # Turned off for Kaggle kernels\n#     if not os.path.exists(cfg.path):\n#         os.makedirs(cfg.path)\n    \n    # Getting the objects\n    torch.cuda.empty_cache()\n    if cfg.testsize != 0.0:\n        trainloader, valloader, testloader = get_loaders(cfg)\n    else:\n        trainloader, valloader = get_loaders(cfg)\n    model = get_model(cfg)\n    optimizer = get_optimizer(model, cfg)\n    if cfg.apex:\n        model, optimizer = amp.initialize(model, optimizer, opt_level=cfg.apexoptlvl, verbosity=0)\n    if cfg.chk and cfg.apex:\n        model.load_state_dict(torch.load(cfg.chk)['model'])\n        optimizer.load_state_dict(torch.load(cfg.chk)['optimizer'])\n        amp.load_state_dict(torch.load(cfg.chk)['amp'])\n    scheduler = get_scheduler(optimizer, cfg)\n    lossfn = get_lossfn(cfg)\n\n    # Initializing metrics\n    trainlosses, vallosses, metrics = [], [], []\n    record = 0\n    stopflag = cfg.stopflag if cfg.stopflag else 0\n    print('Testing \"' + cfg.experiment_name + '\" approach.')\n    if cfg.log:\n        with open(os.path.join(cfg.path, cfg.log), \"w\") as file:\n            file.write('Testing \"' + cfg.experiment_name + '\" approach.\\n')\n    \n    # Training\n    print(\"Have a nice training!\")\n    for epoch in range(1, cfg.numepochs + 1):\n        print(\"Epoch:\", epoch)\n        start_time = time.time()\n        trainloss = train(model, trainloader, optimizer, lossfn)\n        valloss, metric = validation(model, valloader, lossfn)\n        trainlosses.append(trainloss)\n        vallosses.append(valloss)\n        metrics.append(metric)\n        if cfg.scheduler == \"ReduceLROnPlateau\":\n            scheduler.step(valloss)\n        else:\n            scheduler.step()\n        if metric > record:\n            stopflag = 0\n            record = metric\n            savemodel(model, epoch, trainloss, valloss, metric,\n                      optimizer, stopflag, os.path.join(cfg.path, 'thebest.pt'), scheduler)\n            print('New record!')\n        else:\n            stopflag += 1\n            if epoch % cfg.savestep == 0:\n                savemodel(model, epoch, trainloss, valloss, metric,\n                      optimizer, stopflag, os.path.join(cfg.path, f'{epoch}epoch.pt'), scheduler)\n        t = int(time.time() - start_time)\n        printreport(t, trainloss, valloss, metric, record)\n        \n        # Saving to the log\n        if cfg.log:\n            savelog(os.path.join(cfg.path, cfg.log), epoch, trainloss, valloss, metric)\n        \n        torch.cuda.empty_cache()\n        gc.collect()\n        \n        # Early stopping\n        if stopflag == cfg.earlystopping:\n            print(\"Training has been interrupted because of early stopping.\")\n            break\n    \n    # Test\n    if cfg.testsize != 0.0:\n        test(model, testloader, lossfn)\n    \n    # Verbose\n    if cfg.verbose:  \n        drawplot(trainlosses, vallosses, metrics)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"run(cfg)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**If you have read up to this point and found something useful for yourself, then you can leave your upvote.** "}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}