{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"from IPython.display import Image\n## Image(\"../input/idln-temp-files-version-1/ConvSe2SeqArchitecture.png\") ","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Idea\nMy idea is to predict the local waypoint trajectors (starting at origin) using features from IMU sensor alone. The feature generation itself can be found at this [notebook](https://www.kaggle.com/suryajrrafl/interpolated-imu-data). The model consists of encoder-decoder architecture using Convolution blocks with attention mechanism.","metadata":{}},{"cell_type":"code","source":"## Image(\"../input/idln-temp-files-version-1/convSe2SeqModelIdea.jpg\") ","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n\n1. Encoder input - Features are IMU data, considering 2D plane, (i.e), timestamps of IMU data, linear acceleration in x,y directions and yaw rate (gyroscope z axis) along with euler angles w.r.t magnetic north. \n\n2. Decoder input - Timestamp at which waypoint is to be inferred. The inference time is relative to the first imu timestamp.\n\n3. Decoder output - Local waypoints (i.e) actual waypoint - first waypoint of path. The points are translated such that first waypoint is the origin. This is done because, the imu features give local trajectory (~shape of the path) but by itself cannot infer global trajectory directly. \n\nThe encoder and decoder data are made of fixed length. For encoder, the imu data is sampled to be of 100 time sequences. For decoder, the maximum waypoints is fixed at 107 (max number of waypoints in path in train data). More details on data generation can be found at notebook above. Padding is done to ensure equal length in all batches. \n\n\n## Problem\nDue to some issue, the model doesn't seem to learn anything. The loss seems to reduce 0.0x only with each epoch. I have doubts in loss computation (**competitionMetric_Seq2Seq**). I am not sure if the current implementation is the correct way to calculate loss for padded elements. But this is only guess and I am not sure where the mistake is. If somebody can help me out, it would be great.\n\n## References\nMost of the the model implementation is from the wonderful series of notebooks found at the following repo. I just tried adapting the notebook to our problem statement. \n\n1. [seq2seq modelling pytorch](https://github.com/bentrevett/pytorch-seq2seq)\n2. [Pytorch LSTM references](https://www.youtube.com/watch?v=sQUqQddQtB4)\n3. [Variable-length batches](https://towardsdatascience.com/taming-lstms-variable-sized-mini-batches-and-why-pytorch-is-good-for-your-health-61d35642972e)\n    ","metadata":{}},{"cell_type":"markdown","source":"## Library imports","metadata":{"papermill":{"duration":0.029515,"end_time":"2021-05-04T00:08:40.639665","exception":false,"start_time":"2021-05-04T00:08:40.61015","status":"completed"},"tags":[]}},{"cell_type":"code","source":"!pip install pickle5","metadata":{"papermill":{"duration":12.382641,"end_time":"2021-05-04T00:08:53.050236","exception":false,"start_time":"2021-05-04T00:08:40.667595","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## !pip install torch-summary\n## from torchsummary import summary","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# basic imports\nimport os\nimport gc\nimport math\nimport glob\nimport random\nimport itertools\nimport numpy as np\nimport pandas as pd\nimport pickle5 as pickle\n## from tqdm.notebook import tqdm\nfrom tqdm import tqdm\n\n# DL library imports\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport torchvision\nimport torch.optim as optim\nfrom torch.utils.data import Dataset, DataLoader\nfrom torch.optim.lr_scheduler import CosineAnnealingWarmRestarts, CosineAnnealingLR, ReduceLROnPlateau\n## from  torch.cuda.amp import autocast, GradScaler\n\n# metrics calculation\nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn.model_selection import GroupKFold\n\n# basic plotting library\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\n# interactive plots\nimport plotly.express as px\nimport plotly.graph_objects as go\nfrom plotly.subplots import make_subplots\nfrom plotly.offline import iplot\n\nimport warnings  \nwarnings.filterwarnings('ignore')","metadata":{"papermill":{"duration":4.812882,"end_time":"2021-05-04T00:08:57.895251","exception":false,"start_time":"2021-05-04T00:08:53.082369","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Config parameters","metadata":{"papermill":{"duration":0.031754,"end_time":"2021-05-04T00:08:58.031815","exception":false,"start_time":"2021-05-04T00:08:58.000061","status":"completed"},"tags":[]}},{"cell_type":"code","source":"class CFG:\n    # pipeline parameters\n    SEED        = 42\n    TRAIN       = True\n    LR_FIND     = False\n    TEST        = False\n    N_FOLDS     = 5 \n    N_EPOCHS    = 50\n    TEST_BATCH_SIZE  = 128\n    TRAIN_BATCH_SIZE = 16\n    NUM_WORKERS      = 4\n    DATA_FRAC        = 1.0\n    FOLD_TO_TRAIN    = [0,1,2,3,4] # \n\n    # model parameters\n    MODEL_ARCH  = 'seq2seq'\n    MODEL_NAME  = 'ConvSeq_v1'\n    WGT_PATH    = ''\n    WGT_MODEL   = ''\n    PRINT_N_EPOCH = 2\n    RNN_TYPE = 'LSTM'\n    \n    # scheduler variables\n    MAX_LR    = 3e-3\n    MIN_LR    = 1e-6\n    SCHEDULER = 'CosineAnnealingWarmRestarts'  # ['ReduceLROnPlateau', 'None', 'OneCycleLR','CosineAnnealingLR']\n    T_0       = 10      # CosineAnnealingWarmRestarts\n    T_MULT    = 1       # CosineAnnealingWarmRestarts\n    T_MAX     = 10      # CosineAnnealingLR\n\n    # optimizer variables\n    OPTIMIZER     = 'Adam'\n    WEIGHT_DECAY  = 1e-6\n    GRD_ACC_STEPS = 1\n    MAX_GRD_NORM  = 1\n\n    BUILDING_SITES_RANGE = [0,24]\n\nMODEL_INIT_WEIGHT = 0.08\nINPUT_DIM = 7\nDECODER_INPUT_DIM = 1\nOUTPUT_DIM = 2\nENC_EMB_DIM = 16\nDEC_EMB_DIM = 16\nHID_DIM = 32\nN_LAYERS = 10\nKERNEL_SIZE = 3\nFILLER_VALUE = 0.0\nENC_DROPOUT = 0.25\nDEC_DROPOUT = 0.25   ","metadata":{"papermill":{"duration":0.043059,"end_time":"2021-05-04T00:08:58.107606","exception":false,"start_time":"2021-05-04T00:08:58.064547","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"floor_map = {\"B2\": -2, \"B1\": -1, \"F1\": 0, \"F2\": 1, \"F3\": 2, \"F4\": 3, \"F5\": 4, \"F6\": 5, \"F7\": 6, \"F8\": 7, \"F9\": 8,\n             \"1F\": 0, \"2F\": 1, \"3F\": 2, \"4F\": 3, \"5F\": 4, \"6F\": 5, \"7F\": 6, \"8F\": 7, \"9F\": 8}\n\nmodelOutputDir = '.'\nimuFeatures_trainPath = '../input/idln-mlp-wifi-features-dataset/imuSeq2SeqDataLocal_train.pickle'\n## imuFeatures_testPath  = ''\nsampleCsvPath = '../input/indoor-location-navigation/sample_submission.csv'\nssubm = pd.read_csv(sampleCsvPath)\nssubm_df = ssubm[\"site_path_timestamp\"].apply(lambda x: pd.Series(x.split(\"_\")))\n\n## number of time sequences to give as input to encoder\nimuInputSequenceLength = 100\n\n## max number of time sequences in decoder\nwayPointMaxSequenceLength = 107","metadata":{"papermill":{"duration":0.040208,"end_time":"2021-05-04T00:08:58.180327","exception":false,"start_time":"2021-05-04T00:08:58.140119","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Helper functions","metadata":{"papermill":{"duration":0.031519,"end_time":"2021-05-04T00:08:58.243421","exception":false,"start_time":"2021-05-04T00:08:58.211902","status":"completed"},"tags":[]}},{"cell_type":"code","source":"def getBuildingName(buildingDataPath):\n    return buildingDataPath.split('/')[-1].split('_')[0]","metadata":{"papermill":{"duration":0.038906,"end_time":"2021-05-04T00:08:58.314336","exception":false,"start_time":"2021-05-04T00:08:58.27543","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def find_no_of_trainable_params(model):\n    total_trainable_params = sum(p.numel() for p in model.parameters() if p.requires_grad)\n    return total_trainable_params","metadata":{"papermill":{"duration":0.039158,"end_time":"2021-05-04T00:08:58.385788","exception":false,"start_time":"2021-05-04T00:08:58.34663","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def set_seed(seed):\n    random.seed(seed)\n    np.random.seed(seed)\n    os.environ[\"PYTHONHASHSEED\"] = str(seed)\n    torch.manual_seed(seed)\n    torch.cuda.manual_seed(seed)\n    torch.backends.cudnn.deterministic = True\n\nset_seed(CFG.SEED)","metadata":{"_kg_hide-output":true,"papermill":{"duration":0.043922,"end_time":"2021-05-04T00:08:58.461775","exception":false,"start_time":"2021-05-04T00:08:58.417853","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def getSeq2SeqFeatures(imuDataPath):\n    \"\"\" \"\"\"\n    # read features from file \n    with open(imuDataPath, 'rb') as inputFile:\n        imuData = pickle.load(inputFile)        \n    return imuData","metadata":{"papermill":{"duration":0.044629,"end_time":"2021-05-04T00:08:58.539389","exception":false,"start_time":"2021-05-04T00:08:58.49476","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def competitionMetric_Seq2Seq(preds, targets, wayPointsMask):\n    \"\"\" custom metric for seq2seq models \"\"\"\n    ## reshape predicitons and targets to 2D vector\n    outputSize = preds.shape[-1]\n    preds      = preds.view(-1, outputSize)\n    targets    = targets[:,:, 0:outputSize]\n    targets    = targets.view(-1, outputSize)\n    \n    ## unroll into 1D vector\n    wayPointsMask = wayPointsMask.view(-1)\n    totalWayPoints = wayPointsMask.sum()    \n        \n    ## mask predictions\n    xPredictions = preds[:,0] * wayPointsMask\n    yPredictions = preds[:,1] * wayPointsMask\n\n    ## position error\n    meanPosPredictionError = torch.sqrt(torch.square(xPredictions -  targets[:,0]) + \\\n                                        torch.square(yPredictions -  targets[:,1]) + 1e-6).sum()\n    meanPosPredictionError = meanPosPredictionError / totalWayPoints\n    \n    \"\"\"\n    ## floor prediction error\n    floorPredictions = preds[:,2] * wayPointsMask\n    if((preds.shape[1] == 3) and (targets.shape[1] ==3)):\n        meanFloorPredictionError = (15 * (torch.abs(floorPredictions - targets[:,2]))).sum()\n        meanFloorPredictionError = meanFloorPredictionError / totalWayPoints\n    else:\n        meanFloorPredictionError = 0.0\n    \"\"\"\n    meanFloorPredictionError = torch.tensor(0.0).to(device)\n    return meanPosPredictionError, meanFloorPredictionError","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"```python\ndef competitionMetric(preds, targets):\n    \"\"\" The metric used in this competition \"\"\"\n    # position error\n    meanPosPredictionError = torch.mean(torch.sqrt(\n                             torch.pow((preds[:,0] - targets[:,0]), 2) + \n                             torch.pow((preds[:,1] - targets[:,1]), 2)))\n    \n    ## floor prediction error\n    if((preds.shape[1] == 3) and (targets.shape[1] ==3)):\n        meanFloorPredictionError = torch.mean(15 * torch.abs(preds[:,2] - targets[:,2]))\n    else:\n        meanFloorPredictionError = 0.0\n    return meanPosPredictionError, meanFloorPredictionError\n```","metadata":{}},{"cell_type":"code","source":"def getOptimizer(model : nn.Module):    \n    if CFG.OPTIMIZER == 'Adam':\n        optimizer = optim.Adam(model.parameters(), weight_decay=CFG.WEIGHT_DECAY, lr=CFG.MAX_LR)\n    else:\n        optimizer = optim.SGD(model.parameters(), weight_decay=CFG.WEIGHT_DECAY, lr=CFG.MAX_LR, momentum=0.9)\n    return optimizer","metadata":{"papermill":{"duration":0.042186,"end_time":"2021-05-04T00:08:58.833683","exception":false,"start_time":"2021-05-04T00:08:58.791497","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def getScheduler(optimizer, dataloader_train):\n    if CFG.SCHEDULER == 'OneCycleLR':\n        scheduler = optim.lr_scheduler.OneCycleLR(optimizer, max_lr= CFG.MAX_LR, epochs = CFG.N_EPOCHS, \n                          steps_per_epoch = len(dataloader_train), pct_start=0.25, div_factor=10, anneal_strategy='cos')\n    elif CFG.SCHEDULER == 'CosineAnnealingWarmRestarts':\n        scheduler = CosineAnnealingWarmRestarts(optimizer, T_0=CFG.T_0, T_mult=CFG.T_MULT, eta_min=CFG.MIN_LR, last_epoch=-1)\n    elif CFG.SCHEDULER == 'CosineAnnealingLR':\n        scheduler = CosineAnnealingLR(optimizer, T_max=CFG.T_MAX * len(dataloader_train), eta_min=CFG.MIN_LR, last_epoch=-1)\n    else:\n        scheduler = None\n    return scheduler","metadata":{"papermill":{"duration":0.043278,"end_time":"2021-05-04T00:08:58.911468","exception":false,"start_time":"2021-05-04T00:08:58.86819","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def initModelWeights(model):\n    for name, param in model.named_parameters():\n        nn.init.uniform_(param.data, -MODEL_INIT_WEIGHT, MODEL_INIT_WEIGHT)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def getDataLoader(dataset, datasetType : str):\n    if datasetType == 'train':\n        batchSize = CFG.TRAIN_BATCH_SIZE\n        shuffleDataset = True\n    else:\n        batchSize = CFG.TEST_BATCH_SIZE\n        shuffleDataset = False\n    \n    dataLoader = DataLoader(dataset, batch_size= batchSize, shuffle=shuffleDataset,\n                            num_workers=CFG.NUM_WORKERS, pin_memory=False, drop_last=False)\n    return dataLoader","metadata":{"papermill":{"duration":0.041548,"end_time":"2021-05-04T00:08:58.985971","exception":false,"start_time":"2021-05-04T00:08:58.944423","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def plotTrainingResults(resultsDf, buildingName):\n    # subplot to plot\n    fig = make_subplots(rows=1, cols=1)\n    colors = [ ('#d32f2f', '#ef5350'), ('#303f9f', '#5c6bc0'), ('#00796b', '#26a69a'),\n                ('#fbc02d', '#ffeb3b'), ('#5d4037', '#8d6e63')]\n\n    # find number of folds input df\n    numberOfFolds = resultsDf['fold'].nunique()\n    \n    # iterate through folds and plot\n    for i in range(numberOfFolds):\n        data = resultsDf[resultsDf['fold'] == i]\n        fig.add_trace(go.Scatter(x=data['epoch'].values, y=data['trainPosLoss'].values,\n                                mode='lines', visible='legendonly' if i > 0 else True,\n                                line=dict(color=colors[i][0], width=2),\n                                name='{}-trainPossLoss-Fold{}'.format(buildingName, i)),row=1, col=1)\n\n        fig.add_trace(go.Scatter(x=data['epoch'], y=data['valPosLoss'].values,\n                                 mode='lines+markers', visible='legendonly' if i > 0 else True,\n                                 line=dict(color=colors[i][1], width=2),\n                                 name='{}-valPosLoss-Fold{}'.format(buildingName,i)),row=1, col=1)\n    fig.show()","metadata":{"papermill":{"duration":0.043986,"end_time":"2021-05-04T00:08:59.063149","exception":false,"start_time":"2021-05-04T00:08:59.019163","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Dataset class","metadata":{"papermill":{"duration":0.032789,"end_time":"2021-05-04T00:08:59.128673","exception":false,"start_time":"2021-05-04T00:08:59.095884","status":"completed"},"tags":[]}},{"cell_type":"code","source":"class imuFeaturesDataset_train(Dataset):\n    def __init__(self, imuData):\n        self.imuData = imuData \n        \n    def __getitem__(self, index):\n        ## output shape is (imuInputSequenceLength, 7)\n        ## where 7 indicates numbre of features - [ts, lin_ax, lin_ay, gz, roll, pitch, yaw]\n        ## imuInputSequenceLength is max input sequence length\n        encoderData  = self.imuData['encoderData'][index].transpose()\n        encoderData  = torch.from_numpy(encoderData)\n        encoderData[:,0]   = encoderData[:,0]   / 100.0\n        encoderData[:,1:3] = encoderData[:,1:3] / 10.0\n        encoderData[:,3]   = encoderData[:,3]   / 5.0\n        encoderData[:,4:]  = encoderData[:,4:]  / 6.28\n\n        ## output shape is (wayPointMaxSequenceLength, 3)\n        ## where 3 indicates numbere of targets - [x,y,floor]\n        decoderData  = self.imuData['decoderData'][index]  ## .transpose()\n        decoderData  = torch.from_numpy(decoderData)\n\n        ## output shape is (1, wayPointMaxSequenceLength)\n        ## where 1 indicates inference Ts - decoder input\n        inferenceTs  = self.imuData['inferenceTsList'][index]\n        inferenceTs  = torch.from_numpy(inferenceTs)\n        inferenceTs  = inferenceTs  / 100.0\n        \n        ## path name\n        pathName = self.imuData['path'][index]\n        \n        ## output shape is (1, wayPointMaxSequenceLength)\n        wayPointsMask = self.imuData['wayPointMask'][index]\n        wayPointsMask  = torch.from_numpy(wayPointsMask)\n\n        return encoderData, inferenceTs, decoderData, pathName, wayPointsMask\n     \n    def __len__ (self):\n        return len(self.imuData)","metadata":{"papermill":{"duration":0.041938,"end_time":"2021-05-04T00:08:59.203748","exception":false,"start_time":"2021-05-04T00:08:59.16181","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Convolution seq2seq Model classes - Encoder, Decoder and Seq2Seq","metadata":{"papermill":{"duration":0.034297,"end_time":"2021-05-04T00:08:59.346779","exception":false,"start_time":"2021-05-04T00:08:59.312482","status":"completed"},"tags":[]}},{"cell_type":"code","source":"class Encoder(nn.Module):\n    def __init__(self, inputSize, embSize, hidSize, nLayers, \\\n                 kernelSize, dropout, device, maxSeqLen = 100):\n        \n        super().__init__()\n        assert kernelSize % 2 == 1, \"Kernel size must be odd!\"\n        self.scale = torch.sqrt(torch.FloatTensor([0.5])).to(device)\n        \n        ## store variables\n        self.encFeatSize = inputSize\n        self.encEmbSize  = embSize\n        self.encHidSize  = hidSize\n        self.nLayers     = nLayers\n        self.kernelSize  = kernelSize\n        self.device      = device\n        self.encSeqLen   = maxSeqLen\n        \n        ## encoder layers\n        self.tokEmbedding = nn.Linear(self.encFeatSize, self.encEmbSize)\n        self.posEmbedding = nn.Embedding(self.encSeqLen, self.encEmbSize)\n        self.emb2hid = nn.Linear(self.encEmbSize, self.encHidSize)\n        self.hid2emb = nn.Linear(self.encHidSize, self.encEmbSize)\n        self.convs = nn.ModuleList([nn.Conv1d(in_channels = self.encHidSize, out_channels = 2 * self.encHidSize, \\\n                                    kernel_size = self.kernelSize,  padding = (self.kernelSize - 1) // 2)\n                                    for _ in range(self.nLayers)])\n        self.dropout = nn.Dropout(dropout)\n        \n    def forward(self, src):\n        ## src = [batchSize, encSeqLen, encFeatSize]\n        batchSize = src.shape[0]\n        \n        ## 1. create position tensor\n        ## pos = [0, 1, 2, 3, ..., encSeqLen - 1], repeated for batchSize, along rows\n        ## pos = [batchSize, encSeqLen]\n        pos = torch.arange(0, self.encSeqLen).unsqueeze(0).repeat(batchSize, 1).to(self.device)\n              \n        ## 2. embed tokens and positions\n        ## tokEmbedded = posEmbedded = [batchSize, encSeqLen, encEmbSize]        \n        tokEmbedded = self.tokEmbedding(src)\n        posEmbedded = self.posEmbedding(pos)\n        \n        ## 3. combine embeddings by elementwise summing\n        ## embedded = [batchSize, encSeqLen, encEmbSize]\n        embedded = self.dropout(tokEmbedded + posEmbedded)\n        \n        ## 4. pass embedded through linear layer to convert from emb dim to hid dim\n        ## convInput = [batchSize, encSeqLen, encHidSize]\n        convInput = self.emb2hid(embedded)\n        \n        ## 5. permute for convolutional layer\n        ## convInput = [batchSize, encHidSize, encSeqLen]\n        convInput = convInput.permute(0, 2, 1) \n        \n        ## 6. begin convolutional blocks..., repeate for nLayers\n        for i, conv in enumerate(self.convs):\n        \n            ## 7. pass through convolutional layer\n            ## conved = [batchSize, 2*encHidSize, encSeqLen]\n            conved = conv(self.dropout(convInput))\n\n            ## 8. pass through GLU activation function\n            ## conved = [batchSize, encHidSize, encSeqLen]\n            conved = F.glu(conved, dim = 1)\n\n            ## 9. apply residual connection\n            ## conved = [batchSize, encHidSize, encSeqLen]\n            conved = (conved + convInput) * self.scale\n\n            ## 10. set convInput to conved for next loop iteration\n            convInput = conved\n        \n        ## ...end convolutional blocks\n        \n        ## 11. permute and convert back to emb dim\n        ## conved = [batchSize, encSeqLen, encEmbSize]\n        conved = self.hid2emb(conved.permute(0, 2, 1))\n        \n        ## 12. elementwise sum output (conved) and input (embedded) to be used for attention\n        ## combined = [batchSize, encSeqLen, encEmbSize]\n        combined = (conved + embedded) * self.scale\n        return conved, combined","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class Decoder(nn.Module):\n    def __init__(self, inputSize, embSize, hidSize, outputSize, nLayers, \\\n                 kernelSize, dropout, fillerValue, device, maxSeqLen = 107):\n        \n        super().__init__()\n        self.scale = torch.sqrt(torch.FloatTensor([0.5])).to(device)\n        \n        ## store vairables\n        self.decFeatSize = inputSize\n        self.decEmbSize  = embSize\n        self.decHidSize  = hidSize\n        self.decOutSize  = outputSize        \n        self.nLayers     = nLayers\n        self.kernelSize  = kernelSize\n        self.fillerValue = fillerValue\n        self.device      = device        \n        self.decSeqLen   = maxSeqLen\n        \n        \n        ## decoder layers\n        self.tokEmbedding = nn.Linear(self.decFeatSize, self.decEmbSize)\n        self.posEmbedding = nn.Embedding(self.decSeqLen, self.decEmbSize)\n        self.emb2hid = nn.Linear(self.decEmbSize, self.decHidSize)\n        self.hid2emb = nn.Linear(self.decHidSize, self.decEmbSize)\n        \n        ## attention specific layers\n        self.attnHid2emb = nn.Linear(self.decHidSize, self.decEmbSize)\n        self.attnEmb2hid = nn.Linear(self.decEmbSize, self.decHidSize)\n        self.fcOut = nn.Linear(self.decEmbSize, self.decOutSize)\n\n        ## convolution block layers\n        self.convs = nn.ModuleList([nn.Conv1d(in_channels = self.decHidSize, out_channels = 2 * self.decHidSize, \\\n                                    kernel_size = self.kernelSize) for _ in range(self.nLayers)])\n        self.dropout = nn.Dropout(dropout)\n\n      \n    def calculateAttention(self, embedded, conved, encoderConved, encoderCombined):\n        \n        ## embedded = [batchSize, decSeqLen, decEmbSize]\n        ## conved = [batchSize, decHidSize, decSeqLen]\n        ## encoderConved = encoderCombined = [batchSize, encSeqLen, decEmbSize]\n        \n        ## a. permute and convert back to emb dim\n        ## convedEmb = [batchSize, decSeqLen, decEmbSize]\n        convedEmb = self.attnHid2emb(conved.permute(0, 2, 1))\n        \n        ## b. residual connection\n        ## combined = [batchSize, decSeqLen, decEmbSize]\n        combined = (convedEmb + embedded) * self.scale\n        \n        ## c. Energy matrix calculation\n        ## energy = [batchSize, decSeqLen, encSeqLen]\n        energy = torch.matmul(combined, encoderConved.permute(0, 2, 1))\n        \n        ## d. attention is softmax of energy\n        ## attention = [batchSize, decSeqLen, encSeqLen]\n        attention = F.softmax(energy, dim=2)\n        \n        ## e. attention over encoded states \n        ## attendedEncoding = [batchSize, decSeqLen, decEmbSize]\n        attendedEncoding = torch.matmul(attention, encoderCombined)\n        \n        ## f. convert from emb dim -> hid dim\n        ## attendedEncoding = [batchSize, decSeqLen, decHidSize]\n        attendedEncoding = self.attnEmb2hid(attendedEncoding)\n        \n        ## g. apply residual connection with decoder token \n        ## attendedCombined = [batch size, decHidSize, decSeqLen]\n        attendedCombined = (conved + attendedEncoding.permute(0, 2, 1)) * self.scale\n        return attention, attendedCombined\n        \n    \n    def forward(self, trg, encoderConved, encoderCombined):\n        ## trg = [batchSize, decSeqLen]\n        ## encoderConved = encoderCombined = [batchSize, encSeqLen, decEmbSize]\n        batchSize = trg.shape[0]\n            \n        ## 1. create position tensor\n        ## pos = [0, 1, 2, 3, ..., decSeqLen - 1], repeated for batchSize, along rows\n        ## pos = [batchSize, decSeqLen]        \n        pos = torch.arange(0, self.decSeqLen).unsqueeze(0).repeat(batchSize, 1).to(self.device)\n        \n        ## 2. embed tokens and positions\n        ## tokEmbedded = posEmbedded = [batchSize, decSeqLen, decEmbSize] \n        trg = trg.unsqueeze(2)\n        tokEmbedded = self.tokEmbedding(trg)\n        posEmbedded = self.posEmbedding(pos)\n        \n        ## 3. combine embeddings by elementwise summing\n        ## embedded = [batchSize, decSeqLen, decEmbSize]\n        embedded = self.dropout(tokEmbedded + posEmbedded)\n        \n        ## 4. pass embedded through linear layer to convert from emb dim to hid dim\n        ## convInput = [batchSize, decSeqLen, decHidSize]\n        convInput = self.emb2hid(embedded)\n        \n        ## 5. permute for convolutional layer\n        ## convInput = [batchSize, decHidSize, decSeqLen]\n        convInput = convInput.permute(0, 2, 1)         \n        padding = torch.zeros(batchSize, self.decHidSize, self.kernelSize - 1).fill_(\\\n                       self.fillerValue).to(self.device)\n        \n        ## 6. begin of convolution blocks \n        for i, conv in enumerate(self.convs):\n        \n            ## 7. apply dropout\n            convInput = self.dropout(convInput)\n        \n            ## 8. need to pad so decoder can't \"cheat\"\n            ## paddedConvInput = [batchSize, decHidSize, decSeqLen + kernelSize - 1]\n            paddedConvInput = torch.cat((padding, convInput), dim = 2)\n        \n            ## 9. pass through convolutional layer\n            ## conved = [batchSize, 2 * decHidSize, decSeqLen]\n            conved = conv(paddedConvInput)\n\n            ## 10. pass through GLU activation function\n            ## conved = [batchSize, decHidSize, decSeqLen]\n            conved = F.glu(conved, dim = 1)\n\n            ## 11. calculate attention\n            ## attention = [batchSize, decSeqLen, encSeqLen]\n            ## conved = [batch size, decHidSize, decSeqLen]\n            attention, conved = self.calculateAttention(embedded, conved, \n                                encoderConved, encoderCombined)\n            \n            ## 12. apply residual connection\n            ## conved = [batchSize, decHidSize, decSeqLen]\n            conved = (conved + convInput) * self.scale\n                        \n            ## 13. set convInput to conved for next loop iteration\n            convInput = conved\n            \n        ## 14. convert to decEmbSize\n        #conved = [batchSize, decSeqLen, decEmbSize]\n        conved = self.hid2emb(conved.permute(0, 2, 1))\n         \n        ## 15. linear layer to output dimension\n        ## output = [batchSize, decSeqLen, decOutSize]\n        output = self.fcOut(self.dropout(conved))\n        return output, attention","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class Seq2Seq(nn.Module):\n    def __init__(self, encoder, decoder):\n        super().__init__()\n        self.encoder = encoder\n        self.decoder = decoder\n        \n    def forward(self, src, trg):\n        ## src = [batchSize, encSeqLen, encFeatSize]\n        ## trg = [batchSize, decSeqLen]\n        \n        ## 1. calculate z^u (encoderConved) and (z^u + e) (encoderCombined)\n        ## encoderConved is output from final encoder conv. block\n        ## encoderCombined = encoderConved + src embedding + positional embeddings (all elementwise)\n        ## encoderConved = encoderCombined = [batchSize, encSeqLen, decEmbSize]\n        encoderConved, encoderCombined = self.encoder(src)\n                    \n        ## 2. calculate predictions of next words\n        ## output is a batch of predictions for each input in the decoder\n        ## attention a batch of attention scores across the encoder sequence for \n        ## each timestamp in the decoder\n        ## output = [batchSize, decSeqLen, decOutSize]\n        ## attention = [batchSize, decSeqLen, encSeqLen]\n        output, attention = self.decoder(trg, encoderConved, encoderCombined)        \n        return output, attention","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Lr range finder","metadata":{"papermill":{"duration":0.034217,"end_time":"2021-05-04T00:08:59.741822","exception":false,"start_time":"2021-05-04T00:08:59.707605","status":"completed"},"tags":[]}},{"cell_type":"code","source":"def plot_lr_finder_results(lr_finder): \n    # Create subplot grid\n    fig = make_subplots(rows=1, cols=2)\n    # layout ={'title': 'Lr_finder_result'}\n    \n    # Create a line (trace) for the lr vs loss, gradient of loss\n    trace0 = go.Scatter(x=lr_finder['log_lr'], y=lr_finder['smooth_loss'],name='log_lr vs smooth_loss')\n    trace1 = go.Scatter(x=lr_finder['log_lr'], y=lr_finder['grad_loss'],name='log_lr vs loss gradient')\n\n    # Add subplot trace & assign to each grid\n    fig.add_trace(trace0, row=1, col=1);\n    fig.add_trace(trace1, row=1, col=2);\n    iplot(fig, show_link=False)\n    #fig.write_html(CFG.MODEL_NAME + '_lr_find.html');","metadata":{"papermill":{"duration":0.042515,"end_time":"2021-05-04T00:08:59.818806","exception":false,"start_time":"2021-05-04T00:08:59.776291","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def find_lr(model, data_loader, optimizer, criterion, init_value = 1e-8, final_value=100.0, beta = 0.98, num_batches = 200):\n    assert(num_batches > 0)\n    mult = (final_value / init_value) ** (1/num_batches)\n    lr = init_value\n    optimizer.param_groups[0]['lr'] = lr\n    batch_num = 0\n    avg_loss = 0.0\n    best_loss = 0.0\n    smooth_losses = []\n    raw_losses = []\n    log_lrs = []\n    dataloader_it = iter(data_loader)\n    progress_bar = tqdm(range(num_batches))                \n        \n    for idx in progress_bar:\n        batch_num += 1\n        try:\n            encoderInput, decoderInput, decoderOutput, _, wayPointsMask = next(dataloader_it)\n            ## print(encoderInput.shape, decoderInput.shape)\n        except:\n            dataloader_it = iter(data_loader)\n            encoderInput, decoderInput, decoderOutput, _, wayPointsMask = next(dataloader_it)\n\n        # Move input and label tensors to the default device\n        encoderInput = encoderInput.to(device)\n        decoderInput = decoderInput.to(device)\n        decoderOutput = decoderOutput.to(device)\n\n        # handle exception in criterion\n        try:\n            # Forward pass\n            preds, _ = model(encoderInput, decoderInput)\n            posLoss, floorLoss = criterion(preds, decoderOutput, wayPointsMask)\n            loss = posLoss + floorLoss\n        except:\n            if len(smooth_losses) > 1:\n                grad_loss = np.gradient(smooth_losses)\n            else:\n                grad_loss = 0.0\n            lr_finder_results = {'log_lr':log_lrs, 'raw_loss':raw_losses, \n                                 'smooth_loss':smooth_losses, 'grad_loss': grad_loss}\n            return lr_finder_results \n                    \n        #Compute the smoothed loss\n        avg_loss = beta * avg_loss + (1-beta) *loss.item()\n        smoothed_loss = avg_loss / (1 - beta**batch_num)\n        \n        #Stop if the loss is exploding\n        if batch_num > 1 and smoothed_loss > 50 * best_loss:\n            if len(smooth_losses) > 1:\n                grad_loss = np.gradient(smooth_losses)\n            else:\n                grad_loss = 0.0\n            lr_finder_results = {'log_lr':log_lrs, 'raw_loss':raw_losses, \n                                 'smooth_loss':smooth_losses, 'grad_loss': grad_loss}\n            return lr_finder_results\n        \n        #Record the best loss\n        if smoothed_loss < best_loss or batch_num==1:\n            best_loss = smoothed_loss\n        \n        #Store the values\n        raw_losses.append(loss.item())\n        smooth_losses.append(smoothed_loss)\n        log_lrs.append(math.log10(lr))\n        \n        # Backward pass\n        optimizer.zero_grad()\n        loss.backward()\n        torch.nn.utils.clip_grad_norm_(model.parameters(), CFG.MAX_GRD_NORM)\n        optimizer.step()\n        \n        # print info\n        progress_bar.set_description(f\"loss:{loss.item()},smoothLoss: {smoothed_loss},lr:{lr}\")\n\n        #Update the lr for the next step\n        lr *= mult\n        optimizer.param_groups[0]['lr'] = lr\n    \n    grad_loss = np.gradient(smooth_losses)\n    lr_finder_results = {'log_lr':log_lrs, 'raw_loss':raw_losses, \n                         'smooth_loss':smooth_losses, 'grad_loss': grad_loss}\n    return lr_finder_results","metadata":{"papermill":{"duration":0.051403,"end_time":"2021-05-04T00:08:59.904019","exception":false,"start_time":"2021-05-04T00:08:59.852616","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if CFG.LR_FIND == True:\n    # create dataset, dataloader instance\n    data = getSeq2SeqFeatures(imuFeatures_trainPath)\n    tempTrainDataset = imuFeaturesDataset_train(data)\n    tempTrainDataloader = getDataLoader(tempTrainDataset, datasetType='train')\n\n    ## Device as cpu or tpu\n    device = torch.device(\"cuda:0\") if torch.cuda.is_available() else torch.device('cpu')\n    print(device)\n\n    # create model instance   \n    enc = Encoder(INPUT_DIM, ENC_EMB_DIM, HID_DIM, N_LAYERS, KERNEL_SIZE, \\\n                  ENC_DROPOUT, device, imuInputSequenceLength).double()\n    dec = Decoder(DECODER_INPUT_DIM, DEC_EMB_DIM, HID_DIM, OUTPUT_DIM, \\\n                  N_LAYERS, KERNEL_SIZE, DEC_DROPOUT, FILLER_VALUE, device, wayPointMaxSequenceLength).double()\n    model = Seq2Seq(enc, dec)\n    model.to(device);\n\n    ## loss function\n    criterion = competitionMetric_Seq2Seq\n    \n    # optimizer function, lr schedulers and loss function\n    optimizer = getOptimizer(model)\n    lrFinderResults = find_lr(model, tempTrainDataloader, optimizer, criterion)\n    plot_lr_finder_results(lrFinderResults)","metadata":{"papermill":{"duration":0.043117,"end_time":"2021-05-04T00:08:59.98136","exception":false,"start_time":"2021-05-04T00:08:59.938243","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Train & Validate helper functions","metadata":{"papermill":{"duration":0.034429,"end_time":"2021-05-04T00:09:00.051211","exception":false,"start_time":"2021-05-04T00:09:00.016782","status":"completed"},"tags":[]}},{"cell_type":"code","source":"def validateModel(model, validationDataloader):\n    # placeholders to store output\n    inferenceTs = []\n    val_preds = []\n    val_targets = []\n    val_groups = []\n    val_masks = []\n    ## valPosLoss = 0.0\n    ## valFloorLoss = 0.0\n\n    # set model to Validate mode\n    model.eval()\n    dataLoaderIterator = iter(validationDataloader)\n\n    for idx in range(len(validationDataloader)):\n        try:\n            encoderInput, decoderInput, decoderOutput, pathName, wayPointsMask = next(dataLoaderIterator)\n        except StopIteration:\n            dataLoaderIterator = iter(validationDataloader)\n            encoderInput, decoderInput, decoderOutput, pathName, wayPointsMask = next(dataLoaderIterator)\n\n        ## move data to target device\n        encoderInput  = encoderInput.to(device)\n        decoderInput  = decoderInput.to(device)\n        decoderOutput = decoderOutput.to(device)\n        wayPointsMask = wayPointsMask.to(device)\n        \n        # forward prediction\n        with torch.no_grad():    \n            preds, _ = model(encoderInput, decoderInput)\n\n        ## posLoss, floorLoss = criterion(preds, decoderOutput, wayPointsMask)\n        ## valPosLoss += posLoss.item()\n        ## valFloorLoss += floorLoss.item()\n            \n        # store predictions and targets to compute metrics later\n        inferenceTs.append(decoderInput)\n        val_preds.append(preds)\n        val_targets.append(decoderOutput)\n        val_masks.append(wayPointsMask)\n        val_groups.append(pathName)\n\n    # concatenate to get as 1 2d array and find total loss  \n    val_preds   = torch.cat(val_preds, 0)\n    val_targets = torch.cat(val_targets, 0)\n    val_masks   = torch.cat(val_masks, 0)\n    inferenceTs = torch.cat(inferenceTs, 0)\n    \n    ## calculate loss\n    valPosLoss, valFloorLoss = criterion(val_preds, val_targets, val_masks)\n\n    # np array concatenation\n    val_groups = np.concatenate(val_groups, axis=0)\n        \n    # store results\n    validationResults = {'valPosLoss'  : valPosLoss,  'valFloorLoss': valFloorLoss,\\\n                         'val_groups'  : val_groups, \\\n                         'inferenceTs' : inferenceTs.cpu().data.numpy(), \n                         'val_preds'   : val_preds.cpu().data.numpy(), \n                         'val_targets' : val_targets.cpu().data.numpy(),\n                         }\n    return validationResults","metadata":{"papermill":{"duration":0.047314,"end_time":"2021-05-04T00:09:00.132939","exception":false,"start_time":"2021-05-04T00:09:00.085625","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def trainValidateOneFold(i_fold, model, optimizer, scheduler, dataloader_train, dataloader_valid):\n    trainFoldResults = []\n    bestValScore = np.inf\n    bestEpoch = 0\n\n    for epoch in tqdm(range(CFG.N_EPOCHS)):\n    ## for epoch in range(CFG.N_EPOCHS):\n        #print('Epoch {}/{}'.format(epoch + 1, CFG.N_EPOCHS))\n        model.train()\n        trainPosLoss = 0.0\n        trainFloorLoss = 0.0\n\n        # training iterator\n        tr_iterator = iter(dataloader_train)\n\n        for idx in range(len(dataloader_train)):\n            try:\n                encoderInput, decoderInput, decoderOutput, pathName, wayPointsMask = next(tr_iterator)\n            except StopIteration:\n                tr_iterator = iter(dataloader_train)\n                encoderInput, decoderInput, decoderOutput, pathName, wayPointsMask = next(tr_iterator)\n\n            ## move data to target device\n            encoderInput  = encoderInput.to(device)\n            decoderInput  = decoderInput.to(device)\n            decoderOutput = decoderOutput.to(device)\n            wayPointsMask = wayPointsMask.to(device)\n        \n            # builtin package to handle automatic mixed precision\n            ## with autocast():\n            # Forward pass\n            preds, _ = model(encoderInput, decoderInput)   \n            posLoss, floorLoss = criterion(preds, decoderOutput, wayPointsMask)\n            loss = posLoss ## + floorLoss\n            \n        \n            ## Backward pass\n            ## scaler.scale(loss).backward()        \n            ## scaler.step(optimizer)\n            ## scaler.update()\n            loss.backward()\n            torch.nn.utils.clip_grad_norm_(model.parameters(), CFG.MAX_GRD_NORM)\n                \n            optimizer.step()\n            optimizer.zero_grad() \n\n            # log the necessary losses\n            trainPosLoss   += posLoss.item()\n            trainFloorLoss += floorLoss.item()\n\n            if scheduler is not None: \n                if CFG.SCHEDULER == 'CosineAnnealingWarmRestarts':\n                    scheduler.step(epoch + idx / len(dataloader_train)) \n                # onecyle lr scheduler / CosineAnnealingLR scheduler\n                else:\n                    scheduler.step()\n                    \n        # Validate\n        foldValidationResults = validateModel(model, dataloader_valid)\n         \n        # store results\n        trainFoldResults.append({ 'fold': i_fold, 'epoch': epoch, \n                                  'trainPosLoss': trainPosLoss / len(dataloader_train), \n                                  'trainFloorLoss': trainFloorLoss / len(dataloader_train), \n                                  'valPosLoss'  : foldValidationResults['valPosLoss'] , \n                                  'valFloorLoss': foldValidationResults['valFloorLoss']})\n        \n        valScore = foldValidationResults['valPosLoss'] # + foldVal['valFloorLoss']\n        ## print(f'fold = {i_fold}, epoch = {epoch}, valscore = {valScore}')\n        # save best models        \n        if(valScore < bestValScore):\n            # reset variables\n            bestValScore = valScore\n            bestEpoch = epoch\n\n            # save model weights\n            torch.save({'model': model.state_dict(), 'inferenceTs' : foldValidationResults['inferenceTs'], \n                        'val_preds':foldValidationResults['val_preds'], \n                        'val_targets':foldValidationResults['val_targets'],\n                        'val_groups' : foldValidationResults['val_groups']}, \n                        f\"{modelOutputDir}/{CFG.MODEL_NAME}_fold{i_fold}_best.pth\")\n\n    print(f\"For Fold {i_fold}, Best position validation score of {bestValScore} was got at epoch {bestEpoch}\") \n    return trainFoldResults","metadata":{"papermill":{"duration":0.050399,"end_time":"2021-05-04T00:09:00.217682","exception":false,"start_time":"2021-05-04T00:09:00.167283","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def getFoldBestResultsDf(trainResults):\n    bestResults = []\n    numFolds = trainResults['fold'].nunique()\n    \n    for fold in range(numFolds):\n        foldDf = trainResults[trainResults['fold']== fold]\n        bestResults.append(foldDf.iloc[np.argmin(foldDf['valTotalLoss'].values),:])\n    \n    bestResults =pd.DataFrame(bestResults)\n    valPosLossBest = bestResults['valPosLoss'].values\n    print(f\"Best valPosLoss for all folds = {valPosLossBest}\")\n    print(f\"Mean, std ={valPosLossBest.mean()}, {valPosLossBest.std()}\")\n    return bestResults","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Compute Device as CPU or GPU","metadata":{}},{"cell_type":"code","source":"## Device as cpu or tpu\ndevice = torch.device(\"cuda:0\") if torch.cuda.is_available() else torch.device('cpu')\nprint(device)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Preprocessing classes","metadata":{}},{"cell_type":"code","source":"folds = GroupKFold(n_splits=CFG.N_FOLDS)\ncriterion = competitionMetric_Seq2Seq","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Training & Validation main function","metadata":{}},{"cell_type":"code","source":"%%time\nif CFG.TRAIN == True:\n    ## placeholder to store results\n    buildingTrainResults = []\n    \n    ## load data\n    imuSeq2SeqData_train = getSeq2SeqFeatures(imuFeatures_trainPath)\n    print(imuSeq2SeqData_train.shape)\n    totalColumns = list(imuSeq2SeqData_train.columns)\n    ##print(totalColumns)\n\n    ## temporary variables\n    tempX = np.zeros(len(imuSeq2SeqData_train))\n    tempY = np.zeros(len(imuSeq2SeqData_train)) \n    pathGroups = imuSeq2SeqData_train['path'].values\n    \n    for i_fold, (trainIndex, validIndex) in enumerate(folds.split(X=tempX, y=tempY,groups=pathGroups)):\n        if i_fold in CFG.FOLD_TO_TRAIN:\n            ## splitting into train and validataion sets\n            trainSetData = imuSeq2SeqData_train.loc[trainIndex, totalColumns].reset_index(drop=True)\n            validationSetData = imuSeq2SeqData_train.loc[validIndex, totalColumns].reset_index(drop=True)\n            print(f\"Fold {i_fold} -> trainData shape = {trainSetData.shape}, validationData shape = {validationSetData.shape}\")\n            \n            ## create torch Datasets and Dataloader for each fold's train and validation data            \n            dataset_train = imuFeaturesDataset_train(trainSetData)\n            dataset_valid = imuFeaturesDataset_train(validationSetData)\n            dataloader_train = getDataLoader(dataset_train, datasetType= 'train')\n            dataloader_valid = getDataLoader(dataset_valid, datasetType= 'valid')\n            \n            ## supervised model instance and move to compute device\n            enc = Encoder(INPUT_DIM, ENC_EMB_DIM, HID_DIM, N_LAYERS, KERNEL_SIZE, \\\n                          ENC_DROPOUT, device, imuInputSequenceLength).double()\n            dec = Decoder(DECODER_INPUT_DIM, DEC_EMB_DIM, HID_DIM, OUTPUT_DIM, \\\n                          N_LAYERS, KERNEL_SIZE, DEC_DROPOUT, FILLER_VALUE, device, wayPointMaxSequenceLength).double()\n            model = Seq2Seq(enc, dec)\n            model.to(device);    \n            model.apply(initModelWeights)\n\n            \n            ## print(f\"there are {find_no_of_trainable_params(model)} params in model\")\n            \n            ## optimizer function, lr schedulers and loss function\n            optimizer = getOptimizer(model)\n            scheduler = getScheduler(optimizer, dataloader_train)\n            # print(f\"optimizer={optimizer}, scheduler={scheduler}, loss_fn={criterion}\")\n\n            ## train and validate single fold\n            foldResults = trainValidateOneFold(i_fold, model, optimizer, scheduler, dataloader_train, dataloader_valid)\n            buildingTrainResults = buildingTrainResults + foldResults\n    \n    \n    buildingTrainResults = pd.DataFrame(buildingTrainResults)\n    buildingTrainResults['valTotalLoss'] = buildingTrainResults['valPosLoss'] + buildingTrainResults['valFloorLoss']\n    buildingTrainResults['trainTotalLoss'] = buildingTrainResults['trainPosLoss'] + buildingTrainResults['trainFloorLoss']\n    ## bestResults = getFoldBestResultsDf(buildingTrainResults)\n\n    ## generate OOF prediction for building-model combination\n    ## buildingOOF = generateOOF(modelOutputDir, buildingName, CFG.MODEL_NAME)\n    ## buildingOOF.to_pickle(f\"{modelOutputDir}/{buildingName}_{CFG.MODEL_NAME}_OOF.pickle\")    \n    \n    ## prediction for test data too\n    ## generateWiFiSubmission(modelToFit, modelOutputDir, buildingName, CFG.MODEL_NAME)\n\n    ## save training results to output directory\n    ## bestResults.to_pickle(f\"{modelOutputDir}/{CFG.MODEL_NAME}_bestResults.pickle\")\n    ## buildingTrainResults.to_pickle(f\"{modelOutputDir}/{CFG.MODEL_NAME}_trainResults.pickle\")\n    \n    ## plot model training results\n    plotTrainingResults(buildingTrainResults, 'cs2s')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"cp = torch.load('./ConvSeq_v1_fold3_best.pth')\nval_preds = cp['val_preds']\nval_targets = cp['val_targets']\nprint(val_preds.shape, val_targets.shape)\npredsDimension = val_preds.shape[-1]","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}