![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F651378%2F699965e32e4940ab317dbe0f7cbe95d1%2Fpiplinev1.PNG?generation=1687193496974817&alt=media\")
DataSet
\nNetwork Architecture
\nAll models has following architecture
\n`class Wave_Block(nn.Module):
def :\n super(Wave_Block, self).\n self.num_rates = dilation_rates\n self.convs = nn.\n self.filter_convs = nn.\n self.gate_convs = nn.\n\n self.convs.append(nn.)\n dilation_rates = \n dilation_rate dilation_rates:\n self.filter_convs.append(\n nn.)/), dilation=dilation_rate))\n self.gate_convs.append(\n nn.)/), dilation=dilation_rate))\n self.convs.append(nn.)\n\ndef forward(self, x):\n x = self.convs(x)\n res = x\n i range(self.num_rates):\n x = torch.tanh(self.filter_convs(x))torch.sigmoid(self.gate_convs(x))\n x = self.convs(x)\n res = res + x\n return res\n\nclass Classifier(nn.Module):
\n def init(self, inch=3, kernel_size=3):
\n super().init()
\n self.LSTM = nn.GRU(input_size=128, hidden_size=128, num_layers=4,
\n batch_first=True, bidirectional=True)
#self.wave_block1 = \n self.wave_block2 = \n self.wave_block3 = \n self.wave_block4 = \n self.fc1 = nn.\n\ndef forward(self, x):\n x = x.permute(, , )\n #x = self.wave\n x = self.wave\n x = self.wave\n\n x = self.wave\n x = x.permute(, , )\n x, h = self.\n x = self.fc1(x)\n\n\n return x`\nTraining Notebook is found at following link
\nTraining Notebook is found at following link
\nThis notebook is based on pre-training on unlabeled data. In pretraining target is to predict next value in the time series. For data creation each unlabeled series is divided into segment of length 100000.
\nData Creation Notebook is available at : https://www.kaggle.com/code/adityakumarsinha/unlabeled-data-creation/notebook.
\nNote: This notebook will fail in Kaggle kernel as it requires more disk space as default available for kaggle notebooks. Please run on different PC/ Server/ VM
\nPreTraining Notebook is available at : https://www.kaggle.com/code/adityakumarsinha/pretrain-wavenet-4096-v1/notebook
\nThe training will be executed for single fold and best weight will be used as initial weights (without LSTM layer)
\nfor WaveNet-GRU-v3
\nNote: Singe epoch takes around 1-1:30 hours on RTX 3090 so the kernel will timeout. Please run on different PC/ Server/ VM
\nWaveNet-GRU-v3: **Training notebook is available at **: https://www.kaggle.com/code/adityakumarsinha/wavenet-2000-from-pretrain/notebook.
\nUse best weight for each fold in final inference.
\nCV score for this notebook is low as compared to WaveNet-GRU-v1 and WaveNet-GRU-v2 but it improves the final ensemble ( During competition time it improved CV score but due to some bug in inferencing code the final private leader-board score as come down. I will explain this in inferencing section.
As during last week my GPU quota has been exhausted so i need to use CPU for inference. Simple CPU based pytorch inference was exceeding the time limit of 9 hours. So I need to convert pytorch models into ONNX model. Please refer following notebook for model conversion: https://www.kaggle.com/code/adityakumarsinha/openvino-model-converter-all-models-v3/notebook
\nThe converted models are used in final inference. One of the final inference notebook is available at:
\nhttps://www.kaggle.com/adityakumarsinha/gait-openvion-bunch-v2.
After competition I found that, in ensemble, WaveNet-GRU-v3 (model that uses pretrained weight) is overfitting on public leaderboard and in private leaderboard its inclusion had decreased the score. While in local CV ensemble inclusion of this model was increasing the CV score.
\nSo i debugged more and I found that with GPU based inference WaveNet-GRU-v3 is indeed increasing the score. in face simple ensemble of WaveNet-GRU-v1 and WaveNet-GRU-v3 has private leaderboard score of 0.437. More than third position score.
\nThe best GPU based inference notebook is available at
\nhttps://www.kaggle.com/code/adityakumarsinha/wavenet-subm-focal-v2/notebook
Regards
\nAditya
DataSet
\nNetwork Architecture
\nAll models has following architecture
\n`class Wave_Block(nn.Module):
def :\n super(Wave_Block, self).\n self.num_rates = dilation_rates\n self.convs = nn.\n self.filter_convs = nn.\n self.gate_convs = nn.\n\n self.convs.append(nn.)\n dilation_rates = \n dilation_rate dilation_rates:\n self.filter_convs.append(\n nn.)/), dilation=dilation_rate))\n self.gate_convs.append(\n nn.)/), dilation=dilation_rate))\n self.convs.append(nn.)\n\ndef forward(self, x):\n x = self.convs(x)\n res = x\n i range(self.num_rates):\n x = torch.tanh(self.filter_convs(x))torch.sigmoid(self.gate_convs(x))\n x = self.convs(x)\n res = res + x\n return res\n\nclass Classifier(nn.Module):
\n def init(self, inch=3, kernel_size=3):
\n super().init()
\n self.LSTM = nn.GRU(input_size=128, hidden_size=128, num_layers=4,
\n batch_first=True, bidirectional=True)
#self.wave_block1 = \n self.wave_block2 = \n self.wave_block3 = \n self.wave_block4 = \n self.fc1 = nn.\n\ndef forward(self, x):\n x = x.permute(, , )\n #x = self.wave\n x = self.wave\n x = self.wave\n\n x = self.wave\n x = x.permute(, , )\n x, h = self.\n x = self.fc1(x)\n\n\n return x`\nTraining Notebook is found at following link
\nTraining Notebook is found at following link
\nThis notebook is based on pre-training on unlabeled data. In pretraining target is to predict next value in the time series. For data creation each unlabeled series is divided into segment of length 100000.
\nData Creation Notebook is available at : https://www.kaggle.com/code/adityakumarsinha/unlabeled-data-creation/notebook.
\nNote: This notebook will fail in Kaggle kernel as it requires more disk space as default available for kaggle notebooks. Please run on different PC/ Server/ VM
\nPreTraining Notebook is available at : https://www.kaggle.com/code/adityakumarsinha/pretrain-wavenet-4096-v1/notebook
\nThe training will be executed for single fold and best weight will be used as initial weights (without LSTM layer)
\nfor WaveNet-GRU-v3
\nNote: Singe epoch takes around 1-1:30 hours on RTX 3090 so the kernel will timeout. Please run on different PC/ Server/ VM
\nWaveNet-GRU-v3: **Training notebook is available at **: https://www.kaggle.com/code/adityakumarsinha/wavenet-2000-from-pretrain/notebook.
\nUse best weight for each fold in final inference.
\nCV score for this notebook is low as compared to WaveNet-GRU-v1 and WaveNet-GRU-v2 but it improves the final ensemble ( During competition time it improved CV score but due to some bug in inferencing code the final private leader-board score as come down. I will explain this in inferencing section.
As during last week my GPU quota has been exhausted so i need to use CPU for inference. Simple CPU based pytorch inference was exceeding the time limit of 9 hours. So I need to convert pytorch models into ONNX model. Please refer following notebook for model conversion: https://www.kaggle.com/code/adityakumarsinha/openvino-model-converter-all-models-v3/notebook
\nThe converted models are used in final inference. One of the final inference notebook is available at:
\nhttps://www.kaggle.com/adityakumarsinha/gait-openvion-bunch-v2.
After competition I found that, in ensemble, WaveNet-GRU-v3 (model that uses pretrained weight) is overfitting on public leaderboard and in private leaderboard its inclusion had decreased the score. While in local CV ensemble inclusion of this model was increasing the CV score.
\nSo i debugged more and I found that with GPU based inference WaveNet-GRU-v3 is indeed increasing the score. in face simple ensemble of WaveNet-GRU-v1 and WaveNet-GRU-v3 has private leaderboard score of 0.437. More than third position score.
\nThe best GPU based inference notebook is available at
\nhttps://www.kaggle.com/code/adityakumarsinha/wavenet-subm-focal-v2/notebook
Regards
\nAditya