![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1969534%2F18364882e67c41284297f4eaed4ddb47%2Fmodel.svg?generation=1686319620125192&alt=media\")
Thanks to Kaggle and the competition hosts for this competition, and congratulations to the other teams! This was for me the first Kaggle competition in which I invested myself, and it was an awesome experience, I truly learnt a lot.
\nThe model that performed the best for me is a variant of a multi-layer GRU model in which some residual connections and fully connected layers have been added between the GRU layers:
\n
Here are the PyTorch classes corresponding to this model:
\n (nn.Module):\n ():\n (ResidualBiGRU, self).__init__()\n\n self.hidden_size = hidden_size\n self.n_layers = n_layers\n\n self.gru = nn.GRU(\n hidden_size,\n hidden_size,\n n_layers,\n batch_first=,\n bidirectional=bidir,\n )\n dir_factor = bidir \n self.fc1 = nn.Linear(\n hidden_size * dir_factor, hidden_size * dir_factor * \n )\n self.ln1 = nn.LayerNorm(hidden_size * dir_factor * )\n self.fc2 = nn.Linear(hidden_size * dir_factor * , hidden_size)\n self.ln2 = nn.LayerNorm(hidden_size)\n\n ():\n res, new_h = self.gru(x, h)\n \n\n res = self.fc1(res)\n res = self.ln1(res)\n res = nn.functional.relu(res)\n\n res = self.fc2(res)\n res = self.ln2(res)\n res = nn.functional.relu(res)\n\n \n res = res + x\n\n res, new_h\n\n (nn.Module):\n ():\n (MultiResidualBiGRU, self).__init__()\n\n self.input_size = input_size\n self.hidden_size = hidden_size\n self.out_size = out_size\n self.n_layers = n_layers\n\n self.fc_in = nn.Linear(input_size, hidden_size)\n self.ln = nn.LayerNorm(hidden_size)\n self.res_bigrus = nn.ModuleList(\n [\n ResidualBiGRU(hidden_size, n_layers=, bidir=bidir)\n _ (n_layers)\n ]\n )\n self.fc_out = nn.Linear(hidden_size, out_size)\n\n ():\n \n h :\n \n h = [ _ (self.n_layers)]\n\n x = self.fc_in(x)\n x = self.ln(x)\n x = nn.functional.relu(x)\n\n new_h = []\n i, res_bigru (self.res_bigrus):\n x, new_hi = res_bigru(x, h[i])\n new_h.append(new_hi)\n\n x = self.fc_out(x)\n\n x, new_h \nNote that the code can be simplified: for my best model which performed a private lb score of 0.417, the \"h\" was actually always initialized with None.
\nIn terms of data, the choice I made for my model is very simplistic: consider only the accelerometer data (AccV, AccML, AccAP), merge the data from tdcsfog and defog together and train a single model on it. The main steps of my preprocessing pipeline are the following ones:
\nOutside of the unlabeled time steps coming from defog, I did not use any other unlabeled data.
\nFor the prototype of another model i did not have the time to finish, I also began to consider some of the characteristics associated to the person who was producing the sequence, in particular I used \"Visit\", \"Age\", \"Sex\", \"YearsSinceDx\", \"UPDRSIII_On\" and \"NFOGQ\". This prototype was roughly following the same architecture as my best model ; the main idea was to initialize the GRU's hidden states with these characteristics, after using some fully connected layers to project them in the dimension of the hidden states. This prototype was also using 1D convolutions to extract features from the accelerometer data before passing them to the GRU layers, and I also considered adding dropout. I think that with more time for me to tune it, it would have beaten my current best model. The first version achieved a private lb score of 0.398.
\nMy best model - the one which performed 0.417 on the private leaderboard - has been trained without any form of cross-validation, only with a train/validation split of 80% / 20%. To be honest, this model appeared as a prototype in my early experimentation process, and I considered stratified cross-validation only after.
\nIn this solution, I fed each whole downsampled (50Hz) sequence to my model, one after the other ie with a batch size of 1. Note that I would have been unable to feed some of the sequences to my model without downsampling them. I tried multiple different approaches with this architecture, but was unable to produce a better score when increasing the batch size. I tried multiple window sizes for my sequences ; however as I am pretty new in time series and as I also arrived pretty late in the competition, I did not implement any form of overlap and only thought about it too late. This could have probably been a key. Also when increasing the batch size, it seemed apparent that batch normalization was better than layer normalization.
\nFor the loss, I used a simple cross entropy. As the classes are pretty imbalanced (in particular with my 4th artificial one), I also considered using a weighted cross-entropy, using the inverse frequency of each class as a weight. I also considered trying a focal loss ; but these initial tests seemed unable to perform better than the cross entropy in my case. Despite these negative experiments, I still think that dealing with the imbalance nature of the problem in a better way than I did is important.
\nIn terms of optimizer, I used Ranger. I also tried Adam and AdamW and honestly i don't think this choice mattered too much. With Ranger I used a learning rate of 1e-3 with 20 epochs, with a cosine annealing schedule starting at 15.
\nNote that I also used mixed precision training and gradient clipping.
\nThe best parameters I found for the architecture of my model are:
\nLater on, I also tried a stratified k-fold cross-validation in which I stacked the predictions of my k models via a simple average. The architecture and the training details for each fold were the same as for my 0.417 lb model, and this stacking process led to my 2nd best model, performing a score of 0.415 on the private leaderboard (with k=5). I also tried retraining my model on the whole dataset, but this approach did not improve my lb score.
\nIn no particular order, here are a few other architectures that I also tried but did not improve my score:
\nEdit:
\nSubmission Notebook:
\nhttps://www.kaggle.com/zinxira/parkinson-fog-pred-4th-place-submission-notebook
Pretrained models \"dataset\":
\nhttps://www.kaggle.com/datasets/zinxira/models
Full open-source code:
\nhttps://github.com/Zinxira/tlvmc-parkinsons-fog-prediction-4th-place-solution
Thanks to Kaggle and the competition hosts for this competition, and congratulations to the other teams! This was for me the first Kaggle competition in which I invested myself, and it was an awesome experience, I truly learnt a lot.
\nThe model that performed the best for me is a variant of a multi-layer GRU model in which some residual connections and fully connected layers have been added between the GRU layers:
\n
Here are the PyTorch classes corresponding to this model:
\n (nn.Module):\n ():\n (ResidualBiGRU, self).__init__()\n\n self.hidden_size = hidden_size\n self.n_layers = n_layers\n\n self.gru = nn.GRU(\n hidden_size,\n hidden_size,\n n_layers,\n batch_first=,\n bidirectional=bidir,\n )\n dir_factor = bidir \n self.fc1 = nn.Linear(\n hidden_size * dir_factor, hidden_size * dir_factor * \n )\n self.ln1 = nn.LayerNorm(hidden_size * dir_factor * )\n self.fc2 = nn.Linear(hidden_size * dir_factor * , hidden_size)\n self.ln2 = nn.LayerNorm(hidden_size)\n\n ():\n res, new_h = self.gru(x, h)\n \n\n res = self.fc1(res)\n res = self.ln1(res)\n res = nn.functional.relu(res)\n\n res = self.fc2(res)\n res = self.ln2(res)\n res = nn.functional.relu(res)\n\n \n res = res + x\n\n res, new_h\n\n (nn.Module):\n ():\n (MultiResidualBiGRU, self).__init__()\n\n self.input_size = input_size\n self.hidden_size = hidden_size\n self.out_size = out_size\n self.n_layers = n_layers\n\n self.fc_in = nn.Linear(input_size, hidden_size)\n self.ln = nn.LayerNorm(hidden_size)\n self.res_bigrus = nn.ModuleList(\n [\n ResidualBiGRU(hidden_size, n_layers=, bidir=bidir)\n _ (n_layers)\n ]\n )\n self.fc_out = nn.Linear(hidden_size, out_size)\n\n ():\n \n h :\n \n h = [ _ (self.n_layers)]\n\n x = self.fc_in(x)\n x = self.ln(x)\n x = nn.functional.relu(x)\n\n new_h = []\n i, res_bigru (self.res_bigrus):\n x, new_hi = res_bigru(x, h[i])\n new_h.append(new_hi)\n\n x = self.fc_out(x)\n\n x, new_h \nNote that the code can be simplified: for my best model which performed a private lb score of 0.417, the \"h\" was actually always initialized with None.
\nIn terms of data, the choice I made for my model is very simplistic: consider only the accelerometer data (AccV, AccML, AccAP), merge the data from tdcsfog and defog together and train a single model on it. The main steps of my preprocessing pipeline are the following ones:
\nOutside of the unlabeled time steps coming from defog, I did not use any other unlabeled data.
\nFor the prototype of another model i did not have the time to finish, I also began to consider some of the characteristics associated to the person who was producing the sequence, in particular I used \"Visit\", \"Age\", \"Sex\", \"YearsSinceDx\", \"UPDRSIII_On\" and \"NFOGQ\". This prototype was roughly following the same architecture as my best model ; the main idea was to initialize the GRU's hidden states with these characteristics, after using some fully connected layers to project them in the dimension of the hidden states. This prototype was also using 1D convolutions to extract features from the accelerometer data before passing them to the GRU layers, and I also considered adding dropout. I think that with more time for me to tune it, it would have beaten my current best model. The first version achieved a private lb score of 0.398.
\nMy best model - the one which performed 0.417 on the private leaderboard - has been trained without any form of cross-validation, only with a train/validation split of 80% / 20%. To be honest, this model appeared as a prototype in my early experimentation process, and I considered stratified cross-validation only after.
\nIn this solution, I fed each whole downsampled (50Hz) sequence to my model, one after the other ie with a batch size of 1. Note that I would have been unable to feed some of the sequences to my model without downsampling them. I tried multiple different approaches with this architecture, but was unable to produce a better score when increasing the batch size. I tried multiple window sizes for my sequences ; however as I am pretty new in time series and as I also arrived pretty late in the competition, I did not implement any form of overlap and only thought about it too late. This could have probably been a key. Also when increasing the batch size, it seemed apparent that batch normalization was better than layer normalization.
\nFor the loss, I used a simple cross entropy. As the classes are pretty imbalanced (in particular with my 4th artificial one), I also considered using a weighted cross-entropy, using the inverse frequency of each class as a weight. I also considered trying a focal loss ; but these initial tests seemed unable to perform better than the cross entropy in my case. Despite these negative experiments, I still think that dealing with the imbalance nature of the problem in a better way than I did is important.
\nIn terms of optimizer, I used Ranger. I also tried Adam and AdamW and honestly i don't think this choice mattered too much. With Ranger I used a learning rate of 1e-3 with 20 epochs, with a cosine annealing schedule starting at 15.
\nNote that I also used mixed precision training and gradient clipping.
\nThe best parameters I found for the architecture of my model are:
\nLater on, I also tried a stratified k-fold cross-validation in which I stacked the predictions of my k models via a simple average. The architecture and the training details for each fold were the same as for my 0.417 lb model, and this stacking process led to my 2nd best model, performing a score of 0.415 on the private leaderboard (with k=5). I also tried retraining my model on the whole dataset, but this approach did not improve my lb score.
\nIn no particular order, here are a few other architectures that I also tried but did not improve my score:
\nEdit:
\nSubmission Notebook:
\nhttps://www.kaggle.com/zinxira/parkinson-fog-pred-4th-place-submission-notebook
Pretrained models \"dataset\":
\nhttps://www.kaggle.com/datasets/zinxira/models
Full open-source code:
\nhttps://github.com/Zinxira/tlvmc-parkinsons-fog-prediction-4th-place-solution
Such a clean solution and explanation! Thank you for sharing.
", "rawMarkdown": "Such a clean solution and explanation! Thank you for sharing.", "votes": null }, { "id": "2321381", "postDate": "06/28/2023 14:11:24", "content": "Hi and thank you, in particular for the well written code!!
\nIt seems like a residual network but with Recurrent instead of Convolutional layers
\nDid you have a reference for such architecture?
Hi, thank you :) I do not have a reference for this architecture in particular, but I inspired myself from a part of the #1 solution from the \"Google Brain - Ventilator Pressure Prediction\" competition. In their code, they are using a ResidualLSTM module where they also use fully connected layers between the LSTMs. Their architecture is complex, and the ResidualLSTM only represents a small part of it, but that was one of my main starting points in my brainstorming process to build my model.
\nAlso note that I did not have the time to explore all the possible variations one can think about when looking at my architecture: some design choices I made can be seen as arbitrary, and the architecture could probably be refined by a large margin
can you explain how you did best_model and simple_model
\nhow you created data card this is :
\ntlvmc-fog-pred-4th-place-pretrained-models
Such a clean solution and explanation! Thank you for sharing.
", "votes": null, "replies": [] }, { "id": 2321381, "author_name": "albertoannoni", "author_url": "", "post_date": "06/28/2023 14:11:24", "content": "Hi and thank you, in particular for the well written code!!
\nIt seems like a residual network but with Recurrent instead of Convolutional layers
\nDid you have a reference for such architecture?
Hi, thank you :) I do not have a reference for this architecture in particular, but I inspired myself from a part of the #1 solution from the \"Google Brain - Ventilator Pressure Prediction\" competition. In their code, they are using a ResidualLSTM module where they also use fully connected layers between the LSTMs. Their architecture is complex, and the ResidualLSTM only represents a small part of it, but that was one of my main starting points in my brainstorming process to build my model.
\nAlso note that I did not have the time to explore all the possible variations one can think about when looking at my architecture: some design choices I made can be seen as arbitrary, and the architecture could probably be refined by a large margin
can you explain how you did best_model and simple_model
\nhow you created data card this is :
\ntlvmc-fog-pred-4th-place-pretrained-models