![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2208184%2Ff692423c63172494ea4df97b01d65d33%2Fdata.png?generation=1702113769960411&alt=media\")
Unbelievable to write this!
\nAs it is the usage, we first thank the host and Kaggle. These are special thanks because you and us have had a special link in this competition as we gave you more work by reporting data leaks. No doubt you tried to do your best. You are right to animate this community and to trust in it. You are part of it. Please take care of this community that is able to build so much together by sharing. As all of us you have made mistakes and we hope you will learn from them.
\nWe also want to thank all of you, Kagglers. We love and are grateful to be part of our group/community. Thanks for sharing and for the collective learning experience.
\nOur solution is essentially a blend of a XGBoost and a NN models. Both heavily rely on duration that appeared to be a powerful leverage. Time was aggregated in different ways and combined with counts for the GBDT while it is transformed via a custom TimeEmbedding block based on 1D convolutions that produce a representation combined with user event representations for the NN.
\nRobustness and efficiency founded our work. XGBoost models were validated on 10 bags of 5 folds and features incorporated only if the mean of the CV of these 10 bags was greater than the level of noise we quantified while we opted for a majority/consensus strategy to build the NN, i.e. validate choices only if 4 of 5 folds were improved. The 3rd place of the efficiency LB was achieved with a lightweight NN accelerated via TF Lite.
After publishing this write-up we decided to open our code: https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/420332.
\nIt is composed by several parts: how to train the XGBoost models, how to pretrain and train the NN models and the inference notebook used to win this competition.
Looking at the 1st data released showed that there aren't a lot of sessions so not a lot of sequences. Moreover these are long sequences. This is not ideal for a deep learning approach.
\nExploring the Field Day Lab research instructed that the Jo Wilder application was built to help learning to read and that way more than 11,500 learners had played this game.
\nThese 2 ideas led to search for a bigger dataset. In 1 Google search and 3 clicks we came up to the open data portal (https://fielddaylab.wisc.edu/opengamedata/) which contains a lot of sessions. 1 hour and 3 bash commands latter we knew that the train set was in part in the open data. So we took a week to build a pipeline that extracts 98 % of the sessions of the train set perfectly and with minor errors for the last 2 %. Our data are even better than the comp data because we knew before the host confirmation that for the sessions with 2 games the target was skewed (0 if wrong in 1 of the 2 games when we aim at predicting the responses for the 1st game). It seems that fixing these targets can bring a significant boost up to +0.002.
We took 1 more week to build a GBDT/XGBoost baseline that would have scored top 10 given the CV score, with the use of the supplemental data (~20,000 sessions) that gave +0.003/0.004 at that time. As we simulated the API locally (see after), we used some training sessions to infer and noticed that it scored 0.718. We were hoping that the LB sessions were not part of the open data portal but our 1st submission, LB 0.708, immediately showed to us that we had rebuilt about a half of the data and especially the targets in the public LB, because 0.708 = (0.698 + 0.718) / 2. The host and Kaggle have been immediately informed. You know what happened next (https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/415820).
\nAfter the release of the LB data we measured that we perfectly rebuilt ~7000 sessions over the ~11,500 of the LB data.
We spent the first month exploring the data until we understood/knew it pretty well. For example we even reconstituted sessions for what might be schools (several games on 1 IP session), extracted every single session with at least 1 answer, …
\nAfter the update we made a first submission that scored 0.72. This was shocking because this meant that some leaked data were remaining. A few days later we noticed that the open data was not totally similar with the state we found it 1 month before. A file was missing. So we returned to the host and Kaggle to give them more work.
\nThis process/work led us to perfectly understand the data model (that changed since the 1st release of the game). This also allowed us to deeply understand the data itself.
\nNote that we only used the sessions for which we had responses to all questions of the 2 1st level groups. 1) This is more consistant with the sessions we want to predict (game from the beginning to the end) and 2) this approach preserves performance (vs all data) while reducing training time.
\nOur dataset is constituted by 37323 complete sessions (23562 comp + logs) in a total of 66376 sessions.
\nThe supplemental data (that we fully added 1 month ago) gave us consistently CV +0.002.
\nOur solution is mainly an ensemble of GBDT + NN models.
\nWe think that the main reason of the robustness of our solution is that we only relied on CV for decision making. No choice had been made on LB.
\nProbing showed to us that the private test consists in the 1st 1450/1500 sessions served by the API. This is a small set. In our experiments 5,000 sessions is the minimum to guarantee a stable CV/LB alignment. A set less than 2,000 is very noisy so robustness was the way to go.
\nWe only added features that improved the CV for sure. This is not easy to delete features that you believe in but this is needed as science is not a matter of belief. There are several ways to do so: for example monitor all folds in a CV (and accept only on majority or consensus), monitor several bags (composition of CV to not overfit validation), …
\nFor the GBDT approach, we mainly validated on the mean of 10 bags (we defined a bag as a composition of the folds). As we estimated the noise to be ~0.0003, only improvements greater than the noise have been considered. For the NN as we needed to iterate quicker we only used a single bag and only incorporated > 0.0003 overall improvements with at least 3 or 4 (over 5) folds improved.
\nWe experimented a lot on finding a threshold by question but found that this approach is less robust than a single threshold. We mainly used 0.625 as global threshold despite our highest LB scores that were obtained with a threshold per question.
\nWe prototyped a baseline with XGBoost because of the structured/tabular nature of the data. The feature engineering process is interesting to understand what is predictive and to understand the causation, i.e. how the features or decision criteria that enable to predict correctly.
\nGenerally speaking we followed 3 ways to build features: business knowledge, our intuition playing the game and a meticulous exploration of the data.
\nBusiness knowledge refers to using expert knowledge. Reading the papers of the researchers that built this game allow to understand the game beyond usage. For example, Jo Wilder has been built to improve the players reading skills. So this means that the text duration should be important. These are like killer features.
We exclusively made use of Polars because of the CPU constraints and to simply learn it.
\nOur features (663, 1993, 3734 for each level_group) are mainly durations and counts for different aggregations: how much time in a level, in a room, reading a text, interacting in some way (event type), how many events in a level_group, how many events of each type, how many events of each type in a room or a level, …
\nWe also built a few notebook dedicated features: how many type of events on the notebook in a level, …
\nDespite our efforts we weren't able to extract useful information from the coordinates, the only few features of this type had been mean and std for some events in the activities (journal interactions for example).
We considered that injecting targets predicted in the previous level groups was a compression of the signal, meaning a loss of information, so we used, for each session, all interactions from the beginning of the game/session. This led to a +0.002 at the time of this choice.
\nAfter the API needed to order the data, we noticed that models trained both on original order and on index order but validated on index order (inference order) improved our scores. This leads to more variety that was needed to improve stability and robustness. The same goes for the composition of the validation sets: usage of several bags (composition of validation sets) based on the comp data but also on the extracted data improved our scores. We detected late that increasing the number of folds from 5 to 10 could also be leveraged.
\nThe code for GBDT allows to switch from XGBoost to LightGBM and CatBoost with a simple variable parameter but despite the good scores (~0.001 less than XGBoost), this did not bring to ensemble so we sticked to only XGBoost.
\nWe experimented a lot around feature selection but were unable to build a stable strategy. So instead of a top-down approach consisting in deleting useless features, we adopted a bottom-up approach choosing carefully each group of features.
\nOur XGBoost models score CV ~0.7025 +/-0.0003 and blending 5 of them (the only XGBoost we still have with correct score) scores LB 0.704.
\nAfter achieving a good score with gradient boosting and having understood well the data we focused on deep learning.
\nThe first attempt was with Transformers. The 1st results were disappointed: CV 0.685 with 2 hours / fold (as far as we can remember). Transformers are very computationally intensive. Resources: https://arxiv.org/pdf/1912.09363.pdf, https://arxiv.org/pdf/2001.08317.pdf, https://arxiv.org/pdf/1711.03905.pdf, https://arxiv.org/pdf/1907.00235.pdf, …
\nWe then gave a try to Conv1D. In one day we had a very simple model that scored as Transformers but 10x faster allowing to iterate quicker. So we pushed this approach and could seamlessly scaled it beyond our expectations.
\nDifficult to share the tens or hundreds of experimentations needed to achieve the final solution which is both based on a simple architecture and a slightly complex training pipeline.
\nWe browsed the literature based on the question: how to model time in deep learning?
\nThis research made us come to the idea of time-aware events (i.e. https://proceedings.mlr.press/v126/zhang20c/zhang20c.pdf) and back to WaveNet (https://arxiv.org/pdf/1609.03499.pdf) because it uses Conv1D to model long sequences with considerations on causation.
\nOther papers also inspired us: https://arxiv.org/pdf/1703.04691.pdf build on top of WaveNet paper for time series, https://idus.us.es/bitstream/handle/11441/114701/Short-Term%20Load%20Forecasting%20Using%20Encoder-Decoder%20WaveNet.pdf?sequence=1&isAllowed=y also build on top of WaveNet.
\nWe also have to mention the excellent work that @abaojiang shared (https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/398565 and https://www.kaggle.com/code/abaojiang/lb-0-694-tconv-with-4-features-training-part). It inspired our research and maybe successfully biased it.
Let's focus on the model of our efficiency submission that is also one of our final ensemble and which performance is nearly the same as models with a few more features.
\n
5 features as inputs: duration, text_fqid, room_fqid, fqid, event_name + name (this is the event type from the original data model as far as we remember). Each of these information is encoded/embedded into a vector representation (d_model = 24) to be the merged. The 4 categorical features feed a classical Embedding layer and the duration a TimeEmbedding which is a custom block.
\nDeveloping the GBDT solution showed that the duration was crucial, so we put a crucial amount of time trying to model it greatly. The TimeEmbedding layer is a composition of 4x ConvBlock which is inspired by the Transformer main block: Conv1D -> skip connection -> layer norm -> dropout.
\n (tf.keras.layers.):\n ():\n (, ).__init__()\n .conv_blocks = [(d_model, dropout_rate=dropout_rate) _ range(n_blocks)]\n\n ():\n x = tf.expand_dims(inputs, axis=-)\n conv_block .\n x = conv_block(x)\n x\n (tf.keras.layers.):\n ():\n (, ).__init__()\n .conv1d = tf.keras.layers.D(d_model, kernel_size=, padding=, activation=)\n .layer_norm = tf.keras.layers.()\n .dropout = tf.keras.layers.(rate=dropout_rate)\n\n ():\n x = .conv1d(inputs)\n x = x + inputs\n x = .layer_norm(x)\n outputs = .dropout(x)\n outputs\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2208184%2F9407390ffd7f226b7be0f57c37051f77%2Ftime_embedding.png?generation=1702113795144186&alt=media\")
As said, the goal of building these representations was to model time-aware events. We considered the categorical features as events because they represent the user interactions with business entities of the game. We then tried to incorporate duration to make them time-awared. Our main intuition showed to be the best. It is a simple solution based on operation priority to represent that the duration should be associated to each event before associated them together: duration * event_1 + duration * event_2 + … which had been factorized to duration * (event_1 + event_2 + …).
\n (tf.keras.Model):\n ():\n (ConvNet, ).__init__(name=name)\n .input_dims = input_dims\n .n_outputs = n_outputs\n .d_model = d_model\n .n_blocks = n_blocks\n .event_embedding = tf.keras.layers.Embedding(input_dims[], d_model, mask_zero=)\n .room_embedding = tf.keras.layers.Embedding(input_dims[], d_model, mask_zero=)\n .text_embedding = tf.keras.layers.Embedding(input_dims[], d_model, mask_zero=)\n .fqid_embedding = tf.keras.layers.Embedding(input_dims[], d_model, mask_zero=)\n .duration_embedding = TimeEmbedding(n_blocks=n_blocks, d_model=d_model, dropout_rate=)\n .gap = tf.keras.layers.GlobalAveragePooling1D()\n\n ():\n event = .event_embedding(inputs[])\n room = .room_embedding(inputs[])\n text = .text_embedding(inputs[])\n fqid = .fqid_embedding(inputs[])\n duration = .duration_embedding(inputs[])\n x = duration * (event + room + text + fqid)\n outputs = .gap(x)\n outputs\n\n ():\n config = ().get_config().copy()\n config.update({\n : .input_dims,\n : .n_outputs,\n : .d_model,\n : .n_blocks,\n : ._name,\n })\n config\n\n\n ():\n cls(**config)\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2208184%2Fca563daa4f9ef7bf8d8b20e0c85c7951%2Ftime_aware_events_1.png?generation=1702113822015438&alt=media\")
\nThe 2 representations are equivalent: either you can think time-aware events as a combination of time and sub-events or as a combination of sub-events and time.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2208184%2F1f65d399dfd93f128ca6c887d0f0be89%2Ftime_aware_events_2.png?generation=1702113853644045&alt=media\")
The training pipeline is not totally straight forward.
\n@dongyk published great schematics that can be useful to illustrate what is explained bellow: https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/420217#2332166.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2208184%2F970baa66fc37b301bb8960cc92cf8ab4%2Fpre_training.png?generation=1702113879947155&alt=media\")
The best approach for us consists in a kind of backbone that represents the events of a level_group.
\nThis backbone is trained on all the data available for this level_group (i.e. on complete + incomplete sessions). It is associated with a temporary SimpleHead optimizing BCE loss.
\n (tf.keras.):\n ():\n (, ).__init__(name=name)\n .ffs = [tf.keras.layers.(units, activation=) units n_units]\n .out = tf.keras.layers.(n_outputs, activation=)\n\n ():\n x = inputs\n ff .\n x = ff(x)\n outputs = .out(x)\n outputs\nThis approach allows to score CV 0.70025 +/- 0.0005.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2208184%2F98e32800c54ffab63d98d90d220fbbcd%2Fend_2_end.png?generation=1702113900174294&alt=media\")
The weights of each of the 3 backbones (1 by level_group) are freezed for the 2nd level of training to speedup training but also because it is more stable and efficient. These backbones can be thought as \"embedders\".
\nDuring this 2nd step, all the submodels that composed the solution were trained on all complete sessions in an end-to-end setup. The input data are 3 sequences of the 5 features, 1 for each of the 3 level groups. Each \"embedders\" outputs a 24 dim-vector representation. These outputs are the inputs of a head in which enters the representation of level_group '0-4' to predict the 3 first questions and the concatenation of the previous and the current representations for level_groups '5-12' and '13-22' to make use of all information.
\nProceeding like this allows to optimize the overall performance and to monitor it based on the F1 score that is the score of the competition. This means we optimized BCE with F1 score as a metric.
\nOur winning submission uses a simple MLP head but also a skip head (512 -> 512 -> 512 allow it for example). MMoE did not improve the simplest approaches.
\nThis approach allows to score CV 0.70175 +/- 0.0003 which is comparable to the GBDT solution.
\nEarly in the competition we built a simulator of the API. Doing so we never experimented any submission error. Maybe trying to keep ideas and code as simple as possible was also key to debug easily.
\nWe invested the efficiency part of the challenge for GBDT as well as NNs.
\nUsing Treelite for XGBoost allow us to divide by 2 the execution time.
\nOur deep learning models were lights: 400,000 weights for the end-to-end model which combines every parts/sub-models. Having already used TF Lite we knew it could be a game changer. Converting our models led to a significant boost in inference time without any performance loss (we do not remember exactly but we think it is at least 6x faster on our local inference simulator).
\nBeginning to explore pruning as well as hard quantization showed that the performance loss would be significant (which is OK in production but not in a competition) so we sticked to a simple TF Lite conversion.
We have not leveraged what seems to be a problem in the efficiency metric. As we identified the private test sessions to be the 1450/1500 first served by the API we tried to just predict the others to check which time was used (public for public and private for private). Doing so we gain a place but choose to not use this.
\nOur efficiency submission is a NN that scores public LB 0.702 and private LB 0.699 in less than 5 minutes.
\nWe experimented a lot of ensembling alternatives. In the end we sticked to a simple average 50/50 GBDT/NN with:
\nAs our models are lightweight we were able to build a hugh ensemble: 2 x 4 x 10 folds XGBoost + 3 x 4 x 5 folds NNs. The bottleneck for us is the 8 Go RAM constraint.
\nThe winning submission scores CV 0.705, public LB 0.705 and private LB 0.705.
\nThe main achievement of our work is that it is a good solution for the researchers, learners and children that can benefit of it and we hope it will contributes to progresses for a better learning experience. Up to you guys!
\nThanks if you read until here!
\nIf you have any question do not hesitate to ask. We will do our best to respond.
A video presentation to the host has been recorded and can be available on demand. Feel free to ask via PM.
\nBelow are the main sources that we used. More sources can be found in section Details of the submission above.
\nUnbelievable to write this!
\nAs it is the usage, we first thank the host and Kaggle. These are special thanks because you and us have had a special link in this competition as we gave you more work by reporting data leaks. No doubt you tried to do your best. You are right to animate this community and to trust in it. You are part of it. Please take care of this community that is able to build so much together by sharing. As all of us you have made mistakes and we hope you will learn from them.
\nWe also want to thank all of you, Kagglers. We love and are grateful to be part of our group/community. Thanks for sharing and for the collective learning experience.
\nOur solution is essentially a blend of a XGBoost and a NN models. Both heavily rely on duration that appeared to be a powerful leverage. Time was aggregated in different ways and combined with counts for the GBDT while it is transformed via a custom TimeEmbedding block based on 1D convolutions that produce a representation combined with user event representations for the NN.
\nRobustness and efficiency founded our work. XGBoost models were validated on 10 bags of 5 folds and features incorporated only if the mean of the CV of these 10 bags was greater than the level of noise we quantified while we opted for a majority/consensus strategy to build the NN, i.e. validate choices only if 4 of 5 folds were improved. The 3rd place of the efficiency LB was achieved with a lightweight NN accelerated via TF Lite.
After publishing this write-up we decided to open our code: https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/420332.
\nIt is composed by several parts: how to train the XGBoost models, how to pretrain and train the NN models and the inference notebook used to win this competition.
Looking at the 1st data released showed that there aren't a lot of sessions so not a lot of sequences. Moreover these are long sequences. This is not ideal for a deep learning approach.
\nExploring the Field Day Lab research instructed that the Jo Wilder application was built to help learning to read and that way more than 11,500 learners had played this game.
\nThese 2 ideas led to search for a bigger dataset. In 1 Google search and 3 clicks we came up to the open data portal (https://fielddaylab.wisc.edu/opengamedata/) which contains a lot of sessions. 1 hour and 3 bash commands latter we knew that the train set was in part in the open data. So we took a week to build a pipeline that extracts 98 % of the sessions of the train set perfectly and with minor errors for the last 2 %. Our data are even better than the comp data because we knew before the host confirmation that for the sessions with 2 games the target was skewed (0 if wrong in 1 of the 2 games when we aim at predicting the responses for the 1st game). It seems that fixing these targets can bring a significant boost up to +0.002.
We took 1 more week to build a GBDT/XGBoost baseline that would have scored top 10 given the CV score, with the use of the supplemental data (~20,000 sessions) that gave +0.003/0.004 at that time. As we simulated the API locally (see after), we used some training sessions to infer and noticed that it scored 0.718. We were hoping that the LB sessions were not part of the open data portal but our 1st submission, LB 0.708, immediately showed to us that we had rebuilt about a half of the data and especially the targets in the public LB, because 0.708 = (0.698 + 0.718) / 2. The host and Kaggle have been immediately informed. You know what happened next (https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/415820).
\nAfter the release of the LB data we measured that we perfectly rebuilt ~7000 sessions over the ~11,500 of the LB data.
We spent the first month exploring the data until we understood/knew it pretty well. For example we even reconstituted sessions for what might be schools (several games on 1 IP session), extracted every single session with at least 1 answer, …
\nAfter the update we made a first submission that scored 0.72. This was shocking because this meant that some leaked data were remaining. A few days later we noticed that the open data was not totally similar with the state we found it 1 month before. A file was missing. So we returned to the host and Kaggle to give them more work.
\nThis process/work led us to perfectly understand the data model (that changed since the 1st release of the game). This also allowed us to deeply understand the data itself.
\nNote that we only used the sessions for which we had responses to all questions of the 2 1st level groups. 1) This is more consistant with the sessions we want to predict (game from the beginning to the end) and 2) this approach preserves performance (vs all data) while reducing training time.
\nOur dataset is constituted by 37323 complete sessions (23562 comp + logs) in a total of 66376 sessions.
\nThe supplemental data (that we fully added 1 month ago) gave us consistently CV +0.002.
\nOur solution is mainly an ensemble of GBDT + NN models.
\nWe think that the main reason of the robustness of our solution is that we only relied on CV for decision making. No choice had been made on LB.
\nProbing showed to us that the private test consists in the 1st 1450/1500 sessions served by the API. This is a small set. In our experiments 5,000 sessions is the minimum to guarantee a stable CV/LB alignment. A set less than 2,000 is very noisy so robustness was the way to go.
\nWe only added features that improved the CV for sure. This is not easy to delete features that you believe in but this is needed as science is not a matter of belief. There are several ways to do so: for example monitor all folds in a CV (and accept only on majority or consensus), monitor several bags (composition of CV to not overfit validation), …
\nFor the GBDT approach, we mainly validated on the mean of 10 bags (we defined a bag as a composition of the folds). As we estimated the noise to be ~0.0003, only improvements greater than the noise have been considered. For the NN as we needed to iterate quicker we only used a single bag and only incorporated > 0.0003 overall improvements with at least 3 or 4 (over 5) folds improved.
\nWe experimented a lot on finding a threshold by question but found that this approach is less robust than a single threshold. We mainly used 0.625 as global threshold despite our highest LB scores that were obtained with a threshold per question.
\nWe prototyped a baseline with XGBoost because of the structured/tabular nature of the data. The feature engineering process is interesting to understand what is predictive and to understand the causation, i.e. how the features or decision criteria that enable to predict correctly.
\nGenerally speaking we followed 3 ways to build features: business knowledge, our intuition playing the game and a meticulous exploration of the data.
\nBusiness knowledge refers to using expert knowledge. Reading the papers of the researchers that built this game allow to understand the game beyond usage. For example, Jo Wilder has been built to improve the players reading skills. So this means that the text duration should be important. These are like killer features.
We exclusively made use of Polars because of the CPU constraints and to simply learn it.
\nOur features (663, 1993, 3734 for each level_group) are mainly durations and counts for different aggregations: how much time in a level, in a room, reading a text, interacting in some way (event type), how many events in a level_group, how many events of each type, how many events of each type in a room or a level, …
\nWe also built a few notebook dedicated features: how many type of events on the notebook in a level, …
\nDespite our efforts we weren't able to extract useful information from the coordinates, the only few features of this type had been mean and std for some events in the activities (journal interactions for example).
We considered that injecting targets predicted in the previous level groups was a compression of the signal, meaning a loss of information, so we used, for each session, all interactions from the beginning of the game/session. This led to a +0.002 at the time of this choice.
\nAfter the API needed to order the data, we noticed that models trained both on original order and on index order but validated on index order (inference order) improved our scores. This leads to more variety that was needed to improve stability and robustness. The same goes for the composition of the validation sets: usage of several bags (composition of validation sets) based on the comp data but also on the extracted data improved our scores. We detected late that increasing the number of folds from 5 to 10 could also be leveraged.
\nThe code for GBDT allows to switch from XGBoost to LightGBM and CatBoost with a simple variable parameter but despite the good scores (~0.001 less than XGBoost), this did not bring to ensemble so we sticked to only XGBoost.
\nWe experimented a lot around feature selection but were unable to build a stable strategy. So instead of a top-down approach consisting in deleting useless features, we adopted a bottom-up approach choosing carefully each group of features.
\nOur XGBoost models score CV ~0.7025 +/-0.0003 and blending 5 of them (the only XGBoost we still have with correct score) scores LB 0.704.
\nAfter achieving a good score with gradient boosting and having understood well the data we focused on deep learning.
\nThe first attempt was with Transformers. The 1st results were disappointed: CV 0.685 with 2 hours / fold (as far as we can remember). Transformers are very computationally intensive. Resources: https://arxiv.org/pdf/1912.09363.pdf, https://arxiv.org/pdf/2001.08317.pdf, https://arxiv.org/pdf/1711.03905.pdf, https://arxiv.org/pdf/1907.00235.pdf, …
\nWe then gave a try to Conv1D. In one day we had a very simple model that scored as Transformers but 10x faster allowing to iterate quicker. So we pushed this approach and could seamlessly scaled it beyond our expectations.
\nDifficult to share the tens or hundreds of experimentations needed to achieve the final solution which is both based on a simple architecture and a slightly complex training pipeline.
\nWe browsed the literature based on the question: how to model time in deep learning?
\nThis research made us come to the idea of time-aware events (i.e. https://proceedings.mlr.press/v126/zhang20c/zhang20c.pdf) and back to WaveNet (https://arxiv.org/pdf/1609.03499.pdf) because it uses Conv1D to model long sequences with considerations on causation.
\nOther papers also inspired us: https://arxiv.org/pdf/1703.04691.pdf build on top of WaveNet paper for time series, https://idus.us.es/bitstream/handle/11441/114701/Short-Term%20Load%20Forecasting%20Using%20Encoder-Decoder%20WaveNet.pdf?sequence=1&isAllowed=y also build on top of WaveNet.
\nWe also have to mention the excellent work that @abaojiang shared (https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/398565 and https://www.kaggle.com/code/abaojiang/lb-0-694-tconv-with-4-features-training-part). It inspired our research and maybe successfully biased it.
Let's focus on the model of our efficiency submission that is also one of our final ensemble and which performance is nearly the same as models with a few more features.
\n
5 features as inputs: duration, text_fqid, room_fqid, fqid, event_name + name (this is the event type from the original data model as far as we remember). Each of these information is encoded/embedded into a vector representation (d_model = 24) to be the merged. The 4 categorical features feed a classical Embedding layer and the duration a TimeEmbedding which is a custom block.
\nDeveloping the GBDT solution showed that the duration was crucial, so we put a crucial amount of time trying to model it greatly. The TimeEmbedding layer is a composition of 4x ConvBlock which is inspired by the Transformer main block: Conv1D -> skip connection -> layer norm -> dropout.
\n (tf.keras.layers.):\n ():\n (, ).__init__()\n .conv_blocks = [(d_model, dropout_rate=dropout_rate) _ range(n_blocks)]\n\n ():\n x = tf.expand_dims(inputs, axis=-)\n conv_block .\n x = conv_block(x)\n x\n (tf.keras.layers.):\n ():\n (, ).__init__()\n .conv1d = tf.keras.layers.D(d_model, kernel_size=, padding=, activation=)\n .layer_norm = tf.keras.layers.()\n .dropout = tf.keras.layers.(rate=dropout_rate)\n\n ():\n x = .conv1d(inputs)\n x = x + inputs\n x = .layer_norm(x)\n outputs = .dropout(x)\n outputs\n
As said, the goal of building these representations was to model time-aware events. We considered the categorical features as events because they represent the user interactions with business entities of the game. We then tried to incorporate duration to make them time-awared. Our main intuition showed to be the best. It is a simple solution based on operation priority to represent that the duration should be associated to each event before associated them together: duration * event_1 + duration * event_2 + … which had been factorized to duration * (event_1 + event_2 + …).
\n (tf.keras.Model):\n ():\n (ConvNet, ).__init__(name=name)\n .input_dims = input_dims\n .n_outputs = n_outputs\n .d_model = d_model\n .n_blocks = n_blocks\n .event_embedding = tf.keras.layers.Embedding(input_dims[], d_model, mask_zero=)\n .room_embedding = tf.keras.layers.Embedding(input_dims[], d_model, mask_zero=)\n .text_embedding = tf.keras.layers.Embedding(input_dims[], d_model, mask_zero=)\n .fqid_embedding = tf.keras.layers.Embedding(input_dims[], d_model, mask_zero=)\n .duration_embedding = TimeEmbedding(n_blocks=n_blocks, d_model=d_model, dropout_rate=)\n .gap = tf.keras.layers.GlobalAveragePooling1D()\n\n ():\n event = .event_embedding(inputs[])\n room = .room_embedding(inputs[])\n text = .text_embedding(inputs[])\n fqid = .fqid_embedding(inputs[])\n duration = .duration_embedding(inputs[])\n x = duration * (event + room + text + fqid)\n outputs = .gap(x)\n outputs\n\n ():\n config = ().get_config().copy()\n config.update({\n : .input_dims,\n : .n_outputs,\n : .d_model,\n : .n_blocks,\n : ._name,\n })\n config\n\n\n ():\n cls(**config)\n
\nThe 2 representations are equivalent: either you can think time-aware events as a combination of time and sub-events or as a combination of sub-events and time.
\n
The training pipeline is not totally straight forward.
\n@dongyk published great schematics that can be useful to illustrate what is explained bellow: https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/420217#2332166.
\n
The best approach for us consists in a kind of backbone that represents the events of a level_group.
\nThis backbone is trained on all the data available for this level_group (i.e. on complete + incomplete sessions). It is associated with a temporary SimpleHead optimizing BCE loss.
\n (tf.keras.):\n ():\n (, ).__init__(name=name)\n .ffs = [tf.keras.layers.(units, activation=) units n_units]\n .out = tf.keras.layers.(n_outputs, activation=)\n\n ():\n x = inputs\n ff .\n x = ff(x)\n outputs = .out(x)\n outputs\nThis approach allows to score CV 0.70025 +/- 0.0005.
\n
The weights of each of the 3 backbones (1 by level_group) are freezed for the 2nd level of training to speedup training but also because it is more stable and efficient. These backbones can be thought as \"embedders\".
\nDuring this 2nd step, all the submodels that composed the solution were trained on all complete sessions in an end-to-end setup. The input data are 3 sequences of the 5 features, 1 for each of the 3 level groups. Each \"embedders\" outputs a 24 dim-vector representation. These outputs are the inputs of a head in which enters the representation of level_group '0-4' to predict the 3 first questions and the concatenation of the previous and the current representations for level_groups '5-12' and '13-22' to make use of all information.
\nProceeding like this allows to optimize the overall performance and to monitor it based on the F1 score that is the score of the competition. This means we optimized BCE with F1 score as a metric.
\nOur winning submission uses a simple MLP head but also a skip head (512 -> 512 -> 512 allow it for example). MMoE did not improve the simplest approaches.
\nThis approach allows to score CV 0.70175 +/- 0.0003 which is comparable to the GBDT solution.
\nEarly in the competition we built a simulator of the API. Doing so we never experimented any submission error. Maybe trying to keep ideas and code as simple as possible was also key to debug easily.
\nWe invested the efficiency part of the challenge for GBDT as well as NNs.
\nUsing Treelite for XGBoost allow us to divide by 2 the execution time.
\nOur deep learning models were lights: 400,000 weights for the end-to-end model which combines every parts/sub-models. Having already used TF Lite we knew it could be a game changer. Converting our models led to a significant boost in inference time without any performance loss (we do not remember exactly but we think it is at least 6x faster on our local inference simulator).
\nBeginning to explore pruning as well as hard quantization showed that the performance loss would be significant (which is OK in production but not in a competition) so we sticked to a simple TF Lite conversion.
We have not leveraged what seems to be a problem in the efficiency metric. As we identified the private test sessions to be the 1450/1500 first served by the API we tried to just predict the others to check which time was used (public for public and private for private). Doing so we gain a place but choose to not use this.
\nOur efficiency submission is a NN that scores public LB 0.702 and private LB 0.699 in less than 5 minutes.
\nWe experimented a lot of ensembling alternatives. In the end we sticked to a simple average 50/50 GBDT/NN with:
\nAs our models are lightweight we were able to build a hugh ensemble: 2 x 4 x 10 folds XGBoost + 3 x 4 x 5 folds NNs. The bottleneck for us is the 8 Go RAM constraint.
\nThe winning submission scores CV 0.705, public LB 0.705 and private LB 0.705.
\nThe main achievement of our work is that it is a good solution for the researchers, learners and children that can benefit of it and we hope it will contributes to progresses for a better learning experience. Up to you guys!
\nThanks if you read until here!
\nIf you have any question do not hesitate to ask. We will do our best to respond.
A video presentation to the host has been recorded and can be available on demand. Feel free to ask via PM.
\nBelow are the main sources that we used. More sources can be found in section Details of the submission above.
\nHuge congrats @cpmpml and @pdnartreb! Thanks for sharing such a great solution. Double congrats for @cpmpml for finally break the 2nd place curse and get your first winning!
", "rawMarkdown": "Huge congrats @cpmpml and @pdnartreb! Thanks for sharing such a great solution. Double congrats for @cpmpml for finally break the 2nd place curse and get your first winning!", "votes": 7 }, { "id": 2323974, "postDate": "2023-06-30T09:41:15.223Z", "content": "Congrats, impressive solution! Thank you for your detailed explanation.
\nWill you share the kernel? I can't wait to learn.
", "rawMarkdown": "Congrats, impressive solution! Thank you for your detailed explanation. \n\nWill you share the kernel? I can't wait to learn.", "votes": 3, "replies": [ { "id": 2324001, "postDate": "2023-06-30T10:01:06.463Z", "content": "Thanks @zrongchu!
\nWe have decided to share our code: https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/420332.
Thanks! @pdnartreb , @cpmpml
", "rawMarkdown": "Thanks! @pdnartreb , @cpmpml ", "votes": 3 } ] } ] }, { "id": 2339570, "postDate": "2023-07-10T22:35:08.770Z", "content": "Greetings, @pdnartreb and @cpmpml !
\nI wanted to express my sincere appreciation and congratulations for your outstanding work in this recent competition. Your solution is truly impressive and it's evident that you put a tremendous amount of effort and expertise into developing it.
\nYour focus on understanding the data model and extracting valuable insights is impressive. The combination of GBDT and NN models in your ensemble and search of external feature engineering demonstrates your deep understanding of the problem and your ability to leverage different techniques effectively.
\nI have a question regarding your approach: In the training pipeline, you mentioned that the backbone, representing the events of a level group, is trained on all the available data for that group. Could you elaborate on how you handle the incomplete sessions during this training phase? How do you ensure that the backbone captures the most relevant information from both complete and incomplete sessions?
\nOnce again, congratulations on your remarkable achievement, and thank you for your contributions to the Kaggle community. Your solution is an inspiration to fellow data scientists and serves as a testament to your expertise.
\nBest regards,keep up with the great work🦾🚀
", "rawMarkdown": "Greetings, @pdnartreb and @cpmpml !\n\nI wanted to express my sincere appreciation and congratulations for your outstanding work in this recent competition. Your solution is truly impressive and it's evident that you put a tremendous amount of effort and expertise into developing it.\n\nYour focus on understanding the data model and extracting valuable insights is impressive. The combination of GBDT and NN models in your ensemble and search of external feature engineering demonstrates your deep understanding of the problem and your ability to leverage different techniques effectively.\n\nI have a question regarding your approach: In the training pipeline, you mentioned that the backbone, representing the events of a level group, is trained on all the available data for that group. Could you elaborate on how you handle the incomplete sessions during this training phase? How do you ensure that the backbone captures the most relevant information from both complete and incomplete sessions?\n\nOnce again, congratulations on your remarkable achievement, and thank you for your contributions to the Kaggle community. Your solution is an inspiration to fellow data scientists and serves as a testament to your expertise.\n\nBest regards,keep up with the great work🦾🚀", "votes": 1, "replies": [ { "id": 2340147, "postDate": "2023-07-11T09:58:40.450Z", "content": "I am not sure I understand what you don't understand.
\nLet me try still. Maybe this thread can help: https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/420217#2331165
\nFor a given NN there are three NNs trained in the \"pretraining hase\", then they are assembled into a single NN in a second training phase.
\nIncomplete sessions are only used in the pretraining phase.
\nFor instance, if a session has only data for the first level group (i.e. questions 1,2,3), then it is used to train the first of the three NNs, and it is not used for the other two.
", "rawMarkdown": "I am not sure I understand what you don't understand.\n\nLet me try still. Maybe this thread can help: https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/420217#2331165\n\nFor a given NN there are three NNs trained in the \"pretraining hase\", then they are assembled into a single NN in a second training phase. \n\nIncomplete sessions are only used in the pretraining phase.\n\nFor instance, if a session has only data for the first level group (i.e. questions 1,2,3), then it is used to train the first of the three NNs, and it is not used for the other two.", "votes": 3, "replies": [ { "id": 2341136, "postDate": "2023-07-11T21:56:24.593Z", "content": "Thanks for the quick reply!
\nIndeed the following thread has helped to interpret the model training phase. Specially the following schematics:
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8366173%2F960b98da4ee0cefe936edf44d5fec9e9%2F1.png?generation=1688613004356860&alt=media\")
\nKudos to @dongyk , for that.
It seems that you have leveraged the full dataset (complete + incomplete sessions), to create specialized models for each level_group, concatenating the inputs. Which enabled the model to capture specific information of all the level_groups independently and all together, by concatenating the model inputs. Therefore you could optimize BCE with F1 score as metric, since it turned out to be a classification problem….
\nOnce again congratulations @pdnartreb and @cpmpml for the excellent work and thanks for contributing to the Kaggle Community🙏🚀
", "rawMarkdown": "Thanks for the quick reply!\n\nIndeed the following [thread](https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/420217#2331165) has helped to interpret the model training phase. Specially the following schematics:\n\nKudos to @dongyk , for that.\n\nIt seems that you have leveraged the full dataset (complete + incomplete sessions), to create specialized models for each level_group, concatenating the inputs. Which enabled the model to capture specific information of all the level_groups independently and all together, by concatenating the model inputs. Therefore you could optimize BCE with F1 score as metric, since it turned out to be a classification problem....\n\nOnce again congratulations @pdnartreb and @cpmpml for the excellent work and thanks for contributing to the Kaggle Community🙏🚀\n\n\n", "votes": 1 }, { "id": 2341645, "postDate": "2023-07-12T07:59:22.233Z", "content": "yes, that's what we did:
\n\n", "rawMarkdown": "yes, that's what we did:\n\n> It seems that you have leveraged the full dataset (complete + incomplete sessions), to create specialized models for each level_group, concatenating the inputs. ", "votes": 2 } ] } ] }, { "id": 2334085, "postDate": "2023-07-07T12:53:45.227Z", "content": "It seems that you have leveraged the full dataset (complete + incomplete sessions), to create specialized models for each level_group, concatenating the inputs.
\n
Congratulations ! Very interesting !
", "rawMarkdown": "Congratulations ! Very interesting !", "votes": 1 }, { "id": 2333655, "postDate": "2023-07-07T05:31:07.173Z", "content": "Congratulations! very helpful.
", "rawMarkdown": "Congratulations! very helpful. ", "votes": 1 }, { "id": 2332588, "postDate": "2023-07-06T09:47:28.037Z", "content": "Very insightful
", "rawMarkdown": "Very insightful ", "votes": 1 }, { "id": 2331909, "postDate": "2023-07-05T21:38:04.753Z", "content": "Congratulations! Really interesting work
", "rawMarkdown": "Congratulations! Really interesting work", "votes": 1 }, { "id": 2331165, "postDate": "2023-07-05T12:05:07.797Z", "content": "Hi, I was wondering whether I understand it correctly.
\nPSPFGP 1st Place - NN Training
outputs = {}\noutputs = heads(convnet_outputs)\noutputs = heads(\n tf()(, convnet_outputs])\n )\noutputs = heads(\n tf()(, convnet_outputs, convnet_outputs])\n )\nDoes it describe like this??
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8366173%2F5bd337c5017d3ab5606024c830391f40%2F.png?generation=1688556655026917&alt=media\")
\nBackpropagation is performed:
\nfrom outputs['0-4'] to convnets['0-4'], (I will call this as submodel A)
\nfrom outputs['5-12'] to convnets['0-4'] and convnets['5-12'], and (submodel B)
\nfrom outputs['13-22'] to convnets['0-4'], convnets['5-12'], and convnets['13-22']. (submodel C)
\nHowever, this submodels are separated. So, Backpropagation which is performed from outputs['13-22'] to convnets['0-4'] (submodel C) does not affect convnets['0-4'] of submodel A.
\nDid I understand correctly??
+) I was really surprised this approach to compute F1 score during training. Wow…
", "rawMarkdown": "Hi, I was wondering whether I understand it correctly.\n[PSPFGP 1st Place - NN Training](https://www.kaggle.com/code/pdnartreb/pspfgp-1st-place-nn-training)\n```\noutputs = {}\noutputs['0-4'] = heads['0-4'](convnet_outputs['0-4'])\noutputs['5-12'] = heads['5-12'](\n tf.keras.layers.Concatenate()([convnet_outputs['0-4'], convnet_outputs['5-12']])\n )\noutputs['13-22'] = heads['13-22'](\n tf.keras.layers.Concatenate()([convnet_outputs['0-4'], convnet_outputs['5-12'], convnet_outputs['13-22']])\n )\n```\nDoes it describe like this??\n\nBackpropagation is performed: \nfrom outputs['0-4'] to convnets['0-4'], (I will call this as submodel A)\nfrom outputs['5-12'] to convnets['0-4'] and convnets['5-12'], and (submodel B)\nfrom outputs['13-22'] to convnets['0-4'], convnets['5-12'], and convnets['13-22']. (submodel C)\nHowever, this submodels are separated. So, Backpropagation which is performed from outputs['13-22'] to convnets['0-4'] (submodel C) does not affect convnets['0-4'] of submodel A.\nDid I understand correctly??\n\n+) I was really surprised this approach to compute F1 score during training. Wow...", "votes": 1, "replies": [ { "id": 2331653, "postDate": "2023-07-05T17:22:41.923Z", "content": "Hi @dongyk!
\nYou are working hard. Great! This is really satisfying to see that our work is useful for you. With this mindset you are going to learn a lot.
\nYour understanding and your schematic seem nearly OK.
\nKeep in mind that in step 2 of the training the weights of the ConvNets are freezed. This means backpropagation only changes the heads weights.
\nIn this 2nd step we have 3 sequences of data as inputs, 1 for each level_group. Each of these sequences feeds its own ConvNet/embedder that outputs a representation in the form of a 24 dim vector. The representations for level_group 0-4 feeds the SimpleHead for level_group 0-4 which outputs a vector of 3 values that are the probability of the 3 responses of this level_group. The representations for level_group 0-4 and 5-12 are concatenated in a 48 dim vector the feeds the SimpleHead for the 2nd level_group that outputs 10 values that correspond to the 10 questions of the level_group 5-12. Same for level_group 13-22.
You correctly spotted that this had been made to train the whole ensemble to optimize the overall F1 score that is what we want.
\nWe tried every setup: training the whole ensemble from scratch, each part independently, … The setup we presented here corresponds to what worked best for us.
Does this respond to your question(s)? If not feel free to ask.
", "rawMarkdown": "Hi @dongyk!\n\nYou are working hard. Great! This is really satisfying to see that our work is useful for you. With this mindset you are going to learn a lot.\n\nYour understanding and your schematic seem nearly OK. \nKeep in mind that in step 2 of the training the weights of the ConvNets are freezed. This means backpropagation only changes the heads weights. \nIn this 2nd step we have 3 sequences of data as inputs, 1 for each level_group. Each of these sequences feeds its own ConvNet/embedder that outputs a representation in the form of a 24 dim vector. The representations for level_group 0-4 feeds the SimpleHead for level_group 0-4 which outputs a vector of 3 values that are the probability of the 3 responses of this level_group. The representations for level_group 0-4 and 5-12 are concatenated in a 48 dim vector the feeds the SimpleHead for the 2nd level_group that outputs 10 values that correspond to the 10 questions of the level_group 5-12. Same for level_group 13-22.\n\nYou correctly spotted that this had been made to train the whole ensemble to optimize the overall F1 score that is what we want.\nWe tried every setup: training the whole ensemble from scratch, each part independently, ... The setup we presented here corresponds to what worked best for us.\n\nDoes this respond to your question(s)? If not feel free to ask.", "votes": 1, "replies": [ { "id": 2332166, "postDate": "2023-07-06T03:25:15.107Z", "content": "Thank you for your answering!!
\nI'm learning a lot from your code.
\n\nWe tried every setup: training the whole ensemble from scratch, each part independently, … The setup we presented here corresponds to what worked best for us.
\n
I was thinking the same. Did they try the whole ensemble from scratch? Because it's more cumbersome to train seperately. Training separately can be helpful to train.
\nIs the reason why you used only convnet['5-12'] for outputs['5-12'] not contained convnet['0-4']?
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8366173%2F824ba97fb079de1424be6625e9e99f0b%2F2.png?generation=1688620218055724&alt=media\")
And… I editted the figure.
\n
\nI found that you fed train_dataset only once and the same convnet['0-4'] is fed into simplehead['0-4'], simplehead['5-12'], and simplehead['13-22'].
\nIs it more accurate?
\nIf I didn't freeze the convnets, can convnet['0-4'] backpropagated(affected) by not only outputs['0-4'] but outputs['5-12'] and outputs['13-22']?
Wow! Thanks for your schematics that are 100% correct as I understand them! The write-up has been updated to link to your post.
\nIf the weights of the convnets are not freezed then they are updated by the backprop coming from each output they are linked to when training end-to-end.
\nWe first trained separatly which gave a baseline. Then we tried to train the end-to-end setup from scratch. As you might have noticed the output allow to monitor the loss which is composed by the losses of each level group that can also be monitored. We observed that the loss for 0-4 was the first to converge but also that, as the others go down, it tends to increase. We interpreted that as the optimization of previous level groups representation for next level groups. This is not what we wanted. We wanted a good representation for a level group that can be used by the next level groups. So we chose to first optimize each convnet separatly and to exploit its representation by heads in a second time.
\nAs F1 score cannot be optimized directy, the main goal of the end-2-end approach was to be able to monitor the F1 score to select the best weights.
", "rawMarkdown": "Wow! Thanks for your schematics that are 100% correct as I understand them! The write-up has been updated to link to your post.\n\nIf the weights of the convnets are not freezed then they are updated by the backprop coming from each output they are linked to when training end-to-end.\n\nWe first trained separatly which gave a baseline. Then we tried to train the end-to-end setup from scratch. As you might have noticed the output allow to monitor the loss which is composed by the losses of each level group that can also be monitored. We observed that the loss for 0-4 was the first to converge but also that, as the others go down, it tends to increase. We interpreted that as the optimization of previous level groups representation for next level groups. This is not what we wanted. We wanted a good representation for a level group that can be used by the next level groups. So we chose to first optimize each convnet separatly and to exploit its representation by heads in a second time.\n\nAs F1 score cannot be optimized directy, the main goal of the end-2-end approach was to be able to monitor the F1 score to select the best weights.", "votes": 1 }, { "id": 2332884, "postDate": "2023-07-06T13:36:42.110Z", "content": "Cool.
\nI feel good to understand it.
\n\nWe observed that the loss for 0-4 was the first to converge but also that, as the others go down, it tends to increase. We interpreted that as the optimization of previous level groups representation for next level groups. This is not what we wanted. We wanted a good representation for a level group that can be used by the next level groups. So we chose to first optimize each convnet separatly and to exploit its representation by heads in a second time.
\n
I also tried to come up with a way to change the architecture of the model, but I realized that it's the best idea considering the pros and cons of possible models.
\n\n\nAs F1 score cannot be optimized directy, the main goal of the end-2-end approach was to be able to monitor the F1 score to select the best weights.
\n
The way I see it, it's the most important approach in the NN code.
\nAgain, thank you so much.
", "rawMarkdown": "Cool.\nI feel good to understand it.\n\n>We observed that the loss for 0-4 was the first to converge but also that, as the others go down, it tends to increase. We interpreted that as the optimization of previous level groups representation for next level groups. This is not what we wanted. We wanted a good representation for a level group that can be used by the next level groups. So we chose to first optimize each convnet separatly and to exploit its representation by heads in a second time.\n\nI also tried to come up with a way to change the architecture of the model, but I realized that it's the best idea considering the pros and cons of possible models.\n\n>As F1 score cannot be optimized directy, the main goal of the end-2-end approach was to be able to monitor the F1 score to select the best weights.\n\nThe way I see it, it's the most important approach in the NN code.\n\nAgain, thank you so much.", "votes": 1 } ] } ] }, { "id": 2326886, "postDate": "2023-07-02T13:22:02.613Z", "content": "bravo 1St place
", "rawMarkdown": "bravo 1St place", "votes": 1 }, { "id": 2326179, "postDate": "2023-07-02T00:55:58.820Z", "content": "Congratulations on your win🎉 And thanks you also for sharing your solution!
\nIf I may, may I ask a question about the third chapter, the beginning of the Data?
\nDue to the small number of game sessions, the length of the sequence (i.e. the length of the game to be put into the NN) is not long enough. This as not ideal for the Deep Learning approach, I interpreted.
\nIf so, what does it mean by \"Moreover these are long sequences.\"? Does it relevant that in some cases the game is played many times and is actually a long sequence, but what we have as data is a short sequence?
There are too few sequences to enable complex NN model training. But sequence average length is rather long.
", "rawMarkdown": "There are too few sequences to enable complex NN model training. But sequence average length is rather long.", "votes": 2, "replies": [ { "id": 2326787, "postDate": "2023-07-02T11:59:11.803Z", "content": "Thank you. I understood!
", "rawMarkdown": "Thank you. I understood!", "votes": 1 } ] } ] }, { "id": 2325659, "postDate": "2023-07-01T14:02:09.720Z", "content": "@pdnartreb 1st place bravo. Great Victory.
", "rawMarkdown": "@pdnartreb 1st place bravo. Great Victory.", "votes": 1 }, { "id": 2325346, "postDate": "2023-07-01T09:10:11.110Z", "content": "Congratulations on your first place!
", "rawMarkdown": "Congratulations on your first place!", "votes": 1 }, { "id": 2325317, "postDate": "2023-07-01T08:49:29.843Z", "content": "very useful, thank you for your fanscinating work.
", "rawMarkdown": "very useful, thank you for your fanscinating work.", "votes": 1 }, { "id": 2325268, "postDate": "2023-07-01T08:07:48.353Z", "content": "Such an impressive combination of good ideas and hard hard work.
", "rawMarkdown": "Such an impressive combination of good ideas and hard hard work.", "votes": 1 }, { "id": 2325097, "postDate": "2023-07-01T05:51:09.790Z", "content": "Congrats on winning first place! Thanks a lot for your detailed explanation. Impressive! 🔥🔥🔥🔥🔥🔥
", "rawMarkdown": "Congrats on winning first place! Thanks a lot for your detailed explanation. Impressive! 🔥🔥🔥🔥🔥🔥", "votes": 1 }, { "id": 2325053, "postDate": "2023-07-01T05:26:58.413Z", "content": "Congrats on the win! Really love the NN design here. I has a very similar NN design, except that I just one-hot encode everything due to low cardinality of categorical features.
", "rawMarkdown": "Congrats on the win! Really love the NN design here. I has a very similar NN design, except that I just one-hot encode everything due to low cardinality of categorical features.", "votes": 1 }, { "id": 2324735, "postDate": "2023-06-30T20:45:31.277Z", "content": "Wow, just wow! :D
\nSuch an impressive combination of good ideas and hard hard work. Very inspiring!
", "rawMarkdown": "Wow, just wow! :D\n\nSuch an impressive combination of good ideas and hard hard work. Very inspiring!", "votes": 1 }, { "id": 2324715, "postDate": "2023-06-30T20:21:55.020Z", "content": "This is very cool the understanding is very good and I like this lessonThis is very cool the understanding is very good and I like this lessonThis is very cool the understanding is very good and I like this lesson
", "rawMarkdown": "This is very cool the understanding is very good and I like this lessonThis is very cool the understanding is very good and I like this lessonThis is very cool the understanding is very good and I like this lesson", "votes": 1 }, { "id": 2324446, "postDate": "2023-06-30T15:53:44.053Z", "content": "Congrats @cpmpml and @pdnartreb for winning the 1st place! Also, thanks a lot for the detailed explanation!
\nMay I ask how you decide to simply divide the numeric features (e.g., duration) by a constant (60000 in the code you open source), instead of using other normalization methods. I suppose that shrinking the scale of the raw feature could make the training process converge faster and other normalization techniques don't show superiority over the way you eventually adopted. If there's misunderstanding, please put me right. Thanks a lot!
Thanks @abaojiang!
\nThe writeup has been updated to mention your excellent notebook. We first saw the time-aware events and the Conv1D ideas in your work. Finding once more these ideas while exploring the litterature likely made us focus more on them.
\nContinuous features are tricky to preprocess. We knew from our GBDT modeling work that time and especially duration was very important. We put a lot of careful efforts to try to model it correctly. Processing the duration seemed important so we tried a lot of things: scaling, normalization, standardization, clipping, … We explored the data to analyze the distribution of duration times and experimented with values around a baseline that seemed to us reasonnable because of the better distribution on the continuum. What worked best was clipping at 60 seconds + simple scaling.
Hi @pdnartreb,
\nThanks very much for the mention. I'm so glad that my inconspicuous sharing can inspire you and other teams to come up with far better solutions. And, I think that's the reason why I learn to share and gradually enjoy sharing!
\nDuring the first month of the competition, I tried techniques like normalization, standardization, quantile transformation, np.log1p, etc., all of which didn't help improve the CV score. Hence, I sticked to the original processing method that clipped duration at 3.6e6 (i.e., one hour). However, I didn't try to clip it with smaller values. I'll try them out and verify more ideas with late submissions!
I learn so so much from your sharing. (1) Instead of just using beautiful visualization to interpret the data, you try hard to understand the mechanism behind raw data generation and clean the data to improve the data quality. (2) In addition to the model architecture design, robust training process and techniques are also crucial (e.g., the loss criterion to use, whether to share \"embedders\" across different level_group, how to choose checkpoints). (3) Create a robust CV strategy and resist the temptation to select submission based on LB. (4) FE plays an important role all the time, to name a few.
Thanks a lot for the clarification, and good luck with your next competition!!
", "rawMarkdown": "Hi @pdnartreb,\n\nThanks very much for the mention. I'm so glad that my inconspicuous sharing can inspire you and other teams to come up with far better solutions. And, I think that's the reason why I learn to share and gradually enjoy sharing!\n\nDuring the first month of the competition, I tried techniques like normalization, standardization, quantile transformation, `np.log1p`, etc., all of which didn't help improve the CV score. Hence, I sticked to the original processing method that clipped `duration` at `3.6e6` (*i.e.,* one hour). However, I didn't try to clip it with smaller values. I'll try them out and verify more ideas with late submissions!\n\nI learn so so much from your sharing. (1) Instead of just using beautiful visualization to interpret the data, you try hard to understand the mechanism behind raw data generation and clean the data to improve the data quality. (2) In addition to the model architecture design, robust training process and techniques are also crucial (*e.g.,* the loss criterion to use, whether to share \"embedders\" across different `level_group`, how to choose checkpoints). (3) Create a robust CV strategy and resist the temptation to select submission based on LB. (4) FE plays an important role all the time, to name a few.\n\nThanks a lot for the clarification, and good luck with your next competition!!", "votes": 1 } ] } ] }, { "id": 2324246, "postDate": "2023-06-30T13:42:44.707Z", "content": "it is very useful content
", "rawMarkdown": "it is very useful content", "votes": 1 }, { "id": 2324241, "postDate": "2023-06-30T13:40:58.287Z", "content": "Congrats and thank you for sharing 🎉 Impressive solution with many creative ideas!
", "rawMarkdown": "Congrats and thank you for sharing 🎉 Impressive solution with many creative ideas!", "votes": 1 }, { "id": 2324151, "postDate": "2023-06-30T12:21:00.883Z", "content": "Congrats, mate!
", "rawMarkdown": "Congrats, mate!", "votes": 1 }, { "id": 2324107, "postDate": "2023-06-30T11:41:11.563Z", "content": "Congratulations! Keep it up!
", "rawMarkdown": "Congratulations! Keep it up!", "votes": 1 }, { "id": 2324083, "postDate": "2023-06-30T11:20:05.807Z", "content": "Congratulations on winning!
", "rawMarkdown": "Congratulations on winning!", "votes": 1 }, { "id": 2323256, "postDate": "2023-06-29T19:18:23.833Z", "content": "Winner winner chicken dinner! :p
", "rawMarkdown": "Winner winner chicken dinner! :p", "votes": 1 }, { "id": 2323126, "postDate": "2023-06-29T17:21:02.953Z", "content": "Congratulations on coming 1st @pdnartreb 🎉. Keep on achieving new milestones 👍
", "rawMarkdown": "Congratulations on coming 1st @pdnartreb 🎉. Keep on achieving new milestones 👍", "votes": 1, "replies": [ { "id": 2324716, "postDate": "2023-06-30T20:22:44.147Z", "content": "It's true he is very detailed in terms of giving knowledge I like
", "rawMarkdown": "It's true he is very detailed in terms of giving knowledge I like ", "votes": 1 } ] }, { "id": 2327560, "postDate": "2023-07-03T03:34:57.963Z", "content": "Congratulations! What a great work! You and your team deserve the 1st place.
\nI am a newbee. May I ask a question about probing the private test?
\nAs you mentioned here:
\n\nProbing showed to us that the private test consists in the 1st 1450/1500 sessions served by the API.
\n
As far as what I know, the public test data can be probed, because the public learderboard give us the score on the public test dataset. But the private leaderboard seems won't give us feedback before the competition ends.
\nSo how do you probe the private test dataset?
\nThanks a lot.
You can switch some predictions and see if the public LB changes at all. If it does not change then the flipped predictions are on the private test set.
", "rawMarkdown": "You can switch some predictions and see if the public LB changes at all. If it does not change then the flipped predictions are on the private test set.", "votes": 2, "replies": [ { "id": 2329321, "postDate": "2023-07-04T07:52:27.473Z", "content": "Really appreciate for the reply.
\nYou mean the whole test dataset is provided in one \"for loop\", after all predictions are made, some are used for pubic score calculation, some are used for private score calculation.
\nDid I understand correctly?
You understood correctly. Private LB score is computed when you submit, but it is only shown when the competition ends.
", "rawMarkdown": "You understood correctly. Private LB score is computed when you submit, but it is only shown when the competition ends.\n", "votes": 1 } ] } ] }, { "id": 2326751, "postDate": "2023-07-02T11:19:42.630Z", "content": "Congratulations! Great work hope to learn from your excellent work！
", "rawMarkdown": "Congratulations! Great work hope to learn from your excellent work！", "votes": 2 }, { "id": 2325063, "postDate": "2023-07-01T05:31:38.557Z", "content": "Thank you for your posting!
\nI have a question about \"for example monitor all folds in a CV (and accept only on majority or consensus)\".
\nAm I correct in understanding that if adding a certain feature improves accuracy for fold1~4 and decreases accuracy for fold5, and as a result there is no change in the overall average score, then the feature is adopted?
", "rawMarkdown": "Thank you for your posting!\n\nI have a question about \"for example monitor all folds in a CV (and accept only on majority or consensus)\".\n\nAm I correct in understanding that if adding a certain feature improves accuracy for fold1~4 and decreases accuracy for fold5, and as a result there is no change in the overall average score, then the feature is adopted?", "votes": 2, "replies": [ { "id": 2325352, "postDate": "2023-07-01T09:14:42.890Z", "content": "Thanks for your question @aesoptacit!
\nThere is not a single response as there are dependancies with the data, the problem, the metric, the level of performance achieved, …
\nIt is unlikely that if 4 folds are improved except 1 the average won't be improved but it is possible. It probably would require to explore why such a dynamic.
In this competition, for the GBDT approach, we mainly worked with 10 bags (maybe 5 would have been enough) and estimated the noise to be ~0.0003. Only > 0.0003 overall improvements have been considered. See the code to look at the outputs we monitored: https://www.kaggle.com/code/pdnartreb/pspfgp-1st-place-gbdt-training. For the NN as we needed to iterate quicker we only used a single bag and only incorporated > 0.0003 overall improvements with at least 3 or 4 (over 5) folds improved.
", "rawMarkdown": "Thanks for your question @aesoptacit!\n\nThere is not a single response as there are dependancies with the data, the problem, the metric, the level of performance achieved, ...\nIt is unlikely that if 4 folds are improved except 1 the average won't be improved but it is possible. It probably would require to explore why such a dynamic.\n\nIn this competition, for the GBDT approach, we mainly worked with 10 bags (maybe 5 would have been enough) and estimated the noise to be ~0.0003. Only > 0.0003 overall improvements have been considered. See the code to look at the outputs we monitored: https://www.kaggle.com/code/pdnartreb/pspfgp-1st-place-gbdt-training. For the NN as we needed to iterate quicker we only used a single bag and only incorporated > 0.0003 overall improvements with at least 3 or 4 (over 5) folds improved.", "votes": 2, "replies": [ { "id": 2325512, "postDate": "2023-07-01T12:26:03.133Z", "content": "Thank you for your detailed explanation!
", "rawMarkdown": "Thank you for your detailed explanation!", "votes": 1 } ] } ] }, { "id": 2324109, "postDate": "2023-06-30T11:46:36.813Z", "content": "Congrats, impressive solution! Thank you for your detailed explanation.💥
", "rawMarkdown": "Congrats, impressive solution! Thank you for your detailed explanation.💥", "votes": 2, "replies": [ { "id": 2324133, "postDate": "2023-06-30T12:07:02.890Z", "content": "+1 congrats and thank you for the explanation
", "rawMarkdown": "+1 congrats and thank you for the explanation", "votes": 1 } ] }, { "id": 2323791, "postDate": "2023-06-30T06:51:19.437Z", "content": "Congrats!
", "rawMarkdown": "Congrats! ", "votes": 2 }, { "id": 2323653, "postDate": "2023-06-30T05:19:24.717Z", "content": "Congrats!!
\nThank you for providing this wonderful solution.
\nI'm very pleased to be able to dig into this solution.
Would I see your entire code??
", "rawMarkdown": "Would I see your entire code??", "votes": 2, "replies": [ { "id": 2323759, "postDate": "2023-06-30T06:38:54.600Z", "content": "We are considering to release the code but are not alone to decide. Stay tuned in the coming week.
", "rawMarkdown": "We are considering to release the code but are not alone to decide. Stay tuned in the coming week.", "votes": 2 }, { "id": 2324014, "postDate": "2023-06-30T10:23:02.350Z", "content": "😃😃😃😃😃
", "rawMarkdown": "😃😃😃😃😃", "votes": 2 } ] } ] }, { "id": 2323625, "postDate": "2023-06-30T04:56:34.523Z", "content": "@cpmpml and @pdnartreb congrats you with 1st place and gold medals!
", "rawMarkdown": "@cpmpml and @pdnartreb congrats you with 1st place and gold medals!", "votes": 2 }, { "id": 2323517, "postDate": "2023-06-30T02:47:35.463Z", "content": "Hi, Congratulations! How much of an effort your team has put in this competition! Your really deserve to win this competition!
\nI have a question which I hope you won't mind answering. In fact, this is a question I would like to ask every Kaggle competition winner…hope you won't mistake me..
\nIn the spirit of competition, the Kaggle community makes tremendous effort to win over others that leads to fighting for third/fourth decimal accuracy, which I consider unavoidable, and may even be good for innovation and further progress in research..
\nBut as a competition host, suppose I ask you: \"For the competition sake, it's ok, but we are really not interested in third or fourth decimal accuracy for deployment in real world environment, but we want a much simpler but very robust model which can be extended for versions / deployed for long term..\" . What would be your recommended model architecture for a best second decimal accuracy for this specific problem? Given the amount of efforts and experimentations you have done, you must have such a solution in mind…Of course, there were thousands of submissions with 0.7 score, but we know all are not necessarily / equally that robust that can be extended to real world changes or that would meet the client requirement..
", "rawMarkdown": "Hi, Congratulations! How much of an effort your team has put in this competition! Your really deserve to win this competition!\n\nI have a question which I hope you won't mind answering. In fact, this is a question I would like to ask every Kaggle competition winner...hope you won't mistake me..\n\nIn the spirit of competition, the Kaggle community makes tremendous effort to win over others that leads to fighting for third/fourth decimal accuracy, which I consider unavoidable, and may even be good for innovation and further progress in research..\n\nBut as a competition host, suppose I ask you: \"For the competition sake, it's ok, but we are really not interested in third or fourth decimal accuracy for deployment in real world environment, but we want a much simpler but very robust model which can be extended for versions / deployed for long term..\" . What would be your recommended model architecture for a best second decimal accuracy for this specific problem? Given the amount of efforts and experimentations you have done, you must have such a solution in mind...Of course, there were thousands of submissions with 0.7 score, but we know all are not necessarily / equally that robust that can be extended to real world changes or that would meet the client requirement..", "votes": 2, "replies": [ { "id": 2323735, "postDate": "2023-06-30T06:33:36.117Z", "content": "Hi @murugesann!
\nThanks!
\nNo problem with your question.
My recommendation is to use our solution that can be considered real world compliant.
\nGiven the fact that predicting the mean of each question lead to ~0.66 score and that the ML models do not learn that much (+0.04), 3rd decimal (+/- 0.002) is what we want to improve. The final LB shows that 3rd decimal is meaningful: for example even if 2nd place private is +101 positions, it is only 0.002 diff with public. 4th decimal is noise. I do not understand why you state that only the 2nd decimal is meaningful, implying that 3rd is just luck. I disagree.
\nMy first competition was https://www.kaggle.com/competitions/microsoft-malware-prediction. The private dataset was totally skewed because the host did not have all the targets so they were put to 0 (https://www.kaggle.com/competitions/microsoft-malware-prediction/discussion/83946). A lot of competitors, including me, had worked very hard (ask to @cdeotte) and the shakeup was monstruous and very hard to accept. I invite you to have a look at @cpmpml results: strong on the public LB and strong on the private LB! Production-ready models, right? Argue that it is just luck is biased. I learned a lot from my failure as well as from @cpmpml and @titericz success. These guys are true legends.
\nLast year I participated in https://www.kaggle.com/competitions/jigsaw-toxic-severity-rating and only entered in the last few days because of @cpmpml feedback: https://www.kaggle.com/competitions/jigsaw-toxic-severity-rating/discussion/304441#1671696. I only submitted twice. Just luck? I would rather say the private test set was pretty unfair and that robustness (my solution was significantly better than every benchmarks used in the field) ensured a good place that was in fact, for me and without modesty, a hugh disappointment. I let you search for the details if you want to understand why: https://www.kaggle.com/competitions/jigsaw-toxic-severity-rating/discussion/306181.
As the context is set, it is now possible to respond to your question:
\n\n\nWhat would be your recommended model architecture for a best second decimal accuracy for this specific problem?
\n
I would recommand to consider 3rd decimal and the model architecture we used. As it is explained (sorry for the english that is far from perfect) in the writeup, we focused on CV scores and just use the LB as a subsidiary information. It means that the private score corresponds to the real world environment you are referring at. Even if luck is present, keep in mind that our solution scores CV 0.705, public LB 0.705 (the winning solution is not our best LB because we thought our best LB overfitted) and private LB 0.705.
\nIf you want a stable/robust solution, I would recommand to blend GBDT and NN because as you need robustness, you need good but also diverse models. That being said proceeding like this you will limit variance so this choice is a tradeoff between ultimate performance and robustness. Real-world requires tradeoffs and robustness and it is the path we chose.
\nHope it helps.
Kaggle always ask winners to provide a simple mode that provides 90% of the solution quality. Exactly your question.
", "rawMarkdown": "Kaggle always ask winners to provide a simple mode that provides 90% of the solution quality. Exactly your question.", "votes": 1, "replies": [ { "id": 2324279, "postDate": "2023-06-30T14:03:25.947Z", "content": "How would you measure the quality?
\nIf that’s the private score, then it is 90% of 0.705, and this is less than the score of submission with the constant mean correctness for each question.
", "rawMarkdown": "How would you measure the quality?\n\nIf that’s the private score, then it is 90% of 0.705, and this is less than the score of submission with the constant mean correctness for each question." }, { "id": 2327313, "postDate": "2023-07-02T20:48:01.177Z", "content": "I guess we can provide a simple model that gives more thna 90% of the quality.
", "rawMarkdown": "I guess we can provide a simple model that gives more thna 90% of the quality.", "votes": 1 } ] } ] }, { "id": 2323230, "postDate": "2023-06-29T18:59:04.270Z", "content": "I have been lucky to team with you!
", "rawMarkdown": "I have been lucky to team with you!", "votes": 2, "replies": [ { "id": 2323264, "postDate": "2023-06-29T19:28:40.540Z", "content": "I, too, think that I have been lucky to team with you!
", "rawMarkdown": "I, too, think that I have been lucky to team with you!", "votes": 1 } ] }, { "id": 2323912, "postDate": "2023-06-30T08:42:58.907Z", "content": "I need help. i am beginner and do not know what to do I want to collect information about Talent acquisition data science fundamentals to learn about them what to write in parameters?
\nimport requests
\nimport time
\nimport json
url = 'https://www.kaggle.com/'
\nheaders = {'Authorization': 'My API'}
\nparams = {'q': ''}
requests_per_minute = 10
\nseconds_per_request = 60 / requests_per_minute
response = requests.get(url, headers=headers, params=params)
\nif response.status_code == 200:
\n # Parse the response
\n data = json.loads(response.content)
\n item data[]:\n (, item[])\n (, item[])\n (, item[])\n ()\nelse:
\n print('Error:', response.status_code)
time.sleep(seconds_per_request)
\nHow is this related to the competition solution we share?
\nThe first thing to learn is to ask questions in the right forum.
\nThe second is to ask clear questions . Your question is impossible to understand.
", "rawMarkdown": "How is this related to the competition solution we share?\n\nThe first thing to learn is to ask questions in the right forum.\n\nThe second is to ask clear questions . Your question is impossible to understand. ", "votes": 1 } ] }, { "id": 2539675, "postDate": "2023-11-27T08:58:44.050Z", "content": "I still don't understand how \"duration\" in your article is derived? Can you explain to me, thank you
", "rawMarkdown": "I still don't understand how \"duration\" in your article is derived? Can you explain to me, thank you" }, { "id": 2343891, "postDate": "2023-07-14T04:56:27.350Z", "content": "Congratulations. And thanks for sharing the solution. The ML competition is great place to learn.
", "rawMarkdown": "Congratulations. And thanks for sharing the solution. The ML competition is great place to learn." }, { "id": 2327283, "postDate": "2023-07-02T20:08:16.840Z", "content": "Congratulations! Great work hope to learn from your excellent workCongratulations! Great work hope to learn from your excellent work SIP
", "rawMarkdown": "Congratulations! Great work hope to learn from your excellent workCongratulations! Great work hope to learn from your excellent work SIP" }, { "id": 3153970, "postDate": "2025-03-19T11:44:35.700Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2327699, "postDate": "2023-07-03T05:49:09.270Z", "rawMarkdown": "", "votes": -1, "isDeleted": true }, { "id": 2327556, "postDate": "2023-07-03T03:33:50.393Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2327659, "postDate": "2023-07-03T05:09:39.983Z", "content": "Thanks for helping us learn!
", "rawMarkdown": "Thanks for helping us learn!", "votes": 3 }, { "id": 3322537, "postDate": "2025-11-13T15:09:47.883Z", "content": "looks great!
", "rawMarkdown": "looks great!" } ], "comments": [ { "id": 2323315, "author_name": "Giba", "author_url": "", "post_date": "2023-06-29T20:59:06.750000", "content": "Huge congrats @cpmpml and @pdnartreb! Thanks for sharing such a great solution. Double congrats for @cpmpml for finally break the 2nd place curse and get your first winning!
", "votes": 7, "replies": [] }, { "id": 2323974, "author_name": "Rongchu", "author_url": "", "post_date": "2023-06-30T09:41:15.223000", "content": "Congrats, impressive solution! Thank you for your detailed explanation.
\nWill you share the kernel? I can't wait to learn.
", "votes": 3, "replies": [ { "id": 2324001, "author_name": "Bertrand P", "author_url": "", "post_date": "2023-06-30T10:01:06.463000", "content": "Thanks @zrongchu!
\nWe have decided to share our code: https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/420332.
Thanks! @pdnartreb , @cpmpml
", "votes": 3, "replies": [] } ] } ] }, { "id": 2339570, "author_name": "Vladimir Simões da Luz Junior", "author_url": "", "post_date": "2023-07-10T22:35:08.770000", "content": "Greetings, @pdnartreb and @cpmpml !
\nI wanted to express my sincere appreciation and congratulations for your outstanding work in this recent competition. Your solution is truly impressive and it's evident that you put a tremendous amount of effort and expertise into developing it.
\nYour focus on understanding the data model and extracting valuable insights is impressive. The combination of GBDT and NN models in your ensemble and search of external feature engineering demonstrates your deep understanding of the problem and your ability to leverage different techniques effectively.
\nI have a question regarding your approach: In the training pipeline, you mentioned that the backbone, representing the events of a level group, is trained on all the available data for that group. Could you elaborate on how you handle the incomplete sessions during this training phase? How do you ensure that the backbone captures the most relevant information from both complete and incomplete sessions?
\nOnce again, congratulations on your remarkable achievement, and thank you for your contributions to the Kaggle community. Your solution is an inspiration to fellow data scientists and serves as a testament to your expertise.
\nBest regards,keep up with the great work🦾🚀
", "votes": 1, "replies": [ { "id": 2340147, "author_name": "CPMP", "author_url": "", "post_date": "2023-07-11T09:58:40.450000", "content": "I am not sure I understand what you don't understand.
\nLet me try still. Maybe this thread can help: https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/420217#2331165
\nFor a given NN there are three NNs trained in the \"pretraining hase\", then they are assembled into a single NN in a second training phase.
\nIncomplete sessions are only used in the pretraining phase.
\nFor instance, if a session has only data for the first level group (i.e. questions 1,2,3), then it is used to train the first of the three NNs, and it is not used for the other two.
", "votes": 3, "replies": [ { "id": 2341136, "author_name": "Vladimir Simões da Luz Junior", "author_url": "", "post_date": "2023-07-11T21:56:24.593000", "content": "Thanks for the quick reply!
\nIndeed the following thread has helped to interpret the model training phase. Specially the following schematics:
\n
\nKudos to @dongyk , for that.
It seems that you have leveraged the full dataset (complete + incomplete sessions), to create specialized models for each level_group, concatenating the inputs. Which enabled the model to capture specific information of all the level_groups independently and all together, by concatenating the model inputs. Therefore you could optimize BCE with F1 score as metric, since it turned out to be a classification problem….
\nOnce again congratulations @pdnartreb and @cpmpml for the excellent work and thanks for contributing to the Kaggle Community🙏🚀
", "votes": 1, "replies": [] }, { "id": 2341645, "author_name": "CPMP", "author_url": "", "post_date": "2023-07-12T07:59:22.233000", "content": "yes, that's what we did:
\n\n", "votes": 2, "replies": [] } ] } ] }, { "id": 2334085, "author_name": "Hamed JOORATI", "author_url": "", "post_date": "2023-07-07T12:53:45.227000", "content": "It seems that you have leveraged the full dataset (complete + incomplete sessions), to create specialized models for each level_group, concatenating the inputs.
\n
Congratulations ! Very interesting !
", "votes": 1, "replies": [] }, { "id": 2333655, "author_name": "UoM_268345H", "author_url": "", "post_date": "2023-07-07T05:31:07.173000", "content": "Congratulations! very helpful.
", "votes": 1, "replies": [] }, { "id": 2332588, "author_name": "Ansh Tanwar", "author_url": "", "post_date": "2023-07-06T09:47:28.037000", "content": "Very insightful
", "votes": 1, "replies": [] }, { "id": 2331909, "author_name": "MG", "author_url": "", "post_date": "2023-07-05T21:38:04.753000", "content": "Congratulations! Really interesting work
", "votes": 1, "replies": [] }, { "id": 2331165, "author_name": "DongYK", "author_url": "", "post_date": "2023-07-05T12:05:07.797000", "content": "Hi, I was wondering whether I understand it correctly.
\nPSPFGP 1st Place - NN Training
outputs = {}\noutputs = heads(convnet_outputs)\noutputs = heads(\n tf()(, convnet_outputs])\n )\noutputs = heads(\n tf()(, convnet_outputs, convnet_outputs])\n )\nDoes it describe like this??
\n
\nBackpropagation is performed:
\nfrom outputs['0-4'] to convnets['0-4'], (I will call this as submodel A)
\nfrom outputs['5-12'] to convnets['0-4'] and convnets['5-12'], and (submodel B)
\nfrom outputs['13-22'] to convnets['0-4'], convnets['5-12'], and convnets['13-22']. (submodel C)
\nHowever, this submodels are separated. So, Backpropagation which is performed from outputs['13-22'] to convnets['0-4'] (submodel C) does not affect convnets['0-4'] of submodel A.
\nDid I understand correctly??
+) I was really surprised this approach to compute F1 score during training. Wow…
", "votes": 1, "replies": [ { "id": 2331653, "author_name": "Bertrand P", "author_url": "", "post_date": "2023-07-05T17:22:41.923000", "content": "Hi @dongyk!
\nYou are working hard. Great! This is really satisfying to see that our work is useful for you. With this mindset you are going to learn a lot.
\nYour understanding and your schematic seem nearly OK.
\nKeep in mind that in step 2 of the training the weights of the ConvNets are freezed. This means backpropagation only changes the heads weights.
\nIn this 2nd step we have 3 sequences of data as inputs, 1 for each level_group. Each of these sequences feeds its own ConvNet/embedder that outputs a representation in the form of a 24 dim vector. The representations for level_group 0-4 feeds the SimpleHead for level_group 0-4 which outputs a vector of 3 values that are the probability of the 3 responses of this level_group. The representations for level_group 0-4 and 5-12 are concatenated in a 48 dim vector the feeds the SimpleHead for the 2nd level_group that outputs 10 values that correspond to the 10 questions of the level_group 5-12. Same for level_group 13-22.
You correctly spotted that this had been made to train the whole ensemble to optimize the overall F1 score that is what we want.
\nWe tried every setup: training the whole ensemble from scratch, each part independently, … The setup we presented here corresponds to what worked best for us.
Does this respond to your question(s)? If not feel free to ask.
", "votes": 1, "replies": [ { "id": 2332166, "author_name": "DongYK", "author_url": "", "post_date": "2023-07-06T03:25:15.107000", "content": "Thank you for your answering!!
\nI'm learning a lot from your code.
\n\nWe tried every setup: training the whole ensemble from scratch, each part independently, … The setup we presented here corresponds to what worked best for us.
\n
I was thinking the same. Did they try the whole ensemble from scratch? Because it's more cumbersome to train seperately. Training separately can be helpful to train.
\nIs the reason why you used only convnet['5-12'] for outputs['5-12'] not contained convnet['0-4']?
\n
And… I editted the figure.
\n
\nI found that you fed train_dataset only once and the same convnet['0-4'] is fed into simplehead['0-4'], simplehead['5-12'], and simplehead['13-22'].
\nIs it more accurate?
\nIf I didn't freeze the convnets, can convnet['0-4'] backpropagated(affected) by not only outputs['0-4'] but outputs['5-12'] and outputs['13-22']?
Wow! Thanks for your schematics that are 100% correct as I understand them! The write-up has been updated to link to your post.
\nIf the weights of the convnets are not freezed then they are updated by the backprop coming from each output they are linked to when training end-to-end.
\nWe first trained separatly which gave a baseline. Then we tried to train the end-to-end setup from scratch. As you might have noticed the output allow to monitor the loss which is composed by the losses of each level group that can also be monitored. We observed that the loss for 0-4 was the first to converge but also that, as the others go down, it tends to increase. We interpreted that as the optimization of previous level groups representation for next level groups. This is not what we wanted. We wanted a good representation for a level group that can be used by the next level groups. So we chose to first optimize each convnet separatly and to exploit its representation by heads in a second time.
\nAs F1 score cannot be optimized directy, the main goal of the end-2-end approach was to be able to monitor the F1 score to select the best weights.
", "votes": 1, "replies": [] }, { "id": 2332884, "author_name": "DongYK", "author_url": "", "post_date": "2023-07-06T13:36:42.110000", "content": "Cool.
\nI feel good to understand it.
\n\nWe observed that the loss for 0-4 was the first to converge but also that, as the others go down, it tends to increase. We interpreted that as the optimization of previous level groups representation for next level groups. This is not what we wanted. We wanted a good representation for a level group that can be used by the next level groups. So we chose to first optimize each convnet separatly and to exploit its representation by heads in a second time.
\n
I also tried to come up with a way to change the architecture of the model, but I realized that it's the best idea considering the pros and cons of possible models.
\n\n\nAs F1 score cannot be optimized directy, the main goal of the end-2-end approach was to be able to monitor the F1 score to select the best weights.
\n
The way I see it, it's the most important approach in the NN code.
\nAgain, thank you so much.
", "votes": 1, "replies": [] } ] } ] }, { "id": 2326886, "author_name": "Mayur B. Ingole", "author_url": "", "post_date": "2023-07-02T13:22:02.613000", "content": "bravo 1St place
", "votes": 1, "replies": [] }, { "id": 2326179, "author_name": "yCarbon", "author_url": "", "post_date": "2023-07-02T00:55:58.820000", "content": "Congratulations on your win🎉 And thanks you also for sharing your solution!
\nIf I may, may I ask a question about the third chapter, the beginning of the Data?
\nDue to the small number of game sessions, the length of the sequence (i.e. the length of the game to be put into the NN) is not long enough. This as not ideal for the Deep Learning approach, I interpreted.
\nIf so, what does it mean by \"Moreover these are long sequences.\"? Does it relevant that in some cases the game is played many times and is actually a long sequence, but what we have as data is a short sequence?
There are too few sequences to enable complex NN model training. But sequence average length is rather long.
", "votes": 2, "replies": [ { "id": 2326787, "author_name": "yCarbon", "author_url": "", "post_date": "2023-07-02T11:59:11.803000", "content": "Thank you. I understood!
", "votes": 1, "replies": [] } ] } ] }, { "id": 2325659, "author_name": "Pranav Jadhav", "author_url": "", "post_date": "2023-07-01T14:02:09.720000", "content": "@pdnartreb 1st place bravo. Great Victory.
", "votes": 1, "replies": [] }, { "id": 2325346, "author_name": "Sercan Yeşilöz", "author_url": "", "post_date": "2023-07-01T09:10:11.110000", "content": "Congratulations on your first place!
", "votes": 1, "replies": [] }, { "id": 2325317, "author_name": "Minze Li", "author_url": "", "post_date": "2023-07-01T08:49:29.843000", "content": "very useful, thank you for your fanscinating work.
", "votes": 1, "replies": [] }, { "id": 2325268, "author_name": "PRIYANSHU YADAV", "author_url": "", "post_date": "2023-07-01T08:07:48.353000", "content": "Such an impressive combination of good ideas and hard hard work.
", "votes": 1, "replies": [] }, { "id": 2325097, "author_name": "Dimas Mufid", "author_url": "", "post_date": "2023-07-01T05:51:09.790000", "content": "Congrats on winning first place! Thanks a lot for your detailed explanation. Impressive! 🔥🔥🔥🔥🔥🔥
", "votes": 1, "replies": [] }, { "id": 2325053, "author_name": "Mohsin hasan", "author_url": "", "post_date": "2023-07-01T05:26:58.413000", "content": "Congrats on the win! Really love the NN design here. I has a very similar NN design, except that I just one-hot encode everything due to low cardinality of categorical features.
", "votes": 1, "replies": [] }, { "id": 2324735, "author_name": "Robert Hatch", "author_url": "", "post_date": "2023-06-30T20:45:31.277000", "content": "Wow, just wow! :D
\nSuch an impressive combination of good ideas and hard hard work. Very inspiring!
", "votes": 1, "replies": [] }, { "id": 2324715, "author_name": "Andriansyah1", "author_url": "", "post_date": "2023-06-30T20:21:55.020000", "content": "This is very cool the understanding is very good and I like this lessonThis is very cool the understanding is very good and I like this lessonThis is very cool the understanding is very good and I like this lesson
", "votes": 1, "replies": [] }, { "id": 2324446, "author_name": "AbaoJiang", "author_url": "", "post_date": "2023-06-30T15:53:44.053000", "content": "Congrats @cpmpml and @pdnartreb for winning the 1st place! Also, thanks a lot for the detailed explanation!
\nMay I ask how you decide to simply divide the numeric features (e.g., duration) by a constant (60000 in the code you open source), instead of using other normalization methods. I suppose that shrinking the scale of the raw feature could make the training process converge faster and other normalization techniques don't show superiority over the way you eventually adopted. If there's misunderstanding, please put me right. Thanks a lot!
Thanks @abaojiang!
\nThe writeup has been updated to mention your excellent notebook. We first saw the time-aware events and the Conv1D ideas in your work. Finding once more these ideas while exploring the litterature likely made us focus more on them.
\nContinuous features are tricky to preprocess. We knew from our GBDT modeling work that time and especially duration was very important. We put a lot of careful efforts to try to model it correctly. Processing the duration seemed important so we tried a lot of things: scaling, normalization, standardization, clipping, … We explored the data to analyze the distribution of duration times and experimented with values around a baseline that seemed to us reasonnable because of the better distribution on the continuum. What worked best was clipping at 60 seconds + simple scaling.
Hi @pdnartreb,
\nThanks very much for the mention. I'm so glad that my inconspicuous sharing can inspire you and other teams to come up with far better solutions. And, I think that's the reason why I learn to share and gradually enjoy sharing!
\nDuring the first month of the competition, I tried techniques like normalization, standardization, quantile transformation, np.log1p, etc., all of which didn't help improve the CV score. Hence, I sticked to the original processing method that clipped duration at 3.6e6 (i.e., one hour). However, I didn't try to clip it with smaller values. I'll try them out and verify more ideas with late submissions!
I learn so so much from your sharing. (1) Instead of just using beautiful visualization to interpret the data, you try hard to understand the mechanism behind raw data generation and clean the data to improve the data quality. (2) In addition to the model architecture design, robust training process and techniques are also crucial (e.g., the loss criterion to use, whether to share \"embedders\" across different level_group, how to choose checkpoints). (3) Create a robust CV strategy and resist the temptation to select submission based on LB. (4) FE plays an important role all the time, to name a few.
Thanks a lot for the clarification, and good luck with your next competition!!
", "votes": 1, "replies": [] } ] } ] }, { "id": 2324246, "author_name": "ANSH TIWARI Student", "author_url": "", "post_date": "2023-06-30T13:42:44.707000", "content": "it is very useful content
", "votes": 1, "replies": [] }, { "id": 2324241, "author_name": "Joel Erikanders", "author_url": "", "post_date": "2023-06-30T13:40:58.287000", "content": "Congrats and thank you for sharing 🎉 Impressive solution with many creative ideas!
", "votes": 1, "replies": [] }, { "id": 2324151, "author_name": "Fahmi Rizal Kurnia", "author_url": "", "post_date": "2023-06-30T12:21:00.883000", "content": "Congrats, mate!
", "votes": 1, "replies": [] }, { "id": 2324107, "author_name": "BELSONRAJA T", "author_url": "", "post_date": "2023-06-30T11:41:11.563000", "content": "Congratulations! Keep it up!
", "votes": 1, "replies": [] }, { "id": 2324083, "author_name": "Piyush Sukhija", "author_url": "", "post_date": "2023-06-30T11:20:05.807000", "content": "Congratulations on winning!
", "votes": 1, "replies": [] }, { "id": 2323256, "author_name": "Patrick Gendotti", "author_url": "", "post_date": "2023-06-29T19:18:23.833000", "content": "Winner winner chicken dinner! :p
", "votes": 1, "replies": [] }, { "id": 2323126, "author_name": "Swapnil Chowdhury", "author_url": "", "post_date": "2023-06-29T17:21:02.953000", "content": "Congratulations on coming 1st @pdnartreb 🎉. Keep on achieving new milestones 👍
", "votes": 1, "replies": [ { "id": 2324716, "author_name": "Andriansyah1", "author_url": "", "post_date": "2023-06-30T20:22:44.147000", "content": "It's true he is very detailed in terms of giving knowledge I like
", "votes": 1, "replies": [] } ] }, { "id": 2327560, "author_name": "huoyeqianxun", "author_url": "", "post_date": "2023-07-03T03:34:57.963000", "content": "Congratulations! What a great work! You and your team deserve the 1st place.
\nI am a newbee. May I ask a question about probing the private test?
\nAs you mentioned here:
\n\nProbing showed to us that the private test consists in the 1st 1450/1500 sessions served by the API.
\n
As far as what I know, the public test data can be probed, because the public learderboard give us the score on the public test dataset. But the private leaderboard seems won't give us feedback before the competition ends.
\nSo how do you probe the private test dataset?
\nThanks a lot.
You can switch some predictions and see if the public LB changes at all. If it does not change then the flipped predictions are on the private test set.
", "votes": 2, "replies": [ { "id": 2329321, "author_name": "huoyeqianxun", "author_url": "", "post_date": "2023-07-04T07:52:27.473000", "content": "Really appreciate for the reply.
\nYou mean the whole test dataset is provided in one \"for loop\", after all predictions are made, some are used for pubic score calculation, some are used for private score calculation.
\nDid I understand correctly?
You understood correctly. Private LB score is computed when you submit, but it is only shown when the competition ends.
", "votes": 1, "replies": [] } ] } ] }, { "id": 2326751, "author_name": "Hoping", "author_url": "", "post_date": "2023-07-02T11:19:42.630000", "content": "Congratulations! Great work hope to learn from your excellent work！
", "votes": 2, "replies": [] }, { "id": 2325063, "author_name": "wakaka", "author_url": "", "post_date": "2023-07-01T05:31:38.557000", "content": "Thank you for your posting!
\nI have a question about \"for example monitor all folds in a CV (and accept only on majority or consensus)\".
\nAm I correct in understanding that if adding a certain feature improves accuracy for fold1~4 and decreases accuracy for fold5, and as a result there is no change in the overall average score, then the feature is adopted?
", "votes": 2, "replies": [ { "id": 2325352, "author_name": "Bertrand P", "author_url": "", "post_date": "2023-07-01T09:14:42.890000", "content": "Thanks for your question @aesoptacit!
\nThere is not a single response as there are dependancies with the data, the problem, the metric, the level of performance achieved, …
\nIt is unlikely that if 4 folds are improved except 1 the average won't be improved but it is possible. It probably would require to explore why such a dynamic.
In this competition, for the GBDT approach, we mainly worked with 10 bags (maybe 5 would have been enough) and estimated the noise to be ~0.0003. Only > 0.0003 overall improvements have been considered. See the code to look at the outputs we monitored: https://www.kaggle.com/code/pdnartreb/pspfgp-1st-place-gbdt-training. For the NN as we needed to iterate quicker we only used a single bag and only incorporated > 0.0003 overall improvements with at least 3 or 4 (over 5) folds improved.
", "votes": 2, "replies": [ { "id": 2325512, "author_name": "wakaka", "author_url": "", "post_date": "2023-07-01T12:26:03.133000", "content": "Thank you for your detailed explanation!
", "votes": 1, "replies": [] } ] } ] }, { "id": 2324109, "author_name": "", "author_url": "", "post_date": "2023-06-30T11:46:36.813000", "content": "", "votes": 2, "replies": [ { "id": 2324133, "author_name": "", "author_url": "", "post_date": "2023-06-30T12:07:02.890000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 2323791, "author_name": "", "author_url": "", "post_date": "2023-06-30T06:51:19.437000", "content": "", "votes": 2, "replies": [] }, { "id": 2323653, "author_name": "", "author_url": "", "post_date": "2023-06-30T05:19:24.717000", "content": "", "votes": 2, "replies": [ { "id": 2323681, "author_name": "", "author_url": "", "post_date": "2023-06-30T05:43:25.340000", "content": "", "votes": 2, "replies": [ { "id": 2323759, "author_name": "", "author_url": "", "post_date": "2023-06-30T06:38:54.600000", "content": "", "votes": 2, "replies": [] }, { "id": 2324014, "author_name": "", "author_url": "", "post_date": "2023-06-30T10:23:02.350000", "content": "", "votes": 2, "replies": [] } ] } ] }, { "id": 2323625, "author_name": "", "author_url": "", "post_date": "2023-06-30T04:56:34.523000", "content": "", "votes": 2, "replies": [] }, { "id": 2323517, "author_name": "", "author_url": "", "post_date": "2023-06-30T02:47:35.463000", "content": "", "votes": 2, "replies": [ { "id": 2323735, "author_name": "", "author_url": "", "post_date": "2023-06-30T06:33:36.117000", "content": "", "votes": 1, "replies": [] }, { "id": 2324098, "author_name": "", "author_url": "", "post_date": "2023-06-30T11:32:20.727000", "content": "", "votes": 1, "replies": [ { "id": 2324279, "author_name": "", "author_url": "", "post_date": "2023-06-30T14:03:25.947000", "content": "", "votes": 0, "replies": [] }, { "id": 2327313, "author_name": "", "author_url": "", "post_date": "2023-07-02T20:48:01.177000", "content": "", "votes": 1, "replies": [] } ] } ] }, { "id": 2323230, "author_name": "", "author_url": "", "post_date": "2023-06-29T18:59:04.270000", "content": "", "votes": 2, "replies": [ { "id": 2323264, "author_name": "", "author_url": "", "post_date": "2023-06-29T19:28:40.540000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 2323912, "author_name": "", "author_url": "", "post_date": "2023-06-30T08:42:58.907000", "content": "", "votes": -1, "replies": [ { "id": 2324096, "author_name": "", "author_url": "", "post_date": "2023-06-30T11:31:11.873000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 2539675, "author_name": "", "author_url": "", "post_date": "2023-11-27T08:58:44.050000", "content": "", "votes": 0, "replies": [] }, { "id": 2343891, "author_name": "", "author_url": "", "post_date": "2023-07-14T04:56:27.350000", "content": "", "votes": 0, "replies": [] }, { "id": 2327283, "author_name": "", "author_url": "", "post_date": "2023-07-02T20:08:16.840000", "content": "", "votes": 0, "replies": [] }, { "id": 3153970, "author_name": "", "author_url": "", "post_date": "2025-03-19T11:44:35.700000", "content": "", "votes": 0, "replies": [] }, { "id": 2327699, "author_name": "", "author_url": "", "post_date": "2023-07-03T05:49:09.270000", "content": "", "votes": -1, "replies": [] }, { "id": 2327556, "author_name": "", "author_url": "", "post_date": "2023-07-03T03:33:50.393000", "content": "", "votes": 0, "replies": [] }, { "id": 2327659, "author_name": "", "author_url": "", "post_date": "2023-07-03T05:09:39.983000", "content": "", "votes": 3, "replies": [] }, { "id": 3322537, "author_name": "", "author_url": "", "post_date": "2025-11-13T15:09:47.883000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2323109": "Unbelievable to write this!\n\n# Thanks!\n\nAs it is the usage, we first **thank the host and Kaggle**. These are special thanks because you and us have had a special link in this competition as we gave you more work by reporting data leaks. No doubt you tried to do your best. You are right to animate this community and to trust in it. You are part of it. Please take care of this community that is able to build so much together by sharing. As all of us you have made mistakes and we hope you will learn from them.\n\nWe also want to **thank all of you**, Kagglers. We love and are grateful to be part of our group/community. Thanks for sharing and for the collective learning experience.\n\n# Context\n\n- Business context: https://www.kaggle.com/competitions/predict-student-performance-from-game-play/overview,\n- Data context: https://www.kaggle.com/competitions/predict-student-performance-from-game-play/data.\n\n# Overview of the Approach\n\nOur solution is essentially a blend of a XGBoost and a NN models. Both heavily rely on duration that appeared to be a powerful leverage. Time was aggregated in different ways and combined with counts for the GBDT while it is transformed via a custom TimeEmbedding block based on 1D convolutions that produce a representation combined with user event representations for the NN.\nRobustness and efficiency founded our work. XGBoost models were validated on 10 bags of 5 folds and features incorporated only if the mean of the CV of these 10 bags was greater than the level of noise we quantified while we opted for a majority/consensus strategy to build the NN, i.e. validate choices only if 4 of 5 folds were improved. The 3rd place of the efficiency LB was achieved with a lightweight NN accelerated via TF Lite.\n\n# Details of the submission\n\n## Code\n\nAfter publishing this write-up we decided to open our code: https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/420332.\nIt is composed by several parts: [how to train the XGBoost models](https://www.kaggle.com/code/pdnartreb/pspfgp-1st-place-gbdt-training), how to [pretrain](https://www.kaggle.com/code/pdnartreb/pspfgp-1st-place-nn-pretraining) and [train](https://www.kaggle.com/code/pdnartreb/pspfgp-1st-place-nn-training) the NN models and the [inference notebook](https://www.kaggle.com/code/pdnartreb/pspfgp-1st-place-inference) used to win this competition.\n\n## Data\n\nLooking at the 1st data released showed that there aren't a lot of sessions so not a lot of sequences. Moreover these are long sequences. This is not ideal for a deep learning approach. \nExploring the Field Day Lab research instructed that the Jo Wilder application was built to help learning to read and that way more than 11,500 learners had played this game. \nThese 2 ideas led to search for a bigger dataset. In 1 Google search and 3 clicks we came up to the open data portal (https://fielddaylab.wisc.edu/opengamedata/) which contains a lot of sessions. 1 hour and 3 bash commands latter we knew that the train set was in part in the open data. So we took a week to **build a pipeline that extracts 98 % of the sessions of the train set perfectly and with minor errors for the last 2 %**. Our data are even better than the comp data because we knew before the host confirmation that for the sessions with 2 games the target was skewed (0 if wrong in 1 of the 2 games when we aim at predicting the responses for the 1st game). It seems that fixing these targets can bring a significant boost up to +0.002.\n\nWe took 1 more week to build a GBDT/XGBoost baseline that would have scored top 10 given the CV score, with the use of the supplemental data (~20,000 sessions) that gave +0.003/0.004 at that time. As we simulated the API locally (see after), we used some training sessions to infer and noticed that it scored 0.718. We were hoping that the LB sessions were not part of the open data portal but our 1st submission, LB 0.708, immediately showed to us that we had rebuilt about a half of the data and especially the targets in the public LB, because 0.708 = (0.698 + 0.718) / 2. The host and Kaggle have been immediately informed. You know what happened next (https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/415820). \nAfter the release of the LB data we measured that we perfectly rebuilt ~7000 sessions over the ~11,500 of the LB data.\n\nWe spent the first month exploring the data until we understood/knew it pretty well. For example we even reconstituted sessions for what might be schools (several games on 1 IP session), extracted every single session with at least 1 answer, ...\n\nAfter the update we made a first submission that scored 0.72. This was shocking because this meant that some leaked data were remaining. A few days later we noticed that the open data was not totally similar with the state we found it 1 month before. A file was missing. So we returned to the host and Kaggle to give them more work.\n\n**This process/work led us to perfectly understand the data model** (that changed since the 1st release of the game). This also allowed us to deeply understand the data itself.\n\nNote that we only used the sessions for which we had responses to all questions of the 2 1st level groups. 1) This is more consistant with the sessions we want to predict (game from the beginning to the end) and 2) this approach preserves performance (vs all data) while reducing training time.\n\nOur dataset is constituted by **37323 complete sessions (23562 comp + logs) in a total of 66376 sessions**.\n\nThe supplemental data (that we fully added 1 month ago) gave us consistently **CV +0.002**.\n\n## Model\n\nOur solution is mainly an ensemble of GBDT + NN models.\n\n### Trust your validation\n\nWe think that **the main reason of the robustness of our solution is that we only relied on CV** for decision making. No choice had been made on LB.\n\nProbing showed to us that the private test consists in the 1st 1450/1500 sessions served by the API. This is a small set. In our experiments 5,000 sessions is the minimum to guarantee a stable CV/LB alignment. A set less than 2,000 is very noisy so **robustness was the way to go**.\n\nWe **only added features that improved the CV for sure**. This is not easy to delete features that you believe in but this is needed as science is not a matter of belief. There are several ways to do so: for example monitor all folds in a CV (and accept only on majority or consensus), monitor several bags (composition of CV to not overfit validation), ...\n\nFor the GBDT approach, we mainly validated on the mean of 10 bags (we defined a bag as a composition of the folds). As we estimated the noise to be ~0.0003, only improvements greater than the noise have been considered. For the NN as we needed to iterate quicker we only used a single bag and only incorporated > 0.0003 overall improvements with at least 3 or 4 (over 5) folds improved.\n\n### Metric\n\nWe experimented a lot on finding a threshold by question but found that this approach is less robust than a single threshold. We mainly used 0.625 as global threshold despite our highest LB scores that were obtained with a threshold per question.\n\n### GBDT\n\nWe prototyped a baseline with **XGBoost because of the structured/tabular nature of the data**. The feature engineering process is interesting to understand what is predictive and to understand the causation, i.e. how the features or decision criteria that enable to predict correctly.\n\nGenerally speaking we followed 3 ways to build features: **business knowledge**, our **intuition** playing the game and a meticulous **exploration of the data**.\nBusiness knowledge refers to using expert knowledge. Reading the papers of the researchers that built this game allow to understand the game beyond usage. For example, Jo Wilder has been built to improve the players reading skills. So this means that the text duration should be important. These are like killer features.\n\nWe exclusively made use of Polars because of the CPU constraints and to simply learn it.\nOur features (663, 1993, 3734 for each level\\_group) are mainly **durations and counts for different aggregations**: how much time in a level, in a room, reading a text, interacting in some way (event type), how many events in a level_group, how many events of each type, how many events of each type in a room or a level, ... \nWe also built a few notebook dedicated features: how many type of events on the notebook in a level, ... \nDespite our efforts we weren't able to extract useful information from the coordinates, the only few features of this type had been mean and std for some events in the activities (journal interactions for example).\n\nWe considered that injecting targets predicted in the previous level groups was a compression of the signal, meaning a loss of information, so we used, for each session, **all interactions from the beginning of the game/session**. This led to a +0.002 at the time of this choice.\n\nAfter the API needed to order the data, we noticed that **models trained both on original order and on index order** but validated on index order (inference order) improved our scores. This leads to more variety that was needed to **improve stability and robustness**. The same goes for the composition of the validation sets: usage of several bags (composition of validation sets) based on the comp data but also on the extracted data improved our scores. We detected late that increasing the number of folds from 5 to 10 could also be leveraged.\n\nThe code for GBDT allows to switch from XGBoost to LightGBM and CatBoost with a simple variable parameter but despite the good scores (~0.001 less than XGBoost), this did not bring to ensemble so we sticked to only XGBoost.\n\nWe experimented a lot around feature selection but were unable to build a stable strategy. So instead of a top-down approach consisting in deleting useless features, we adopted a bottom-up approach choosing carefully each group of features.\n\nOur **XGBoost models score CV ~0.7025 +/-0.0003** and blending 5 of them (the only XGBoost we still have with correct score) scores **LB 0.704**.\n\n### NN\n\nAfter achieving a good score with gradient boosting and having understood well the data we focused on deep learning.\n\nThe **first attempt was with Transformers**. The 1st results were disappointed: CV 0.685 with 2 hours / fold (as far as we can remember). Transformers are very computationally intensive. Resources: https://arxiv.org/pdf/1912.09363.pdf, https://arxiv.org/pdf/2001.08317.pdf, https://arxiv.org/pdf/1711.03905.pdf, https://arxiv.org/pdf/1907.00235.pdf, ...\n\nWe then gave a try to **Conv1D**. In one day we had a very simple model that scored as Transformers but **10x faster** allowing to iterate quicker. So we pushed this approach and could seamlessly scaled it beyond our expectations.\n\nDifficult to share the **tens or hundreds of experimentations** needed to achieve the final solution which is both based on a simple architecture and a slightly complex training pipeline.\n\n#### Architecture roots\n\nWe browsed the literature based on the question: how to model time in deep learning? \nThis research made us come to the idea of **time-aware events** (i.e. https://proceedings.mlr.press/v126/zhang20c/zhang20c.pdf) and back to **WaveNet** (https://arxiv.org/pdf/1609.03499.pdf) because it uses **Conv1D to model long sequences with considerations on causation**. \nOther papers also inspired us: https://arxiv.org/pdf/1703.04691.pdf build on top of WaveNet paper for time series, https://idus.us.es/bitstream/handle/11441/114701/Short-Term%20Load%20Forecasting%20Using%20Encoder-Decoder%20WaveNet.pdf?sequence=1&isAllowed=y also build on top of WaveNet. \nWe also have to mention the excellent work that @abaojiang shared (https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/398565 and https://www.kaggle.com/code/abaojiang/lb-0-694-tconv-with-4-features-training-part). It inspired our research and maybe successfully biased it.\n\nLet's focus on the model of our efficiency submission that is also one of our final ensemble and which performance is nearly the same as models with a few more features.\n\n#### Feature representations\n\n\n\n**5 features as inputs: duration, text\\_fqid, room\\_fqid, fqid, event\\_name + name** (this is the event type from the original data model as far as we remember). Each of these information is encoded/embedded into a vector representation (d_model = 24) to be the merged. The 4 **categorical features feed a classical Embedding layer and the duration a TimeEmbedding** which is a custom block.\n\nDeveloping the GBDT solution showed that the **duration** was crucial, so we put a crucial amount of time trying to model it greatly. The TimeEmbedding layer is a composition of 4x ConvBlock which is inspired by the Transformer main block: Conv1D -> skip connection -> layer norm -> dropout.\n\n```\nclass TimeEmbedding(tf.keras.layers.Layer):\n def __init__(self, n_blocks, d_model, dropout_rate):\n super(TimeEmbedding, self).__init__()\n self.conv_blocks = [ConvBlock(d_model, dropout_rate=dropout_rate) for _ in range(n_blocks)]\n \n def call(self, inputs):\n x = tf.expand_dims(inputs, axis=-1)\n for conv_block in self.conv_blocks:\n x = conv_block(x)\n return x\n```\n\n```\nclass ConvBlock(tf.keras.layers.Layer):\n def __init__(self, d_model, dropout_rate):\n super(ConvBlock, self).__init__()\n self.conv1d = tf.keras.layers.Conv1D(d_model, kernel_size=5, padding='same', activation='gelu')\n self.layer_norm = tf.keras.layers.LayerNormalization()\n self.dropout = tf.keras.layers.Dropout(rate=dropout_rate)\n \n def call(self, inputs):\n x = self.conv1d(inputs)\n x = x + inputs\n x = self.layer_norm(x)\n outputs = self.dropout(x)\n return outputs\n```\n\n\n\n#### Time-aware events\n\nAs said, the goal of building these representations was to model time-aware events. We considered the **categorical features as events** because they represent the user interactions with business entities of the game. We then tried to incorporate duration to make them time-awared. Our main intuition showed to be the best. It is a **simple solution based on operation priority to represent that the duration should be associated to each event before associated them together**: duration * event_1 + duration * event_2 + ... which had been factorized to duration * (event_1 + event_2 + ...).\n\n```\nclass ConvNet(tf.keras.Model):\n def __init__(self, input_dims, n_outputs, d_model, n_blocks=4, name=None):\n super(ConvNet, self).__init__(name=name)\n self.input_dims = input_dims\n self.n_outputs = n_outputs\n self.d_model = d_model\n self.n_blocks = n_blocks\n self.event_embedding = tf.keras.layers.Embedding(input_dims['event_name_name'], d_model, mask_zero=True)\n self.room_embedding = tf.keras.layers.Embedding(input_dims['room_fqid'], d_model, mask_zero=True)\n self.text_embedding = tf.keras.layers.Embedding(input_dims['text'], d_model, mask_zero=True)\n self.fqid_embedding = tf.keras.layers.Embedding(input_dims['fqid'], d_model, mask_zero=True)\n self.duration_embedding = TimeEmbedding(n_blocks=n_blocks, d_model=d_model, dropout_rate=0.2)\n self.gap = tf.keras.layers.GlobalAveragePooling1D()\n \n def call(self, inputs):\n event = self.event_embedding(inputs['event_name_name'])\n room = self.room_embedding(inputs['room_fqid'])\n text = self.text_embedding(inputs['text'])\n fqid = self.fqid_embedding(inputs['fqid'])\n duration = self.duration_embedding(inputs['duration'])\n x = duration * (event + room + text + fqid)\n outputs = self.gap(x)\n return outputs\n\n def get_config(self):\n config = super().get_config().copy()\n config.update({\n 'input_dims': self.input_dims,\n 'n_outputs': self.n_outputs,\n 'd_model': self.d_model,\n 'n_blocks': self.n_blocks,\n 'name': self._name,\n })\n return config\n\n @classmethod\n def from_config(cls, config):\n return cls(**config)\n```\n\n\nThe 2 representations are equivalent: either you can think time-aware events as a combination of time and sub-events or as a combination of sub-events and time.\n\n \n#### Training pipeline\n\nThe training pipeline is not totally straight forward.\n\n@dongyk published great schematics that can be useful to illustrate what is explained bellow: https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/420217#2332166.\n\n##### 1st step (pre-training?)\n\n\n\nThe best approach for us consists in a kind of **backbone that represents the events of a level_group**.\n\nThis backbone is trained on all the data available for this level_group (i.e. on complete + incomplete sessions). It is associated with a temporary SimpleHead optimizing BCE loss.\n\n```\nclass SimpleHead(tf.keras.Model):\n def __init__(self, n_units, n_outputs, name=None):\n super(SimpleHead, self).__init__(name=name)\n self.ffs = [tf.keras.layers.Dense(units, activation='gelu') for units in n_units]\n self.out = tf.keras.layers.Dense(n_outputs, activation='sigmoid')\n \n def call(self, inputs):\n x = inputs\n for ff in self.ffs:\n x = ff(x)\n outputs = self.out(x)\n return outputs\n```\n\nThis approach allows to score **CV 0.70025 +/- 0.0005**.\n\n##### 2nd step (training?)\n\n\n\n**The weights of each of the 3 backbones (1 by level_group) are freezed** for the 2nd level of training to speedup training but also because it is more stable and efficient. These backbones can be thought as \"embedders\".\n\nDuring this 2nd step, **all the submodels that composed the solution were trained on all complete sessions in an end-to-end setup**. The input data are 3 sequences of the 5 features, 1 for each of the 3 level groups. Each \"embedders\" outputs a 24 dim-vector representation. These outputs are the inputs of a head in which enters the representation of level\\_group '0-4' to predict the 3 first questions and the concatenation of the previous and the current representations for level_groups '5-12' and '13-22' to make use of all information.\n\nProceeding like this allows to optimize the overall performance and to monitor it based on the F1 score that is the score of the competition. This means we optimized BCE with F1 score as a metric.\n\nOur winning submission uses a simple **MLP head** but also a **skip head** (512 -> 512 -> 512 allow it for example). **MMoE** did not improve the simplest approaches.\n\nThis approach allows to score **CV 0.70175 +/- 0.0003** which is **comparable to the GBDT solution**.\n\n### Inference\n\n#### Build a simulator\n\nEarly in the competition we built a simulator of the API. Doing so we never experimented any submission error. Maybe trying to keep ideas and code as simple as possible was also key to debug easily.\n\n#### Efficiency\n\nWe invested the efficiency part of the challenge for GBDT as well as NNs. \nUsing **Treelite** for XGBoost allow us to divide by 2 the execution time. \nOur deep learning models were lights: **400,000 weights** for the end-to-end model which combines every parts/sub-models. Having already used **TF Lite** we knew it could be a game changer. Converting our models led to a significant boost in inference time without any performance loss (we do not remember exactly but we think it is at least **6x faster** on our local inference simulator). \nBeginning to explore pruning as well as hard quantization showed that the performance loss would be significant (which is OK in production but not in a competition) so we sticked to a simple TF Lite conversion.\n\nWe have not leveraged what seems to be a problem in the efficiency metric. As we identified the private test sessions to be the 1450/1500 first served by the API we tried to just predict the others to check which time was used (public for public and private for private). Doing so we gain a place but choose to not use this.\n\nOur **efficiency submission is a NN that scores public LB 0.702 and private LB 0.699 in less than 5 minutes**.\n\n#### Ensemble\n\nWe experimented a lot of ensembling alternatives. In the end we sticked to a simple average 50/50 GBDT/NN with:\n\n * 2 kinds of GBDT: trained on original order + trained on index order (validated on index order that is the inference case),\n * 3 kinds of NNs: trained on original order + trained on index order with 5 or all features.\n \nAs our models are lightweight we were able to build a hugh ensemble: **2 x 4 x 10 folds XGBoost + 3 x 4 x 5 folds NNs**. The bottleneck for us is the 8 Go RAM constraint.\n\nThe winning submission scores **CV 0.705, public LB 0.705 and private LB 0.705**.\n\n## Conclusion\n\nThe main achievement of our work is that it is a good solution for the researchers, learners and children that can benefit of it and we hope it will contributes to progresses for a better learning experience. Up to you guys!\n\nThanks if you read until here!\nIf you have any question do not hesitate to ask. We will do our best to respond.\n\n## Presentation to the host\n\nA video presentation to the host has been recorded and can be available on demand. Feel free to ask via PM.\n\n# Sources\n\nBelow are the main sources that we used. More sources can be found in section *Details of the submission* above.\n\n- https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/420332,\n- https://fielddaylab.wisc.edu/opengamedata/,\n- https://arxiv.org/pdf/1609.03499.pdf,\n- https://www.tensorflow.org/lite/guide", "2323315": "Huge congrats @cpmpml and @pdnartreb! Thanks for sharing such a great solution. Double congrats for @cpmpml for finally break the 2nd place curse and get your first winning!", "2323974": "Congrats, impressive solution! Thank you for your detailed explanation. \n\nWill you share the kernel? I can't wait to learn.", "2339570": "Greetings, @pdnartreb and @cpmpml !\n\nI wanted to express my sincere appreciation and congratulations for your outstanding work in this recent competition. Your solution is truly impressive and it's evident that you put a tremendous amount of effort and expertise into developing it.\n\nYour focus on understanding the data model and extracting valuable insights is impressive. The combination of GBDT and NN models in your ensemble and search of external feature engineering demonstrates your deep understanding of the problem and your ability to leverage different techniques effectively.\n\nI have a question regarding your approach: In the training pipeline, you mentioned that the backbone, representing the events of a level group, is trained on all the available data for that group. Could you elaborate on how you handle the incomplete sessions during this training phase? How do you ensure that the backbone captures the most relevant information from both complete and incomplete sessions?\n\nOnce again, congratulations on your remarkable achievement, and thank you for your contributions to the Kaggle community. Your solution is an inspiration to fellow data scientists and serves as a testament to your expertise.\n\nBest regards,keep up with the great work🦾🚀", "2334085": "Congratulations ! Very interesting !", "2333655": "Congratulations! very helpful. ", "2332588": "Very insightful ", "2331909": "Congratulations! Really interesting work", "2331165": "Hi, I was wondering whether I understand it correctly.\n[PSPFGP 1st Place - NN Training](https://www.kaggle.com/code/pdnartreb/pspfgp-1st-place-nn-training)\n```\noutputs = {}\noutputs['0-4'] = heads['0-4'](convnet_outputs['0-4'])\noutputs['5-12'] = heads['5-12'](\n tf.keras.layers.Concatenate()([convnet_outputs['0-4'], convnet_outputs['5-12']])\n )\noutputs['13-22'] = heads['13-22'](\n tf.keras.layers.Concatenate()([convnet_outputs['0-4'], convnet_outputs['5-12'], convnet_outputs['13-22']])\n )\n```\nDoes it describe like this??\n\nBackpropagation is performed: \nfrom outputs['0-4'] to convnets['0-4'], (I will call this as submodel A)\nfrom outputs['5-12'] to convnets['0-4'] and convnets['5-12'], and (submodel B)\nfrom outputs['13-22'] to convnets['0-4'], convnets['5-12'], and convnets['13-22']. (submodel C)\nHowever, this submodels are separated. So, Backpropagation which is performed from outputs['13-22'] to convnets['0-4'] (submodel C) does not affect convnets['0-4'] of submodel A.\nDid I understand correctly??\n\n+) I was really surprised this approach to compute F1 score during training. Wow...", "2326886": "bravo 1St place", "2326179": "Congratulations on your win🎉 And thanks you also for sharing your solution!\n\nIf I may, may I ask a question about the third chapter, the beginning of the **Data**?\n\nDue to the small number of game sessions, the length of the sequence (i.e. the length of the game to be put into the NN) is not long enough. This as not ideal for the Deep Learning approach, I interpreted.\nIf so, what does it mean by \"Moreover these are long sequences.\"? Does it relevant that in some cases the game is played many times and is actually a long sequence, but what we have as data is a short sequence?", "2325659": "@pdnartreb 1st place bravo. Great Victory.", "2325346": "Congratulations on your first place!", "2325317": "very useful, thank you for your fanscinating work.", "2325268": "Such an impressive combination of good ideas and hard hard work.", "2325097": "Congrats on winning first place! Thanks a lot for your detailed explanation. Impressive! 🔥🔥🔥🔥🔥🔥", "2325053": "Congrats on the win! Really love the NN design here. I has a very similar NN design, except that I just one-hot encode everything due to low cardinality of categorical features.", "2324735": "Wow, just wow! :D\n\nSuch an impressive combination of good ideas and hard hard work. Very inspiring!", "2324715": "This is very cool the understanding is very good and I like this lessonThis is very cool the understanding is very good and I like this lessonThis is very cool the understanding is very good and I like this lesson", "2324446": "Congrats @cpmpml and @pdnartreb for winning the 1st place! Also, thanks a lot for the detailed explanation!\n\nMay I ask how you decide to simply divide the numeric features (*e.g.,* `duration`) by a constant (60000 in the code you open source), instead of using other normalization methods. I suppose that shrinking the scale of the raw feature could make the training process converge faster and other normalization techniques don't show superiority over the way you eventually adopted. If there's misunderstanding, please put me right. Thanks a lot!", "2324246": "it is very useful content", "2324241": "Congrats and thank you for sharing 🎉 Impressive solution with many creative ideas!", "2324151": "Congrats, mate!", "2324107": "Congratulations! Keep it up!", "2324083": "Congratulations on winning!", "2323256": "Winner winner chicken dinner! :p", "2323126": "Congratulations on coming 1st @pdnartreb 🎉. Keep on achieving new milestones 👍", "2327560": "Congratulations! What a great work! You and your team deserve the 1st place.\nI am a newbee. May I ask a question about probing the private test?\nAs you mentioned here:\n>Probing showed to us that the private test consists in the 1st 1450/1500 sessions served by the API.\n\nAs far as what I know, the public test data can be probed, because the public learderboard give us the score on the public test dataset. But the private leaderboard seems won't give us feedback before the competition ends.\nSo how do you probe the private test dataset?\nThanks a lot.", "2326751": "Congratulations! Great work hope to learn from your excellent work！", "2325063": "Thank you for your posting!\n\nI have a question about \"for example monitor all folds in a CV (and accept only on majority or consensus)\".\n\nAm I correct in understanding that if adding a certain feature improves accuracy for fold1~4 and decreases accuracy for fold5, and as a result there is no change in the overall average score, then the feature is adopted?", "2324109": "Congrats, impressive solution! Thank you for your detailed explanation.💥", "2323791": "Congrats! ", "2323653": "Congrats!!\nThank you for providing this wonderful solution.\nI'm very pleased to be able to dig into this solution.", "2323625": "@cpmpml and @pdnartreb congrats you with 1st place and gold medals!", "2323517": "Hi, Congratulations! How much of an effort your team has put in this competition! Your really deserve to win this competition!\n\nI have a question which I hope you won't mind answering. In fact, this is a question I would like to ask every Kaggle competition winner...hope you won't mistake me..\n\nIn the spirit of competition, the Kaggle community makes tremendous effort to win over others that leads to fighting for third/fourth decimal accuracy, which I consider unavoidable, and may even be good for innovation and further progress in research..\n\nBut as a competition host, suppose I ask you: \"For the competition sake, it's ok, but we are really not interested in third or fourth decimal accuracy for deployment in real world environment, but we want a much simpler but very robust model which can be extended for versions / deployed for long term..\" . What would be your recommended model architecture for a best second decimal accuracy for this specific problem? Given the amount of efforts and experimentations you have done, you must have such a solution in mind...Of course, there were thousands of submissions with 0.7 score, but we know all are not necessarily / equally that robust that can be extended to real world changes or that would meet the client requirement..", "2323230": "I have been lucky to team with you!", "2323912": "I need help. i am beginner and do not know what to do I want to collect information about Talent acquisition data science fundamentals to learn about them what to write in parameters?\nimport requests\nimport time\nimport json\n\n# Set up API request parameters\nurl = 'https://www.kaggle.com/'\nheaders = {'Authorization': 'My API'}\nparams = {'q': ''}\n\n# Set up rate limiting parameters\nrequests_per_minute = 10\nseconds_per_request = 60 / requests_per_minute\n\n# Send API request and get response\nresponse = requests.get(url, headers=headers, params=params)\n\n# Check if the request was successful\nif response.status_code == 200:\n # Parse the response\n data = json.loads(response.content)\n\n # Extract relevant information\n for item in data['items']:\n print('Title:', item['title'])\n print('Link:', item['link'])\n print('Description:', item['snippet'])\n print('---')\nelse:\n print('Error:', response.status_code)\n\n# Wait to ensure rate limiting compliance\ntime.sleep(seconds_per_request)\n# This script scrapes data from example.com for research purposes.\n# All data is used with permission and proper attribution is given.", "2539675": "I still don't understand how \"duration\" in your article is derived? Can you explain to me, thank you", "2343891": "Congratulations. And thanks for sharing the solution. The ML competition is great place to learn.", "2327283": "Congratulations! Great work hope to learn from your excellent workCongratulations! Great work hope to learn from your excellent work SIP", "3153970": "", "2327699": "", "2327556": "", "2327659": "Thanks for helping us learn!", "3322537": "looks great!" } }