{ "id": 395250, "title": "Dangers in Data", "url": "/competitions/predict-student-performance-from-game-play/discussion/395250", "author_name": "", "post_date": "2023-03-16T14:36:28.638139600Z", "votes": 73, "comment_count": 20, "views": 0, "content": "

Hi everyone,

\n

(Update, 03/21): Because the competition host has announced the data update policy here, I decide to re-run this notebook to make exploration results aligned with the newest data.

\n

The competition has lasted for over one month, but some of the important data issues remain unsolved. Hence, I decide to restate these problems to facilitate more discussions. Also, I hope this topic could help those not joining at the start of the competition have a quick understanding about the potential dangers in data. For detailed implementation, please refer to EDA on Game Progress\n.

\n

index and elapsed_time

\n

As shown in the data tab, the meaning of columns index and elapsed_time are:

\n\n

It's reasonable to assume that the (session_id, index) pairs can uniquely identify data samples, but this isn't the case. Duplicated (session_id, index) pairs exist. In addition, elapsed_time by definition should be monotonically increasing, which doesn't always hold in data. Lastly, as the indicator of the event triggered during the game play, index should increase by one for each step. Again, it fails.

\n

Concretely, issues related to index and elapsed_time are summarized below with the host's answers here:

\n
    \n
  1. 142 sessions have duplicated (session_id, index) pairs.
    2. \n
  2. elapsed_time isn't always monotonically increasing.
    3. \n
  3. 258 sessions have reversed index phenomenon.
    \n\"reversed-idx.png\"
    4. \n
  4. index can increase more than one for one step (98.46% jumps occur after checkpoint).
    5. \n
  5. index == 0 doesn't always exist in the session.
    6. \n
\n

level and level_group

\n

As the game progress, level should gradually increase, which indicates level-ups. But, there exist some sessions with reversed level and reversed level group phenomenon. Specifically, issues related to level and level_group are introduced as follows:

\n
    \n
  1. 486 out of 23562(about 2.06%) sessions have reversed level phenomenon.
    2. \n
  2. 442 out of 23562 (about 1.88%) sessions have reversed level group phenomenon.
    \n\"reversed-lv-gp.png\"
    3. \n
  3. Some sessions have missing level, details can be found here.
    4. \n
\n

So, the reason why these two numbers are different is that reversed index doesn't always imply reversed index group. In other words, level can fall back to some specific points within the same level_group, which is illustrated as follows:
\n\"reversed-lv-only.png\"

\n

One of the reasonable assumption is that these sessions have multiple game plays in a row. We should note the potential data leakage caused by these phenomena. For more detailed information, please refer to the original issue report by @cdeotte here. So far, we haven't got response from the host.

\n

checkpoint

\n

checkpoint is an indicator of question prompts at the end of each level_group. For a normal game play, we can assume there are 3 checkpoints, each of which corresponds to one of the level_group (i.e., \"0-4\", \"5-12\" and \"13-22\"). However, there exist sessions with less than or more than 3 checkpoints. To be concrete, issues related to checkpoint are summarized as follows:

\n
    \n
  1. 99.29% of the sessions have 3 checkpoints.
    2. \n
  2. 1 session (i.e., session 22090108192456930) has only 2 checkpoints, missing level_group == \"0-4\".
    3. \n
  3. 167 sessions have more than 3 checkpoints.
    \n\"n-ckpts-2.png\"
    4. \n
\n

Combined with issues related to level and level_group, we can see that even if one session has exactly 3 checkpoints, level or level_group can still fall back. Hence, checking number of checkpoints in one session may not be a good way to ensure the validity of the session.

\n

Hope this brief writeup could help you quickly understand the potential dangers in the data. If there's any question, please feel free to comment below. Thanks a lot!

", "messages": [ { "id": "2184672", "postDate": "03/16/2023 14:36:28", "content": "

Hi everyone,

\n

(Update, 03/21): Because the competition host has announced the data update policy here, I decide to re-run this notebook to make exploration results aligned with the newest data.

\n

The competition has lasted for over one month, but some of the important data issues remain unsolved. Hence, I decide to restate these problems to facilitate more discussions. Also, I hope this topic could help those not joining at the start of the competition have a quick understanding about the potential dangers in data. For detailed implementation, please refer to EDA on Game Progress\n.

\n

index and elapsed_time

\n

As shown in the data tab, the meaning of columns index and elapsed_time are:

\n\n

It's reasonable to assume that the (session_id, index) pairs can uniquely identify data samples, but this isn't the case. Duplicated (session_id, index) pairs exist. In addition, elapsed_time by definition should be monotonically increasing, which doesn't always hold in data. Lastly, as the indicator of the event triggered during the game play, index should increase by one for each step. Again, it fails.

\n

Concretely, issues related to index and elapsed_time are summarized below with the host's answers here:

\n
    \n
  1. 142 sessions have duplicated (session_id, index) pairs.
    2. \n
  2. elapsed_time isn't always monotonically increasing.
    3. \n
  3. 258 sessions have reversed index phenomenon.
    \n\"reversed-idx.png\"
    4. \n
  4. index can increase more than one for one step (98.46% jumps occur after checkpoint).
    5. \n
  5. index == 0 doesn't always exist in the session.
    6. \n
\n

level and level_group

\n

As the game progress, level should gradually increase, which indicates level-ups. But, there exist some sessions with reversed level and reversed level group phenomenon. Specifically, issues related to level and level_group are introduced as follows:

\n
    \n
  1. 486 out of 23562(about 2.06%) sessions have reversed level phenomenon.
    2. \n
  2. 442 out of 23562 (about 1.88%) sessions have reversed level group phenomenon.
    \n\"reversed-lv-gp.png\"
    3. \n
  3. Some sessions have missing level, details can be found here.
    4. \n
\n

So, the reason why these two numbers are different is that reversed index doesn't always imply reversed index group. In other words, level can fall back to some specific points within the same level_group, which is illustrated as follows:
\n\"reversed-lv-only.png\"

\n

One of the reasonable assumption is that these sessions have multiple game plays in a row. We should note the potential data leakage caused by these phenomena. For more detailed information, please refer to the original issue report by @cdeotte here. So far, we haven't got response from the host.

\n

checkpoint

\n

checkpoint is an indicator of question prompts at the end of each level_group. For a normal game play, we can assume there are 3 checkpoints, each of which corresponds to one of the level_group (i.e., \"0-4\", \"5-12\" and \"13-22\"). However, there exist sessions with less than or more than 3 checkpoints. To be concrete, issues related to checkpoint are summarized as follows:

\n
    \n
  1. 99.29% of the sessions have 3 checkpoints.
    2. \n
  2. 1 session (i.e., session 22090108192456930) has only 2 checkpoints, missing level_group == \"0-4\".
    3. \n
  3. 167 sessions have more than 3 checkpoints.
    \n\"n-ckpts-2.png\"
    4. \n
\n

Combined with issues related to level and level_group, we can see that even if one session has exactly 3 checkpoints, level or level_group can still fall back. Hence, checking number of checkpoints in one session may not be a good way to ensure the validity of the session.

\n

Hope this brief writeup could help you quickly understand the potential dangers in the data. If there's any question, please feel free to comment below. Thanks a lot!

", "rawMarkdown": "Hi everyone,\n\n**(Update, 03/21):** Because the competition host has announced the data update policy [here](https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/396202), I decide to re-run this notebook to make exploration results aligned with the newest data.\n\nThe competition has lasted for over one month, but some of the important data issues remain unsolved. Hence, I decide to restate these problems to facilitate more discussions. Also, I hope this topic could help those not joining at the start of the competition have a quick understanding about the potential dangers in data. For detailed implementation, please refer to [EDA on Game Progress\n](https://www.kaggle.com/code/abaojiang/eda-on-game-progress/notebook?scriptVersionId=122864479).\n\n### `index` and `elapsed_time`\nAs shown in the [data tab](https://www.kaggle.com/competitions/predict-student-performance-from-game-play/data), the meaning of columns `index` and `elapsed_time` are:\n* `index`: the index of the event for the session\n* `elapsed_time` - how much time has passed (in milliseconds) between the start of the session and when the event was recorded\n\nIt's reasonable to assume that the (`session_id`, `index`) pairs can uniquely identify data samples, but this isn't the case. Duplicated (`session_id`, `index`) pairs exist. In addition, `elapsed_time` by definition should be **monotonically increasing**, which doesn't always hold in data. Lastly, as the indicator of the event triggered during the game play, `index` should increase by one for each step. Again, it fails.\n\nConcretely, issues related to `index` and `elapsed_time` are summarized below with the host's answers [here](https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/384342#2134312):\n1. 142 sessions have duplicated (`session_id`, `index`) pairs.\n * The reason why duplication exists remains unknown.\n2. `elapsed_time` isn't always monotonically increasing.\n * The `index` gives the correct order of events in cases where the `elapsed_time` is inconsistent\n3. 258 sessions have **reversed index** phenomenon.\n[![reversed-idx.png](https://i.postimg.cc/Vs0z9j25/reversed-idx.png)](https://postimg.cc/jCKVtJxr)\n4. `index` can increase more than one for one step (98.46% jumps occur after `checkpoint`).\n5. `index == 0` doesn't always exist in the session.\n * `index == 0` should indicate the start of the session.\n\n### `level` and `level_group`\nAs the game progress, `level` should gradually increase, which indicates level-ups. But, there exist some sessions with **reversed level** and **reversed level group** phenomenon. Specifically, issues related to `level` and `level_group` are introduced as follows:\n1. 486 out of 23562(about 2.06%) sessions have **reversed level** phenomenon.\n2. 442 out of 23562 (about 1.88%) sessions have **reversed level group** phenomenon.\n[![reversed-lv-gp.png](https://i.postimg.cc/nrL274Hk/reversed-lv-gp.png)](https://postimg.cc/phgKMn5h)\n3. Some sessions have missing `level`, details can be found [here](https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/390339).\n * The host has confirmed that **all gameplay sessions don't use save codes in both training and test data**.\n\nSo, the reason why these two numbers are different is that **reversed index** doesn't always imply **reversed index group**. In other words, `level` can fall back to some specific points within the same `level_group`, which is illustrated as follows:\n[![reversed-lv-only.png](https://i.postimg.cc/fRzFC9B6/reversed-lv-only.png)](https://postimg.cc/6yPY5ys0)\n\nOne of the reasonable assumption is that these sessions have **multiple game plays** in a row. We should note the potential **data leakage** caused by these phenomena. For more detailed information, please refer to the original issue report by [@cdeotte](https://www.kaggle.com/cdeotte) [here](https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/388479#2149959). So far, we haven't got response from the host.\n\n### `checkpoint`\n`checkpoint` is an indicator of question prompts at the end of each `level_group`. For a normal game play, we can assume there are 3 checkpoints, each of which corresponds to one of the `level_group` (*i.e.,* `\"0-4\"`, `\"5-12\"` and `\"13-22\"`). However, there exist sessions with less than or more than 3 checkpoints. To be concrete, issues related to `checkpoint` are summarized as follows:\n1. 99.29% of the sessions have 3 checkpoints.\n2. 1 session (*i.e.,* session `22090108192456930`) has only 2 checkpoints, missing `level_group == \"0-4\"`.\n3. 167 sessions have more than 3 checkpoints.\n[![n-ckpts-2.png](https://i.postimg.cc/mgHZXZcq/n-ckpts-2.png)](https://postimg.cc/BjJGt0Bc)\n\nCombined with issues related to `level` and `level_group`, we can see that even if one session has exactly 3 checkpoints, `level` or `level_group` can still fall back. Hence, checking number of checkpoints in one session may not be a good way to ensure the validity of the session.\n\nHope this brief writeup could help you quickly understand the potential dangers in the data. If there's any question, please feel free to comment below. Thanks a lot!", "votes": null }, { "id": "2186338", "postDate": "03/17/2023 18:06:51", "content": "

Hi, this discussion and your notebook both are very helpful. Thank you.

\n

From what I've observed, elapsed_time is a really important feature in this competiton (Maybe even the magic feature :0). So how are you dealing with the error of the values not monotonically increasing?

", "rawMarkdown": "Hi, this discussion and your notebook both are very helpful. Thank you.\n\nFrom what I've observed, elapsed_time is a really important feature in this competiton (Maybe even the magic feature :0). So how are you dealing with the error of the values not monotonically increasing?", "votes": null }, { "id": "2187229", "postDate": "03/18/2023 14:14:52", "content": "

Hi @shashwatraman,

\n

Thanks for your appreciation! So far, I've built some DL-based models to try to capture temporal patterns hidden behind the sequence of elapsed_time difference. Just as you mention, elapsed_time seems to be a key feature. As for the negative elapsed_time difference, I've only tried two methods so far:

\n
    \n
  1. Clip all negative difference to zero.
    2. \n
  2. Use the median derived from each (level, event_name) group to replace the negative entries.
    3. \n
\n

For experimental results, they are nearly the same. There must exist better ways to handle this issue, so I'll keep exploring. Hope I can share something effective in the near future. Thanks a lot!

", "rawMarkdown": "Hi @shashwatraman,\n\nThanks for your appreciation! So far, I've built some DL-based models to try to capture temporal patterns hidden behind the sequence of `elapsed_time` difference. Just as you mention, `elapsed_time` seems to be a key feature. As for the **negative** `elapsed_time` difference, I've only tried two methods so far:\n1. Clip all negative difference to zero.\n2. Use the median derived from each (`level`, `event_name`) group to replace the negative entries.\n\nFor experimental results, they are nearly the same. There must exist better ways to handle this issue, so I'll keep exploring. Hope I can share something effective in the near future. Thanks a lot!", "votes": null }, { "id": "2187240", "postDate": "03/18/2023 14:34:17", "content": "

Thank you for the reply!
\nI tried to find if these problems are also present in the test data. I think they are. Please check out my discussion here.
\nThanks!!

", "rawMarkdown": "Thank you for the reply! \nI tried to find if these problems are also present in the test data. I think they are. Please check out my discussion [here.](https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/395686)\nThanks!!", "votes": null }, { "id": "2187938", "postDate": "03/19/2023 06:39:36", "content": "

Hi @shashwatraman,

\n

I've checked out your works. Very helpful! I still focus on local development to find out how to clean the data better. Now, it seems that we should pay more attention to how test data will be presented (maybe wait for response from the host๐Ÿ‘€). Any way, thanks a lot for your hard works and sharing!

", "rawMarkdown": "Hi @shashwatraman,\n\nI've checked out your works. Very helpful! I still focus on local development to find out how to clean the data better. Now, it seems that we should pay more attention to how test data will be presented (maybe wait for response from the host๐Ÿ‘€). Any way, thanks a lot for your hard works and sharing!", "votes": null }, { "id": "2188277", "postDate": "03/19/2023 12:59:38", "content": "

Thanks @abaojiang, dear for letting us informed. This post is full of deep insights and it can create lot of ways to solve the remaining problems in the notebook. It is interesting to work in. Thanks๐Ÿ‘

", "rawMarkdown": "Thanks @abaojiang, dear for letting us informed. This post is full of deep insights and it can create lot of ways to solve the remaining problems in the notebook. It is interesting to work in. Thanks๐Ÿ‘", "votes": null }, { "id": "2188861", "postDate": "03/20/2023 03:52:39", "content": "

Hi @tariqbashir,

\n

Thanks for your reply. Glad it helps!๐Ÿ˜Š

", "rawMarkdown": "Hi @tariqbashir,\n\nThanks for your reply. Glad it helps!๐Ÿ˜Š", "votes": null }, { "id": "2189395", "postDate": "03/20/2023 13:11:15", "content": "

Thanks for the Information. As I found same issue,I was looking for people who experienced same issue. It was really helpful! @abaojiang

", "rawMarkdown": "Thanks for the Information. As I found same issue,I was looking for people who experienced same issue. It was really helpful! @abaojiang", "votes": null }, { "id": "2190017", "postDate": "03/21/2023 00:55:13", "content": "

Thank you @abaojiang for your summarizing data quality issue! Hope competition hosts announce their plans on this issue.

", "rawMarkdown": "Thank you @abaojiang for your summarizing data quality issue! Hope competition hosts announce their plans on this issue.", "votes": null }, { "id": "2190124", "postDate": "03/21/2023 04:00:11", "content": "

Interesting! thank you for sharing your opinions! @abaojiang

", "rawMarkdown": "Interesting! thank you for sharing your opinions! @abaojiang", "votes": null }, { "id": "2190246", "postDate": "03/21/2023 06:17:24", "content": "

Hi @johnthethrid,

\n

Thanks for your appreciation, and please see the update from the host!

", "rawMarkdown": "Hi @johnthethrid,\n\nThanks for your appreciation, and please see the [update](https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/396202) from the host!", "votes": null }, { "id": "2190248", "postDate": "03/21/2023 06:18:32", "content": "

Hi @arti1117,

\n

Glad it helps, and please see the update from the host!

", "rawMarkdown": "Hi @arti1117,\n\nGlad it helps, and please see the [update](https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/396202) from the host!", "votes": null }, { "id": "2190251", "postDate": "03/21/2023 06:20:28", "content": "

Hi @tomokikmogura,

\n

Thanks for your reply. The competition host has announced the data update here, but we still need to re-check the aforementioned issues!

", "rawMarkdown": "Hi @tomokikmogura,\n\nThanks for your reply. The competition host has announced the data update [here](https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/396202), but we still need to re-check the aforementioned issues!", "votes": null }, { "id": "2191745", "postDate": "03/22/2023 07:18:00", "content": "

thank you for sharing your opinions! @abaojiang

", "rawMarkdown": "thank you for sharing your opinions! @abaojiang", "votes": null }, { "id": "2194748", "postDate": "03/24/2023 06:12:43", "content": "

Hi @thitiwat,

\n

Thanks for your appreciation!

", "rawMarkdown": "Hi @thitiwat,\n\nThanks for your appreciation!", "votes": null }, { "id": "2218155", "postDate": "04/11/2023 13:17:54", "content": "

Interesting! thank you for sharing your opinions!

", "rawMarkdown": "Interesting! thank you for sharing your opinions!", "votes": null }, { "id": "2219982", "postDate": "04/13/2023 03:53:02", "content": "

Thank you for your valuable information. I am impressed with the easy to read graph you created. Was this created in python? The colors are beautiful and I was able to discover that it is possible to do such a thing.

", "rawMarkdown": "Thank you for your valuable information. I am impressed with the easy to read graph you created. Was this created in python? The colors are beautiful and I was able to discover that it is possible to do such a thing.", "votes": null }, { "id": "2221062", "postDate": "04/14/2023 00:16:59", "content": "

thank you for the great EDA.
\nI had been confused by the session_id and index.
\nReally helpful:)

", "rawMarkdown": "thank you for the great EDA.\nI had been confused by the session_id and index.\nReally helpful:)", "votes": null }, { "id": "2221163", "postDate": "04/14/2023 04:13:22", "content": "

Hi @risakashiwabara,

\n

These graphs are plotted with matplotlib and seaborn packages in python. The implementation can be found here. Glad it helps, thanks a lot !

", "rawMarkdown": "Hi @risakashiwabara,\n\nThese graphs are plotted with `matplotlib` and `seaborn` packages in python. The implementation can be found [here](https://www.kaggle.com/code/abaojiang/eda-on-game-progress/notebook?scriptVersionId=122864479). Glad it helps, thanks a lot !", "votes": null }, { "id": "2221164", "postDate": "04/14/2023 04:14:37", "content": "

Hi @yutodennou,

\n

Thanks for your appreciation!

", "rawMarkdown": "Hi @yutodennou,\n\nThanks for your appreciation!", "votes": null }, { "id": "2221166", "postDate": "04/14/2023 04:15:10", "content": "

Thanks @byungeunhwang ๐Ÿ˜Š

", "rawMarkdown": "Thanks @byungeunhwang ๐Ÿ˜Š", "votes": null } ], "comments": [ { "id": 2186338, "author_name": "shashwatraman", "author_url": "", "post_date": "03/17/2023 18:06:51", "content": "

Hi, this discussion and your notebook both are very helpful. Thank you.

\n

From what I've observed, elapsed_time is a really important feature in this competiton (Maybe even the magic feature :0). So how are you dealing with the error of the values not monotonically increasing?

", "votes": null, "replies": [ { "id": 2187229, "author_name": "abaojiang", "author_url": "", "post_date": "03/18/2023 14:14:52", "content": "

Hi @shashwatraman,

\n

Thanks for your appreciation! So far, I've built some DL-based models to try to capture temporal patterns hidden behind the sequence of elapsed_time difference. Just as you mention, elapsed_time seems to be a key feature. As for the negative elapsed_time difference, I've only tried two methods so far:

\n
    \n
  1. Clip all negative difference to zero.
    2. \n
  2. Use the median derived from each (level, event_name) group to replace the negative entries.
    3. \n
\n

For experimental results, they are nearly the same. There must exist better ways to handle this issue, so I'll keep exploring. Hope I can share something effective in the near future. Thanks a lot!

", "votes": null, "replies": [ { "id": 2187240, "author_name": "shashwatraman", "author_url": "", "post_date": "03/18/2023 14:34:17", "content": "

Thank you for the reply!
\nI tried to find if these problems are also present in the test data. I think they are. Please check out my discussion here.
\nThanks!!

", "votes": null, "replies": [ { "id": 2187938, "author_name": "abaojiang", "author_url": "", "post_date": "03/19/2023 06:39:36", "content": "

Hi @shashwatraman,

\n

I've checked out your works. Very helpful! I still focus on local development to find out how to clean the data better. Now, it seems that we should pay more attention to how test data will be presented (maybe wait for response from the host๐Ÿ‘€). Any way, thanks a lot for your hard works and sharing!

", "votes": null, "replies": [] } ] } ] } ] }, { "id": 2188277, "author_name": "tariqbashir", "author_url": "", "post_date": "03/19/2023 12:59:38", "content": "

Thanks @abaojiang, dear for letting us informed. This post is full of deep insights and it can create lot of ways to solve the remaining problems in the notebook. It is interesting to work in. Thanks๐Ÿ‘

", "votes": null, "replies": [ { "id": 2188861, "author_name": "abaojiang", "author_url": "", "post_date": "03/20/2023 03:52:39", "content": "

Hi @tariqbashir,

\n

Thanks for your reply. Glad it helps!๐Ÿ˜Š

", "votes": null, "replies": [] } ] }, { "id": 2189395, "author_name": "johnthethrid", "author_url": "", "post_date": "03/20/2023 13:11:15", "content": "

Thanks for the Information. As I found same issue,I was looking for people who experienced same issue. It was really helpful! @abaojiang

", "votes": null, "replies": [ { "id": 2190246, "author_name": "abaojiang", "author_url": "", "post_date": "03/21/2023 06:17:24", "content": "

Hi @johnthethrid,

\n

Thanks for your appreciation, and please see the update from the host!

", "votes": null, "replies": [] } ] }, { "id": 2190017, "author_name": "tomokikmogura", "author_url": "", "post_date": "03/21/2023 00:55:13", "content": "

Thank you @abaojiang for your summarizing data quality issue! Hope competition hosts announce their plans on this issue.

", "votes": null, "replies": [ { "id": 2190251, "author_name": "abaojiang", "author_url": "", "post_date": "03/21/2023 06:20:28", "content": "

Hi @tomokikmogura,

\n

Thanks for your reply. The competition host has announced the data update here, but we still need to re-check the aforementioned issues!

", "votes": null, "replies": [] } ] }, { "id": 2190124, "author_name": "arti1117", "author_url": "", "post_date": "03/21/2023 04:00:11", "content": "

Interesting! thank you for sharing your opinions! @abaojiang

", "votes": null, "replies": [ { "id": 2190248, "author_name": "abaojiang", "author_url": "", "post_date": "03/21/2023 06:18:32", "content": "

Hi @arti1117,

\n

Glad it helps, and please see the update from the host!

", "votes": null, "replies": [] } ] }, { "id": 2191745, "author_name": "thitiwat", "author_url": "", "post_date": "03/22/2023 07:18:00", "content": "

thank you for sharing your opinions! @abaojiang

", "votes": null, "replies": [ { "id": 2194748, "author_name": "abaojiang", "author_url": "", "post_date": "03/24/2023 06:12:43", "content": "

Hi @thitiwat,

\n

Thanks for your appreciation!

", "votes": null, "replies": [] } ] }, { "id": 2218155, "author_name": "byungeunhwang", "author_url": "", "post_date": "04/11/2023 13:17:54", "content": "

Interesting! thank you for sharing your opinions!

", "votes": null, "replies": [ { "id": 2221166, "author_name": "abaojiang", "author_url": "", "post_date": "04/14/2023 04:15:10", "content": "

Thanks @byungeunhwang ๐Ÿ˜Š

", "votes": null, "replies": [] } ] }, { "id": 2219982, "author_name": "risakashiwabara", "author_url": "", "post_date": "04/13/2023 03:53:02", "content": "

Thank you for your valuable information. I am impressed with the easy to read graph you created. Was this created in python? The colors are beautiful and I was able to discover that it is possible to do such a thing.

", "votes": null, "replies": [ { "id": 2221163, "author_name": "abaojiang", "author_url": "", "post_date": "04/14/2023 04:13:22", "content": "

Hi @risakashiwabara,

\n

These graphs are plotted with matplotlib and seaborn packages in python. The implementation can be found here. Glad it helps, thanks a lot !

", "votes": null, "replies": [] } ] }, { "id": 2221062, "author_name": "yutodennou", "author_url": "", "post_date": "04/14/2023 00:16:59", "content": "

thank you for the great EDA.
\nI had been confused by the session_id and index.
\nReally helpful:)

", "votes": null, "replies": [ { "id": 2221164, "author_name": "abaojiang", "author_url": "", "post_date": "04/14/2023 04:14:37", "content": "

Hi @yutodennou,

\n

Thanks for your appreciation!

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2184672": "Hi everyone,\n\n**(Update, 03/21):** Because the competition host has announced the data update policy [here](https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/396202), I decide to re-run this notebook to make exploration results aligned with the newest data.\n\nThe competition has lasted for over one month, but some of the important data issues remain unsolved. Hence, I decide to restate these problems to facilitate more discussions. Also, I hope this topic could help those not joining at the start of the competition have a quick understanding about the potential dangers in data. For detailed implementation, please refer to [EDA on Game Progress\n](https://www.kaggle.com/code/abaojiang/eda-on-game-progress/notebook?scriptVersionId=122864479).\n\n### `index` and `elapsed_time`\nAs shown in the [data tab](https://www.kaggle.com/competitions/predict-student-performance-from-game-play/data), the meaning of columns `index` and `elapsed_time` are:\n* `index`: the index of the event for the session\n* `elapsed_time` - how much time has passed (in milliseconds) between the start of the session and when the event was recorded\n\nIt's reasonable to assume that the (`session_id`, `index`) pairs can uniquely identify data samples, but this isn't the case. Duplicated (`session_id`, `index`) pairs exist. In addition, `elapsed_time` by definition should be **monotonically increasing**, which doesn't always hold in data. Lastly, as the indicator of the event triggered during the game play, `index` should increase by one for each step. Again, it fails.\n\nConcretely, issues related to `index` and `elapsed_time` are summarized below with the host's answers [here](https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/384342#2134312):\n1. 142 sessions have duplicated (`session_id`, `index`) pairs.\n * The reason why duplication exists remains unknown.\n2. `elapsed_time` isn't always monotonically increasing.\n * The `index` gives the correct order of events in cases where the `elapsed_time` is inconsistent\n3. 258 sessions have **reversed index** phenomenon.\n[![reversed-idx.png](https://i.postimg.cc/Vs0z9j25/reversed-idx.png)](https://postimg.cc/jCKVtJxr)\n4. `index` can increase more than one for one step (98.46% jumps occur after `checkpoint`).\n5. `index == 0` doesn't always exist in the session.\n * `index == 0` should indicate the start of the session.\n\n### `level` and `level_group`\nAs the game progress, `level` should gradually increase, which indicates level-ups. But, there exist some sessions with **reversed level** and **reversed level group** phenomenon. Specifically, issues related to `level` and `level_group` are introduced as follows:\n1. 486 out of 23562(about 2.06%) sessions have **reversed level** phenomenon.\n2. 442 out of 23562 (about 1.88%) sessions have **reversed level group** phenomenon.\n[![reversed-lv-gp.png](https://i.postimg.cc/nrL274Hk/reversed-lv-gp.png)](https://postimg.cc/phgKMn5h)\n3. Some sessions have missing `level`, details can be found [here](https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/390339).\n * The host has confirmed that **all gameplay sessions don't use save codes in both training and test data**.\n\nSo, the reason why these two numbers are different is that **reversed index** doesn't always imply **reversed index group**. In other words, `level` can fall back to some specific points within the same `level_group`, which is illustrated as follows:\n[![reversed-lv-only.png](https://i.postimg.cc/fRzFC9B6/reversed-lv-only.png)](https://postimg.cc/6yPY5ys0)\n\nOne of the reasonable assumption is that these sessions have **multiple game plays** in a row. We should note the potential **data leakage** caused by these phenomena. For more detailed information, please refer to the original issue report by [@cdeotte](https://www.kaggle.com/cdeotte) [here](https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/388479#2149959). So far, we haven't got response from the host.\n\n### `checkpoint`\n`checkpoint` is an indicator of question prompts at the end of each `level_group`. For a normal game play, we can assume there are 3 checkpoints, each of which corresponds to one of the `level_group` (*i.e.,* `\"0-4\"`, `\"5-12\"` and `\"13-22\"`). However, there exist sessions with less than or more than 3 checkpoints. To be concrete, issues related to `checkpoint` are summarized as follows:\n1. 99.29% of the sessions have 3 checkpoints.\n2. 1 session (*i.e.,* session `22090108192456930`) has only 2 checkpoints, missing `level_group == \"0-4\"`.\n3. 167 sessions have more than 3 checkpoints.\n[![n-ckpts-2.png](https://i.postimg.cc/mgHZXZcq/n-ckpts-2.png)](https://postimg.cc/BjJGt0Bc)\n\nCombined with issues related to `level` and `level_group`, we can see that even if one session has exactly 3 checkpoints, `level` or `level_group` can still fall back. Hence, checking number of checkpoints in one session may not be a good way to ensure the validity of the session.\n\nHope this brief writeup could help you quickly understand the potential dangers in the data. If there's any question, please feel free to comment below. Thanks a lot!", "2186338": "Hi, this discussion and your notebook both are very helpful. Thank you.\n\nFrom what I've observed, elapsed_time is a really important feature in this competiton (Maybe even the magic feature :0). So how are you dealing with the error of the values not monotonically increasing?", "2187229": "Hi @shashwatraman,\n\nThanks for your appreciation! So far, I've built some DL-based models to try to capture temporal patterns hidden behind the sequence of `elapsed_time` difference. Just as you mention, `elapsed_time` seems to be a key feature. As for the **negative** `elapsed_time` difference, I've only tried two methods so far:\n1. Clip all negative difference to zero.\n2. Use the median derived from each (`level`, `event_name`) group to replace the negative entries.\n\nFor experimental results, they are nearly the same. There must exist better ways to handle this issue, so I'll keep exploring. Hope I can share something effective in the near future. Thanks a lot!", "2187240": "Thank you for the reply! \nI tried to find if these problems are also present in the test data. I think they are. Please check out my discussion [here.](https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/395686)\nThanks!!", "2187938": "Hi @shashwatraman,\n\nI've checked out your works. Very helpful! I still focus on local development to find out how to clean the data better. Now, it seems that we should pay more attention to how test data will be presented (maybe wait for response from the host๐Ÿ‘€). Any way, thanks a lot for your hard works and sharing!", "2188277": "Thanks @abaojiang, dear for letting us informed. This post is full of deep insights and it can create lot of ways to solve the remaining problems in the notebook. It is interesting to work in. Thanks๐Ÿ‘", "2188861": "Hi @tariqbashir,\n\nThanks for your reply. Glad it helps!๐Ÿ˜Š", "2189395": "Thanks for the Information. As I found same issue,I was looking for people who experienced same issue. It was really helpful! @abaojiang", "2190017": "Thank you @abaojiang for your summarizing data quality issue! Hope competition hosts announce their plans on this issue.", "2190124": "Interesting! thank you for sharing your opinions! @abaojiang", "2190246": "Hi @johnthethrid,\n\nThanks for your appreciation, and please see the [update](https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/396202) from the host!", "2190248": "Hi @arti1117,\n\nGlad it helps, and please see the [update](https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/396202) from the host!", "2190251": "Hi @tomokikmogura,\n\nThanks for your reply. The competition host has announced the data update [here](https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/396202), but we still need to re-check the aforementioned issues!", "2191745": "thank you for sharing your opinions! @abaojiang", "2194748": "Hi @thitiwat,\n\nThanks for your appreciation!", "2218155": "Interesting! thank you for sharing your opinions!", "2219982": "Thank you for your valuable information. I am impressed with the easy to read graph you created. Was this created in python? The colors are beautiful and I was able to discover that it is possible to do such a thing.", "2221062": "thank you for the great EDA.\nI had been confused by the session_id and index.\nReally helpful:)", "2221163": "Hi @risakashiwabara,\n\nThese graphs are plotted with `matplotlib` and `seaborn` packages in python. The implementation can be found [here](https://www.kaggle.com/code/abaojiang/eda-on-game-progress/notebook?scriptVersionId=122864479). Glad it helps, thanks a lot !", "2221164": "Hi @yutodennou,\n\nThanks for your appreciation!", "2221166": "Thanks @byungeunhwang ๐Ÿ˜Š" }, "source": "meta" }