\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3110858%2Fe49b6d8581547e3a9477a626647d36cc%2FWX20230629-0924562x.png?generation=1688002004825453&alt=media\")
Thank my teammates for their efforts, I have learned a lot from them. Luckily we don't shake-down too much. Now I would like to introduce my solution to you.
\nMy modeling method is like a 'cumulative' one: using 0-4 part data to generate the train set of q1-q3, using 0-4 and 5-12 part data to generate the train set of q4-q13, using 0-4, 5-12 and 13-22 part data to generate the train set of q14-q18. Question is also a feature. It has merit that I don't need to save 'historical' data. And this one is time-saving and costs about 40min for inference.
\n
There are my FE ideas:
\nbasic agg features: eclipse_time_diff sum, count and max of each group, each level, each event_name, …, each text; eclipse_time_diff sum, count under a particular room_fqid and an event_name, etc.
behavior-change features: the number of room change, and the number of room change under each level; the number of text_fqid change, and the number of text_fqid change under each level, etc.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3110858%2F2f61e63e58a3bfab1944b9af16d0e589%2F1688005632589.jpg?generation=1688005685285390&alt=media\")
\nIn this picture, we can see a room change behavior, we calculate the change times and average to characterize one's ability to understand and reason. Some of them have pretty high feature importance.
Meta features: Besides basic groupby feature engineering, I add the meta feature for 5-12 and 13-22 groups. There are two way to use them:
predict_proba as a feature, 5-12's model includes features q1_proba, q2_proba, and q3_proba, 13-22's model includes features q1_proba, q2_proba, … q13_proba.For my part, I use 9 models for my ensemble. They are 4 xgboost, 1 lightgbm, 2 dart, 2 catboost. The private-best single model is a dart, which achieved Public 0.704 and Private 0.704. The public-best model is a xgboost, which achieved Public 0.705 and Private 0.698. My dart notebook Game-Play-LGBDart[INFER] Private LB 0.704
\nI think the most difficult part of this comp is to establish CV and choose the threshold and the submissions. As you can see, my dart model and xgboost model in the same CV strategy vary wildly. It's beyond my expectation. I even have no confidence to give my dart models a bigger weight. I believe many teams didn't choose their best results.
\nFinally, I would like to pay tribute to all kagglers who share their ideas. See you next game.
", "messages": [ { "id": "2321952", "postDate": "06/29/2023 02:34:58", "content": "Thank my teammates for their efforts, I have learned a lot from them. Luckily we don't shake-down too much. Now I would like to introduce my solution to you.
\nMy modeling method is like a 'cumulative' one: using 0-4 part data to generate the train set of q1-q3, using 0-4 and 5-12 part data to generate the train set of q4-q13, using 0-4, 5-12 and 13-22 part data to generate the train set of q14-q18. Question is also a feature. It has merit that I don't need to save 'historical' data. And this one is time-saving and costs about 40min for inference.
\n
There are my FE ideas:
\nbasic agg features: eclipse_time_diff sum, count and max of each group, each level, each event_name, …, each text; eclipse_time_diff sum, count under a particular room_fqid and an event_name, etc.
behavior-change features: the number of room change, and the number of room change under each level; the number of text_fqid change, and the number of text_fqid change under each level, etc.
\n
\nIn this picture, we can see a room change behavior, we calculate the change times and average to characterize one's ability to understand and reason. Some of them have pretty high feature importance.
Meta features: Besides basic groupby feature engineering, I add the meta feature for 5-12 and 13-22 groups. There are two way to use them:
predict_proba as a feature, 5-12's model includes features q1_proba, q2_proba, and q3_proba, 13-22's model includes features q1_proba, q2_proba, … q13_proba.For my part, I use 9 models for my ensemble. They are 4 xgboost, 1 lightgbm, 2 dart, 2 catboost. The private-best single model is a dart, which achieved Public 0.704 and Private 0.704. The public-best model is a xgboost, which achieved Public 0.705 and Private 0.698. My dart notebook Game-Play-LGBDart[INFER] Private LB 0.704
\nI think the most difficult part of this comp is to establish CV and choose the threshold and the submissions. As you can see, my dart model and xgboost model in the same CV strategy vary wildly. It's beyond my expectation. I even have no confidence to give my dart models a bigger weight. I believe many teams didn't choose their best results.
\nFinally, I would like to pay tribute to all kagglers who share their ideas. See you next game.
", "rawMarkdown": "Thank my teammates for their efforts, I have learned a lot from them. Luckily we don't shake-down too much. Now I would like to introduce my solution to you. \n# Modeling\nMy modeling method is like a 'cumulative' one: using 0-4 part data to generate the train set of q1-q3, using 0-4 and 5-12 part data to generate the train set of q4-q13, using 0-4, 5-12 and 13-22 part data to generate the train set of q14-q18. **Question is also a feature**. It has merit that I don't need to save 'historical' data. And this one is time-saving and costs about 40min for inference.\n\n# Feature Engineering\nThere are my FE ideas:\n- **basic agg** features: eclipse_time_diff sum, count and max of each group, each level, each event_name, ..., each text; eclipse_time_diff sum, count under a particular room_fqid and an event_name, etc.\n- **behavior-change** features: the number of room change, and the number of room change under each level; the number of text_fqid change, and the number of text_fqid change under each level, etc.\n\nIn this picture, we can see a **room change** behavior, we calculate the change times and average to characterize one's ability to understand and reason. Some of them have pretty high feature importance.\n\n- **Meta** features: Besides basic `groupby` feature engineering, I add the meta feature for 5-12 and 13-22 groups. There are two way to use them:\n1. each question's `predict_proba` as a feature, 5-12's model includes features **q1_proba**, **q2_proba**, and **q3_proba**, 13-22's model includes features **q1_proba**, **q2_proba**, ... **q13_proba**.\n2. mean of all question in one group as a feature, for instance, 5-12's model includes a feature **mean_of_q1-q3_proba**, 13-22's model includes features **mean_of_q1-q3_proba**, **mean_of_q4-q13_proba**.\n\n# Models\nFor my part, I use 9 models for my ensemble. They are 4 xgboost, 1 lightgbm, 2 dart, 2 catboost. The private-best single model is a dart, which achieved **Public 0.704** and **Private 0.704**. The public-best model is a xgboost, which achieved **Public 0.705** and **Private 0.698**. My dart notebook [Game-Play-LGBDart[INFER] Private LB 0.704](https://www.kaggle.com/code/takanashihumbert/game-play-lgbdart-infer/notebook)\n# The difficulty\nI think the most difficult part of this comp is to establish CV and choose the threshold and the submissions. As you can see, my dart model and xgboost model in the same CV strategy vary wildly. It's beyond my expectation. I even have no confidence to give my dart models a bigger weight. **I believe many teams didn't choose their best results.**\n\nFinally, I would like to pay tribute to all kagglers who share their ideas. See you next game.", "votes": null }, { "id": "2321967", "postDate": "06/29/2023 02:48:45", "content": "By the way, I want to share some thoughts on leaked data here. From this data, we can get about 7000+ extra session_id with complete logs(i.e. the user answers all the 18 question). By using this data, private score and local score can boost by 0.001. But it didn't work on the public LB. It really confused me during this competition. I tried many ways to use this extra data and all failed in public score. It's glad to see it work on private data although we didn't select those subs with private data. We have almost 20 subs can get 0.703 in private score but we didn't select it. Next time I would trust more on local cv😂
", "rawMarkdown": "Very very very glad to team with you guys !! @takanashihumbert @zui0711 @max2020 \n------------------\n\nBy the way, I want to share some thoughts on [leaked data](https://fielddaylab.wisc.edu/opengamedata/) here. From this data, we can get about 7000+ extra session_id with complete logs(i.e. the user answers all the 18 question). By using this data, private score and local score can boost by 0.001. But it didn't work on the public LB. It really confused me during this competition. I tried many ways to use this extra data and all failed in public score. It's glad to see it work on private data although we didn't select those subs with private data. We have almost 20 subs can get 0.703 in private score but we didn't select it. Next time I would trust more on local cv😂", "votes": null }, { "id": "2321974", "postDate": "06/29/2023 02:54:18", "content": "Congratulations on the gold medal
\nour ensemble brings very low Private score, and our best submission is a single model with meta features Public 0.702 and Private 0.704. did your team have success in feature selection?
It's also my pleasure, Professor A.
", "rawMarkdown": "It's also my pleasure, Professor A.", "votes": null }, { "id": "2321978", "postDate": "06/29/2023 02:59:08", "content": "Thanks! And sadly I didn't do any feature selection.🤕
", "rawMarkdown": "Thanks! And sadly I didn't do any feature selection.🤕", "votes": null }, { "id": "2321985", "postDate": "06/29/2023 03:05:45", "content": "I did some feature selection by holdout cv in very early stage. Because of several reruns, I cannot compare the effects of feature selection on private score. But for public lb, they are almost same and feature selection can save a lot of time for training and inference.
", "rawMarkdown": "I did some feature selection by holdout cv in very early stage. Because of several reruns, I cannot compare the effects of feature selection on private score. But for public lb, they are almost same and feature selection can save a lot of time for training and inference.", "votes": null }, { "id": "2321988", "postDate": "06/29/2023 03:06:49", "content": "Congrat on your gold medal.
\nDid your team use the external data (raw data) from the fielddaylab site? I think that external data could change this game but it turns out you do not need it to get gold place
The external data, as Adam mentions above, helps my local cv. The F1-score improved by nearly 0.001, but public LB didn't change. What confuses me the most is that xgboost models are far lower than dart on private LB.
", "rawMarkdown": "The external data, as Adam mentions above, helps my local cv. The F1-score improved by nearly 0.001, but public LB didn't change. What confuses me the most is that xgboost models are far lower than dart on private LB.", "votes": null }, { "id": "2321997", "postDate": "06/29/2023 03:17:26", "content": "Congratulations Joseph and team! Well done achieving 14th place Gold !!
", "rawMarkdown": "Congratulations Joseph and team! Well done achieving 14th place Gold !!", "votes": null }, { "id": "2322011", "postDate": "06/29/2023 03:24:41", "content": "Thank you, Chris. I have learned a lot from you in many competitions!
", "rawMarkdown": "Thank you, Chris. I have learned a lot from you in many competitions!", "votes": null }, { "id": "2322015", "postDate": "06/29/2023 03:25:56", "content": "On the leak data and private score, same here. I spent weeks pre-processing those external data trying to make it align with original training data only to receive public LB < CV. We didn't choose the submissions with external data, and then it turns out the 7000 sessions could have pumped us to money prize ranks.
\nWhat a roller coaster of feelings.
", "rawMarkdown": "On the leak data and private score, same here. I spent weeks pre-processing those external data trying to make it align with original training data only to receive public LB < CV. We didn't choose the submissions with external data, and then it turns out the 7000 sessions could have pumped us to money prize ranks.\n\nWhat a roller coaster of feelings.", "votes": null }, { "id": "2322017", "postDate": "06/29/2023 03:27:13", "content": "Btw congratulations on your 3rd gold medal in a row! You're well on your track to GM title!
", "rawMarkdown": "Btw congratulations on your 3rd gold medal in a row! You're well on your track to GM title!", "votes": null }, { "id": "2322021", "postDate": "06/29/2023 03:28:46", "content": "Thank you for your information. I did crawl and parse the external data and get the same result as you (slightly increase CV, LB does not change). Thought something wrong with my pipeline
", "rawMarkdown": "Thank you for your information. I did crawl and parse the external data and get the same result as you (slightly increase CV, LB does not change). Thought something wrong with my pipeline", "votes": null }, { "id": "2322022", "postDate": "06/29/2023 03:29:11", "content": "haha. Thanks. Still long way to get a solo medal.😂
", "rawMarkdown": "haha. Thanks. Still long way to get a solo medal.😂", "votes": null }, { "id": "2322024", "postDate": "06/29/2023 03:32:14", "content": "we also got extra 7000 sessions, but the lb score drop and didn't keep going
", "rawMarkdown": "we also got extra 7000 sessions, but the lb score drop and didn't keep going", "votes": null }, { "id": "2322025", "postDate": "06/29/2023 03:34:04", "content": "Congratulations on the 14th place finish and also for posting such a clean solution.
\nI have a couple of questions regarding your approach if you don't mind:
I observed the same thing. After many validation rounds, I believed my preprocessing had aligned with the original training data's preprocessing, and started to suspect a distribution shift between test set and train set, and potentially huge shake-ups/downs. Fortunately we were the lucky ones to get pumped up.
", "rawMarkdown": "I observed the same thing. After many validation rounds, I believed my preprocessing had aligned with the original training data's preprocessing, and started to suspect a distribution shift between test set and train set, and potentially huge shake-ups/downs. Fortunately we were the lucky ones to get pumped up.", "votes": null }, { "id": "2322037", "postDate": "06/29/2023 03:43:18", "content": "The second question is difficult for me to answer; because I didn't tweak the parameters too much.🤕
", "rawMarkdown": "dart cv: one is 0.70199, the other is 0.70203\n- catboost cv: one is 0.70122, the other is 0.70095\n\nThe second question is difficult for me to answer; because I didn't tweak the parameters too much.🤕", "votes": null }, { "id": "2322089", "postDate": "06/29/2023 04:36:54", "content": "@takanashihumbert congrats with gold medal!
", "rawMarkdown": "takanashihumbert congrats with gold medal!", "votes": null }, { "id": "2322132", "postDate": "06/29/2023 05:08:22", "content": "Congrats 👏 we had used exactly same meta features 😊 just final selection faltered
", "rawMarkdown": "Congrats 👏 we had used exactly same meta features 😊 just final selection faltered", "votes": null }, { "id": "2322206", "postDate": "06/29/2023 06:29:16", "content": "Thanks @serangu !
", "rawMarkdown": "Thanks @serangu !", "votes": null }, { "id": "2322207", "postDate": "06/29/2023 06:30:38", "content": "Thanks! We indeed need some luck in this comp.
", "rawMarkdown": "Thanks! We indeed need some luck in this comp.", "votes": null }, { "id": "2322771", "postDate": "06/29/2023 13:39:31", "content": "@takanashihumbert Congratulations on your 14th place finish! Your approach of using a cumulative modeling method for different question sets shows creativity and efficiency. The ensemble of different models further enhances your solution.
", "rawMarkdown": "takanashihumbert Congratulations on your 14th place finish! Your approach of using a cumulative modeling method for different question sets shows creativity and efficiency. The ensemble of different models further enhances your solution.", "votes": null }, { "id": "2324125", "postDate": "06/30/2023 12:01:55", "content": "@takanashihumbert congrats with the achievementl!
", "rawMarkdown": "takanashihumbert congrats with the achievementl!", "votes": null }, { "id": "2324142", "postDate": "06/30/2023 12:10:48", "content": "thank you and congrats on gold
", "rawMarkdown": "thank you and congrats on gold", "votes": null }, { "id": "2324434", "postDate": "06/30/2023 15:45:02", "content": "Congratulations on your achievement
", "rawMarkdown": "Congratulations on your achievement", "votes": null }, { "id": "2325496", "postDate": "07/01/2023 12:11:18", "content": "Congratulations on your 14th place solution! It's impressive how you employed cumulative modeling and various feature engineering techniques to create a successful solution. Ensembling 9 models, including XGBoost, LightGBM, Dart, and Catboost, shows your dedication to finding the best combination. The challenge of establishing CV, choosing thresholds, and making submissions was indeed difficult. Congratulations once again, and best of luck in future competitions!
", "rawMarkdown": "Congratulations on your 14th place solution! It's impressive how you employed cumulative modeling and various feature engineering techniques to create a successful solution. Ensembling 9 models, including XGBoost, LightGBM, Dart, and Catboost, shows your dedication to finding the best combination. The challenge of establishing CV, choosing thresholds, and making submissions was indeed difficult. Congratulations once again, and best of luck in future competitions!", "votes": null } ], "comments": [ { "id": 2321967, "author_name": "hookman", "author_url": "", "post_date": "06/29/2023 02:48:45", "content": "By the way, I want to share some thoughts on leaked data here. From this data, we can get about 7000+ extra session_id with complete logs(i.e. the user answers all the 18 question). By using this data, private score and local score can boost by 0.001. But it didn't work on the public LB. It really confused me during this competition. I tried many ways to use this extra data and all failed in public score. It's glad to see it work on private data although we didn't select those subs with private data. We have almost 20 subs can get 0.703 in private score but we didn't select it. Next time I would trust more on local cv😂
", "votes": null, "replies": [ { "id": 2321977, "author_name": "takanashihumbert", "author_url": "", "post_date": "06/29/2023 02:57:02", "content": "It's also my pleasure, Professor A.
", "votes": null, "replies": [] }, { "id": 2322015, "author_name": "hoangnguyen719", "author_url": "", "post_date": "06/29/2023 03:25:56", "content": "On the leak data and private score, same here. I spent weeks pre-processing those external data trying to make it align with original training data only to receive public LB < CV. We didn't choose the submissions with external data, and then it turns out the 7000 sessions could have pumped us to money prize ranks.
\nWhat a roller coaster of feelings.
", "votes": null, "replies": [ { "id": 2322024, "author_name": "tonymarkchris", "author_url": "", "post_date": "06/29/2023 03:32:14", "content": "we also got extra 7000 sessions, but the lb score drop and didn't keep going
", "votes": null, "replies": [ { "id": 2322035, "author_name": "hoangnguyen719", "author_url": "", "post_date": "06/29/2023 03:38:56", "content": "I observed the same thing. After many validation rounds, I believed my preprocessing had aligned with the original training data's preprocessing, and started to suspect a distribution shift between test set and train set, and potentially huge shake-ups/downs. Fortunately we were the lucky ones to get pumped up.
", "votes": null, "replies": [] } ] } ] }, { "id": 2322017, "author_name": "hoangnguyen719", "author_url": "", "post_date": "06/29/2023 03:27:13", "content": "Btw congratulations on your 3rd gold medal in a row! You're well on your track to GM title!
", "votes": null, "replies": [ { "id": 2322022, "author_name": "hookman", "author_url": "", "post_date": "06/29/2023 03:29:11", "content": "haha. Thanks. Still long way to get a solo medal.😂
", "votes": null, "replies": [] } ] } ] }, { "id": 2321974, "author_name": "tonymarkchris", "author_url": "", "post_date": "06/29/2023 02:54:18", "content": "Congratulations on the gold medal
\nour ensemble brings very low Private score, and our best submission is a single model with meta features Public 0.702 and Private 0.704. did your team have success in feature selection?
Thanks! And sadly I didn't do any feature selection.🤕
", "votes": null, "replies": [] }, { "id": 2321985, "author_name": "hookman", "author_url": "", "post_date": "06/29/2023 03:05:45", "content": "I did some feature selection by holdout cv in very early stage. Because of several reruns, I cannot compare the effects of feature selection on private score. But for public lb, they are almost same and feature selection can save a lot of time for training and inference.
", "votes": null, "replies": [] } ] }, { "id": 2321988, "author_name": "minhtu123", "author_url": "", "post_date": "06/29/2023 03:06:49", "content": "Congrat on your gold medal.
\nDid your team use the external data (raw data) from the fielddaylab site? I think that external data could change this game but it turns out you do not need it to get gold place
The external data, as Adam mentions above, helps my local cv. The F1-score improved by nearly 0.001, but public LB didn't change. What confuses me the most is that xgboost models are far lower than dart on private LB.
", "votes": null, "replies": [ { "id": 2322021, "author_name": "minhtu123", "author_url": "", "post_date": "06/29/2023 03:28:46", "content": "Thank you for your information. I did crawl and parse the external data and get the same result as you (slightly increase CV, LB does not change). Thought something wrong with my pipeline
", "votes": null, "replies": [] } ] } ] }, { "id": 2321997, "author_name": "cdeotte", "author_url": "", "post_date": "06/29/2023 03:17:26", "content": "Congratulations Joseph and team! Well done achieving 14th place Gold !!
", "votes": null, "replies": [ { "id": 2322011, "author_name": "takanashihumbert", "author_url": "", "post_date": "06/29/2023 03:24:41", "content": "Thank you, Chris. I have learned a lot from you in many competitions!
", "votes": null, "replies": [] } ] }, { "id": 2322025, "author_name": "chaudharypriyanshu", "author_url": "", "post_date": "06/29/2023 03:34:04", "content": "Congratulations on the 14th place finish and also for posting such a clean solution.
\nI have a couple of questions regarding your approach if you don't mind:
The second question is difficult for me to answer; because I didn't tweak the parameters too much.🤕
", "votes": null, "replies": [] } ] }, { "id": 2322089, "author_name": "serangu", "author_url": "", "post_date": "06/29/2023 04:36:54", "content": "@takanashihumbert congrats with gold medal!
", "votes": null, "replies": [ { "id": 2322206, "author_name": "takanashihumbert", "author_url": "", "post_date": "06/29/2023 06:29:16", "content": "Thanks @serangu !
", "votes": null, "replies": [] } ] }, { "id": 2322132, "author_name": "gauravbrills", "author_url": "", "post_date": "06/29/2023 05:08:22", "content": "Congrats 👏 we had used exactly same meta features 😊 just final selection faltered
", "votes": null, "replies": [ { "id": 2322207, "author_name": "takanashihumbert", "author_url": "", "post_date": "06/29/2023 06:30:38", "content": "Thanks! We indeed need some luck in this comp.
", "votes": null, "replies": [] } ] }, { "id": 2322771, "author_name": "akshayvyas02", "author_url": "", "post_date": "06/29/2023 13:39:31", "content": "@takanashihumbert Congratulations on your 14th place finish! Your approach of using a cumulative modeling method for different question sets shows creativity and efficiency. The ensemble of different models further enhances your solution.
", "votes": null, "replies": [] }, { "id": 2324125, "author_name": "sergueiivanov", "author_url": "", "post_date": "06/30/2023 12:01:55", "content": "@takanashihumbert congrats with the achievementl!
", "votes": null, "replies": [ { "id": 2324142, "author_name": "joek47", "author_url": "", "post_date": "06/30/2023 12:10:48", "content": "thank you and congrats on gold
", "votes": null, "replies": [] } ] }, { "id": 2324434, "author_name": "jinukhaggs", "author_url": "", "post_date": "06/30/2023 15:45:02", "content": "Congratulations on your achievement
", "votes": null, "replies": [] }, { "id": 2325496, "author_name": "poojach7611", "author_url": "", "post_date": "07/01/2023 12:11:18", "content": "Congratulations on your 14th place solution! It's impressive how you employed cumulative modeling and various feature engineering techniques to create a successful solution. Ensembling 9 models, including XGBoost, LightGBM, Dart, and Catboost, shows your dedication to finding the best combination. The challenge of establishing CV, choosing thresholds, and making submissions was indeed difficult. Congratulations once again, and best of luck in future competitions!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2321952": "Thank my teammates for their efforts, I have learned a lot from them. Luckily we don't shake-down too much. Now I would like to introduce my solution to you. \n# Modeling\nMy modeling method is like a 'cumulative' one: using 0-4 part data to generate the train set of q1-q3, using 0-4 and 5-12 part data to generate the train set of q4-q13, using 0-4, 5-12 and 13-22 part data to generate the train set of q14-q18. **Question is also a feature**. It has merit that I don't need to save 'historical' data. And this one is time-saving and costs about 40min for inference.\n\n# Feature Engineering\nThere are my FE ideas:\n- **basic agg** features: eclipse_time_diff sum, count and max of each group, each level, each event_name, ..., each text; eclipse_time_diff sum, count under a particular room_fqid and an event_name, etc.\n- **behavior-change** features: the number of room change, and the number of room change under each level; the number of text_fqid change, and the number of text_fqid change under each level, etc.\n\nIn this picture, we can see a **room change** behavior, we calculate the change times and average to characterize one's ability to understand and reason. Some of them have pretty high feature importance.\n\n- **Meta** features: Besides basic `groupby` feature engineering, I add the meta feature for 5-12 and 13-22 groups. There are two way to use them:\n1. each question's `predict_proba` as a feature, 5-12's model includes features **q1_proba**, **q2_proba**, and **q3_proba**, 13-22's model includes features **q1_proba**, **q2_proba**, ... **q13_proba**.\n2. mean of all question in one group as a feature, for instance, 5-12's model includes a feature **mean_of_q1-q3_proba**, 13-22's model includes features **mean_of_q1-q3_proba**, **mean_of_q4-q13_proba**.\n\n# Models\nFor my part, I use 9 models for my ensemble. They are 4 xgboost, 1 lightgbm, 2 dart, 2 catboost. The private-best single model is a dart, which achieved **Public 0.704** and **Private 0.704**. The public-best model is a xgboost, which achieved **Public 0.705** and **Private 0.698**. My dart notebook [Game-Play-LGBDart[INFER] Private LB 0.704](https://www.kaggle.com/code/takanashihumbert/game-play-lgbdart-infer/notebook)\n# The difficulty\nI think the most difficult part of this comp is to establish CV and choose the threshold and the submissions. As you can see, my dart model and xgboost model in the same CV strategy vary wildly. It's beyond my expectation. I even have no confidence to give my dart models a bigger weight. **I believe many teams didn't choose their best results.**\n\nFinally, I would like to pay tribute to all kagglers who share their ideas. See you next game.", "2321967": "Very very very glad to team with you guys !! @takanashihumbert @zui0711 @max2020 \n------------------\n\nBy the way, I want to share some thoughts on [leaked data](https://fielddaylab.wisc.edu/opengamedata/) here. From this data, we can get about 7000+ extra session_id with complete logs(i.e. the user answers all the 18 question). By using this data, private score and local score can boost by 0.001. But it didn't work on the public LB. It really confused me during this competition. I tried many ways to use this extra data and all failed in public score. It's glad to see it work on private data although we didn't select those subs with private data. We have almost 20 subs can get 0.703 in private score but we didn't select it. Next time I would trust more on local cv😂", "2321974": "Congratulations on the gold medal\nour ensemble brings very low Private score, and our best submission is a single model with meta features **Public 0.702** and **Private 0.704**. did your team have success in feature selection?", "2321977": "It's also my pleasure, Professor A.", "2321978": "Thanks! And sadly I didn't do any feature selection.🤕", "2321985": "I did some feature selection by holdout cv in very early stage. Because of several reruns, I cannot compare the effects of feature selection on private score. But for public lb, they are almost same and feature selection can save a lot of time for training and inference.", "2321988": "Congrat on your gold medal.\nDid your team use the external data (raw data) from the fielddaylab site? I think that external data could change this game but it turns out you do not need it to get gold place", "2321995": "The external data, as Adam mentions above, helps my local cv. The F1-score improved by nearly 0.001, but public LB didn't change. What confuses me the most is that xgboost models are far lower than dart on private LB.", "2321997": "Congratulations Joseph and team! Well done achieving 14th place Gold !!", "2322011": "Thank you, Chris. I have learned a lot from you in many competitions!", "2322015": "On the leak data and private score, same here. I spent weeks pre-processing those external data trying to make it align with original training data only to receive public LB < CV. We didn't choose the submissions with external data, and then it turns out the 7000 sessions could have pumped us to money prize ranks.\n\nWhat a roller coaster of feelings.", "2322017": "Btw congratulations on your 3rd gold medal in a row! You're well on your track to GM title!", "2322021": "Thank you for your information. I did crawl and parse the external data and get the same result as you (slightly increase CV, LB does not change). Thought something wrong with my pipeline", "2322022": "haha. Thanks. Still long way to get a solo medal.😂", "2322024": "we also got extra 7000 sessions, but the lb score drop and didn't keep going", "2322025": "Congratulations on the 14th place finish and also for posting such a clean solution.\nI have a couple of questions regarding your approach if you don't mind:\n\n1. Could you please share the CV score of the LGBM dart and CatBoost models?\n2. I'm curious to know which parameters had the most significant impact on the performance of these two models (e.g., colsample_by_tree, etc.).", "2322035": "I observed the same thing. After many validation rounds, I believed my preprocessing had aligned with the original training data's preprocessing, and started to suspect a distribution shift between test set and train set, and potentially huge shake-ups/downs. Fortunately we were the lucky ones to get pumped up.", "2322037": "dart cv: one is 0.70199, the other is 0.70203\n- catboost cv: one is 0.70122, the other is 0.70095\n\nThe second question is difficult for me to answer; because I didn't tweak the parameters too much.🤕", "2322089": "takanashihumbert congrats with gold medal!", "2322132": "Congrats 👏 we had used exactly same meta features 😊 just final selection faltered", "2322206": "Thanks @serangu !", "2322207": "Thanks! We indeed need some luck in this comp.", "2322771": "takanashihumbert Congratulations on your 14th place finish! Your approach of using a cumulative modeling method for different question sets shows creativity and efficiency. The ensemble of different models further enhances your solution.", "2324125": "takanashihumbert congrats with the achievementl!", "2324142": "thank you and congrats on gold", "2324434": "Congratulations on your achievement", "2325496": "Congratulations on your 14th place solution! It's impressive how you employed cumulative modeling and various feature engineering techniques to create a successful solution. Ensembling 9 models, including XGBoost, LightGBM, Dart, and Catboost, shows your dedication to finding the best combination. The challenge of establishing CV, choosing thresholds, and making submissions was indeed difficult. Congratulations once again, and best of luck in future competitions!" }, "source": "meta" }