I was checking current high score kernels so far, many kernels are using user_id for feature of the prediction model. That’s right, all of user_ids on public test data set are included on the user_id on train dataset.
\nHowever how about private test dataset? I think this is risky. What do you think?
\n[Edit]
\nI found the user_id on test data that is not contained on train data. Thank you for pointing out, @aquatic
\nEither way, I think user_id should be treated carefully to predict for unseen data.
I was checking current high score kernels so far, many kernels are using user_id for feature of the prediction model. That’s right, all of user_ids on public test data set are included on the user_id on train dataset.
\nHowever how about private test dataset? I think this is risky. What do you think?
\n[Edit]
\nI found the user_id on test data that is not contained on train data. Thank you for pointing out, @aquatic
\nEither way, I think user_id should be treated carefully to predict for unseen data.
I think this is a baseline approach. Later people will use features based on user_id and not user_id itself
", "rawMarkdown": "I think this is a baseline approach. Later people will use features based on user_id and not user_id itself", "votes": null }, { "id": "1041073", "postDate": "10/07/2020 14:36:09", "content": "I think it a good topic to discuss. In my LGBM model which uses features including user_id, the biggest feature importance is given by user_id. I, as well as you, am somewhat suspicious of it.
Yes, I also think these are baseline at really first phase of this competition, however, my concern is if the user_id is useful for only public test data, not private test data, in that case public LB will be not helpful to evaluate models. (Should we truct local CV? lol)
", "rawMarkdown": "Yes, I also think these are baseline at really first phase of this competition, however, my concern is if the user_id is useful for only public test data, not private test data, in that case public LB will be not helpful to evaluate models. (Should we truct local CV? lol)", "votes": null }, { "id": "1041133", "postDate": "10/07/2020 15:20:17", "content": "\n\nThat’s right, all of user_ids on public test data set are included on the user_id on train dataset.
\n
How do you know this -- was it tested or demonstrated somewhere, if you don't mind sharing?
\nI wonder if all test users do actually show up in train. In the data description we get:
\n\n\nSome questions will appear in the hidden test set that have NOT been presented in the train set, emulating the challenge of quickly adapting to modeling newly introduced questions. Their metadata is still in question.csv as usual
\n
They emphasize the presence of new questions, but not of new users. Why not both if that's really the case?
\nRegardless, representing users with other features instead of their user_id is likely a good idea. Gradient boosted trees and neural nets both don't particularly like high cardinality category inputs and id type features seem particularly likely to cause overfitting. It may be much better to figure out how to encode the user behavior patterns that matter as features so the model can focus on more general patterns instead of memorizing users.
It's likely true that user_id helps a lot to predict on training data, but keep in mind that it's also likely to see spurious feature importances driven by a feature having a lot of distinct values, especially if you're looking at frequency importance. E.g. try using an index column as a feature and see what happens in feature importance.
Oh, I'm wrong, I found the user_id on test data that is not contained on train data!
\nThank you for pointing out.
Interesting, thanks for sharing!
", "rawMarkdown": "Interesting, thanks for sharing!", "votes": null }, { "id": "1041255", "postDate": "10/07/2020 16:42:12", "content": "I agree with you in that LGBM feature importance is affected by cardinalities. Thank you for your comments.
", "rawMarkdown": "I agree with you in that LGBM feature importance is affected by cardinalities. Thank you for your comments.", "votes": null }, { "id": "1041555", "postDate": "10/07/2020 20:06:49", "content": "in the example_test.csv, the same user id is seen at different groups. I thought we are supposed to track the user performance as the group number increases. hence the user id is supposed to be used. am I wrong on that?
", "rawMarkdown": "in the example_test.csv, the same user id is seen at different groups. I thought we are supposed to track the user performance as the group number increases. hence the user id is supposed to be used. am I wrong on that?", "votes": null }, { "id": "1041866", "postDate": "10/08/2020 00:22:29", "content": "Well, you can think of this competition as recommending items to users (here, the recommendable items are given to the Agent at prediction time but the whole items set is fixed). Hence, the goal is to track the agent/content over time and that could hardly be done without using agent/content Ids.
\nBut, your concerns are legitimate because new contents or new users could appear over time, this is known as cold start problem in the the recommendation jargon and there are several techniques to deal with it (content based, user meta data, top items …).
", "rawMarkdown": "Well, you can think of this competition as recommending items to users (here, the recommendable items are given to the *Agent* at prediction time but the whole items set is fixed). Hence, the goal is to track the agent/content over time and that could hardly be done without using agent/content Ids.\n\nBut, your concerns are legitimate because new contents or new users could appear over time, this is known as **cold start problem** in the the recommendation jargon and there are several techniques to deal with it (content based, user meta data, top items ...).", "votes": null } ], "comments": [ { "id": 1041043, "author_name": "artgor", "author_url": "", "post_date": "10/07/2020 14:13:48", "content": "I think this is a baseline approach. Later people will use features based on user_id and not user_id itself
", "votes": null, "replies": [ { "id": 1041098, "author_name": "kenmatsu4", "author_url": "", "post_date": "10/07/2020 15:02:12", "content": "Yes, I also think these are baseline at really first phase of this competition, however, my concern is if the user_id is useful for only public test data, not private test data, in that case public LB will be not helpful to evaluate models. (Should we truct local CV? lol)
", "votes": null, "replies": [] } ] }, { "id": 1041073, "author_name": "sishihara", "author_url": "", "post_date": "10/07/2020 14:36:09", "content": "I think it a good topic to discuss. In my LGBM model which uses features including user_id, the biggest feature importance is given by user_id. I, as well as you, am somewhat suspicious of it.
It's likely true that user_id helps a lot to predict on training data, but keep in mind that it's also likely to see spurious feature importances driven by a feature having a lot of distinct values, especially if you're looking at frequency importance. E.g. try using an index column as a feature and see what happens in feature importance.
I agree with you in that LGBM feature importance is affected by cardinalities. Thank you for your comments.
", "votes": null, "replies": [] } ] }, { "id": 1041133, "author_name": "aquatic", "author_url": "", "post_date": "10/07/2020 15:20:17", "content": "\n\nThat’s right, all of user_ids on public test data set are included on the user_id on train dataset.
\n
How do you know this -- was it tested or demonstrated somewhere, if you don't mind sharing?
\nI wonder if all test users do actually show up in train. In the data description we get:
\n\n\nSome questions will appear in the hidden test set that have NOT been presented in the train set, emulating the challenge of quickly adapting to modeling newly introduced questions. Their metadata is still in question.csv as usual
\n
They emphasize the presence of new questions, but not of new users. Why not both if that's really the case?
\nRegardless, representing users with other features instead of their user_id is likely a good idea. Gradient boosted trees and neural nets both don't particularly like high cardinality category inputs and id type features seem particularly likely to cause overfitting. It may be much better to figure out how to encode the user behavior patterns that matter as features so the model can focus on more general patterns instead of memorizing users.
Oh, I'm wrong, I found the user_id on test data that is not contained on train data!
\nThank you for pointing out.
Interesting, thanks for sharing!
", "votes": null, "replies": [] } ] }, { "id": 1041555, "author_name": "hengck23", "author_url": "", "post_date": "10/07/2020 20:06:49", "content": "in the example_test.csv, the same user id is seen at different groups. I thought we are supposed to track the user performance as the group number increases. hence the user id is supposed to be used. am I wrong on that?
", "votes": null, "replies": [] }, { "id": 1041866, "author_name": "kneroma", "author_url": "", "post_date": "10/08/2020 00:22:29", "content": "Well, you can think of this competition as recommending items to users (here, the recommendable items are given to the Agent at prediction time but the whole items set is fixed). Hence, the goal is to track the agent/content over time and that could hardly be done without using agent/content Ids.
\nBut, your concerns are legitimate because new contents or new users could appear over time, this is known as cold start problem in the the recommendation jargon and there are several techniques to deal with it (content based, user meta data, top items …).
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1040918": "I was checking current high score kernels so far, many kernels are using user_id for feature of the prediction model. That’s right, all of user_ids on public test data set are included on the user_id on train dataset. \n\nHowever how about private test dataset? I think this is risky. What do you think?\n\n[Edit]\nI found the user_id on test data that is not contained on train data. Thank you for pointing out, @aquatic \nEither way, I think user_id should be treated carefully to predict for unseen data.", "1041043": "I think this is a baseline approach. Later people will use features based on user_id and not user_id itself", "1041073": "I think it a good topic to discuss. In my LGBM model which uses features including `user_id`, the biggest feature importance is given by `user_id`. I, as well as you, am somewhat suspicious of it.", "1041098": "Yes, I also think these are baseline at really first phase of this competition, however, my concern is if the user_id is useful for only public test data, not private test data, in that case public LB will be not helpful to evaluate models. (Should we truct local CV? lol)", "1041133": "> That’s right, all of user_ids on public test data set are included on the user_id on train dataset. \n\nHow do you know this -- was it tested or demonstrated somewhere, if you don't mind sharing? \n\nI wonder if all test users do actually show up in train. In the data description we get:\n\n> Some questions will appear in the hidden test set that have NOT been presented in the train set, emulating the challenge of quickly adapting to modeling newly introduced questions. Their metadata is still in question.csv as usual\n\nThey emphasize the presence of new questions, but not of new users. Why not both if that's really the case? \n\nRegardless, representing users with other features instead of their `user_id` is likely a good idea. Gradient boosted trees and neural nets both don't particularly like high cardinality category inputs and id type features seem particularly likely to cause overfitting. It may be much better to figure out how to encode the user behavior patterns that matter as features so the model can focus on more general patterns instead of memorizing users.", "1041138": "It's likely true that `user_id` helps a lot to predict on training data, but keep in mind that it's also likely to see spurious feature importances driven by a feature having a lot of distinct values, especially if you're looking at frequency importance. E.g. try using an index column as a feature and see what happens in feature importance.", "1041216": "Oh, I'm wrong, I found the user_id on test data that is not contained on train data!\nThank you for pointing out.", "1041236": "Interesting, thanks for sharing!", "1041255": "I agree with you in that LGBM feature importance is affected by cardinalities. Thank you for your comments.", "1041555": "in the example_test.csv, the same user id is seen at different groups. I thought we are supposed to track the user performance as the group number increases. hence the user id is supposed to be used. am I wrong on that?", "1041866": "Well, you can think of this competition as recommending items to users (here, the recommendable items are given to the *Agent* at prediction time but the whole items set is fixed). Hence, the goal is to track the agent/content over time and that could hardly be done without using agent/content Ids.\n\nBut, your concerns are legitimate because new contents or new users could appear over time, this is known as **cold start problem** in the the recommendation jargon and there are several techniques to deal with it (content based, user meta data, top items ...)." }, "source": "meta" }