How to find out whether data distribution in train and test are similar? I used correlation matrix but it is not of great use.
", "messages": [ { "id": "302950", "postDate": "03/25/2018 04:04:27", "content": "How to find out whether data distribution in train and test are similar? I used correlation matrix but it is not of great use.
", "rawMarkdown": "How to find out whether data distribution in train and test are similar? I used correlation matrix but it is not of great use.", "votes": null }, { "id": "302969", "postDate": "03/25/2018 05:58:00", "content": "Distribution of what? Distribution of is_attributed you won't be able to know as we have no knowledge for that for the test data (it has to be predicted). Distribution of the hour of the day is pretty much similiar, note however that in test there are only a few hours out there and not a whole day (on top of my head it was something like 4,5,9,10,14,15 hours)
", "rawMarkdown": "Distribution of what? Distribution of is_attributed you won't be able to know as we have no knowledge for that for the test data (it has to be predicted). Distribution of the hour of the day is pretty much similiar, note however that in test there are only a few hours out there and not a whole day (on top of my head it was something like 4,5,9,10,14,15 hours)", "votes": null }, { "id": "302975", "postDate": "03/25/2018 06:03:47", "content": "I am referrring to similarity in train and test features data.
", "rawMarkdown": "I am referrring to similarity in train and test features data.", "votes": null }, { "id": "302995", "postDate": "03/25/2018 07:35:32", "content": "I would go with adversarial validation - along the lines of
\n\nhttp://fastml.com/adversarial-validation-part-one/
", "rawMarkdown": "I would go with adversarial validation - along the lines of \n\nhttp://fastml.com/adversarial-validation-part-one/", "votes": null }, { "id": "303023", "postDate": "03/25/2018 09:26:21", "content": "thank you.
", "rawMarkdown": "thank you.", "votes": null }, { "id": "303035", "postDate": "03/25/2018 10:13:54", "content": "Check this cool kernel on adversarial validation :
\n\nhttps://www.kaggle.com/ogrellier/adversarial-validation-and-lb-shakeup
", "rawMarkdown": "Check this cool kernel on adversarial validation :\n\n https://www.kaggle.com/ogrellier/adversarial-validation-and-lb-shakeup", "votes": null }, { "id": "303040", "postDate": "03/25/2018 10:34:27", "content": "There is one difference that was identified by several people, see for instance: https://www.kaggle.com/cpmpml/ip-download-rates/notebook
", "rawMarkdown": "There is one difference that was identified by several people, see for instance: https://www.kaggle.com/cpmpml/ip-download-rates/notebook", "votes": null }, { "id": "303113", "postDate": "03/25/2018 14:55:23", "content": "sure ...thank you
", "rawMarkdown": "sure ...thank you", "votes": null }, { "id": "303400", "postDate": "03/26/2018 05:30:22", "content": "I, first of all, looked at the individual feature distributions.\nFor example, ip and click_time have very different distributions in train vs test set. See my kernel of distributions where I use a 10,000,000-row training set.
", "rawMarkdown": "I, first of all, looked at the individual feature distributions.\nFor example, ip and click_time have very different distributions in train vs test set. See my [kernel of distributions][1] where I use a 10,000,000-row training set.\n\n\n [1]: https://www.kaggle.com/araksstepanyan/feature-distributions-10mm-row-train-set", "votes": null }, { "id": "855976", "postDate": "05/21/2020 11:35:50", "content": "I know 2 ways of checking this:
\n\nBuild a classifier that tries to predict whether a data point belongs to the training or test set. If it succeeds, the distributions are different ;). Here's a tutorial.
Do a statistical test like Kolmogorov-Smirnov. This needs to be done in a per column level. If you are working with column-dependant data (like work embeddings, for example), there are some workarounds - basically, you transform the data so that it is uni dimensional, losing some information in the process, of course. Here's a tutorial for both the statistical test and dealing with word embeddings.
Distribution of what? Distribution of is_attributed you won't be able to know as we have no knowledge for that for the test data (it has to be predicted). Distribution of the hour of the day is pretty much similiar, note however that in test there are only a few hours out there and not a whole day (on top of my head it was something like 4,5,9,10,14,15 hours)
", "votes": null, "replies": [ { "id": 302975, "author_name": "maheshak04", "author_url": "", "post_date": "03/25/2018 06:03:47", "content": "I am referrring to similarity in train and test features data.
", "votes": null, "replies": [] } ] }, { "id": 302995, "author_name": "konradb", "author_url": "", "post_date": "03/25/2018 07:35:32", "content": "I would go with adversarial validation - along the lines of
\n\nhttp://fastml.com/adversarial-validation-part-one/
", "votes": null, "replies": [ { "id": 303023, "author_name": "maheshak04", "author_url": "", "post_date": "03/25/2018 09:26:21", "content": "thank you.
", "votes": null, "replies": [] } ] }, { "id": 303035, "author_name": "actuben", "author_url": "", "post_date": "03/25/2018 10:13:54", "content": "Check this cool kernel on adversarial validation :
\n\nhttps://www.kaggle.com/ogrellier/adversarial-validation-and-lb-shakeup
", "votes": null, "replies": [ { "id": 303113, "author_name": "maheshak04", "author_url": "", "post_date": "03/25/2018 14:55:23", "content": "sure ...thank you
", "votes": null, "replies": [] } ] }, { "id": 303040, "author_name": "cpmpml", "author_url": "", "post_date": "03/25/2018 10:34:27", "content": "There is one difference that was identified by several people, see for instance: https://www.kaggle.com/cpmpml/ip-download-rates/notebook
", "votes": null, "replies": [] }, { "id": 303400, "author_name": "araksstepanyan", "author_url": "", "post_date": "03/26/2018 05:30:22", "content": "I, first of all, looked at the individual feature distributions.\nFor example, ip and click_time have very different distributions in train vs test set. See my kernel of distributions where I use a 10,000,000-row training set.
", "votes": null, "replies": [] }, { "id": 855976, "author_name": "gmosse", "author_url": "", "post_date": "05/21/2020 11:35:50", "content": "I know 2 ways of checking this:
\n\nBuild a classifier that tries to predict whether a data point belongs to the training or test set. If it succeeds, the distributions are different ;). Here's a tutorial.
Do a statistical test like Kolmogorov-Smirnov. This needs to be done in a per column level. If you are working with column-dependant data (like work embeddings, for example), there are some workarounds - basically, you transform the data so that it is uni dimensional, losing some information in the process, of course. Here's a tutorial for both the statistical test and dealing with word embeddings.