Because of lack of consideration, I run a training with the attribution rate about 0.25% in train set (similar to the rate of the universe) and a validation set of attribution rate about 0.12%.
\n\nIn the whole train time, firstly quite odd, the auc for validation set is higher than that for training set. Of course, the auc on LB is much less.
\n\nMy ballpark theory is because of less positives in validation set, the result for true positive looks higher than it should be and on the other hand more negatives in validation set artificially makes false positive lower. The overall effect is pulling the roc curve to the top left corner, therefore resulting higher auc.
\n\nThe take-away can be always create validation set with stratifying the classes.
", "messages": [ { "id": "315441", "postDate": "04/17/2018 05:47:15", "content": "Because of lack of consideration, I run a training with the attribution rate about 0.25% in train set (similar to the rate of the universe) and a validation set of attribution rate about 0.12%.
\n\nIn the whole train time, firstly quite odd, the auc for validation set is higher than that for training set. Of course, the auc on LB is much less.
\n\nMy ballpark theory is because of less positives in validation set, the result for true positive looks higher than it should be and on the other hand more negatives in validation set artificially makes false positive lower. The overall effect is pulling the roc curve to the top left corner, therefore resulting higher auc.
\n\nThe take-away can be always create validation set with stratifying the classes.
", "rawMarkdown": "Because of lack of consideration, I run a training with the attribution rate about 0.25% in train set (similar to the rate of the universe) and a validation set of attribution rate about 0.12%.\n\nIn the whole train time, firstly quite odd, the auc for validation set is higher than that for training set. Of course, the auc on LB is much less.\n\nMy ballpark theory is because of less positives in validation set, the result for true positive looks higher than it should be and on the other hand more negatives in validation set artificially makes false positive lower. The overall effect is pulling the roc curve to the top left corner, therefore resulting higher auc.\n\nThe take-away can be always create validation set with stratifying the classes.", "votes": null }, { "id": "315454", "postDate": "04/17/2018 06:17:08", "content": "I had the same sometimes with a validation set with 10m rows from the training set for training and the next 2m for validation. I did not check the attribution rate though.
", "rawMarkdown": "I had the same sometimes with a validation set with 10m rows from the training set for training and the next 2m for validation. I did not check the attribution rate though.", "votes": null }, { "id": "315457", "postDate": "04/17/2018 06:21:20", "content": "Don't have time to verify what I said. But it's highly likely true. In the next training, I stratified the classes so the validation set has same attribution rate as the training set. Validation auc then comes under training auc during training.
", "rawMarkdown": "Don't have time to verify what I said. But it's highly likely true. In the next training, I stratified the classes so the validation set has same attribution rate as the training set. Validation auc then comes under training auc during training.", "votes": null }, { "id": "315461", "postDate": "04/17/2018 06:27:46", "content": "How did you stratify ?\nWhenever I get more free tm I'll tryto check more thoroughly your theory ;)
", "rawMarkdown": "How did you stratify ?\nWhenever I get more free tm I'll tryto check more thoroughly your theory ;)", "votes": null }, { "id": "315472", "postDate": "04/17/2018 06:38:54", "content": "I set the stratify parameter in sklearn train_test_split:
\n\n\ntrain, valid = train_test_split(dat, test_size=0.2, stratify=dat.is_attributed)\n
As @Fei specified you need to stratify the test/validation set.
\n\nAlternatively, if you only use pandas, you can shuffle the data before picking a test/validation set - it would give a similar percentage of is_attributed as of the entire set (0.247% of data has i_attributed = 1)\nI'm doing something like this
\n\ntest_size = len(train_df) // 5 #(I have this fixed to 2000000)
\n\ntrain_df = train_df.sample(frac=1.0)
\n\ntrain_data = train_df[:-test_size]
\n\ntest_data = train_df[len(train_df)-test_size:]
\n\nObviously if you do not fully retrain the model you'll have to persist the generated train/test_data and reload if needed.
", "rawMarkdown": "As @Fei specified you need to stratify the test/validation set.\n\nAlternatively, if you only use pandas, you can shuffle the data before picking a test/validation set - it would give a similar percentage of is_attributed as of the entire set (0.247% of data has i_attributed = 1)\nI'm doing something like this\n\ntest_size = len(train_df) // 5 #(I have this fixed to 2000000)\n\ntrain_df = train_df.sample(frac=1.0)\n\ntrain_data = train_df[:-test_size]\n\ntest_data = train_df[len(train_df)-test_size:]\n\nObviously if you do not fully retrain the model you'll have to persist the generated train/test_data and reload if needed.", "votes": null }, { "id": "316919", "postDate": "04/20/2018 07:29:00", "content": "A double check with value_counts would be handy:
\n\n\nvalid.is_attributed.value_counts(normalize=True)\n
Which gives the percentage of each category based on counts.
", "rawMarkdown": "A double check with value_counts would be handy:\n\n\n```\nvalid.is_attributed.value_counts(normalize=True)\n```\n\nWhich gives the percentage of each category based on counts.", "votes": null } ], "comments": [ { "id": 315454, "author_name": "thepathofd", "author_url": "", "post_date": "04/17/2018 06:17:08", "content": "I had the same sometimes with a validation set with 10m rows from the training set for training and the next 2m for validation. I did not check the attribution rate though.
", "votes": null, "replies": [ { "id": 315457, "author_name": "enfeizhan", "author_url": "", "post_date": "04/17/2018 06:21:20", "content": "Don't have time to verify what I said. But it's highly likely true. In the next training, I stratified the classes so the validation set has same attribution rate as the training set. Validation auc then comes under training auc during training.
", "votes": null, "replies": [] }, { "id": 315461, "author_name": "propanon", "author_url": "", "post_date": "04/17/2018 06:27:46", "content": "How did you stratify ?\nWhenever I get more free tm I'll tryto check more thoroughly your theory ;)
", "votes": null, "replies": [] }, { "id": 315472, "author_name": "enfeizhan", "author_url": "", "post_date": "04/17/2018 06:38:54", "content": "I set the stratify parameter in sklearn train_test_split:
\n\n\ntrain, valid = train_test_split(dat, test_size=0.2, stratify=dat.is_attributed)\n
As @Fei specified you need to stratify the test/validation set.
\n\nAlternatively, if you only use pandas, you can shuffle the data before picking a test/validation set - it would give a similar percentage of is_attributed as of the entire set (0.247% of data has i_attributed = 1)\nI'm doing something like this
\n\ntest_size = len(train_df) // 5 #(I have this fixed to 2000000)
\n\ntrain_df = train_df.sample(frac=1.0)
\n\ntrain_data = train_df[:-test_size]
\n\ntest_data = train_df[len(train_df)-test_size:]
\n\nObviously if you do not fully retrain the model you'll have to persist the generated train/test_data and reload if needed.
", "votes": null, "replies": [] }, { "id": 316919, "author_name": "enfeizhan", "author_url": "", "post_date": "04/20/2018 07:29:00", "content": "A double check with value_counts would be handy:
\n\n\nvalid.is_attributed.value_counts(normalize=True)\n
Which gives the percentage of each category based on counts.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "315441": "Because of lack of consideration, I run a training with the attribution rate about 0.25% in train set (similar to the rate of the universe) and a validation set of attribution rate about 0.12%.\n\nIn the whole train time, firstly quite odd, the auc for validation set is higher than that for training set. Of course, the auc on LB is much less.\n\nMy ballpark theory is because of less positives in validation set, the result for true positive looks higher than it should be and on the other hand more negatives in validation set artificially makes false positive lower. The overall effect is pulling the roc curve to the top left corner, therefore resulting higher auc.\n\nThe take-away can be always create validation set with stratifying the classes.", "315454": "I had the same sometimes with a validation set with 10m rows from the training set for training and the next 2m for validation. I did not check the attribution rate though.", "315457": "Don't have time to verify what I said. But it's highly likely true. In the next training, I stratified the classes so the validation set has same attribution rate as the training set. Validation auc then comes under training auc during training.", "315461": "How did you stratify ?\nWhenever I get more free tm I'll tryto check more thoroughly your theory ;)", "315472": "I set the stratify parameter in sklearn train_test_split:\n\n```\ntrain, valid = train_test_split(dat, test_size=0.2, stratify=dat.is_attributed)\n```", "316331": "As @Fei specified you need to stratify the test/validation set.\n\nAlternatively, if you only use pandas, you can shuffle the data before picking a test/validation set - it would give a similar percentage of is_attributed as of the entire set (0.247% of data has i_attributed = 1)\nI'm doing something like this\n\ntest_size = len(train_df) // 5 #(I have this fixed to 2000000)\n\ntrain_df = train_df.sample(frac=1.0)\n\ntrain_data = train_df[:-test_size]\n\ntest_data = train_df[len(train_df)-test_size:]\n\nObviously if you do not fully retrain the model you'll have to persist the generated train/test_data and reload if needed.", "316919": "A double check with value_counts would be handy:\n\n\n```\nvalid.is_attributed.value_counts(normalize=True)\n```\n\nWhich gives the percentage of each category based on counts." }, "source": "meta" }