I performed the adversarial validation on the data generated in this notebook.
\nhttps://www.kaggle.com/code/roberthatch/amex-feature-engg-gpu-or-cpu-process-in-chunks
\nI used a simple cat boost and performed it in two ways:
The most distinguishing features were R_1 and B_29 out of all the features.
\n**
\nThis suggests that both public test and private test data are different, however, still there is a good correlation between public test and train data.**
for the 2nd set:
\nI removed those features and performed it again and still found the AUC of 0.99999. I think the reason for the shakeup (if it happened) can be only two features just because they are significantly different not the other features as other notebooks are trying to tell.
\nThat's my hypothesis, this forum is open to discussion and I will be glad to hear others' input.😄
I performed the adversarial validation on the data generated in this notebook.
\nhttps://www.kaggle.com/code/roberthatch/amex-feature-engg-gpu-or-cpu-process-in-chunks
\nI used a simple cat boost and performed it in two ways:
The most distinguishing features were R_1 and B_29 out of all the features.
\n**
\nThis suggests that both public test and private test data are different, however, still there is a good correlation between public test and train data.**
for the 2nd set:
\nI removed those features and performed it again and still found the AUC of 0.99999. I think the reason for the shakeup (if it happened) can be only two features just because they are significantly different not the other features as other notebooks are trying to tell.
\nThat's my hypothesis, this forum is open to discussion and I will be glad to hear others' input.😄
Do I understand it correctly, you merged the training and test data sets and labeled them 0 and 1. Then you trained a classifier and got AUC of 0.999.
\nAs I understand it, this means that your dataset contains features whose distribution in the training dataset is very different from the distribution in the test dataset. If the features of the two datasets were similarly distributed, you would get an AUC around 0.5.
\nOr have I misunderstood something?
", "rawMarkdown": "Do I understand it correctly, you merged the training and test data sets and labeled them 0 and 1. Then you trained a classifier and got AUC of 0.999.\n\nAs I understand it, this means that your dataset contains features whose distribution in the training dataset is very different from the distribution in the test dataset. If the features of the two datasets were similarly distributed, you would get an AUC around 0.5.\n\nOr have I misunderstood something?", "votes": null }, { "id": "1906155", "postDate": "08/19/2022 16:23:06", "content": "yes exactly, i have taken the help from this notebook
\nhttps://www.kaggle.com/code/zakopur0/adversarial-validation-private-vs-public
\nas a baseline and perform these experiments on the above-mentioned data.
Of all the features the most differently distributed features were min , std, which if i am not wrong every data is going to use. still, CV/LB is correlating.
", "rawMarkdown": "yes exactly, i have taken the help from this notebook\n[https://www.kaggle.com/code/zakopur0/adversarial-validation-private-vs-public](url)\nas a baseline and perform these experiments on the above-mentioned data.\n\nOf all the features the most differently distributed features were min , std, which if i am not wrong every data is going to use. still, CV/LB is correlating.", "votes": null }, { "id": "1908621", "postDate": "08/21/2022 20:32:06", "content": "Thanks for the clarification and for your interesting post!
", "rawMarkdown": "Thanks for the clarification and for your interesting post!", "votes": null } ], "comments": [ { "id": 1906147, "author_name": "viktorfairuschin", "author_url": "", "post_date": "08/19/2022 16:11:32", "content": "Do I understand it correctly, you merged the training and test data sets and labeled them 0 and 1. Then you trained a classifier and got AUC of 0.999.
\nAs I understand it, this means that your dataset contains features whose distribution in the training dataset is very different from the distribution in the test dataset. If the features of the two datasets were similarly distributed, you would get an AUC around 0.5.
\nOr have I misunderstood something?
", "votes": null, "replies": [ { "id": 1906155, "author_name": "chaudharypriyanshu", "author_url": "", "post_date": "08/19/2022 16:23:06", "content": "yes exactly, i have taken the help from this notebook
\nhttps://www.kaggle.com/code/zakopur0/adversarial-validation-private-vs-public
\nas a baseline and perform these experiments on the above-mentioned data.
Of all the features the most differently distributed features were min , std, which if i am not wrong every data is going to use. still, CV/LB is correlating.
", "votes": null, "replies": [] }, { "id": 1908621, "author_name": "viktorfairuschin", "author_url": "", "post_date": "08/21/2022 20:32:06", "content": "Thanks for the clarification and for your interesting post!
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1904924": "I performed the adversarial validation on the data generated in this notebook.\n[https://www.kaggle.com/code/roberthatch/amex-feature-engg-gpu-or-cpu-process-in-chunks](url)\nI used a simple cat boost and performed it in two ways:\n\n1. Train and Public Test: **AUC: 0.9999** \n2. Train and Private Test: **AUC: 0.999999**\n\n**The most distinguishing features were R_1 and B_29** out of all the features.\n\n\n\n\n**\nThis suggests that both public test and private test data are different, however, still there is a good correlation between public test and train data.**\n\nfor the 2nd set:\nI removed those features and performed it again and still found the AUC of 0.99999. I think the reason for the shakeup (if it happened) can be only two features just because they are significantly different not the other features as other notebooks are trying to tell.\n**That's my hypothesis, this forum is open to discussion and I will be glad to hear others' input.**😄", "1906147": "Do I understand it correctly, you merged the training and test data sets and labeled them 0 and 1. Then you trained a classifier and got AUC of 0.999.\n\nAs I understand it, this means that your dataset contains features whose distribution in the training dataset is very different from the distribution in the test dataset. If the features of the two datasets were similarly distributed, you would get an AUC around 0.5.\n\nOr have I misunderstood something?", "1906155": "yes exactly, i have taken the help from this notebook\n[https://www.kaggle.com/code/zakopur0/adversarial-validation-private-vs-public](url)\nas a baseline and perform these experiments on the above-mentioned data.\n\nOf all the features the most differently distributed features were min , std, which if i am not wrong every data is going to use. still, CV/LB is correlating.", "1908621": "Thanks for the clarification and for your interesting post!" }, "source": "meta" }