{"cells":[{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5"},"cell_type":"markdown","source":"Code taken from https://www.kaggle.com/ogrellier/adversarial-validation-and-lb-shakeup kernel in toxic classification challenge\n> \nAdversarial validation is a mean to check if train and test datasets have significant differences. The idea is to use the dataset features to try and separate train and test samples.\n\nSo you would create a binary target that would be 1 for train samples and 0 for test samples and fit a classifier on the features to predict if a given sample is in train or test datasets!\n\nHere we will use a LogisticRegression and a TF-IDF vectorizer to check if text features distributions are different and see if we can separate the samples. \n\nThe best kernel on this is certainly [here](https://www.kaggle.com/konradb/adversarial-validation-and-other-scary-terms) by [Konrad Banachewicz](https://www.kaggle.com/konradb)\n\nOther resources can be found [on fastML](http://fastml.com/adversarial-validation-part-one/)\n"},{"metadata":{"_uuid":"0eefbd8a8d3c7c327ff334763fb76f5095c90519"},"cell_type":"markdown","source":"### Objective: We want to find if distribution of train data and test data is similar or not?\n\nIf it is not we are gonna face huge leaderboard shakeup in second stage. So lets see"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\ntrn = pd.read_csv(\"../input/train.csv\", encoding=\"utf-8\")\nsub = pd.read_csv(\"../input/test.csv\", encoding=\"utf-8\")","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"356ea9e9-126b-4cf3-abe5-9eb4f9a0f5af","_uuid":"a9d76507e3a7805b16b2bf6e902223ee48ff1d8b","trusted":true},"cell_type":"code","source":"from sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer\nimport regex\nvectorizer = TfidfVectorizer(\n    sublinear_tf=True,\n    strip_accents='unicode',\n    tokenizer=lambda x: regex.findall(r'[^\\p{P}\\W]+', x),\n    analyzer='word',\n    token_pattern=None,\n    stop_words='english',\n    ngram_range=(1, 1), \n    max_features=50000\n)\ntrn_idf = vectorizer.fit_transform(trn.question_text)\ntrn_vocab = vectorizer.vocabulary_\nsub_idf = vectorizer.fit_transform(sub.question_text)\nsub_vocab = vectorizer.vocabulary_\nall_idf = vectorizer.fit_transform(pd.concat([trn.question_text, sub.question_text], axis=0))\nall_vocab = vectorizer.vocabulary_","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"61f32cac-cd19-445c-bd04-3ffd9ccdcc1d","_uuid":"de3f629a93be69f4079ad1de465482ec9de4ba05"},"cell_type":"markdown","source":"Convert vocab dictionnaries to list of words"},{"metadata":{"_cell_guid":"c8539cd9-b7b2-47fa-9cdd-0de5498ac0c6","_uuid":"578f39976d38377dd72adc9062bf200d7e715e77","trusted":true},"cell_type":"code","source":"trn_words = [word for word in trn_vocab.keys()]\nsub_words = [word for word in sub_vocab.keys()]\nall_words = [word for word in all_vocab.keys()]","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"c1d48036-0873-48ea-b712-db51b607684f","_uuid":"2681c121a8366fdda44d681c6708e809e8e1da2f"},"cell_type":"markdown","source":"Check a few figures on words not in train or test"},{"metadata":{"_cell_guid":"8dc68da6-c0ef-44ab-9ce8-4aa6a4425574","_uuid":"acc03c89faa3578c8b7c6a249a384f4b443ce4d7","_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"common_words = set(trn_words).intersection(set(sub_words)) \nprint(\"number of words in both train and test : %d \"\n      % len(common_words))\nprint(\"number of words in all_words not in train : %d \"\n      % (len(trn_words) - len(set(trn_words).intersection(set(all_words)))))\nprint(\"number of words in all_words not in test : %d \"\n      % (len(sub_words) - len(set(sub_words).intersection(set(all_words)))))","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"819aee86-98a6-4f24-82cb-4133a8d0dd13","_uuid":"5d17d63b9c61e83516b77c74a0a208171ec38848"},"cell_type":"markdown","source":"This means there is not much substantial differences between train and test vocabularies or term frequencies\n\nLet's check if a LinearRegression can make a difference between train and test using this.\n\nWe would take the output of the TF-IDF vectorizer fitted on train + test."},{"metadata":{"_cell_guid":"92863c3d-4600-4fe5-8287-e2777b693f70","_uuid":"94884ef5129ad259ecc0034fa8bc328da14aa47c","trusted":true},"cell_type":"code","source":"from sklearn.linear_model import LogisticRegression\nfrom sklearn.model_selection import StratifiedKFold\nfrom sklearn.metrics import roc_auc_score\n# Create target where all train samples are ones and all test samples are zeros\ntarget = np.hstack((np.ones(trn.shape[0]), np.zeros(sub.shape[0])))\n# Shuffle samples to mix zeros and ones\nidx = np.arange(all_idf.shape[0])\nnp.random.seed(1)\nnp.random.shuffle(idx)\nall_idf = all_idf[idx]\ntarget = target[idx]\n# Train a Logistic Regression\nfolds = StratifiedKFold(5, True, 1)\nfor trn_idx, val_idx in folds.split(all_idf, target):\n    lr = LogisticRegression(solver = 'saga')\n    lr.fit(all_idf[trn_idx], target[trn_idx])\n    print(roc_auc_score(target[val_idx], lr.predict_proba(all_idf[val_idx])[:, 1]))","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"00977781-2ae1-45f0-9fcd-a1e461a18573","_uuid":"1df84a6b320e7aff77356b497077e3f924cf200c"},"cell_type":"markdown","source":"So Based on AUC score we cannot really find much difference between the train and test data.** Good for us. **\n\nI did not see any wrong doing in the code so far but if you do please shout as loud as possible !\n\n** This to me means we might not have many surprises on the private LB. **\n\n** Although with the size of the test set so low we cannot be sure. **\n"},{"metadata":{"_uuid":"dbca68987f00217b6fc2b53f47ac42e24218f639"},"cell_type":"markdown","source":"You may want to push all this further and see the impact of a cleaner dataset."}],"metadata":{"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"}},"nbformat":4,"nbformat_minor":1}