{"cells":[{"metadata":{"_uuid":"20edda61f0c0b884a0f9ac97938a5accb7a04909"},"cell_type":"markdown","source":"I found some interesting [public](http://https://www.kaggle.com/him4318/avito-lightgbm-with-ridge-feature-v-2-0/code) [kernals](http://). I tried to fork one of the kernals cause I found the aggregated feature file from [this](http://https://www.kaggle.com/bminixhofer/aggregated-features-lightgbm) one is quite useful. Here, I'll try to add image blurrness score features I extracted from one of my other [kernals](http://https://www.kaggle.com/sukhyun9673/image-processing-600000-to-750000). (Later, I'll also try adding [regional information I scraped from Wikipedia.](http://https://www.kaggle.com/sukhyun9673/scraping-regional-info-population-time-zone-etc))\n\nThis kernal includes forks and reference from : https://www.kaggle.com/him4318/avito-lightgbm-with-ridge-feature-v-2-0/code, and https://www.kaggle.com/bminixhofer/aggregated-features-lightgbm\nThanks to original authors for inspiration!"},{"metadata":{"_cell_guid":"033852ed-2dd3-4bc3-bec1-ac3481ab175f","_kg_hide-input":true,"_uuid":"af7422df064b7f16934cca9e86e8213bc9c667e8","trusted":true,"collapsed":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport gc\nfrom tqdm import tqdm\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom matplotlib_venn import venn2, venn2_circles\nimport string\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import mean_squared_error\nfrom sklearn.preprocessing import LabelEncoder\nfrom sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer\nimport nltk\nfrom nltk.corpus import stopwords\nimport scipy\nimport lightgbm as lgb\nfrom sklearn.linear_model import Ridge\nfrom sklearn.cross_validation import KFold\nfrom math import sqrt\n\nsns.set()\n%matplotlib inline","execution_count":1,"outputs":[]},{"metadata":{"_cell_guid":"5f037998-0ef3-44a8-816c-d6fd79fb08dd","_uuid":"3a30dfe38892ca287f1a1d6de4b0dfe0cde5c309","trusted":true,"collapsed":true},"cell_type":"code","source":"train = pd.read_csv('../input/avito-demand-prediction/train.csv')\ntest = pd.read_csv('../input/avito-demand-prediction/test.csv')\n\ntrain_sorted = train.sort_values(by = [\"image\"])\ntest_sorted = test.sort_values(by = [\"image\"])\n\n#Load image blurrness for train\n\ntr_1 = pd.read_csv(\"../input/train-data-image-blurrness/1.csv\")\ntr_2 = pd.read_csv(\"../input/train-data-image-blurrness/2.csv\")\ntr_3 = pd.read_csv(\"../input/train-data-image-blurrness/3.csv\")\ntr_4 = pd.read_csv(\"../input/train-data-image-blurrness/4.csv\")\ntr_5 = pd.read_csv(\"../input/train-data-image-blurrness/5.csv\")\ntr_6 = pd.read_csv(\"../input/train-data-image-blurrness/6.csv\")\ntr_7 = pd.read_csv(\"../input/train-data-image-blurrness/7.csv\")\ntr_8 = pd.read_csv(\"../input/train-data-image-blurrness/8.csv\")\ntr_9 = pd.read_csv(\"../input/train-data-image-blurrness/9.csv\")\ntr_10 = pd.read_csv(\"../input/train-data-image-blurrness/10.csv\")\ntr_11 = pd.read_csv(\"../input/train-data-image-blurrness/11(12_13.5).csv\")\ntr_12 = pd.read_csv(\"../input/train-data-image-blurrness/last.csv\")\n\nframes = [tr_1, tr_2, tr_3, tr_4, tr_5, tr_6, tr_7, tr_8, tr_9, tr_10, tr_11, tr_12]\nnew = pd.concat(frames)\nnew[\"File\"] = new[\"File\"].apply(lambda x : x.split(\"/\")[-1].split(\".\")[0])\nnew = new.sort_values(by = [\"File\"])\nscores = list(new[\"Score\"].values) + [-1] * (len(train)-len(new))\ntrain_sorted[\"image_blurrness_score\"] = scores\ntrain = train_sorted.sort_index()\n\n\n##Testing\nte_1 = pd.read_csv(\"../input/image-blurrness-test/test_1.csv\")\nte_2 = pd.read_csv(\"../input/image-blurrness-test/test_2.csv\")\nte_3 = pd.read_csv(\"../input/image-blurrness-test/test_3.csv\")\nte_4 = pd.read_csv(\"../input/image-blurrness-test/test_4.csv\")\nte_5 = pd.read_csv(\"../input/image-blurrness-test/test_5.csv\")\n\nframes_te = [te_1, te_2, te_3, te_4, te_5]\nnew_te = pd.concat(frames_te)\nnew_te[\"File\"] = new_te[\"File\"].apply(lambda x : x.split(\"/\")[-1].split(\".\")[0])\nnew_te = new_te.sort_values(by = [\"File\"])\nscores_te = list(new_te[\"Score\"].values) + [-1] * (len(test)-len(new_te))\n\ntest_sorted[\"image_blurrness_score\"] = scores_te\ntest = test_sorted.sort_index()\n","execution_count":2,"outputs":[]},{"metadata":{"_cell_guid":"bc719fcf-f418-4db5-a4fa-ebfa27716521","_uuid":"02aad0b977053435e089bd189ea45e4a88731e87","trusted":true,"collapsed":true},"cell_type":"code","source":"gp = pd.read_csv(\"../input/aggregate/aggregated_features.csv\")\ntrain = train.merge(gp, on='user_id', how='left')\ntest = test.merge(gp, on='user_id', how='left')\n\nagg_cols = list(gp.columns)[1:]\n\ndel gp\ngc.collect()\n","execution_count":3,"outputs":[]},{"metadata":{"_uuid":"4530cc5211658a4796fa390327bbabe17b97e668"},"cell_type":"markdown","source":"Now, I have  the train / test dataset to use. "},{"metadata":{"_cell_guid":"41e1111d-674e-4c86-b232-c0086832ee6d","_uuid":"9c60409e1dc6c618f229e15c2b368660385c2ec1","trusted":true,"collapsed":true},"cell_type":"code","source":"count = lambda l1,l2: sum([1 for x in l1 if x in l2])\n\n\nfor df in [train, test]:\n    df[\"price\"] = np.log(df[\"price\"]+0.001)\n    df[\"image_blurrness_score\"] = np.log(df[\"image_blurrness_score\"]+0.001)\n    df['description'].fillna('unknowndescription', inplace=True)\n    df['title'].fillna('unknowntitle', inplace=True)\n    \n    df['weekday'] = pd.to_datetime(df['activation_date']).dt.day\n    for col in ['description', 'title']:\n        df['num_words_' + col] = df[col].apply(lambda comment: len(comment.split()))\n        df['num_unique_words_' + col] = df[col].apply(lambda comment: len(set(w for w in comment.split())))\n\n    df['words_vs_unique_title'] = df['num_unique_words_title'] / df['num_words_title'] * 100\n    df['words_vs_unique_description'] = df['num_unique_words_description'] / df['num_words_description'] * 100\n    \n    df['city'] = df['region'] + '_' + df['city']\n    df['num_desc_punct'] = df['description'].apply(lambda x: count(x, set(string.punctuation)))\n    \n    for col in agg_cols:\n        df[col].fillna(-1, inplace=True)\n\nfor df in [train, test]:\n    df.price.replace(to_replace=[np.inf, -np.inf,np.nan], value=-1,inplace=True)\n    df.image_blurrness_score.replace(to_replace=[np.inf, -np.inf,np.nan], value=-1,inplace=True)","execution_count":4,"outputs":[]},{"metadata":{"_cell_guid":"8d61ada6-8f3a-4b58-baad-199589122985","_uuid":"cc558a03cbddc3954b01783636785cdee6cf954a","trusted":true,"collapsed":true},"cell_type":"code","source":"count_vectorizer_title = CountVectorizer(stop_words=stopwords.words('russian'), lowercase=True, min_df=25)\n\ntitle_counts = count_vectorizer_title.fit_transform(train['title'].append(test['title']))\n\ntrain_title_counts = title_counts[:len(train)]\ntest_title_counts = title_counts[len(train):]\n\n\ncount_vectorizer_desc = TfidfVectorizer(stop_words=stopwords.words('russian'), \n                                        lowercase=True, ngram_range=(1, 2),\n                                        max_features=15000)\n\ndesc_counts = count_vectorizer_desc.fit_transform(train['description'].append(test['description']))\n\ntrain_desc_counts = desc_counts[:len(train)]\ntest_desc_counts = desc_counts[len(train):]\n\ntrain_title_counts.shape, train_desc_counts.shape\n\ntrain","execution_count":5,"outputs":[]},{"metadata":{"_cell_guid":"56338e56-c266-4342-a8f3-10621db49dee","_uuid":"cc6f15f981f05ad9b1e89ab7acb54ea2350278c6","collapsed":true,"trusted":true},"cell_type":"code","source":"target = 'deal_probability'\npredictors = [\n    'num_desc_punct', \n    'words_vs_unique_description', 'num_unique_words_description', 'num_unique_words_title', 'num_words_description', 'num_words_title',\n    'avg_times_up_user', 'avg_days_up_user', 'n_user_items', \n    'price', 'item_seq_number', \"image_blurrness_score\"\n]\ncategorical = [\n    'image_top_1', 'param_1', 'param_2', 'param_3', \n    'city', 'region', 'category_name', 'parent_category_name', 'user_type'\n]\n\npredictors = predictors + categorical","execution_count":8,"outputs":[]},{"metadata":{"_cell_guid":"7695c1a1-06cf-40e6-8792-77b62f48262a","_uuid":"0bcde9bb9b9b26347c7db7da47123c6fec40860e","trusted":true,"collapsed":true},"cell_type":"code","source":"for feature in categorical:\n    print(f'Transforming {feature}...')\n    encoder = LabelEncoder()\n    encoder.fit(train[feature].append(test[feature]).astype(str))\n    \n    train[feature] = encoder.transform(train[feature].astype(str))\n    test[feature] = encoder.transform(test[feature].astype(str))\n    \ntrain","execution_count":9,"outputs":[]},{"metadata":{"_uuid":"1f66fe8fa0c37b8f648e8f5380ff9507e5df4b43"},"cell_type":"markdown","source":"Ridge feature here\n"},{"metadata":{"trusted":true,"_uuid":"2879fddcf12ef23f77cc43454b4b5f45c2195ace","scrolled":true,"collapsed":true},"cell_type":"code","source":"df = pd.concat([train[predictors], test[predictors]], axis = 0)\nNFOLDS = 5\nSEED = 42\nVALID = False\n\ntraindex = train.index\ntestdex = test.index\nntrain = train.shape[0]\nntest = test.shape[0]\nkf = KFold(ntrain, n_folds=NFOLDS, shuffle=True, random_state=SEED)\n\n\n\ny = train.deal_probability.copy()\n\nclass SklearnWrapper(object):\n    def __init__(self, clf, seed=0, params=None, seed_bool = True):\n        if(seed_bool == True):\n            params['random_state'] = seed\n        self.clf = clf(**params)\n\n    def train(self, x_train, y_train):\n        self.clf.fit(x_train, y_train)\n\n    def predict(self, x):\n        return self.clf.predict(x)\n        \ndef get_oof(clf, x_train, y, x_test):\n    oof_train = np.zeros((ntrain,))\n    oof_test = np.zeros((ntest,))\n    oof_test_skf = np.empty((NFOLDS, ntest))\n\n    for i, (train_index, test_index) in enumerate(kf):\n        print('\\nFold {}'.format(i))\n        x_tr = x_train[train_index]\n        y_tr = y[train_index]\n        x_te = x_train[test_index]\n\n        clf.train(x_tr, y_tr)\n\n        oof_train[test_index] = clf.predict(x_te)\n        oof_test_skf[i, :] = clf.predict(x_test)\n\n    oof_test[:] = oof_test_skf.mean(axis=0)\n    return oof_train.reshape(-1, 1), oof_test.reshape(-1, 1)\n\nridge_params = {'alpha':20.0, 'fit_intercept':True, 'normalize':False, 'copy_X':True,\n                'max_iter':None, 'tol':0.001, 'solver':'auto', 'random_state':SEED}\nridge = SklearnWrapper(clf=Ridge, seed = SEED, params = ridge_params)\nridge_oof_train, ridge_oof_test = get_oof(ridge, np.array(df[:ntrain]), y, np.array(df[ntrain:]))\n\n\ndef rmse(y, y0):\n    assert len(y) == len(y0)\n    return np.sqrt(np.mean(np.power((y - y0), 2)))\nrms = sqrt(mean_squared_error(y, ridge_oof_train))\nprint('Ridge OOF RMSE: {}'.format(rms))\n\nridge_preds = np.concatenate([ridge_oof_train, ridge_oof_test])\ndf['ridge_preds'] = ridge_preds\n\ntrain = df[:ntrain]\ntrain[\"deal_probability\"] = y\ntest = df[ntrain:]\npredictors.append(\"ridge_preds\")","execution_count":53,"outputs":[]},{"metadata":{"_cell_guid":"7e94bcc4-ab56-4a38-bdcf-218791e95c2c","_uuid":"ac57e30cf4452005a2cb20db424d7bcf1f11d1df","trusted":true,"collapsed":true},"cell_type":"code","source":"feature_names = np.hstack([\n    count_vectorizer_desc.get_feature_names(),\n    count_vectorizer_title.get_feature_names(),\n    predictors\n])\nprint('Number of features:', len(feature_names))","execution_count":25,"outputs":[]},{"metadata":{"_cell_guid":"9e910115-312f-4437-8ba3-9fb6f7426af8","_uuid":"1a50b3e34d76019fdbfd45c3566eec09556f24cb","trusted":true,"collapsed":true},"cell_type":"code","source":"train_index, valid_index = train_test_split(np.arange(len(train)), test_size=0.1, random_state=42)\n\nx_train = scipy.sparse.hstack([\n        train_desc_counts[train_index],\n        train_title_counts[train_index],\n        train.loc[train_index, predictors]\n], format='csr')\ny_train = train.loc[train_index, target]\n\nx_valid = scipy.sparse.hstack([\n    train_desc_counts[valid_index],\n    train_title_counts[valid_index],\n    train.loc[valid_index, predictors]\n], format='csr')\ny_valid = train.loc[valid_index, target]\n\nx_test = scipy.sparse.hstack([\n    test_desc_counts,\n    test_title_counts,\n    test.loc[:, predictors]\n], format='csr')\n\ndtrain = lgb.Dataset(x_train, label=y_train,\n                     feature_name=list(feature_names), \n                     categorical_feature=categorical)\ndvalid = lgb.Dataset(x_valid, label=y_valid,\n                     feature_name=list(feature_names), \n                     categorical_feature=categorical)","execution_count":60,"outputs":[]},{"metadata":{"_cell_guid":"cfc958c3-f045-49d4-8017-be7a60d7e18b","_uuid":"40209bda4fe8e803d8e56e0c98ed4ec0375eac94","trusted":true,"collapsed":true},"cell_type":"code","source":"rounds = 16000\nearly_stop_rounds = 500\nparams = {\n    'objective' : 'regression',\n    'metric' : 'rmse',\n    'num_leaves' : 32,\n    'max_depth': 15,\n    'learning_rate' : 0.02,\n    'feature_fraction' : 0.6,\n    'verbosity' : -1\n}\n\n\nevals_result = {}\nmodel = lgb.train(params, dtrain, \n                  valid_sets=[dtrain, dvalid], \n                  valid_names=['train', 'valid'],\n                  num_boost_round=rounds, \n                  early_stopping_rounds=early_stop_rounds, \n                  verbose_eval=500)","execution_count":65,"outputs":[]},{"metadata":{"_cell_guid":"c1870612-7d3a-4871-a1de-27b379c9e970","_uuid":"723d20c1ee33b7ccbeb3e0a428048300f0a86944"},"cell_type":"markdown","source":"That looks good. But the model is kind of a black box. It is a good idea to plot the feature importances for our model now."},{"metadata":{"_cell_guid":"5827b4fb-0bfc-4ef2-ab43-5be682151de8","_uuid":"37cbdd3d10742d44e169a2db4386f7bb3c7b09bc","trusted":true,"collapsed":true},"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(10, 14))\nlgb.plot_importance(model, max_num_features=50, ax=ax)\nplt.title(\"Light GBM Feature Importance\")","execution_count":30,"outputs":[]},{"metadata":{"_cell_guid":"9d49a8cc-4b52-47f4-868a-f7b3006cd9f3","_uuid":"3c93c6232801704e05d2a94c38b7df77d7ed14f4","trusted":true,"collapsed":true},"cell_type":"code","source":"subm = pd.read_csv('../input/avito-demand-prediction/sample_submission.csv')\nsubm['deal_probability'] = np.clip(model.predict(x_test), 0, 1)\nsubm.to_csv('Aggregate_Ridge.csv', index=False)","execution_count":32,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.5","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}