{"cells":[{"metadata":{"_cell_guid":"a83bf0d3-fac7-4204-be85-ce7d7a7e5318","_uuid":"6c312474f35f552a05c0be7d86a3d2fd11d7026c"},"cell_type":"markdown","source":"In this kernel we will take a look at engineering aggregated features. Specifically, we will engineer 3 aggregated features using `train_active.csv`, `test_active.csv`, `periods_test.csv` and `periods_test.csv`. Those features will be:\n- `avg_times_up_user` - how often the average item of the user has been put up for sale.\n- `avg_days_up_user` - the average number of days an item from the user has been put up for sale.\n- `n_user_items` - the number of items the user has put up for sale.\n\nLet's see if they help :)"},{"metadata":{"_cell_guid":"daa83617-9966-4c15-ae6d-37f5c7e0c693","_uuid":"fd6890865a5d2c9e1e94bc76a159d62a312a30af"},"cell_type":"markdown","source":"# Engineering aggregated features"},{"metadata":{"_cell_guid":"033852ed-2dd3-4bc3-bec1-ac3481ab175f","_kg_hide-input":true,"_uuid":"af7422df064b7f16934cca9e86e8213bc9c667e8","collapsed":true,"trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport gc\nfrom tqdm import tqdm\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom matplotlib_venn import venn2, venn2_circles\nimport string\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import mean_squared_error\nfrom sklearn.preprocessing import LabelEncoder\nfrom sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer\nimport nltk\nfrom nltk.corpus import stopwords\nimport scipy\nimport lightgbm as lgb\n\nsns.set()\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"e5717975-f70f-48fe-b09d-1d1b088e2855","_uuid":"9c5f66436f0108a7a19179c65a8153f072fd4618"},"cell_type":"markdown","source":"Start with loading the data. To save some memory, we only load `item_id` and `user_id`, as that's all we need for the proposed features."},{"metadata":{"_cell_guid":"5f037998-0ef3-44a8-816c-d6fd79fb08dd","_uuid":"3a30dfe38892ca287f1a1d6de4b0dfe0cde5c309","collapsed":true,"trusted":true},"cell_type":"code","source":"used_cols = ['item_id', 'user_id']\n\ntrain = pd.read_csv('../input/train.csv', usecols=used_cols)\ntrain_active = pd.read_csv('../input/train_active.csv', usecols=used_cols)\ntest = pd.read_csv('../input/test.csv', usecols=used_cols)\ntest_active = pd.read_csv('../input/test_active.csv', usecols=used_cols)\n\ntrain_periods = pd.read_csv('../input/periods_train.csv', parse_dates=['date_from', 'date_to'])\ntest_periods = pd.read_csv('../input/periods_test.csv', parse_dates=['date_from', 'date_to'])\n\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"b3320539-ff8f-4b9f-86db-5c9aec03e2cf","_uuid":"f52e08c3356976134bf45e8f3b343e8aa14e1694"},"cell_type":"markdown","source":"It's time for some visualizations. The following venn diagrams show the overlap of the user ID between the relevant dataframes. If this overlap is reasonably large, it might be a good idea to use the aggregated features."},{"metadata":{"_cell_guid":"9c6fe7ab-1a01-44aa-af5e-c0679159b94e","_kg_hide-input":true,"_uuid":"71e3765fa171b50f3c515d1b8d5ad678aaff3913","collapsed":true,"trusted":true},"cell_type":"code","source":"fig, axarr = plt.subplots(2, 2, figsize=(14, 7))\n\ndef get_venn(axarr, feature):\n    axarr[0, 0].set_title(f'Overlap between {feature} in train and train_active')\n    venn2([\n        set(train[feature].values), \n        set(train_active[feature].values)\n    ], set_labels = ('train', 'train_active'), ax=axarr[0, 0])\n\n    axarr[0, 1].set_title(f'Overlap between {feature} in test and test_active')\n    venn2([\n        set(test[feature].values), \n        set(test_active[feature].values)\n    ], set_labels = ('test', 'test_active'), ax=axarr[0, 1])\n\n    axarr[1, 0].set_title(f'Overlap between {feature} in train and test')\n    venn2([\n        set(train[feature].values), \n        set(test[feature].values)\n    ], set_labels = ('train', 'test'), ax=axarr[1, 0])\n\n    axarr[1, 1].set_title(f'Overlap between {feature} in train_active and test_active')\n    venn2([\n        set(train_active[feature].values), \n        set(test_active[feature].values)\n    ], set_labels = ('train_active', 'test_active'), ax=axarr[1, 1])\n    \nget_venn(axarr, 'user_id')","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"5f20cf26-18e4-4c0e-aad5-9f2b8ba0be57","_uuid":"b462d2fd16ed326e3aed3519ccbc1918a5a6e858"},"cell_type":"markdown","source":"We're lucky! There is a huge overlap between the IDs of `train` / `train_active` and `test` / `test_active`. Out of curiosity, we'll also take a look at the overlap of item ID. This should (hopefully) not overlap at all."},{"metadata":{"_cell_guid":"7b470f63-1ed4-4dc5-9bf9-8c62b07c1565","_kg_hide-input":true,"_uuid":"c7b9ab8df8929db55e292b3c86c3835365026dd6","collapsed":true,"trusted":true},"cell_type":"code","source":"fig, axarr = plt.subplots(2, 2, figsize=(14, 7))\n\nget_venn(axarr, 'item_id')","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"611c1af8-92cd-4860-96d6-d5a6df7cd794","_uuid":"69977e8dad0525ea2530b37f1f124f3f1699838a"},"cell_type":"markdown","source":"As suspected, there is no overlap between the dataframes. Except of `train_active` and `test_active`. These might be duplicated rows or items that have been put up for sale multiple times. We will have to filter these duplicated IDs for our engineered features.\n\nAnyway, we will now merge the data into one dataframe and, as mentioned, drop the duplicate item IDs. At this point we can also delete `train_active` and `test_active` to free up some memory."},{"metadata":{"_cell_guid":"919c8404-8aeb-4e47-9fed-4dc861bb9e5f","_kg_hide-input":false,"_kg_hide-output":true,"_uuid":"f9fe381d5064e44be5847c81bd1b575f25d096dd","collapsed":true,"trusted":true},"cell_type":"code","source":"all_samples = pd.concat([\n    train,\n    train_active,\n    test,\n    test_active\n]).reset_index(drop=True)\nall_samples.drop_duplicates(['item_id'], inplace=True)\n\ndel train_active\ndel test_active\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"e5b026eb-c1aa-44d1-a1d5-83e2cffb268b","_uuid":"e104c5748b46833328f152747afcbb37fd4c9534"},"cell_type":"markdown","source":"We will also concatenate the train and test period data to one dataframe for easier processing."},{"metadata":{"_cell_guid":"250ae468-8176-41e9-bfb5-802b5a6f1cf9","_uuid":"b68957f7f487eb155da75aeb6e54eb7a47666b8b","collapsed":true,"trusted":true},"cell_type":"code","source":"all_periods = pd.concat([\n    train_periods,\n    test_periods\n])\n\ndel train_periods\ndel test_periods\ngc.collect()\n\nall_periods.head()","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"58606458-c901-44c1-af9b-ecfa4ddbc132","_uuid":"ec346f51f4df2574560f0f1774dabb34f6b03f3f"},"cell_type":"markdown","source":"Now the interesting part begins! For our feature `avg_days_up_user`, we first have to calculate the number of days every item has been put up. This can easily be done with pandas's `dt` API."},{"metadata":{"_cell_guid":"ee553d0c-0142-4f60-8d56-d0c0110501c5","_uuid":"20d4b8eb2831660155e18f53aae6e07225d69d93","collapsed":true,"trusted":true},"cell_type":"code","source":"all_periods['days_up'] = all_periods['date_to'].dt.dayofyear - all_periods['date_from'].dt.dayofyear","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"53545672-fd45-4f3e-80f8-fd29efb12e06","_uuid":"b209601a09d97d2da4db692781fb8d06d138d0b3"},"cell_type":"markdown","source":"Because we want the sum of days one item has been put up for sale, we will group by `item_id` and sum the `days_up` column. We will also count the number of items in an item ID group for our second feature, `avg_times_up_user`."},{"metadata":{"_cell_guid":"0fc31fb0-f28f-4694-9efe-5dc0e204d95c","_uuid":"948194fc5532f404ffb559a8acc57735756a5ad5","collapsed":true,"trusted":true},"cell_type":"code","source":"gp = all_periods.groupby(['item_id'])[['days_up']]\n\ngp_df = pd.DataFrame()\ngp_df['days_up_sum'] = gp.sum()['days_up']\ngp_df['times_put_up'] = gp.count()['days_up']\ngp_df.reset_index(inplace=True)\ngp_df.rename(index=str, columns={'index': 'item_id'})\n\ngp_df.head()","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"3741ae4c-cb9c-4dd7-80d3-f09551ad759c","_uuid":"99e1a732686f0ab6cfc8b42a99f7393ba2cd58a8"},"cell_type":"markdown","source":"At this point, we have 2 scalars associated with every `item_id` appearing in train and test periods. We can now savely drop the duplicate item IDs in the `all_periods` dataframe and merge the features back into `all_periods`."},{"metadata":{"_cell_guid":"426f6a1a-a1cf-4eec-b368-4c76f47fa71b","_uuid":"12d89f448d1956996b61b9ab5f17ac350947d611","collapsed":true,"trusted":true},"cell_type":"code","source":"all_periods.drop_duplicates(['item_id'], inplace=True)\nall_periods = all_periods.merge(gp_df, on='item_id', how='left')\nall_periods.head()","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"acf91783-7938-455a-8a31-e171a2aefaf4","_kg_hide-input":true,"_kg_hide-output":true,"_uuid":"1334f7eb4b40527243fb92de0e4b9b3d7ff1b81f","collapsed":true,"trusted":true},"cell_type":"code","source":"del gp\ndel gp_df\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"f370ff7d-f014-4242-8804-1878b4ba0d3d","_uuid":"de8507310745edefe60ffcb9cfcb330413f6b62c"},"cell_type":"markdown","source":"We have an interesting but kind of useless feature now. As seen in the second venn diagram, there is no overlap at all between `train_active` (and with that `train_periods`) and `train` concerning *item* IDs. For the feature to become useful, we somehow have to associate an item ID with a user ID.\n\nSo the next step is to merge `all_samples` into `all_periods`. This will get us a user ID for every item ID in the periods dataframes. Now there is an overlap!"},{"metadata":{"_cell_guid":"b49b8f80-bc74-4b18-b934-7e0a60b2cfa3","_uuid":"c24f2167dcd8d85a740418a5c3e2d5d09926b659","collapsed":true,"trusted":true},"cell_type":"code","source":"all_periods = all_periods.merge(all_samples, on='item_id', how='left')\nall_periods.head()","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"96c19581-1fa2-4e72-b5d5-30a2713dae4d","_uuid":"a3feae7571a6e25b0369644de5371e38ae4e6241"},"cell_type":"markdown","source":"The next problem is that there are multiple features for a user if that user has put up more than one item that appears in `train_active` / `test_active`. We will have to somehow reduce this to one feature.\n\nHere they are averaged, but you can try something else like median or modus too."},{"metadata":{"_cell_guid":"4abe1fbb-0e28-48c3-a849-f6860a1ee164","_uuid":"afef398b80af41c478296dda561deb78d02c2bbb","collapsed":true,"trusted":true},"cell_type":"code","source":"gp = all_periods.groupby(['user_id'])[['days_up_sum', 'times_put_up']].mean().reset_index() \\\n    .rename(index=str, columns={\n        'days_up_sum': 'avg_days_up_user',\n        'times_put_up': 'avg_times_up_user'\n    })\ngp.head()","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"589c1734-4c4d-4d83-8b07-8b6404cd45d3","_uuid":"ee0e759eded9ba1e1cc4743f1ebd1cf26b08f960"},"cell_type":"markdown","source":"For our last feature, `n_user_items`, we just group by user ID and count the number of items. We have to be careful to use `all_samples` instead of `all_periods` here because the latter does not contain the `train.csv` and `test.csv` samples."},{"metadata":{"_cell_guid":"772b3a50-0b74-4822-9a4c-b01bfe612afb","_uuid":"5067101e74c7b1e98778457b9c6e4d3b39e579c2","collapsed":true,"trusted":true},"cell_type":"code","source":"n_user_items = all_samples.groupby(['user_id'])[['item_id']].count().reset_index() \\\n    .rename(index=str, columns={\n        'item_id': 'n_user_items'\n    })\ngp = gp.merge(n_user_items, on='user_id', how='outer')\n\ngp.head()","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"1d3bed01-9f20-4f25-b850-e079dfc55d85","_uuid":"946a126873114ed35fcdb4bb3955338d22d39e3b"},"cell_type":"markdown","source":"I'll save the features to a CSV so you don't have to run the entire code yourself if you want to try them in your model."},{"metadata":{"_cell_guid":"4f36dc3c-d24e-4823-9bfe-accff883c869","_uuid":"d4f16e0ebffc27b2eda29a97c64a5343c45ba2b0","collapsed":true,"trusted":true},"cell_type":"code","source":"gp.to_csv('aggregated_features.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"fc0baa8f-82c3-49d3-9265-f6259cc4d976","_kg_hide-output":true,"_uuid":"ca1af4070e69b5d143005d6d0cb44eb0d0e3ac42","collapsed":true,"trusted":true},"cell_type":"code","source":"del all_samples\ndel all_periods\ndel train\ndel test\n\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"94a03718-1718-4981-88ef-532efcb435ee","_uuid":"5d8e0a1a86fe5fbfa5162d3180d0b1ebea94f97d"},"cell_type":"markdown","source":"# Training a LightGBM model"},{"metadata":{"_cell_guid":"bc719fcf-f418-4db5-a4fa-ebfa27716521","_uuid":"02aad0b977053435e089bd189ea45e4a88731e87","collapsed":true,"trusted":true},"cell_type":"code","source":"train = pd.read_csv('../input/train.csv')\ntest = pd.read_csv('../input/test.csv')\n\ntrain = train.merge(gp, on='user_id', how='left')\ntest = test.merge(gp, on='user_id', how='left')\n\nagg_cols = list(gp.columns)[1:]\n\ndel gp\ngc.collect()\n\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"e624308f-c957-48e8-87f6-f9e78fc5b69f","_uuid":"abbecd367f012736903c150e6fb989de3e135cca"},"cell_type":"markdown","source":"One more thing about the approach that I haven't mentioned yet is that we will have quite some NaN values because not every ID in `train` and `test` occurs in `train_active` and `test_active`. Let's check how big that problem is."},{"metadata":{"_cell_guid":"d6a65f65-04a0-4d0d-81ec-4ed0551308bc","_uuid":"65a2f2fe6ca76cc9f742affe57d4bd491f29c3bc","collapsed":true,"trusted":true},"cell_type":"code","source":"train[agg_cols].isna().any(axis=1).sum() / len(train) * 100","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"5e26f756-3f9e-4eb7-bcbe-5eaedc7d9f35","_uuid":"e088f55ab84dbd80c5a466e551d9eeb44ca9e623","collapsed":true,"trusted":true},"cell_type":"code","source":"test[agg_cols].isna().any(axis=1).sum() / len(test) * 100","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"06a9a6c2-a602-48c5-bcb5-d7ed40cc463d","_uuid":"813b1e3a93f96e3bc7d9f2375e452f8b109112fa"},"cell_type":"markdown","source":"We have missing features for 22.41% of train and 24.35% of test data. That's not perfect but certainly acceptable. Onto some more basic feature engineering with ideas from [a great kernel](https://www.kaggle.com/tunguz/bow-meta-text-and-dense-features-lb-0-2241?scriptVersionId=3603709)."},{"metadata":{"_cell_guid":"41e1111d-674e-4c86-b232-c0086832ee6d","_uuid":"9c60409e1dc6c618f229e15c2b368660385c2ec1","collapsed":true,"trusted":true},"cell_type":"code","source":"count = lambda l1,l2: sum([1 for x in l1 if x in l2])\n\n\nfor df in [train, test]:\n    df['description'].fillna('unknowndescription', inplace=True)\n    df['title'].fillna('unknowntitle', inplace=True)\n\n    df['weekday'] = pd.to_datetime(df['activation_date']).dt.day\n    \n    for col in ['description', 'title']:\n        df['num_words_' + col] = df[col].apply(lambda comment: len(comment.split()))\n        df['num_unique_words_' + col] = df[col].apply(lambda comment: len(set(w for w in comment.split())))\n\n    df['words_vs_unique_title'] = df['num_unique_words_title'] / df['num_words_title'] * 100\n    df['words_vs_unique_description'] = df['num_unique_words_description'] / df['num_words_description'] * 100\n    \n    df['city'] = df['region'] + '_' + df['city']\n    df['num_desc_punct'] = df['description'].apply(lambda x: count(x, set(string.punctuation)))\n    \n    for col in agg_cols:\n        df[col].fillna(-1, inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"8d61ada6-8f3a-4b58-baad-199589122985","_uuid":"cc558a03cbddc3954b01783636785cdee6cf954a","collapsed":true,"trusted":true},"cell_type":"code","source":"count_vectorizer_title = CountVectorizer(stop_words=stopwords.words('russian'), lowercase=True, min_df=25)\n\ntitle_counts = count_vectorizer_title.fit_transform(train['title'].append(test['title']))\n\ntrain_title_counts = title_counts[:len(train)]\ntest_title_counts = title_counts[len(train):]\n\n\ncount_vectorizer_desc = TfidfVectorizer(stop_words=stopwords.words('russian'), \n                                        lowercase=True, ngram_range=(1, 2),\n                                        max_features=15000)\n\ndesc_counts = count_vectorizer_desc.fit_transform(train['description'].append(test['description']))\n\ntrain_desc_counts = desc_counts[:len(train)]\ntest_desc_counts = desc_counts[len(train):]\n\ntrain_title_counts.shape, train_desc_counts.shape","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"56338e56-c266-4342-a8f3-10621db49dee","_uuid":"cc6f15f981f05ad9b1e89ab7acb54ea2350278c6","collapsed":true,"trusted":true},"cell_type":"code","source":"target = 'deal_probability'\npredictors = [\n    'num_desc_punct', \n    'words_vs_unique_description', 'num_unique_words_description', 'num_unique_words_title', 'num_words_description', 'num_words_title',\n    'avg_times_up_user', 'avg_days_up_user', 'n_user_items', \n    'price', 'item_seq_number'\n]\ncategorical = [\n    'image_top_1', 'param_1', 'param_2', 'param_3', \n    'city', 'region', 'category_name', 'parent_category_name', 'user_type'\n]\n\npredictors = predictors + categorical","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"7695c1a1-06cf-40e6-8792-77b62f48262a","_uuid":"0bcde9bb9b9b26347c7db7da47123c6fec40860e","collapsed":true,"trusted":true},"cell_type":"code","source":"for feature in categorical:\n    print(f'Transforming {feature}...')\n    encoder = LabelEncoder()\n    encoder.fit(train[feature].append(test[feature]).astype(str))\n    \n    train[feature] = encoder.transform(train[feature].astype(str))\n    test[feature] = encoder.transform(test[feature].astype(str))","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"edf2ac02-764e-42d3-bd00-8c1f79878d52","_uuid":"312ce17bcca12df526c85cb9c85e7ac67c9246c7"},"cell_type":"markdown","source":"After some hyperparameter definitions and creating train / valid / test matrices, we can finally train the model. Let's see if the aggregated features helped.\n\n*Note: For further feature engineering, I would recommend restricting the max_depth further (5 worked well for me) and increasing the learning rate (to ~ 0.1) so you don't have to wait forever for the training to finish.*"},{"metadata":{"_cell_guid":"7e94bcc4-ab56-4a38-bdcf-218791e95c2c","_uuid":"ac57e30cf4452005a2cb20db424d7bcf1f11d1df","collapsed":true,"trusted":true},"cell_type":"code","source":"rounds = 16000\nearly_stop_rounds = 500\nparams = {\n    'objective' : 'regression',\n    'metric' : 'rmse',\n    'num_leaves' : 32,\n    'max_depth': 15,\n    'learning_rate' : 0.02,\n    'feature_fraction' : 0.6,\n    'verbosity' : -1\n}\n\nfeature_names = np.hstack([\n    count_vectorizer_desc.get_feature_names(),\n    count_vectorizer_title.get_feature_names(),\n    predictors\n])\nprint('Number of features:', len(feature_names))","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"9e910115-312f-4437-8ba3-9fb6f7426af8","_uuid":"1a50b3e34d76019fdbfd45c3566eec09556f24cb","collapsed":true,"trusted":true},"cell_type":"code","source":"train_index, valid_index = train_test_split(np.arange(len(train)), test_size=0.1, random_state=42)\n\nx_train = scipy.sparse.hstack([\n        train_desc_counts[train_index],\n        train_title_counts[train_index],\n        train.loc[train_index, predictors]\n], format='csr')\ny_train = train.loc[train_index, target]\n\nx_valid = scipy.sparse.hstack([\n    train_desc_counts[valid_index],\n    train_title_counts[valid_index],\n    train.loc[valid_index, predictors]\n], format='csr')\ny_valid = train.loc[valid_index, target]\n\nx_test = scipy.sparse.hstack([\n    test_desc_counts,\n    test_title_counts,\n    test.loc[:, predictors]\n], format='csr')\n\ndtrain = lgb.Dataset(x_train, label=y_train,\n                     feature_name=list(feature_names), \n                     categorical_feature=categorical)\ndvalid = lgb.Dataset(x_valid, label=y_valid,\n                     feature_name=list(feature_names), \n                     categorical_feature=categorical)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"cfc958c3-f045-49d4-8017-be7a60d7e18b","_uuid":"40209bda4fe8e803d8e56e0c98ed4ec0375eac94","collapsed":true,"trusted":true},"cell_type":"code","source":"evals_result = {}\nmodel = lgb.train(params, dtrain, \n                  valid_sets=[dtrain, dvalid], \n                  valid_names=['train', 'valid'],\n                  num_boost_round=rounds, \n                  early_stopping_rounds=early_stop_rounds, \n                  verbose_eval=500)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"c1870612-7d3a-4871-a1de-27b379c9e970","_uuid":"723d20c1ee33b7ccbeb3e0a428048300f0a86944"},"cell_type":"markdown","source":"That looks good. But the model is kind of a black box. It is a good idea to plot the feature importances for our model now."},{"metadata":{"_cell_guid":"5827b4fb-0bfc-4ef2-ab43-5be682151de8","_uuid":"37cbdd3d10742d44e169a2db4386f7bb3c7b09bc","collapsed":true,"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(10, 14))\nlgb.plot_importance(model, max_num_features=50, ax=ax)\nplt.title(\"Light GBM Feature Importance\")","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"959449c6-6712-4447-9223-9ac8dd094e22","_uuid":"1b9e07f74d5599578910c4d7486f60dc1f6e5688"},"cell_type":"markdown","source":"`avg_days_up`, `avg_times_up_user` and `n_user_items` are our most important engineered features! Looks like we were successful. Now we just have to predict the test matrix and submit!"},{"metadata":{"_cell_guid":"9d49a8cc-4b52-47f4-868a-f7b3006cd9f3","_uuid":"3c93c6232801704e05d2a94c38b7df77d7ed14f4","collapsed":true,"trusted":true},"cell_type":"code","source":"subm = pd.read_csv('../input/sample_submission.csv')\nsubm['deal_probability'] = np.clip(model.predict(x_test), 0, 1)\nsubm.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"e41f2802-c21c-471b-a7b3-f190aec1a7f2","_uuid":"478c61d627e87de36aa762a69b1c10ff6c718788"},"cell_type":"markdown","source":"I'll end this kernel with some ideas to improve it:\n- Use K-Fold cross validation.\n- Try other methods than mean for reducing the aggregated features to one per user (e. g. modus or median).\n- Try other gradient boosting libraries like CatBoost or XGBoost.\n- Add a temporal dimension to engineered features (e. g. # of items a user put up for sale *per day*).\n- Add more advanced text features like pretrained word embeddings.\n- Add image features. At the moment we completely ignore images! (as discussed [here](https://www.kaggle.com/c/avito-demand-prediction/discussion/56678), two promising approaches could be [NIMA: Neural Image Assessment](https://arxiv.org/abs/1709.05424) and [Multimedia Features for Click Prediction](https://storage.googleapis.com/kaggle-forum-message-attachments/328059/9411/dimitri-clickadvert.pdf)).\n- Normalize text before creating the Tf-Idf matrix (e. g. using [stemming](http://www.nltk.org/howto/stem.html)).\n- ~~Learn russian and do in-depth text analysis.~~\n\nThanks for reading and have fun in this competition!"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.5","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}