{"cells":[{"metadata":{"_uuid":"e551059d4d292873b899e49dd701c136346eb9f0"},"cell_type":"markdown","source":"**Avito Demand Prediction Challenge:**\n\nDescription of the challenge taken from the competition page:\n\nWhen selling used goods online, a combination of tiny, nuanced details in a product description can make a big difference in drumming up interest. \n\nAnd, even with an optimized product listing, demand for a product may simply not exist–frustrating sellers who may have over-invested in marketing.\n\nAvito, Russia’s largest classified advertisements website, is deeply familiar with this problem. Sellers on their platform sometimes feel frustrated with both too little demand (indicating something is wrong with the product or the product listing) or too much demand (indicating a hot item with a good description was underpriced).\n\nIn their fourth Kaggle competition, Avito is challenging you to predict demand for an online advertisement based on its full description (title, description, images, etc.), its context (geographically where it was posted, similar ads already posted) and historical demand for similar ads in similar contexts. With this information, Avito can inform sellers on how to best optimize their listing and provide some indication of how much interest they should realistically expect to receive.\n\n**What are we doing in this kernel?**\n\nThe aim of this kernel is to perform EDA of Avito demand prediction challenge competition as well add new features which would be used by the LightGBM library for performing prediction. "},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import os\nimport pandas_profiling as pp\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n%matplotlib inline\nplt.style.use('ggplot')\n\nimport datetime\nimport pandas_profiling as pp\n\nfrom scipy import stats\nfrom scipy.sparse import hstack, csr_matrix\nfrom sklearn.model_selection import train_test_split\nfrom wordcloud import WordCloud\nfrom collections import Counter\nfrom nltk.corpus import stopwords\nfrom nltk.util import ngrams\n\nfrom sklearn.feature_extraction.text import TfidfVectorizer\nstop = set(stopwords.words('russian'))\n\nimport lightgbm as lgb\nfrom sklearn.linear_model import Ridge\nfrom sklearn.model_selection import cross_val_predict\nfrom sklearn.model_selection import KFold\n\nkf = KFold(n_splits=5)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"df_train = pd.read_csv(\"../input/train.csv\")\ndf_test  = pd.read_csv(\"../input/test.csv\")\ndf_sub   = pd.read_csv(\"../input/sample_submission.csv\")\ndf_periods_train = pd.read_csv(\"../input/periods_train.csv\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"398e2546647edf97dd34b643b1b914b029541066"},"cell_type":"markdown","source":"**Data Overview:**\n\nLets understand the data that we have for this kernel."},{"metadata":{"trusted":true,"_uuid":"6168003c29dddbb0eedd89c1dc453a933e5e8ecc"},"cell_type":"code","source":"df_train.head(5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"076cd091fb817cab0836c54fb772bdb472c32b76"},"cell_type":"code","source":"df_train.info()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"97160e066e5821a34545e128aa961437ddb57279"},"cell_type":"markdown","source":"Columns:\n- item_id\n- user_id\n- region\n- city\n- parent_category_name\n- category_name\n- param_1 \n- param_2\n- param_3\n- title\n- description\n- price\n- item_seq_number\n- activation_date\n- user_type\n- image\n- image_top_1\n- deal_probability\n"},{"metadata":{"trusted":true,"_uuid":"abf798a37f53fa61872db4cf338ff19aca5aca84"},"cell_type":"code","source":"df_train.describe(include='all')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"435959e86966538dae36a87df30dd8accda8df5e"},"cell_type":"markdown","source":"As majority of the columns are categorical variables, we have very few numerical variables. "},{"metadata":{"trusted":true,"_uuid":"25ad683ad7899edc9449eb487badaec9583b74b4"},"cell_type":"code","source":"df_periods_train.head(5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"cc52a8433225df00c16a8b44b3e517d49bb2b48a"},"cell_type":"code","source":"df_periods_train.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"db3d2e6b185cb582b87bfca0b6b868e5f731458e"},"cell_type":"code","source":"df_periods_train.describe(include='all')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6f86c5d4d2d1483c757c48bc3abffbd5e0f756f2"},"cell_type":"markdown","source":"**Pandas profiling:**\n\nGenerates profile reports from a pandas DataFrame. The pandas df.describe() function is great but a little basic for serious exploratory data analysis.\n\nFor each column the following statistics - if relevant for the column type - are presented in an interactive HTML report:\n\n- **Essentials**: type, unique values, missing values\n- **Quantile statistics** like minimum value, Q1, median, Q3, maximum, range, interquartile range\n- **Descriptive statistics** like mean, mode, standard deviation, sum, median absolute deviation, coefficient of variation, kurtosis, skewness\n- **Most frequent values**\n- **Histogram**\n- **Correlations** highlighting of highly correlated variables, Spearman and Pearson matrixes\n"},{"metadata":{"trusted":true,"_uuid":"c8cc8408aaf2c983348d06b30c58d2658bb25d19"},"cell_type":"code","source":"pp.ProfileReport(df_train)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c7fedb0d727c4d25e4118c0ccfecf868f0e56091"},"cell_type":"markdown","source":"Let's try to understand the output of the pandas profiling:\n\n- Types of features: we have different types of features i.e. numerical, categorical, text, date\n- Missing values: some columns have missing values like some times description text is missing for a particular ad.\n- Users: We have over 51% unique users, most of them don't post lot of ads. \n- Categories: There are 9 categories in parent_category_name\n- price is skewed and hence will require careful processing. "},{"metadata":{"_uuid":"18eb05aeb2072fb2e909331f4790fed0a8285b85"},"cell_type":"markdown","source":"**Feature Analysis:**\n\nLets analyze some of the features in more details. \n\n**activation_date:**\nLets create new features based on activation_date: date, weekday, day of month.\n\n"},{"metadata":{"trusted":true,"_uuid":"23f24e043472b2cc9e20730f2427ee43dbf4738c"},"cell_type":"code","source":"df_train[\"activation_date\"] = pd.to_datetime(df_train[\"activation_date\"])\ndf_train[\"date\"]            = df_train[\"activation_date\"].dt.date\ndf_train[\"weekday\"]         = df_train[\"activation_date\"].dt.weekday\ndf_train[\"day\"]             = df_train[\"activation_date\"].dt.day\n\ncount_by_date_train         = df_train.groupby(\"date\")[\"deal_probability\"].count()\nmean_by_date_train          = df_train.groupby(\"date\")[\"deal_probability\"].mean()\n\ndf_test[\"activation_date\"]  = pd.to_datetime(df_test[\"activation_date\"])\ndf_test[\"date\"]             = df_test[\"activation_date\"].dt.date\ndf_test[\"weekday\"]          = df_test[\"activation_date\"].dt.weekday\ndf_test[\"day\"]              = df_test[\"activation_date\"].dt.day\n\ncount_by_date_test          = df_test.groupby('date')['item_id'].count()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7a5c06deb44884852ac5671a7838a0ac7816446a"},"cell_type":"markdown","source":"Lets chart out the Counts and Means for deal probability and try to find relation between number of ads and mean of deal_probability to find any patterns. "},{"metadata":{"trusted":true,"_uuid":"de32ea3b9af84e4e5ed46e8c475627106d4c229d"},"cell_type":"code","source":"fig, (ax1, ax3) = plt.subplots(figsize=(26, 8), \n                              ncols=2,\n                              sharey=True)\ncount_by_date_train.plot(ax=ax1, \n                        legend=False,\n                        label=\"Ads Count\")\n\nax2 = ax1.twinx()\n\nmean_by_date_train.plot(ax=ax2,\n                       color=\"g\",\n                       legend=False,\n                       label=\"Mean deal_probability\")\n\nax2.set_ylabel(\"Mean deal_probabilit\", color=\"g\")\n\ncount_by_date_test.plot(ax=ax3,\n                       color=\"r\",\n                       legend=False,\n                       label=\"Ads count test\")\n\nplt.grid(False)\n\nax1.title.set_text(\"Trends of deal_probability and number of ads\")\nax3.title.set_text(\"Trends of number of ads for test data\")\nax1.legend(loc=(0.8, 0.35))\nax2.legend(loc=(0.8, 0.2))\nax3.legend(loc=\"upper right\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9678cd81c0f30278096f50ebe694b0c5c2802824"},"cell_type":"markdown","source":"As we can see, we have few weeks of data in training set and one week of data in the test dataset. \n\n- For most of the data in training dataset, the number of ads is quite high ( 100,000 or more) and mean deal_probability is low around 0.14, but after March 28, the number of ads drop off and so the deal_probability fluctuates.\n- In test data set, we have few ads going up to April 18th and then it falls off. \n\nFor training dataset, lets remove the data after March 29 because number of ads fall  to 0."},{"metadata":{"_uuid":"1591fdef593fb57748ae23806685606bfdbfece1"},"cell_type":"markdown","source":"Lets create a box plot between Ads Count and Deal probability by day of the week to find some patterns. "},{"metadata":{"trusted":true,"_uuid":"3928fbbc98efa9b00339e20a22ec78f48a32e894"},"cell_type":"code","source":"fig, ax1 = plt.subplots(figsize=(16, 8))\n\nplt.title(\"Ads Count and Deal Probability by day of the week\")\n\nsns.countplot(x   = \"weekday\",\n             data = df_train,\n             ax   = ax1)\n\nax1.set_ylabel(\"Ads Count\", color=\"b\")\n\nplt.legend([\"Projects Count\"])\n\nax2 = ax1.twinx()\n\nsns.pointplot(x   = \"weekday\",\n             y    = \"deal_probability\",\n             data = df_train,\n             ci   = 99,\n             ax   = ax2,\n             color = \"black\")\n\nax2.set_ylabel(\"deal_probability\", color=\"g\")\nplt.legend([\"deal_probability\"], loc=(0.875, 0.9))\nplt.grid(False)\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"77888c7d84fabeb723edccd30771eff91cdd88af"},"cell_type":"markdown","source":"Learnings from above plots:\n- Deal probability increases as the ads count fall and deal probability reaches a maximum on day 6th of the week. \n- Ads count are highest on weekends (assuming day 0 and day 6 are the weekend).\n- deal probability fall off during mid-week, potentially due to mid of the week effect \n\n"},{"metadata":{"_uuid":"1968cbfedd7936621079273556e7f76363be6f28"},"cell_type":"markdown","source":"**Categories:**\n\nLets spend some time looking at the categories. "},{"metadata":{"trusted":true,"_uuid":"951d3a7af5839a95c650a228ea0cb06460322a49"},"cell_type":"code","source":"a = df_train.groupby([\"parent_category_name\", \"category_name\"]).agg({\"deal_probability\": [\"mean\", \"count\"]}).reset_index().sort_values([(\"deal_probability\", \"mean\")], ascending=False).reset_index(drop=True)\n\na","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6287c6b4f96aa3b8e5d50f4d6a95e62b3a30b09b"},"cell_type":"markdown","source":"We can see that \"Услуги\" (services) is the category with the highest deal_probability. Other \"good\" categories are about animals or electronics/cars. Least successful are various accessories or expensive things."},{"metadata":{"_uuid":"9832f39e7154ca82d0c326a695fb53f00a97c510"},"cell_type":"markdown","source":"**City: **\n\nLets take a look at the city feature in depth. "},{"metadata":{"trusted":true,"_uuid":"232a9ebd1b9677c3e3fe7ea45f9cfc32af4671c2"},"cell_type":"code","source":"city_ads = df_train.groupby(\"city\").agg({\"deal_probability\": [\"mean\", \"count\"]}).reset_index().sort_values([(\"deal_probability\", \"mean\")], ascending=False).reset_index(drop=True)\n\nprint(\"There are {0} cities in total\".format(len(df_train.city.unique())))\n\nprint(\"There are {1} cities with more than {0} ads\".format(100, city_ads[city_ads[\"deal_probability\"][\"count\"] > 100].shape[0]))\n\nprint('There are {1} cities with more that {0} ads.'.format(1000, city_ads[city_ads['deal_probability']['count'] > 1000].shape[0]))\n\nprint('There are {1} cities with more that {0} ads.'.format(10000, city_ads[city_ads['deal_probability']['count'] > 10000].shape[0]))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7cc1023a52b606415ef767f272474838e6aeb926"},"cell_type":"markdown","source":"It seems that most of the cities have little ads posted and only in 33 of them there are a lot of ads. Let's see the best and the worst cities by mean deal_probability."},{"metadata":{"trusted":true,"_uuid":"d618a0dad463dad046ff8de60a361b63d1ea689c"},"cell_type":"code","source":"city_ads[city_ads[\"deal_probability\"][\"count\"] > 1000].head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1a2149b870246642cd97eca5d14c783f7aeb46e2"},"cell_type":"code","source":"city_ads[city_ads['deal_probability']['count'] > 1000].tail()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2011645301c71d9ba104cd3874105681a48fe081"},"cell_type":"markdown","source":"I think that it could be interesting to see what is sold in Лабинск and Миллерово"},{"metadata":{"trusted":true,"_uuid":"7811e4633c20e72456257ba62debd1640a4c0cac"},"cell_type":"code","source":"print(\"Лабинск\")\n\ndf_train.loc[df_train.city == \"Лабинск\"].groupby('category_name').agg({'deal_probability': ['mean', 'count']}).reset_index().sort_values([('deal_probability', 'count')], ascending=False).reset_index(drop=True).head(5)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"333a01e61666159dbdedcc28212370286b6d178d"},"cell_type":"markdown","source":"Most popular categories are \"Автомобили\" (cars) and \"Телефоны\" (telephones).\n\n"},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"0693cf43c3b86351bd49c3886c2e27c100e49677"},"cell_type":"code","source":"print('Миллерово')\ndf_train.loc[df_train.city == 'Миллерово'].groupby('category_name').agg({'deal_probability': ['mean', 'count']}).reset_index().sort_values([('deal_probability', 'count')], ascending=False).reset_index(drop=True).head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ab26a31183fe886996a5f6f237617ccd1f8bf179"},"cell_type":"markdown","source":"Most popular categories are \"Автомобили\" (cars). And it seems that second-hand wares are least popular.\n\n"},{"metadata":{"trusted":true,"_uuid":"024e8886427ed9a06700ed110a2c6282238051b2"},"cell_type":"markdown","source":"**deal_probability**\n\n"},{"metadata":{"trusted":true,"_uuid":"f6bab82c7c87694049b9ef09e69eb4458cfb214d"},"cell_type":"code","source":"plt.hist(df_train[\"deal_probability\"]);\nplt.title(\"deal_probability\");","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"264b44979eccfe699ec03e35d1eb827b83543c17"},"cell_type":"markdown","source":"On the one hand the distribution of the target value is highly skewered towards zero, on the other hand, there is a spike at about 0.8."},{"metadata":{"_uuid":"6f4759b7a5da48e413e978acc0ee50dbca4e16ce"},"cell_type":"markdown","source":"**title:**"},{"metadata":{"trusted":true,"_uuid":"ddb8dad935e731b936bcc00138c20a0dee959eee"},"cell_type":"code","source":"text = ' '.join(df_train[\"title\"].values)\nwordCloud = WordCloud(max_font_size = None,\n                      stopwords = stop,\n                      background_color = \"white\",\n                      width = 1200,\n                      height = 1000).generate(text)\n\nplt.figure(figsize=(12, 8))\nplt.imshow(wordCloud)\nplt.title('Top words for title')\nplt.axis(\"off\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0c2f915172efda8c6e22008c1e402b35b2f0c3ce"},"cell_type":"markdown","source":"**description:**"},{"metadata":{"trusted":true,"_uuid":"5da81a3b535abecb98b458b3c3c2675263ff4a1c"},"cell_type":"code","source":"df_train[\"description\"] = df_train[\"description\"].apply(\n    lambda x: str(x).replace('/\\n', ' ').replace('\\xa0', ' ')\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f0d0f5db45cf6ebd0b27d633d0300b7bd4d1a92d"},"cell_type":"code","source":"text = ' '.join(df_train['description'].values)\ntext = [i for i in ngrams(text.lower().split(), 3)]\nprint('Common trigrams.')\nCounter(text).most_common(40)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a68b8faaf2fc90d7d91e328cbc895868a90e25c8"},"cell_type":"markdown","source":"We can see that sellers try to tell buyers that their wares are great and also tell about possibilities of delivery. But there are some strange values, let's have a look...\n\n"},{"metadata":{"trusted":true,"_uuid":"c30cf401acfb583ae1dd2c855f8cf2e75962ba01"},"cell_type":"code","source":"df_train[df_train.description.str.contains('↓')]['description'].head(10).values","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2c4c64860e5c5b544f5b6399562746b535ce232a"},"cell_type":"markdown","source":"Some of the authors are using emoticons for their ads. "},{"metadata":{"_uuid":"69f38d5ac211bb2a8c5b5c015c8ad9c48ee736c5"},"cell_type":"markdown","source":"**image:**\n\nIn this kernel I won't use the images themselves, but I'll create a feature showing wheather there is an image or not"},{"metadata":{"trusted":true,"_uuid":"3d499deea49115b97e3450bd7458a8f7f3586ddb"},"cell_type":"code","source":"df_train['has_image'] = 1\ndf_train.loc[df_train['image'].isnull(),'has_image'] = 0\n\nprint('There are {} ads with images. Mean deal_probability is {:.3}.'.format(len(df_train.loc[df_train['has_image'] == 1]), df_train.loc[df_train['has_image'] == 1, 'deal_probability'].mean()))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4a6c10e23117ec742149acb9c52c21cf7990a852"},"cell_type":"code","source":"print('There are {} ads without images. Mean deal_probability is {:.3}.'.format(len(df_train.loc[df_train['has_image'] == 0]), df_train.loc[df_train['has_image'] == 0, 'deal_probability'].mean()))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5ecefe05b0d2a9f413933eb30932dcc372a60f67"},"cell_type":"markdown","source":"**item_seq_number:**"},{"metadata":{"trusted":true,"_uuid":"d5bc829fe28f56b6db5c0d4b184e2ae4b2c57635"},"cell_type":"code","source":"plt.scatter(df_train.item_seq_number, df_train.deal_probability, label=\"item_seq_number vs deal_probability\");\n\nplt.xlabel(\"item_seq_number\");\nplt.ylabel(\"deal_probability\");","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"603f0897176d6906e0535e6e3650090bd91d533c"},"cell_type":"markdown","source":"It seems like there are many users who post a lot of ads and number of ads posted isn't really correlated with deal_probability. "},{"metadata":{"_uuid":"c70147a39ce7849c348828a63ff417f824effdab"},"cell_type":"markdown","source":"**Params:**\n\nThere are three fields with additional information, let's combine it into one. Technically it is possible to treat these features as categorical, but there would be too many of them."},{"metadata":{"trusted":true,"_uuid":"8a5b8cce98f21492a5067284cd501842f4066870"},"cell_type":"code","source":"df_train[\"params\"] = df_train[\"param_1\"].fillna('') + ' ' + df_train[\"param_2\"].fillna('') + ' ' + df_train[\"param_3\"].fillna('')\n\ndf_train[\"params\"] = df_train[\"params\"].str.strip()\n\ntext = ' '.join(df_train[\"params\"].values)\ntext = [i for i in ngrams(text.lower().split(), 3)]\n\nprint(\"common trigrams\")\n\nCounter(text).most_common(40)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9f32b4849f293489c3e30e8e3c3b171d6c1e4a49"},"cell_type":"markdown","source":"Most of params belong to clothes or cars.\n\n"},{"metadata":{"_uuid":"bee29ac7d2c1b765419ca959a597d05e9bfb8c11"},"cell_type":"markdown","source":"**user_type:**\n\nthere are three main user_Types. Let's see prices of their wares, where prices are below 100,000. "},{"metadata":{"trusted":true,"_uuid":"09cc69b6562b4a55a30e48278533e8de057f063f"},"cell_type":"code","source":"sns.set(rc = {'figure.figsize': (15, 8)})\n\ndf_train_ = df_train[df_train.price.isnull() == False]\ndf_train_ = df_train.loc[df_train.price < 100000.0]\n\nsns.boxplot(x = \"parent_category_name\",\n           y = \"price\",\n           hue = \"user_type\",\n           data = df_train_)\n\nplt.title(\"Price by parent gategory and user type\")\nplt.xticks(rotation = \"vertical\")\nplt.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f6bc7be53ff950a1ca109e9a4e9a36aa762f48b6"},"cell_type":"markdown","source":"We can see that shops usually have higher prices than companies and private sellers usually have the lowest price - maybe because they are usually second-hand.\n\n"},{"metadata":{"_uuid":"502ee923466c656d2f97318340e415b8314b23bd"},"cell_type":"markdown","source":"**price:**\n\nThe first question is how to deal with missing values. I have decided to do the following:\n- at first fill missing values with median by city and category.\n- then missing values which are left are filled with region by region and category\n- the remaining missing values are filled with median by category\n"},{"metadata":{"trusted":true,"_uuid":"a8e2150fee81388bcaf4b6cd85948a38a97581ec"},"cell_type":"code","source":"df_train[\"price\"] = df_train.groupby([\"city\", \"category_name\"])[\"price\"].apply(\n    lambda x: x.fillna(x.median())\n)\n\ndf_train[\"price\"] = df_train.groupby([\"region\", \"category_name\"])[\"price\"].apply(\n    lambda x: x.fillna(x.median())\n)\n\ndf_train[\"price\"] = df_train.groupby([\"category_name\"])[\"price\"].apply(\n    lambda x: x.fillna(x.median())\n)\n\nplt.hist(df_train[\"price\"]);","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8c98d1821f0db5a01d40a6d34d378416a4d71dcb"},"cell_type":"markdown","source":"Let's use boxcox transformation to get rid of skewness\n\n"},{"metadata":{"trusted":true,"_uuid":"95af0fdbc67f69995b32adc7949daa7f4403012b"},"cell_type":"code","source":"plt.hist(stats.boxcox(df_train[\"price\"] + 1)[0]);","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3be095b94e4d3431c73766df3766389a5df3507e"},"cell_type":"markdown","source":"Imagine that you are watching a race and that you are located close to the finish line. When the first and fastest runners complete the race, the differences in times between them will probably be quite small.\n\nNow wait until the last runners arrive and consider their finishing times. For these slowest runners, the differences in completion times will be extremely large. This is due to the fact that for longer racing times a small difference in speed will have a significant impact on completion times, whereas for the fastest runners, small differences in speed will have a small (but decisive) impact on arrival times.\n\nThis phenomenon is called “heteroscedasticity” (non-constant variance). In this example, the amount of Variation depends on the average value (small variations for shorter completion times, large variations for longer times).\n\nThis distribution of running times data will probably not follow the familiar bell-shaped curve (a.k.a. the normal distribution). The resulting distribution will be asymmetrical with a longer tail on the right side. This is because there's small variability on the left side with a short tail for smaller running times, and larger variability for longer running times on the right side, hence the longer tail.\n\n\nWhy does this matter?\n\nModel bias and spurious interactions: If you are performing a regression or a design of experiments (any statistical modelling), this asymmetrical behavior may lead to a bias in the model. If a factor has a significant effect on the average speed, because the variability is much larger for a larger average running time, many factors will seem to have a stronger effect when the mean is larger. This is not due, however, to a true factor effect but rather to an increased amount of variability that affects all factor effect estimates when the mean gets larger. This will probably generate spurious interactions due to a non-constant variation, resulting in a very complex model with many (spurious and unrealistic) interactions.\n\nIf you are performing a standard capability analysis, this analysis is based on the normality assumption. A substantial departure from normality will bias your capability estimates.\n\nhttp://blog.minitab.com/blog/applying-statistics-in-quality-projects/how-could-you-benefit-from-a-box-cox-transformation"},{"metadata":{"_uuid":"cda9289a89f615b86000fa23a3aed5dbe1d1379e"},"cell_type":"markdown","source":"**Feature Engineering:**\n"},{"metadata":{"trusted":true,"_uuid":"a6daf9113e592de021637c29576df1454602ef0e"},"cell_type":"code","source":"# lets transform the test in the same way as train\n\ndf_test[\"params\"] = df_test[\"param_1\"].fillna('') + ' ' + df_test[\"param_2\"].fillna('') + ' ' + df_test[\"param_3\"].fillna('')\ndf_test[\"params\"] = df_test[\"params\"].str.strip()\n\ndf_test[\"description\"] = df_test[\"description\"].apply( lambda x: str(x).replace(\"/\\n\", ' ').replace(\"\\xa0\", \" \"))\n\ndf_test[\"has_image\"] = 1\ndf_test.loc[df_test[\"image\"].isnull(), \"has_image\"] = 0\n\ndf_test[\"price\"] = df_test.groupby([\"city\", \"category_name\"])[\"price\"].apply(lambda x: x.fillna(x.median()))\ndf_test[\"price\"] = df_test.groupby([\"region\", \"category_name\"])[\"price\"].apply(lambda x: x.fillna(x.median()))\ndf_test[\"price\"] = df_test.groupby([\"category_name\"])[\"price\"].apply(lambda x: x.fillna(x.median()))\n\ndf_train[\"price\"] = stats.boxcox(df_train.price + 1)[0]\ndf_test[\"price\"]  = stats.boxcox(df_test.price + 1)[0]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e92b79e0381f16ebbf9433683b3db88529503c11"},"cell_type":"markdown","source":"**Aggregate features:**\n\nI'll create a number of aggregate features. \n\n- user_price_mean\n- user_ad_count\n- region_price_mean\n- region_price_median\n- region_price_max\n- city_price_mean\n- city_price_median\n- city_price_max\n- parent_category_name_price_mean\n- parent_category_name_price_median\n- parent_category_name_price_max\n- category_name_price_mean\n- category_name_price_median\n- category_name_price_max\n- user_type_category_price_mean\n- user_type_category_price_median\n- user_type_category_price_nax\n\n"},{"metadata":{"trusted":true,"_uuid":"1e37391743a0488f1e3cdc430e7fa904624a764b"},"cell_type":"code","source":"df_train[\"user_price_mean\"] = df_train.groupby(\"user_id\")[\"price\"].transform(\"mean\")\ndf_train[\"user_ad_count\"]   = df_train.groupby(\"user_id\")[\"price\"].transform(\"sum\")\n\ndf_train[\"region_price_mean\"]   = df_train.groupby(\"region\")[\"price\"].transform(\"mean\")\ndf_train[\"region_price_median\"] = df_train.groupby(\"region\")[\"price\"].transform(\"median\")\ndf_train[\"region_price_max\"]    = df_train.groupby(\"region\")[\"price\"].transform(\"max\")\n\ndf_train[\"city_price_mean\"]   = df_train.groupby(\"region\")[\"price\"].transform(\"mean\")\ndf_train[\"city_price_median\"] = df_train.groupby(\"region\")[\"price\"].transform(\"median\")\ndf_train[\"city_price_max\"]    = df_train.groupby(\"region\")[\"price\"].transform(\"max\")\n\ndf_train[\"parent_category_name_price_mean\"]   = df_train.groupby(\"parent_category_name\")[\"price\"].transform(\"mean\")\ndf_train[\"parent_category_name_price_median\"] = df_train.groupby(\"parent_category_name\")[\"price\"].transform(\"median\")\ndf_train[\"parent_category_name_price_max\"]    = df_train.groupby(\"parent_category_name\")[\"price\"].transform(\"max\")\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"69cabf97e4dd07c787ba9da63c53872beb22f7ec"},"cell_type":"code","source":"df_train[\"category_name_price_mean\"]   = df_train.groupby(\"category_name\")[\"price\"].transform(\"mean\")\ndf_train[\"category_name_price_median\"] = df_train.groupby(\"category_name\")[\"price\"].transform(\"median\")\ndf_train[\"category_name_price_max\"]    = df_train.groupby(\"category_name\")[\"price\"].transform(\"max\")\n\ndf_train[\"user_type_category_price_mean\"]   = df_train.groupby([\"user_type\", \"parent_category_name\"])[\"price\"].transform(\"mean\")\ndf_train[\"user_type_category_price_median\"] = df_train.groupby([\"user_type\", \"parent_category_name\"])[\"price\"].transform(\"mean\")\ndf_train[\"user_type_category_price_mean\"]   = df_train.groupby([\"user_type\", \"parent_category_name\"])[\"price\"].transform(\"mean\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a5b145c2d7f3c95cb0e1be4cfea0b694d1278f2e"},"cell_type":"code","source":"df_test[\"user_price_mean\"] = df_test.groupby(\"user_id\")[\"price\"].transform(\"mean\")\ndf_test[\"user_ad_count\"]   = df_test.groupby(\"user_id\")[\"price\"].transform(\"sum\")\n\ndf_test[\"region_price_mean\"]   = df_test.groupby(\"region\")[\"price\"].transform(\"mean\")\ndf_test[\"region_price_median\"] = df_test.groupby(\"region\")[\"price\"].transform(\"median\")\ndf_test[\"region_price_max\"]    = df_test.groupby(\"region\")[\"price\"].transform(\"max\")\n\ndf_test[\"city_price_mean\"]   = df_test.groupby(\"region\")[\"price\"].transform(\"mean\")\ndf_test[\"city_price_median\"] = df_test.groupby(\"region\")[\"price\"].transform(\"median\")\ndf_test[\"city_price_max\"]    = df_test.groupby(\"region\")[\"price\"].transform(\"max\")\n\ndf_test[\"parent_category_name_price_mean\"]   = df_test.groupby(\"parent_category_name\")[\"price\"].transform(\"mean\")\ndf_test[\"parent_category_name_price_median\"] = df_test.groupby(\"parent_category_name\")[\"price\"].transform(\"median\")\ndf_test[\"parent_category_name_price_max\"]    = df_test.groupby(\"parent_category_name\")[\"price\"].transform(\"max\")\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"052c9a357da05e8134ca50b294bb514f08b5cf33"},"cell_type":"code","source":"df_test[\"category_name_price_mean\"]   = df_test.groupby(\"category_name\")[\"price\"].transform(\"mean\")\ndf_test[\"category_name_price_median\"] = df_test.groupby(\"category_name\")[\"price\"].transform(\"median\")\ndf_test[\"category_name_price_max\"]    = df_test.groupby(\"category_name\")[\"price\"].transform(\"max\")\n\ndf_test[\"user_type_category_price_mean\"]   = df_test.groupby([\"user_type\", \"parent_category_name\"])[\"price\"].transform(\"mean\")\ndf_test[\"user_type_category_price_median\"] = df_test.groupby([\"user_type\", \"parent_category_name\"])[\"price\"].transform(\"mean\")\ndf_test[\"user_type_category_price_mean\"]   = df_test.groupby([\"user_type\", \"parent_category_name\"])[\"price\"].transform(\"mean\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8282ae32735e3f21cefa31bb09dc3fca5c32ee96"},"cell_type":"markdown","source":"**Categorical features:**\n\nI'll use the target encoding to deal with categorical features."},{"metadata":{"trusted":true,"_uuid":"0f32a61bdc274f5459715216d090b103e81d380e"},"cell_type":"code","source":"def target_encode(trn_series = None,\n                 tst_series  = None, \n                 target      = None,\n                 min_samples_leaf = 1,\n                 smoothing   = 1,\n                 noise_level = 0):\n    \"\"\"\n    \n    https://www.kaggle.com/ogrellier/python-target-encoding-for-categorical-features\n    Smoothing is computed like in the following paper by Daniele Micci-Barreca\n    https://kaggle2.blob.core.windows.net/forum-message-attachments/225952/7441/high%20cardinality%20categoricals.pdf\n    trn_series : training categorical feature as a pd.Series\n    tst_series : test categorical feature as a pd.Series\n    target : target data as a pd.Series\n    min_samples_leaf (int) : minimum samples to take category average into account\n    smoothing (int) : smoothing effect to balance categorical average vs prior  \n    \"\"\" \n    \n    assert len(trn_series) == len(target)\n    assert trn_series.name == tst_series.name\n    temp = pd.concat([trn_series, target], axis=1)\n    # Compute target mean \n    averages = temp.groupby(by=trn_series.name)[target.name].agg([\"mean\", \"count\"])\n    \n    # Compute smoothing\n    smoothing = 1 / (1 + np.exp(-(averages[\"count\"] - min_samples_leaf) / smoothing))\n    \n    # Apply average function to all target data\n    prior = target.mean()\n    \n    # The bigger the count the less full_avg is taken into account\n    averages[target.name] = prior * (1 - smoothing) + averages[\"mean\"] * smoothing\n    averages.drop([\"mean\", \"count\"], axis=1, inplace=True)\n    \n    # Apply averages to trn and tst series\n    ft_trn_series = pd.merge(\n        trn_series.to_frame(trn_series.name),\n        averages.reset_index().rename(columns={'index': target.name, target.name: 'average'}),\n        on=trn_series.name,\n        how='left')['average'].rename(trn_series.name + '_mean').fillna(prior)\n    \n    # pd.merge does not keep the index so restore it\n    ft_trn_series.index = trn_series.index \n    ft_tst_series = pd.merge(\n        tst_series.to_frame(tst_series.name),\n        averages.reset_index().rename(columns={'index': target.name, target.name: 'average'}),\n        on=tst_series.name,\n        how='left')['average'].rename(trn_series.name + '_mean').fillna(prior)\n    \n    # pd.merge does not keep the index so restore it\n    ft_tst_series.index = tst_series.index\n    return ft_trn_series, ft_tst_series\n    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"36ee843bdd5d27f6317c6459fb63c018a9c05199"},"cell_type":"code","source":"df_train['parent_category_name'], df_test['parent_category_name'] = target_encode(df_train['parent_category_name'], df_test['parent_category_name'], df_train['deal_probability'])\n\ndf_train['category_name'], df_test['category_name'] = target_encode(df_train['category_name'], df_test['category_name'], df_train['deal_probability'])\n\ndf_train['region'], df_test['region'] = target_encode(df_train['region'], df_test['region'], df_train['deal_probability'])\ndf_train['image_top_1'], df_test['image_top_1'] = target_encode(df_train['image_top_1'], df_test['image_top_1'], df_train['deal_probability'])\n\ndf_train['city'], df_test['city'] = target_encode(df_train['city'], df_test['city'], df_train['deal_probability'])\n\ndf_train['param_1'], df_test['param_1'] = target_encode(df_train['param_1'], df_test['param_1'], df_train['deal_probability'])\ndf_train['param_2'], df_test['param_2'] = target_encode(df_train['param_2'], df_test['param_2'], df_train['deal_probability'])\ndf_train['param_3'], df_test['param_3'] = target_encode(df_train['param_3'], df_test['param_3'], df_train['deal_probability'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ea80871f71fc255360c66e4c3bdc3d58fc778506"},"cell_type":"code","source":"df_train.drop(['date', 'day', 'user_id'], axis=1, inplace=True)\ndf_test.drop(['date', 'day', 'user_id'], axis=1, inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a5a0c0966bb0183f690dec6b9e8a9ece60d6f870"},"cell_type":"markdown","source":"**Text Features:**\n\nWe have several features with text data and they need to be processed in different ways. But at first let's create new features based on texts:\n- length of text (symbols)\n- number of words\n- counts of punctuation\n- counts of strange symbols ( emoticons)"},{"metadata":{"trusted":true,"_uuid":"6d17fe43019c5817842f9f91f65d314dbfd6e549"},"cell_type":"code","source":"df_train[\"len_title\"] = df_train[\"title\"].apply(lambda x: len(x))\ndf_train[\"words_title\"] = df_train[\"title\"].apply(lambda x: len(x.split()))\n\ndf_train[\"len_description\"] = df_train[\"description\"].apply(lambda x: len(x))\ndf_train[\"words_description\"] = df_train[\"description\"].apply(lambda x: len(x.split()))\n\ndf_train[\"len_params\"] = df_train[\"params\"].apply(lambda x: len(x))\ndf_train[\"words_params\"] = df_train[\"params\"].apply(lambda x: len(x.split()))\n\ndf_train['symbol1_count'] = df_train['description'].str.count('↓')\ndf_train['symbol2_count'] = df_train['description'].str.count('\\*')\ndf_train['symbol3_count'] = df_train['description'].str.count('✔')\ndf_train['symbol4_count'] = df_train['description'].str.count('❀')\ndf_train['symbol5_count'] = df_train['description'].str.count('➚')\ndf_train['symbol6_count'] = df_train['description'].str.count('ஜ')\ndf_train['symbol7_count'] = df_train['description'].str.count('.')\ndf_train['symbol8_count'] = df_train['description'].str.count('!')\ndf_train['symbol9_count'] = df_train['description'].str.count('\\?')\ndf_train['symbol10_count'] = df_train['description'].str.count('  ')\ndf_train['symbol11_count'] = df_train['description'].str.count('-')\ndf_train['symbol12_count'] = df_train['description'].str.count(',')\n\ndf_test['len_title']         = df_test['title'].apply(lambda x: len(x))\ndf_test['words_title']       = df_test['title'].apply(lambda x: len(x.split()))\ndf_test['len_description']   = df_test['description'].apply(lambda x: len(x))\ndf_test['words_description'] = df_test['description'].apply(lambda x: len(x.split()))\ndf_test['len_params']        = df_test['params'].apply(lambda x: len(x))\ndf_test['words_params']      = df_test['params'].apply(lambda x: len(x.split()))\n\ndf_test['symbol1_count'] = df_test['description'].str.count('↓')\ndf_test['symbol2_count'] = df_test['description'].str.count('\\*')\ndf_test['symbol3_count'] = df_test['description'].str.count('✔')\ndf_test['symbol4_count'] = df_test['description'].str.count('❀')\ndf_test['symbol5_count'] = df_test['description'].str.count('➚')\ndf_test['symbol6_count'] = df_test['description'].str.count('ஜ')\ndf_test['symbol7_count'] = df_test['description'].str.count('.')\ndf_test['symbol8_count'] = df_test['description'].str.count('!')\ndf_test['symbol9_count'] = df_test['description'].str.count('\\?')\ndf_test['symbol10_count'] = df_test['description'].str.count('  ')\ndf_test['symbol11_count'] = df_test['description'].str.count('-')\ndf_test['symbol12_count'] = df_test['description'].str.count(',')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"321989354ad666b28d6be5e8de3b312e3cf9069c"},"cell_type":"markdown","source":"Now let's start transforming texts. Titles have little number of unique words, so we can use default values for TfidfVectorizer (only add stopwords). I have to limit max_features due to memory constraints. I won't use descriptions and parameters due to kernel limits. "},{"metadata":{"trusted":true,"_uuid":"466a1c531696807aed7ff1f61b314b0e7ae7ce0c"},"cell_type":"code","source":"vectorizer = TfidfVectorizer(stop_words = stop, max_features = 6000)\nvectorizer.fit(df_train[\"title\"])\n\ndf_train_title = vectorizer.transform(df_train[\"title\"])\ndf_test_title  = vectorizer.transform(df_test[\"title\"])\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7cb998df71d107c83e5cd91a467f725fac292c56"},"cell_type":"code","source":"df_train.drop([\"title\", \"params\", \"description\", \"user_type\", \"activation_date\"], axis=1, inplace=True)\ndf_test.drop([\"title\", \"params\", \"description\", \"user_type\", \"activation_date\"], axis=1, inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"37841be5a2f5386367a5ada67d38ce59194792e9"},"cell_type":"code","source":"pd.set_option('max_columns', 60)\ndf_train.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f62bdd89e7ef37d8933d1d09ecce05e5e44b85d5"},"cell_type":"markdown","source":"**Meta-features:**\n\nOne of the features is used to build a model A and the prediction of model A is used as a feature in building model B.\n\nOne of possible ideas is creating meta-features. It means that we use some features to build a model and use the predictions in another model. I'll use ridge regression to create a new feature based on tokenized title and then I'll combine it with other features."},{"metadata":{"trusted":true,"_uuid":"8838a3c0543ece2984298825096c039b9a0be5ad"},"cell_type":"code","source":"%%time\n\nX_meta = np.zeros((df_train_title.shape[0], 1))\nX_test_meta = []\n\nfor fold_i, (train_i, test_i) in enumerate(kf.split(df_train_title)):\n    print(fold_i)\n    model = Ridge()\n    model.fit(df_train_title.tocsr()[train_i], df_train[\"deal_probability\"][train_i])\n    X_meta[test_i, :] = model.predict(df_train_title.tocsr()[test_i]).reshape(-1, 1)\n    X_test_meta.append(model.predict(df_test_title))\n    \nX_test_meta = np.stack(X_test_meta)\nX_test_meta_mean = np.mean(X_test_meta, axis=0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"aaed55e000b1a956f8ea6f9a8d9429600bd5e274"},"cell_type":"code","source":"X_full = csr_matrix(hstack([df_train.drop(['item_id', 'deal_probability', 'image'], axis=1), X_meta]))\nX_test_full = csr_matrix(hstack([df_test.drop(['item_id', 'image'], axis=1), X_test_meta_mean.reshape(-1, 1)]))\n\nX_train, X_valid, y_train, y_valid = train_test_split(X_full, df_train[\"deal_probability\"], test_size=0.20, random_state=42)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6c00e813f4a46cd42b7496f82bbdda6a892542fa"},"cell_type":"markdown","source":"**Building a simple model:**"},{"metadata":{"trusted":true,"_uuid":"fed02c1918c4cb5940bb7b6dedbde4c3fb89e135"},"cell_type":"code","source":"def rmse(predictions, targets):\n    return np.sqrt( ( (predictions - targets) ** 2).mean() )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"14e105aafa1702ad40238dbb945d3e5503cebc99"},"cell_type":"code","source":"# took parameters from this kernel:  https://www.kaggle.com/the1owl/beep-beep\n\nparams = {\"learning_rate\": 0.08,\n          \"max_depth\": 8,\n          \"boosting\": \"gbdt\",\n          \"objective\": \"regression\",\n          \"metric\": [\"auc\", \"rmse\"],\n          \"is_training_metric\": True,\n          \"seed\": 19,\n          \"num_leaves\": 63,\n          \"feature_fraction\": 0.9,\n          \"bagging_fraction\": 0.8,\n          \"bagging_freq\": 5\n         }\n\nmodel = lgb.train(params,\n                 lgb.Dataset(X_train, label=y_train),\n                 2000,\n                 lgb.Dataset(X_valid, label=y_valid),\n                 verbose_eval=50,\n                 early_stopping_rounds=20)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"531b2555c3fe9c85b996240725f3b90224b31ca1"},"cell_type":"code","source":"pred = model.predict(X_test_full)\n\n#clipping is necessary.\ndf_sub['deal_probability'] = np.clip(pred, 0, 1)\ndf_sub.to_csv('sub.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6d19addb3254c9363714d974f4c3e85a5c83dc95"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}