{"cells":[{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","collapsed":true,"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":false},"cell_type":"markdown","source":"# Defining the Problem\nAvito is Russia’s largest classified advertisement website. Sellers on their platform sometimes feel frustrated with both too little demand (indicating something is wrong with the product or the product listing) or too much demand (indicating a hot item with a good description was underpriced).\n\nOur aim is to predict demand for an online advertisement based on its full description (title, description, etc.), its context (geographically where it was posted, similar ads already posted) and historical demand for similar ads in similar contexts. With this information, Avito can inform sellers on how to best optimize their listing and provide some indication of how much interest they should realistically expect to receive.\n"},{"metadata":{},"cell_type":"markdown","source":"# Exploratory Data Analysis"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer\nfrom sklearn.decomposition import TruncatedSVD\nfrom sklearn import preprocessing, model_selection, metrics\nimport lightgbm as lgb\n\ncolor = sns.color_palette()\n%matplotlib inline\n\nimport plotly.offline as py\npy.init_notebook_mode(connected=True)\nimport plotly.graph_objs as go\nimport plotly.tools as tls\n\npd.options.mode.chained_assignment = None\npd.options.display.max_columns = 999\n\n\ntrain_df = pd.read_csv(\"../input/avito-demand-prediction/train.csv\", parse_dates=[\"activation_date\"])\ntest_df = pd.read_csv(\"../input/avito-demand-prediction/test.csv\", parse_dates=[\"activation_date\"])\nprint(\"Train file rows and columns are : \", train_df.shape)\nprint(\"Test file rows and columns are : \", test_df.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's try to get a glimpse of the training data."},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"After looking at different kernels and the actual Avito competition. It looks the columns represent:\n1. item_id - Ad id.\n1. user_id - User id.\n1. region - Ad region.\n1. city - Ad city.\n1. parent_category_name - Top level ad category as classified by Avito's ad model.\n1. category_name - Fine grain ad category as classified by Avito's ad model.\n1. param_1 - Optional parameter from Avito's ad model.\n1. param_2 - Optional parameter from Avito's ad model.\n1. param_3 - Optional parameter from Avito's ad model.\n1. title - Ad title.\n1. description - Ad description.\n1. price - Ad price.\n1. item_seq_number - Ad sequential number for user.\n1. activation_date- Date ad was placed.\n1. user_type - User type.\n1. image - Id code of image. Ties to a jpg file in train_jpg. Not every ad has an image.\n1. image_top_1 - Avito's classification code for the image.\n1. deal_probability - The target variable. This is the likelihood that an ad actually sold something. It's not possible to verify every transaction with certainty, so this column's value can be any float from zero to one.\nSo deal probability is our target variable and is a float value between 0 and 1 as per the data page. Let us have a look at it."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"plt.figure(figsize=(8,6))\nplt.scatter(range(train_df.shape[0]), np.sort(train_df['deal_probability'].values))\nplt.xlabel('index', fontsize=12)\nplt.ylabel('deal probability', fontsize=12)\nplt.title(\"Deal Probability Distribution\", fontsize=14)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As the data is given in 1e6, this means that approximately 100K Ads are with 0 deal_probability and very very few Ads are high probability like 1.0 \n"},{"metadata":{},"cell_type":"markdown","source":"## Distribution of Parent Category\n\nP.S: The Russian to English Translate maps were used by fellow Kagglers."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"#MAPS\nfrom plotly.offline import init_notebook_mode, iplot\n\nparent_category_name_map = {\"Личные вещи\" : \"Personal belongings\",\n                            \"Для дома и дачи\" : \"For the home and garden\",\n                            \"Бытовая электроника\" : \"Consumer electronics\",\n                            \"Недвижимость\" : \"Real estate\",\n                            \"Хобби и отдых\" : \"Hobbies & leisure\",\n                            \"Транспорт\" : \"Transport\",\n                            \"Услуги\" : \"Services\",\n                            \"Животные\" : \"Animals\",\n                            \"Для бизнеса\" : \"For business\"}\n\nregion_map = {\"Свердловская область\" : \"Sverdlovsk oblast\",\n            \"Самарская область\" : \"Samara oblast\",\n            \"Ростовская область\" : \"Rostov oblast\",\n            \"Татарстан\" : \"Tatarstan\",\n            \"Волгоградская область\" : \"Volgograd oblast\",\n            \"Нижегородская область\" : \"Nizhny Novgorod oblast\",\n            \"Пермский край\" : \"Perm Krai\",\n            \"Оренбургская область\" : \"Orenburg oblast\",\n            \"Ханты-Мансийский АО\" : \"Khanty-Mansi Autonomous Okrug\",\n            \"Тюменская область\" : \"Tyumen oblast\",\n            \"Башкортостан\" : \"Bashkortostan\",\n            \"Краснодарский край\" : \"Krasnodar Krai\",\n            \"Новосибирская область\" : \"Novosibirsk oblast\",\n            \"Омская область\" : \"Omsk oblast\",\n            \"Белгородская область\" : \"Belgorod oblast\",\n            \"Челябинская область\" : \"Chelyabinsk oblast\",\n            \"Воронежская область\" : \"Voronezh oblast\",\n            \"Кемеровская область\" : \"Kemerovo oblast\",\n            \"Саратовская область\" : \"Saratov oblast\",\n            \"Владимирская область\" : \"Vladimir oblast\",\n            \"Калининградская область\" : \"Kaliningrad oblast\",\n            \"Красноярский край\" : \"Krasnoyarsk Krai\",\n            \"Ярославская область\" : \"Yaroslavl oblast\",\n            \"Удмуртия\" : \"Udmurtia\",\n            \"Алтайский край\" : \"Altai Krai\",\n            \"Иркутская область\" : \"Irkutsk oblast\",\n            \"Ставропольский край\" : \"Stavropol Krai\",\n            \"Тульская область\" : \"Tula oblast\"}\n\n\ncategory_map = {\"Одежда, обувь, аксессуары\":\"Clothing, shoes, accessories\",\n\"Детская одежда и обувь\":\"Children's clothing and shoes\",\n\"Товары для детей и игрушки\":\"Children's products and toys\",\n\"Квартиры\":\"Apartments\",\n\"Телефоны\":\"Phones\",\n\"Мебель и интерьер\":\"Furniture and interior\",\n\"Предложение услуг\":\"Offer services\",\n\"Автомобили\":\"Cars\",\n\"Ремонт и строительство\":\"Repair and construction\",\n\"Бытовая техника\":\"Appliances\",\n\"Товары для компьютера\":\"Products for computer\",\n\"Дома, дачи, коттеджи\":\"Houses, villas, cottages\",\n\"Красота и здоровье\":\"Health and beauty\",\n\"Аудио и видео\":\"Audio and video\",\n\"Спорт и отдых\":\"Sports and recreation\",\n\"Коллекционирование\":\"Collecting\",\n\"Оборудование для бизнеса\":\"Equipment for business\",\n\"Земельные участки\":\"Land\",\n\"Часы и украшения\":\"Watches and jewelry\",\n\"Книги и журналы\":\"Books and magazines\",\n\"Собаки\":\"Dogs\",\n\"Игры, приставки и программы\":\"Games, consoles and software\",\n\"Другие животные\":\"Other animals\",\n\"Велосипеды\":\"Bikes\",\n\"Ноутбуки\":\"Laptops\",\n\"Кошки\":\"Cats\",\n\"Грузовики и спецтехника\":\"Trucks and buses\",\n\"Посуда и товары для кухни\":\"Tableware and goods for kitchen\",\n\"Растения\":\"Plants\",\n\"Планшеты и электронные книги\":\"Tablets and e-books\",\n\"Товары для животных\":\"Pet products\",\n\"Комнаты\":\"Room\",\n\"Фототехника\":\"Photo\",\n\"Коммерческая недвижимость\":\"Commercial property\",\n\"Гаражи и машиноместа\":\"Garages and Parking spaces\",\n\"Музыкальные инструменты\":\"Musical instruments\",\n\"Оргтехника и расходники\":\"Office equipment and consumables\",\n\"Птицы\":\"Birds\",\n\"Продукты питания\":\"Food\",\n\"Мотоциклы и мототехника\":\"Motorcycles and bikes\",\n\"Настольные компьютеры\":\"Desktop computers\",\n\"Аквариум\":\"Aquarium\",\n\"Охота и рыбалка\":\"Hunting and fishing\",\n\"Билеты и путешествия\":\"Tickets and travel\",\n\"Водный транспорт\":\"Water transport\",\n\"Готовый бизнес\":\"Ready business\",\n\"Недвижимость за рубежом\":\"Property abroad\"}","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"train_df['region_en'] = train_df['region'].apply(lambda x : region_map[x])\ntrain_df['parent_category_name_en'] = train_df['parent_category_name'].apply(lambda x : parent_category_name_map[x])\ntrain_df['category_name_en'] = train_df['category_name'].apply(lambda x : category_map[x])\n\ndef _generate_bar_plot_hor(df, col, title, color, w=None, h=None, lm=0, limit=100):\n    cnt_srs = df[col].value_counts()[:limit]\n    trace = go.Bar(y=cnt_srs.index[::-1], x=cnt_srs.values[::-1], orientation = 'h',\n        marker=dict(color=color))\n\n    layout = dict(title=title, margin=dict(l=lm), width=w, height=h)\n    data = [trace]\n    fig = go.Figure(data=data, layout=layout)\n    iplot(fig)\n    \ncols = ['parent_category_name_en', 'category_name_en', 'region_en', 'city', 'param_1', 'param_2', 'param_3', 'weekday', 'day','title_len', 'description_len', 'image_top_1', 'user_id']\n_generate_bar_plot_hor(train_df, cols[0], \"Distribution of Parent Category\", '#f2b5bc', 600, 400, 200)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The largest amount of data/genre category appears to be in \"Personal Belonging\" at about 700K Items.\nThen, the priority is for home and garden, consumer electronics, real estate etc.\nBut, ads made for business seems to appear very low with only 18K ads.\n\nNow, let's take a look at the different regions where the Ads are actually originating from."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def _generate_bar_plot_hor(df, col, title, color, w=None, h=None, lm=0, limit=100):\n    cnt_srs = df[col].value_counts()[:limit]\n    trace = go.Bar(y=cnt_srs.index[::-1], x=cnt_srs.values[::-1], orientation = 'h',\n        marker=dict(color=color))\n\n    layout = dict(title=title, margin=dict(l=lm), width=w, height=h)\n    data = [trace]\n    fig = go.Figure(data=data, layout=layout)\n    iplot(fig)\n_generate_bar_plot_hor(train_df, cols[2], \"Distribution of Region\", '#71c8e8', 600, 600, 200, limit=30)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The Top three regions appears to be:\n1. Krasnodar Krai: With a major amounts of Ads at 141K\n1. Sverdlovsk Oblast: With about 94K Ads\n1. Rostov Oblast: With about 89K ads\n\nThe region with the fewest Ads seems to be Tula Oblast with about 25K ads.\nNow, let's check what is the Distribution based on Cities."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"_generate_bar_plot_hor(train_df, cols[3], \"Distribution of City\", '#c2e2a3', 600, 600, 200, limit=30)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Top Cities are:\n1. Krasnodar (63638 items)\n1. Yekaterinburg, (63602 items)\n1. Novosibirsk (56929 items)\n\n"},{"metadata":{},"cell_type":"markdown","source":"## Deal Probability by Parent Category and User Type"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"sns.set(rc={'figure.figsize':(15, 8)})\nsns.boxplot(x=\"parent_category_name_en\", y=\"deal_probability\", hue=\"user_type\",  palette=\"PRGn\", data=train_df)\nplt.title(\"Deal probability by parent category and User Type\")\nplt.xticks(rotation='vertical')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* From the above boxplot graph, Users having \"shop\" user types are only related with items of Real Estate, Transport, and Services categories. Service Category is the one having largest deal probability, while Real estate have lowest.\n* Users having \"Company\" as user type also have highest deal probability in the Services category followed by Transport and Animals. Unlike Shop user types, Company user types are related with almost all parent categoies\n* Users having \"Private\" as user type are scattered in the range of 0.1 to 0.6 in the Services category."},{"metadata":{},"cell_type":"markdown","source":"## Distribution of User-Type in the dataset"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def _create_pie_chart(df, col):\n    tm = df[col].value_counts()\n    labels = list(tm.index)\n    values = list(tm.values)\n    trace = go.Pie(labels=labels, values=values, marker=dict(colors=['#f9c968', '#75e575', '#d693b3']))\n    return trace\ntrace1 = _create_pie_chart(train_df, 'user_type')\nlayout = go.Layout(title='Distribution of User Type', width=600, height=400, margin=dict(l=100))\ndata = [trace1]\nfig = go.Figure(data=data, layout=layout)\niplot(fig)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* Majority of the items are posted by users which belong to Private User type while minority of the items are posted by users which belong to \"Shop\" user type.\n* About 72% of the users are of Private Type while about 5% of the users are of Shop user type"},{"metadata":{},"cell_type":"markdown","source":"## Activation Dates"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"cnt_srs = train_df['activation_date'].value_counts()\n\ntrace = go.Bar(\n    x=cnt_srs.index,\n    y=cnt_srs.values,\n    marker=dict(\n        color=cnt_srs.values,\n        colorscale = 'Picnic',\n        reversescale = True\n    ),\n)\n\nlayout = go.Layout(\n    title='Activation Dates in Train'\n)\n\ndata = [trace]\nfig = go.Figure(data=data, layout=layout)\npy.iplot(fig, filename=\"ActivationDate\")\n\n# Activation dates in test\ncnt_srs = test_df['activation_date'].value_counts()\n\ntrace = go.Bar(\n    x=cnt_srs.index,\n    y=cnt_srs.values,\n    marker=dict(\n        color=cnt_srs.values,\n        colorscale = 'Picnic',\n        reversescale = True\n    ),\n)\n\nlayout = go.Layout(\n    title='Activation Dates in Test'\n)\n\ndata = [trace]\nfig = go.Figure(data=data, layout=layout)\npy.iplot(fig, filename=\"ActivationDate\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Inferences:\n\n1. So the dates are not different between train and test sets. So we need to be careful while doing our validation. May be time based validation is a good option.\n1. We are given two weeks data for training (March 15 to March 28) and one week data for testing (April 12 to April 18, 2017).\n1. There is a gap of two weeks in between training and testing data.\n1. We can probably use weekday as a feature since all the days are present in both train and test sets.\n\n## User id:\n\nNow we can have a look at the number of unique users in train & test and also the number of common users if an"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"from matplotlib_venn import venn2\n\nplt.figure(figsize=(10,7))\nvenn2([set(train_df.user_id.unique()), set(test_df.user_id.unique())], set_labels = ('Train set', 'Test set') )\nplt.title(\"Number of users in train and test\", fontsize=15)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So out of the 306K users in test, about 68K users are there in train and the rest are new.\n\n## Title:\n\nFirst let us look at the number of common titles between train and test set"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"from matplotlib_venn import venn2\n\nplt.figure(figsize=(10,7))\nvenn2([set(train_df.title.unique()), set(test_df.title.unique())], set_labels = ('Train set', 'Test set') )\nplt.title(\"Number of titles in train and test\", fontsize=15)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We have around 64K common titles between train and test set. Now let us look at the number of words present in the title column."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"train_df[\"title_nwords\"] = train_df[\"title\"].apply(lambda x: len(x.split()))\ntest_df[\"title_nwords\"] = test_df[\"title\"].apply(lambda x: len(x.split()))\n\ncnt_srs = train_df['title_nwords'].value_counts()\n\ntrace = go.Bar(\n    x=cnt_srs.index,\n    y=cnt_srs.values,\n    marker=dict(\n        color=\"blue\",\n        #colorscale = 'Blues',\n        reversescale = True\n    ),\n)\n\nlayout = go.Layout(\n    title='Number of words in title column'\n)\n\ndata = [trace]\nfig = go.Figure(data=data, layout=layout)\npy.iplot(fig, filename=\"title_nwords\")     ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Majority of the tiles have 1, 2 or 3 words and has a long tail.\n\nNow we will do the following:\n\n1. Take the TF-IDF of the title column and this will be a sparse matrix with huge dimesnions.\n1. Get the top SVD components fof this TF-IDF\n1. Plot the distribution of SVD components with Deal probability to see if these variables help."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"### TFIDF Vectorizer ###\ntfidf_vec = TfidfVectorizer(ngram_range=(1,1))\nfull_tfidf = tfidf_vec.fit_transform(train_df['title'].values.tolist() + test_df['title'].values.tolist())\ntrain_tfidf = tfidf_vec.transform(train_df['title'].values.tolist())\ntest_tfidf = tfidf_vec.transform(test_df['title'].values.tolist())\n\n### SVD Components ###\nn_comp = 3\nsvd_obj = TruncatedSVD(n_components=n_comp, algorithm='arpack')\nsvd_obj.fit(full_tfidf)\ntrain_svd = pd.DataFrame(svd_obj.transform(train_tfidf))\ntest_svd = pd.DataFrame(svd_obj.transform(test_tfidf))\ntrain_svd.columns = ['svd_title_'+str(i+1) for i in range(n_comp)]\ntest_svd.columns = ['svd_title_'+str(i+1) for i in range(n_comp)]\ntrain_df = pd.concat([train_df, train_svd], axis=1)\ntest_df = pd.concat([test_df, test_svd], axis=1)\ndel full_tfidf, train_tfidf, test_tfidf, train_svd, test_svd","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# 1st svd comp #\nplt.figure(figsize=(8,8))\nsns.jointplot(x=train_df[\"svd_title_1\"].values, y=train_df[\"deal_probability\"].values, size=10)\nplt.ylabel('Deal Probability', fontsize=12)\nplt.xlabel('First SVD component on Title', fontsize=12)\nplt.title(\"Deal Probability distribution for First SVD component on title\", fontsize=15)\nplt.show()\n\n# 2nd svd comp #\nplt.figure(figsize=(8,8))\nsns.jointplot(x=train_df[\"svd_title_2\"].values, y=train_df[\"deal_probability\"].values, size=10)\nplt.ylabel('Deal Probability', fontsize=12)\nplt.xlabel('Second SVD component on Title', fontsize=12)\nplt.title(\"Deal Probability distribution for Second SVD component on title\", fontsize=15)\nplt.show()\n\n# 3rd svd comp #\nplt.figure(figsize=(8,8))\nsns.jointplot(x=train_df[\"svd_title_3\"].values, y=train_df[\"deal_probability\"].values, size=10)\nplt.ylabel('Deal Probability', fontsize=12)\nplt.xlabel('Third SVD component on Title', fontsize=12)\nplt.title(\"Deal Probability distribution for Third SVD component on title\", fontsize=15)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As we can see, the top SVD components capture quite an amount of variation in the data. So this might be helpful features in our modeling process.\n\n**Description:**\n\nLet us first check the number of words in the description column."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"## Filling missing values ##\ntrain_df[\"description\"].fillna(\"NA\", inplace=True)\ntest_df[\"description\"].fillna(\"NA\", inplace=True)\n\ntrain_df[\"desc_nwords\"] = train_df[\"description\"].apply(lambda x: len(x.split()))\ntest_df[\"desc_nwords\"] = test_df[\"description\"].apply(lambda x: len(x.split()))\n\ncnt_srs = train_df['desc_nwords'].value_counts().head(100)\n\ntrace = go.Bar(\n    x=cnt_srs.index,\n    y=cnt_srs.values,\n    marker=dict(\n        color=\"blue\",\n        #colorscale = 'Blues',\n        reversescale = True\n    ),\n)\n\nlayout = go.Layout(\n    title='Number of words in Description column'\n)\n\ndata = [trace]\nfig = go.Figure(data=data, layout=layout)\npy.iplot(fig, filename=\"desc_nwords\")  ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Description column has a huge right tail till about 700 words. I have cut the same till top 100 for better visualization.\n\nNow let us create the first 3 SVD components for description column (just like title) and plot the same against deal probability."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"### TFIDF Vectorizer ###\ntfidf_vec = TfidfVectorizer(ngram_range=(1,1), max_features=100000)\nfull_tfidf = tfidf_vec.fit_transform(train_df['description'].values.tolist() + test_df['description'].values.tolist())\ntrain_tfidf = tfidf_vec.transform(train_df['description'].values.tolist())\ntest_tfidf = tfidf_vec.transform(test_df['description'].values.tolist())\n\n### SVD Components ###\nn_comp = 3\nsvd_obj = TruncatedSVD(n_components=n_comp, algorithm='arpack')\nsvd_obj.fit(full_tfidf)\ntrain_svd = pd.DataFrame(svd_obj.transform(train_tfidf))\ntest_svd = pd.DataFrame(svd_obj.transform(test_tfidf))\ntrain_svd.columns = ['svd_desc_'+str(i+1) for i in range(n_comp)]\ntest_svd.columns = ['svd_desc_'+str(i+1) for i in range(n_comp)]\ntrain_df = pd.concat([train_df, train_svd], axis=1)\ntest_df = pd.concat([test_df, test_svd], axis=1)\ndel full_tfidf, train_tfidf, test_tfidf, train_svd, test_svd","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# 1st svd comp #\nplt.figure(figsize=(8,8))\nsns.jointplot(x=train_df[\"svd_desc_1\"].values, y=train_df[\"deal_probability\"].values, size=10)\nplt.ylabel('Deal Probability', fontsize=12)\nplt.xlabel('First SVD component on Description', fontsize=12)\nplt.title(\"Deal Probability distribution for First SVD component on Description\", fontsize=15)\nplt.show()\n\n# 2nd svd comp #\nplt.figure(figsize=(8,8))\nsns.jointplot(x=train_df[\"svd_desc_2\"].values, y=train_df[\"deal_probability\"].values, size=10)\nplt.ylabel('Deal Probability', fontsize=12)\nplt.xlabel('Second SVD component on Description', fontsize=12)\nplt.title(\"Deal Probability distribution for Second SVD component on title\", fontsize=15)\nplt.show()\n\n# 3rd svd comp #\nplt.figure(figsize=(8,8))\nsns.jointplot(x=train_df[\"svd_desc_3\"].values, y=train_df[\"deal_probability\"].values, size=10)\nplt.ylabel('Deal Probability', fontsize=12)\nplt.xlabel('Second SVD component on Description', fontsize=12)\nplt.title(\"Deal Probability distribution for Third SVD component on Description\", fontsize=15)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Looks like it also captures some good amount of variability.\n\n## LightGBM Model:\n\nNow let us build a baseline model using the given features and newly created features.\n\nIn the following block, we will do :\n\n* Create a new feature for week day\n* Label encode the cateforical variables\n* Drop the columns that are not needed in the model."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# Target and ID variables #\ntrain_y = train_df[\"deal_probability\"].values\ntest_id = test_df[\"item_id\"].values\n\n# New variable on weekday #\ntrain_df[\"activation_weekday\"] = train_df[\"activation_date\"].dt.weekday\ntest_df[\"activation_weekday\"] = test_df[\"activation_date\"].dt.weekday\n\n# Label encode the categorical variables #\ncat_vars = [\"region\", \"city\", \"parent_category_name\", \"category_name\", \"user_type\", \"param_1\", \"param_2\", \"param_3\"]\nfor col in cat_vars:\n    lbl = preprocessing.LabelEncoder()\n    lbl.fit(list(train_df[col].values.astype('str')) + list(test_df[col].values.astype('str')))\n    train_df[col] = lbl.transform(list(train_df[col].values.astype('str')))\n    test_df[col] = lbl.transform(list(test_df[col].values.astype('str')))\n\ncols_to_drop = [\"item_id\", \"user_id\", \"title\", \"description\", \"activation_date\", \"image\"]\ntrain_X = train_df.drop(cols_to_drop + [\"region_en\", \"parent_category_name_en\", \"category_name_en\",\"deal_probability\"], axis=1)\ntest_X = test_df.drop(cols_to_drop, axis=1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now let us create a custom function to build the LightGBM model."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def run_lgb(train_X, train_y, val_X, val_y, test_X):\n    params = {\n        \"objective\" : \"regression\",\n        \"metric\" : \"rmse\",\n        \"num_leaves\" : 30,\n        \"learning_rate\" : 0.1,\n        \"bagging_fraction\" : 0.7,\n        \"feature_fraction\" : 0.7,\n        \"bagging_frequency\" : 5,\n        \"bagging_seed\" : 2018,\n        \"verbosity\" : -1\n    }\n    \n    lgtrain = lgb.Dataset(train_X, label=train_y)\n    lgval = lgb.Dataset(val_X, label=val_y)\n    evals_result = {}\n    model = lgb.train(params, lgtrain, 1000, valid_sets=[lgval], early_stopping_rounds=100, verbose_eval=20, evals_result=evals_result)\n    \n    pred_test_y = model.predict(test_X, num_iteration=model.best_iteration)\n    return pred_test_y, model, evals_result","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We will now split the train into development and validation sample and build our models. We will take the last 200K rows as validation sample."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# Splitting the data for model training#\ndev_X = train_X.iloc[:-200000,:]\nval_X = train_X.iloc[-200000:,:]\ndev_y = train_y[:-200000]\nval_y = train_y[-200000:]\nprint(dev_X.shape, val_X.shape, test_X.shape)\n\n# Training the model #\npred_test, model, evals_result = run_lgb(dev_X, dev_y, val_X, val_y, test_X)\n\n# Making a submission file #\npred_test[pred_test>1] = 1\npred_test[pred_test<0] = 0\nsub_df = pd.DataFrame({\"item_id\":test_id})\nsub_df[\"deal_probability\"] = pred_test\nsub_df.to_csv(\"baseline_lgb.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Feature Importance:\n\nNow let us look at the top features from the model."},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(12,18))\nlgb.plot_importance(model, max_num_features=50, height=0.8, ax=ax)\nax.grid(False)\nplt.title(\"LightGBM - Feature Importance\", fontsize=15)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Price seems to be the most important feature followed by image_top_1 and param_1.\n\n**Ways to improve the model:**\n\n* Adding more new features\n* Tuning the parameters more\n* Blending with other models."}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}