{"cells":[{"metadata":{"_cell_guid":"357cf094-2ea2-4a44-8aeb-d0717696bd2f","_uuid":"b588970d9f1d6304ee9e363ce8580507b389814f"},"cell_type":"markdown","source":"**About the Competition:**\n\n[Avito](https://www.avito.ru/), Russia’s largest classified advertisements website, is hosting its fourth Kaggle competition. The challenge is to predict demand for an online advertisement based on its full description (title, description, images, etc.), its context (geographically where it was posted, similar ads already posted) and historical demand for similar ads in similar contexts. \n\n**About the Notebook:**\n\nOne more exciting competition ahead and this involves both NLP (text data in Russian) and Image data along with numerical . In this notebook, let us get into the basic data exploration using python. \n\nThanks to [Yandex Translate](https://translate.yandex.com/), I was able to get english names for the russian names and used them whenever possible. Most of the plots are in plotly and so please hover over them to see more details. "},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer\nfrom sklearn.decomposition import TruncatedSVD\nfrom sklearn import preprocessing, model_selection, metrics\nimport lightgbm as lgb\n\ncolor = sns.color_palette()\n%matplotlib inline\n\nimport plotly.offline as py\npy.init_notebook_mode(connected=True)\nimport plotly.graph_objs as go\nimport plotly.tools as tls\n\npd.options.mode.chained_assignment = None\npd.options.display.max_columns = 999","execution_count":4,"outputs":[]},{"metadata":{"_cell_guid":"649eea04-9cf2-429e-9378-e7daf948326a","_uuid":"af92759c050311a049b129ec2fb308bc206f9447"},"cell_type":"markdown","source":"Now let us look at the input files present in the dataset. "},{"metadata":{"_cell_guid":"38fe4b05-4391-4520-9346-76ec3a079b22","_uuid":"a63a82670f8c459459066c5d7ebda778ce6d8626","trusted":true},"cell_type":"code","source":"from subprocess import check_output\nprint(check_output([\"ls\", \"../input/\"]).decode(\"utf8\"))","execution_count":5,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"27be232cb3e1d4aec87c8a296f4bc021af78a518"},"cell_type":"code","source":"!ls ../input/","execution_count":6,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","collapsed":true},"cell_type":"markdown","source":"The description of the data files from the data page:\n\n* train.csv - Train data.\n* test.csv - Test data. Same schema as the train data, minus deal_probability.\n* train_active.csv - Supplemental data from ads that were displayed during the same period as train.csv. Same schema as the train data, minus deal_probability.\n* test_active.csv - Supplemental data from ads that were displayed during the same period as test.csv. Same schema as the train data, minus deal_probability.\n* periods_train.csv - Supplemental data showing the dates when the ads from train_active.csv were activated and when they where displayed.\n* periods_test.csv - Supplemental data showing the dates when the ads from test_active.csv were activated and when they where displayed. Same schema as periods_train.csv, except that the item ids map to an ad in test_active.csv.\n* train_jpg.zip - Images from the ads in train.csv.\n* test_jpg.zip - Images from the ads in test.csv.\n* sample_submission.csv - A sample submission in the correct format.\n\nLet us start with the train file."},{"metadata":{"_cell_guid":"02081c71-977c-4f5a-b990-b1d0dc8bb858","_uuid":"67e924346e31d55b07a38a1038da815d76e28356","trusted":true},"cell_type":"code","source":"train_df = pd.read_csv(\"../input/train.csv\", parse_dates=[\"activation_date\"])\ntest_df = pd.read_csv(\"../input/test.csv\", parse_dates=[\"activation_date\"])\nprint(\"Train file rows and columns are : \", train_df.shape)\nprint(\"Test file rows and columns are : \", test_df.shape)","execution_count":7,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1027598c627b18413f58012fc1edc4fe52cced89"},"cell_type":"code","source":"(train_df.image.loc[0])","execution_count":8,"outputs":[]},{"metadata":{"_cell_guid":"23399ebb-1137-4982-bd32-89e85c00dd70","_uuid":"5fd384836d7725f81b638eabf52e8643d4e478d9","trusted":true},"cell_type":"code","source":"train_df.head()","execution_count":9,"outputs":[]},{"metadata":{"_cell_guid":"8e735a3e-e68e-446f-a926-4690132e9fc5","_uuid":"3706ed011cdb10b40c23fcdf3007adf21908f4c6"},"cell_type":"markdown","source":"The train dataset description is as follows:\n\n* item_id - Ad id.\n* user_id - User id.\n* region - Ad region.\n* city - Ad city.\n* parent_category_name - Top level ad category as classified by Avito's ad model.\n* category_name - Fine grain ad category as classified by Avito's ad model.\n* param_1 - Optional parameter from Avito's ad model.\n* param_2 - Optional parameter from Avito's ad model.\n* param_3 - Optional parameter from Avito's ad model.\n* title - Ad title.\n* description - Ad description.\n* price - Ad price.\n* item_seq_number - Ad sequential number for user.\n* activation_date- Date ad was placed.\n* user_type - User type.\n* image - Id code of image. Ties to a jpg file in train_jpg. Not every ad has an image.\n* image_top_1 - Avito's classification code for the image.\n* deal_probability - The target variable. This is the likelihood that an ad actually sold something. It's not possible to verify every transaction with certainty, so this column's value can be any float from zero to one.\n\nSo deal probability is our target variable and  is a float value between 0 and 1 as per the data page. Let us have a look at it. "},{"metadata":{"_cell_guid":"ed12720d-7c7f-48b7-a8e2-852808a91c98","_kg_hide-input":true,"_uuid":"1185a4a591621e9ccb4df4f46fbc45e6a9c3d11e","trusted":true},"cell_type":"code","source":"plt.figure(figsize=(12,8))\nsns.distplot(train_df[\"deal_probability\"].values, bins=100, kde=False)\nplt.xlabel('Deal Probility', fontsize=12)\nplt.title(\"Deal Probability Histogram\", fontsize=14)\nplt.show()\n\nplt.figure(figsize=(8,6))\nplt.scatter(range(train_df.shape[0]), np.sort(train_df['deal_probability'].values))\nplt.xlabel('index', fontsize=12)\nplt.ylabel('deal probability', fontsize=12)\nplt.title(\"Deal Probability Distribution\", fontsize=14)\nplt.show()","execution_count":10,"outputs":[]},{"metadata":{"_cell_guid":"54abad60-32a5-42bb-b724-3900ed1a27a8","_uuid":"214fecd31317940513ebf7e800802fcac688db5b"},"cell_type":"markdown","source":"So almost 100K Ads has 0 probaility (which means it did not sell anything) and few ads have a probability of 1. Rest of the deal probabilities have values in between. \n\n**Region wise distribution of Ads:**\n\nLet us look at the region wise distribution of ads. "},{"metadata":{"_cell_guid":"7c3ad529-9e3a-4f8c-87ed-10576962aa4e","_kg_hide-input":true,"_uuid":"e5e7c28f647efd43b386d82f7988538b6d04be2a","collapsed":true,"trusted":true},"cell_type":"code","source":"from io import StringIO\n\ntemp_data = StringIO(\"\"\"\nregion,region_en\nСвердловская область, Sverdlovsk oblast\nСамарская область, Samara oblast\nРостовская область, Rostov oblast\nТатарстан, Tatarstan\nВолгоградская область, Volgograd oblast\nНижегородская область, Nizhny Novgorod oblast\nПермский край, Perm Krai\nОренбургская область, Orenburg oblast\nХанты-Мансийский АО, Khanty-Mansi Autonomous Okrug\nТюменская область, Tyumen oblast\nБашкортостан, Bashkortostan\nКраснодарский край, Krasnodar Krai\nНовосибирская область, Novosibirsk oblast\nОмская область, Omsk oblast\nБелгородская область, Belgorod oblast\nЧелябинская область, Chelyabinsk oblast\nВоронежская область, Voronezh oblast\nКемеровская область, Kemerovo oblast\nСаратовская область, Saratov oblast\nВладимирская область, Vladimir oblast\nКалининградская область, Kaliningrad oblast\nКрасноярский край, Krasnoyarsk Krai\nЯрославская область, Yaroslavl oblast\nУдмуртия, Udmurtia\nАлтайский край, Altai Krai\nИркутская область, Irkutsk oblast\nСтавропольский край, Stavropol Krai\nТульская область, Tula oblast\n\"\"\")\n\nregion_df = pd.read_csv(temp_data)\ntrain_df = pd.merge(train_df, region_df, how=\"left\", on=\"region\")","execution_count":11,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"aa9640fe30984d86ac8d501a24a88916417a8f30"},"cell_type":"code","source":"region_df.head()","execution_count":12,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"10d651e60279a5302322719228fdeaf3514524f7"},"cell_type":"code","source":"train_df.head()","execution_count":13,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8d164657102b20fe75aee049170fd71ae9e01fbd"},"cell_type":"code","source":"temp_series = train_df['region_en'].value_counts()\n(temp_series[:5])","execution_count":14,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"d6377fa10234fd38d6ad94f86a92c9f9a786b431"},"cell_type":"code","source":"temp_series.index","execution_count":15,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"9ef2b3bb6860db0e2ab4d333aadb6d626e3f0bf7"},"cell_type":"code","source":"sizes = (np.array((temp_series / temp_series.sum())*100))","execution_count":16,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"15af5ae686deb4788de3a18cdee8e936e0254b84"},"cell_type":"code","source":"(sizes[:5])","execution_count":17,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"852f14af2a87fd7b207a4e853f84149e57e6cbd1"},"cell_type":"code","source":"go?","execution_count":18,"outputs":[]},{"metadata":{"_cell_guid":"9719351d-0fb1-4817-9982-1b7ea4e7d0f0","_kg_hide-input":true,"_uuid":"8ea1f92f35123bcc3fc8b7b32fb58e89ce40dea2","trusted":true},"cell_type":"code","source":"temp_series = train_df['region_en'].value_counts()\nlabels = (np.array(temp_series.index))\nsizes = (np.array((temp_series / temp_series.sum())*100))\n\ntrace = go.Pie(labels=labels, values=sizes)\nlayout = go.Layout(\n    title='Region distribution',\n    width=900,\n    height=900,\n)\ndata = [trace]\nfig = go.Figure(data=data, layout=layout)\npy.iplot(fig, filename=\"region\")","execution_count":19,"outputs":[]},{"metadata":{"_cell_guid":"a9376c47-abe1-458a-8d6e-e3274ed92ed1","_uuid":"bc7bde8303ef4e84681312f1a535ce7feeb48db5"},"cell_type":"markdown","source":"The regions have percentage of ads between 1.71% to 9.41%. So the top regions are:\n1. Krasnodar region - 9.41%\n2. Sverdlovsk region - 6.28%\n3. Rostov region - 5.99%\n"},{"metadata":{"_cell_guid":"86624ab8-3fec-4c89-aace-bc0952421c4c","_kg_hide-input":true,"_uuid":"5fa1f0472c9ee994ec18d039192948508a30b398","trusted":true},"cell_type":"code","source":"plt.figure(figsize=(12,8))\nsns.boxplot(y=\"region_en\", x=\"deal_probability\", data=train_df)\nplt.xlabel('Deal probability', fontsize=12)\nplt.ylabel('Region', fontsize=12)\nplt.title(\"Deal probability by region\")\nplt.xticks(rotation='vertical')\nplt.show()","execution_count":20,"outputs":[]},{"metadata":{"_cell_guid":"54cc5a3b-18c5-4af7-ad87-472830f07af3","_uuid":"8e644697dc52117dd85778550f108deb6fbdc1d2"},"cell_type":"markdown","source":"**City wise distribution of Ads:**\n\nNow let us have a look at the top 20 cities present in the dataset."},{"metadata":{"trusted":true,"_uuid":"4db5001bd4f4815f36cb8dc02df1578e62fbbeff"},"cell_type":"code","source":"cnt_srs = train_df['city'].value_counts().head(20)\ntype(cnt_srs)","execution_count":21,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"16dd20ec8bb1847448d26994596efccd0d0b8344"},"cell_type":"code","source":"cnt_srs[:5]","execution_count":22,"outputs":[]},{"metadata":{"_cell_guid":"24d6c30c-4bec-4bf0-a669-df6da219d214","_kg_hide-input":true,"_uuid":"7b2bf819d55d8832f52d98de92ce9b81902e1f64","trusted":true},"cell_type":"code","source":"cnt_srs = train_df['city'].value_counts().head(20)\ntrace = go.Bar(\n    y=cnt_srs.index[::-1],\n    x=cnt_srs.values[::-1],\n    orientation = 'h',\n    marker=dict(\n        color=cnt_srs.values[::-1],\n        colorscale = 'Blues',\n        reversescale = True\n    ),\n)\n\nlayout = dict(\n    title='City distribution of Ads',\n    )\ndata = [trace]\nfig = go.Figure(data=data, layout=layout)\npy.iplot(fig, filename=\"CityAds\")","execution_count":23,"outputs":[]},{"metadata":{"_cell_guid":"b52935e3-5d8f-41c7-bee9-6fbaed7304f4","_uuid":"2e23a7fc1a6c656011d3e65f14df1e412301e2ca"},"cell_type":"markdown","source":"So the top cities where the ads are shown are\n1. Krasnodar\n2. Ekaterinburg\n3. Novosibirsk"},{"metadata":{"_cell_guid":"e4cefc89-d06c-41e6-bd1e-a8c847278ad3","_uuid":"ed40f2802a52608b6750dae893f4bea6e65d3182"},"cell_type":"markdown","source":"**Parent Category Name:**\n\nNow let us look at the distribution of parent cateory names."},{"metadata":{"_cell_guid":"6e87267d-a73c-40ac-bd0f-15472fcb7eea","_kg_hide-input":true,"_uuid":"996f4292fb6054ef38bd3552ec6f5ecc2199ba72","trusted":true},"cell_type":"code","source":"temp_data = StringIO(\"\"\"\nparent_category_name,parent_category_name_en\nЛичные вещи,Personal belongings\nДля дома и дачи,For the home and garden\nБытовая электроника,Consumer electronics\nНедвижимость,Real estate\nХобби и отдых,Hobbies & leisure\nТранспорт,Transport\nУслуги,Services\nЖивотные,Animals\nДля бизнеса,For business\n\"\"\")\n\ntemp_df = pd.read_csv(temp_data)\ntrain_df = pd.merge(train_df, temp_df, on=\"parent_category_name\", how=\"left\")","execution_count":24,"outputs":[]},{"metadata":{"_cell_guid":"3273198f-583e-457a-985c-5ee730331a8b","_kg_hide-input":true,"_uuid":"f15f336da01b67016cecaa609766ab6d1cb4899e","trusted":true},"cell_type":"code","source":"temp_series = train_df['parent_category_name_en'].value_counts()\nlabels = (np.array(temp_series.index))\nsizes = (np.array((temp_series / temp_series.sum())*100))\n\ntrace = go.Pie(labels=labels, values=sizes)\nlayout = go.Layout(\n    title='Parent Category distribution',\n    width=900,\n    height=900,\n)\ndata = [trace]\nfig = go.Figure(data=data, layout=layout)\npy.iplot(fig, filename=\"parentcategory\")","execution_count":25,"outputs":[]},{"metadata":{"_cell_guid":"c5153673-584e-4bfc-a81f-5b76ab926259","_uuid":"655c21b72f139151af599c95e624e2ea4d78b0c2"},"cell_type":"markdown","source":"So 46.4% of the ads are for Personal belongings, 11.9% are for home and garden and 11.5% for consumer electronics."},{"metadata":{"_cell_guid":"31afe8b2-07c4-4efe-aa0f-7d4dd416e26b","_kg_hide-input":true,"_uuid":"7e0492379016632137f45bcc6351bf283182d83e","trusted":true},"cell_type":"code","source":"plt.figure(figsize=(12,8))\nsns.boxplot(x=\"parent_category_name_en\", y=\"deal_probability\", data=train_df)\nplt.ylabel('Deal probability', fontsize=12)\nplt.xlabel('Parent Category', fontsize=12)\nplt.title(\"Deal probability by parent category\", fontsize=14)\nplt.xticks(rotation='vertical')\nplt.show()","execution_count":26,"outputs":[]},{"metadata":{"_cell_guid":"fc9526b2-ebcb-48c6-a75d-a5b60999792d","_uuid":"41f3bff88183dbad30caa1ae2d59417d28a0da9e"},"cell_type":"markdown","source":"\nServices category seems to have slightly higher deal probability compared to others. "},{"metadata":{"_cell_guid":"5a55649d-e15b-4984-97f3-dba2fa0df751","_uuid":"2d8c704f3888afb344b135f231fdc3b3968d1e11"},"cell_type":"markdown","source":"** Category of Ads:**\n\nNow let us look at the category of ads."},{"metadata":{"_cell_guid":"8f56b56d-dbc5-41c5-97b5-1fd452ad99ff","_kg_hide-input":true,"_uuid":"894c0dae35ff6bdde2152ff2614f3c39fbb682ee","collapsed":true,"trusted":true},"cell_type":"code","source":"temp_data = StringIO(\"\"\"\ncategory_name,category_name_en\n\"Одежда, обувь, аксессуары\",\"Clothing, shoes, accessories\"\nДетская одежда и обувь,Children's clothing and shoes\nТовары для детей и игрушки,Children's products and toys\nКвартиры,Apartments\nТелефоны,Phones\nМебель и интерьер,Furniture and interior\nПредложение услуг,Offer services\nАвтомобили,Cars\nРемонт и строительство,Repair and construction\nБытовая техника,Appliances\nТовары для компьютера,Products for computer\n\"Дома, дачи, коттеджи\",\"Houses, villas, cottages\"\nКрасота и здоровье,Health and beauty\nАудио и видео,Audio and video\nСпорт и отдых,Sports and recreation\nКоллекционирование,Collecting\nОборудование для бизнеса,Equipment for business\nЗемельные участки,Land\nЧасы и украшения,Watches and jewelry\nКниги и журналы,Books and magazines\nСобаки,Dogs\n\"Игры, приставки и программы\",\"Games, consoles and software\"\nДругие животные,Other animals\nВелосипеды,Bikes\nНоутбуки,Laptops\nКошки,Cats\nГрузовики и спецтехника,Trucks and buses\nПосуда и товары для кухни,Tableware and goods for kitchen\nРастения,Plants\nПланшеты и электронные книги,Tablets and e-books\nТовары для животных,Pet products\nКомнаты,Room\nФототехника,Photo\nКоммерческая недвижимость,Commercial property\nГаражи и машиноместа,Garages and Parking spaces\nМузыкальные инструменты,Musical instruments\nОргтехника и расходники,Office equipment and consumables\nПтицы,Birds\nПродукты питания,Food\nМотоциклы и мототехника,Motorcycles and bikes\nНастольные компьютеры,Desktop computers\nАквариум,Aquarium\nОхота и рыбалка,Hunting and fishing\nБилеты и путешествия,Tickets and travel\nВодный транспорт,Water transport\nГотовый бизнес,Ready business\nНедвижимость за рубежом,Property abroad\n\"\"\")\n\ntemp_df = pd.read_csv(temp_data)\ntrain_df = pd.merge(train_df, temp_df, on=\"category_name\", how=\"left\")","execution_count":27,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7946e97048941ab2f8eb41856bec4ad5f6bc701c"},"cell_type":"code","source":"(cnt_srs.index[::1]).shape","execution_count":28,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3e1af60c32aa118f39fd924bbb30651815572b4f"},"cell_type":"code","source":"(cnt_srs.index[::-1]).shape","execution_count":29,"outputs":[]},{"metadata":{"_cell_guid":"8f73a8e7-da9f-49e7-8cf8-ceb64b78d791","_kg_hide-input":true,"_uuid":"71768ad41fe38851a0e6c990ba30166c3b37e584","trusted":true},"cell_type":"code","source":"cnt_srs = train_df['category_name_en'].value_counts()\ntrace = go.Bar(\n    y=cnt_srs.index[::-1],\n    x=cnt_srs.values[::-1],\n    orientation = 'h',\n    marker=dict(\n        color=cnt_srs.values[::-1],\n        colorscale = 'Blues',\n        reversescale = True\n    ),\n)\n\nlayout = dict(\n    title='Category Name of Ads - Count',\n    height=900\n    )\ndata = [trace]\nfig = go.Figure(data=data, layout=layout)\npy.iplot(fig, filename=\"category name\")","execution_count":30,"outputs":[]},{"metadata":{"_cell_guid":"3969bee4-d7cf-4eed-b536-172ebf242033","_uuid":"54bcae87d7e90a49aabb23d7600bd391977368a3"},"cell_type":"markdown","source":"So the top 3 categories are:\n1. Clothes, shoes, accessories\n2. Children's clothing and footwear\n3. Goods for children and toys\n"},{"metadata":{"_cell_guid":"26503524-68a1-4dc1-be5e-bb9a7256e6d7","_uuid":"5ef55d3b291cf2e026ff52eac10c89f8d21ff765"},"cell_type":"markdown","source":"** User Type:**\n\nNow let us look at the user type. "},{"metadata":{"_cell_guid":"ba263be0-6c71-49a1-97df-a85e046365ae","_kg_hide-input":true,"_uuid":"1cb60a19d6da78ba93dd97a7ee5a50c04582cdea","trusted":true},"cell_type":"code","source":"temp_series = train_df['user_type'].value_counts()\nlabels = (np.array(temp_series.index))\nsizes = (np.array((temp_series / temp_series.sum())*100))\n\ntrace = go.Pie(labels=labels, values=sizes)\nlayout = go.Layout(\n    title='User Type distribution',\n    width=600,\n    height=600,\n)\ndata = [trace]\nfig = go.Figure(data=data, layout=layout)\npy.iplot(fig, filename=\"usertype\")","execution_count":31,"outputs":[]},{"metadata":{"_cell_guid":"f7a5ddf8-9c31-4e60-9d29-13106f7e5b82","_uuid":"6387e62bf114a4c1b72220ec10f699fbe2126e51"},"cell_type":"markdown","source":"Private users constitute 72% of the data followed by company and shop.\n\n**Price:**\n\nThis is the price shown in the Ad. "},{"metadata":{"_cell_guid":"49dbcd6e-e125-425d-acf0-8f9eebd24d40","_kg_hide-input":true,"_uuid":"c59a8692b97b97d0e169715f53d7c73d3445792d","trusted":true},"cell_type":"code","source":"train_df[\"price_new\"] = train_df[\"price\"].values\ntrain_df[\"price_new\"].fillna(np.nanmean(train_df[\"price\"].values), inplace=True)\n\nplt.figure(figsize=(12,8))\nsns.distplot(np.log1p(train_df[\"price_new\"].values), bins=100, kde=False)\nplt.xlabel('Log of price', fontsize=12)\nplt.title(\"Log of Price Histogram\", fontsize=14)\nplt.show()","execution_count":32,"outputs":[]},{"metadata":{"_cell_guid":"4c8ac15c-bc30-4a8d-9bb4-170f2c5b3dfd","_uuid":"76eb5e4d3c814b1760d3551d469df0c10db79ef2"},"cell_type":"markdown","source":"**Activation Date:**"},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"3044fca3fc9f5147af6073b89dd2c992ecf515b2"},"cell_type":"code","source":"cnt_srs = train_df['activation_date']","execution_count":33,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"729a3f7e3b60d57bbc19027c53d381187a003e2d"},"cell_type":"code","source":"(cnt_srs[:5])","execution_count":34,"outputs":[]},{"metadata":{"_cell_guid":"e5b5c36b-e750-48dc-b0a9-6832e951275e","_kg_hide-input":true,"_uuid":"e930d03a411b47e1c76733d34bc9211e9c87fce1","trusted":true},"cell_type":"code","source":"cnt_srs = train_df['activation_date'].value_counts()\n\ntrace = go.Bar(\n    x=cnt_srs.index,\n    y=cnt_srs.values,\n    marker=dict(\n        color=cnt_srs.values,\n        colorscale = 'Picnic',\n        reversescale = True\n    ),\n)\n\nlayout = go.Layout(\n    title='Activation Dates in Train'\n)\n\ndata = [trace]\nfig = go.Figure(data=data, layout=layout)\npy.iplot(fig, filename=\"ActivationDate\")\n\n# Activation dates in test\ncnt_srs = test_df['activation_date'].value_counts()\n\ntrace = go.Bar(\n    x=cnt_srs.index,\n    y=cnt_srs.values,\n    marker=dict(\n        color=cnt_srs.values,\n        colorscale = 'Picnic',\n        reversescale = True\n    ),\n)\n\nlayout = go.Layout(\n    title='Activation Dates in Test'\n)\n\ndata = [trace]\nfig = go.Figure(data=data, layout=layout)\npy.iplot(fig, filename=\"ActivationDate\")","execution_count":35,"outputs":[]},{"metadata":{"_cell_guid":"d3609c4e-a6ca-4913-8ca2-290e64a06624","_uuid":"88fc62fadc8a34a33dd1074d350a893b02e8e3c4"},"cell_type":"markdown","source":"Inferences:\n\n1. So the dates are not different between train and test sets. So we need to be careful while doing our validation. May be time based validation is a good option.\n2. We are given two weeks data for training (March 15 to March 28) and one week data for testing (April 12 to April 18, 2017).\n3. There is a gap of two weeks in between training and testing data. \n4. We can probably use weekday as a feature since all the days are present in both train and test sets. \n\n\n**User id:**\n\nNow we can have a look at the number of unique users in train & test and also the number of common users if any."},{"metadata":{"_cell_guid":"941cee98-9a2c-43f6-813b-6ae5e5f6cb07","_kg_hide-input":true,"_uuid":"92353e06ed1a03a6f78611f514b89703b9c4f1bc","trusted":true},"cell_type":"code","source":"from matplotlib_venn import venn2\n\nplt.figure(figsize=(10,7))\nvenn2([set(train_df.user_id.unique()), set(test_df.user_id.unique())], set_labels = ('Train set', 'Test set') )\nplt.title(\"Number of users in train and test\", fontsize=15)\nplt.show()","execution_count":36,"outputs":[]},{"metadata":{"_cell_guid":"90292fcb-facc-4b4b-8fef-811361410797","_uuid":"3b271c43d2f14fddded57267b5efdce5d7975e23"},"cell_type":"markdown","source":"So out of the 306K users in test, about 68K users are there in train and the rest are new.\n\n** Title: **\n\nFirst let us look at the number of common titles between train and test set"},{"metadata":{"trusted":true,"_uuid":"46b1d467cec6021c5877bf4d5337113d6245aa64"},"cell_type":"code","source":"train_df.title[:5]","execution_count":37,"outputs":[]},{"metadata":{"_cell_guid":"8b932137-829c-46e0-8a81-4c050d8a6a5e","_kg_hide-input":true,"_uuid":"d6fc6de20d442fea87243d76c1a07ed8fef9d7d8","trusted":true},"cell_type":"code","source":"from matplotlib_venn import venn2\n\nplt.figure(figsize=(10,7))\nvenn2([set(train_df.title.unique()), set(test_df.title.unique())], set_labels = ('Train set', 'Test set') )\nplt.title(\"Number of titles in train and test\", fontsize=15)\nplt.show()","execution_count":38,"outputs":[]},{"metadata":{"_cell_guid":"178201a2-561c-4d3a-912c-7f62c5d929a3","_uuid":"23f2f3cf6b8441938cc588509c08269e82f25c59"},"cell_type":"markdown","source":"We have around 64K common titles between train and test set. Now let us look at the number of words present in the title column."},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"c09e1f1f02e7c2b802bffd3e80fdc3bfe54b8ede"},"cell_type":"code","source":"train_df[\"title_nwords\"] = train_df[\"title\"].apply(lambda x: len(x.split()))\ntest_df[\"title_nwords\"] = test_df[\"title\"].apply(lambda x: len(x.split()))","execution_count":39,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"08043e8bbed3f108cf211c99678f578e907701e1","collapsed":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"ee78ef12-bbca-46f3-b2ca-c01780e320e0","_kg_hide-input":true,"_uuid":"d839c73ea40ccecc0dd31dc292b883502510de6e","trusted":true},"cell_type":"code","source":"train_df[\"title_nwords\"] = train_df[\"title\"].apply(lambda x: len(x.split()))\ntest_df[\"title_nwords\"] = test_df[\"title\"].apply(lambda x: len(x.split()))\n\ncnt_srs = train_df['title_nwords'].value_counts()\n\ntrace = go.Bar(\n    x=cnt_srs.index,\n    y=cnt_srs.values,\n    marker=dict(\n        color=\"blue\",\n        #colorscale = 'Blues',\n        reversescale = True\n    ),\n)\n\nlayout = go.Layout(\n    title='Number of words in title column'\n)\n\ndata = [trace]\nfig = go.Figure(data=data, layout=layout)\npy.iplot(fig, filename=\"title_nwords\")           ","execution_count":40,"outputs":[]},{"metadata":{"_cell_guid":"f74fea23-7145-4f8c-97f2-6a552fc242ac","_uuid":"f07c7566d3430226a6f8f3d0098fa2243700f112"},"cell_type":"markdown","source":"Majority of the tiles have 1, 2 or 3 words and has a long tail.\n\nNow we will do the following:\n1. Take the TF-IDF of the title column and this will be a sparse matrix with huge dimesnions.\n2. Get the top SVD components fof this TF-IDF \n3. Plot the distribution of SVD components with Deal probability to see if these variables help."},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"46b9067299aef5325dc62e8787e15ccb40f4b973"},"cell_type":"code","source":"tfidf_vec = TfidfVectorizer(ngram_range=(1,1))\nfull_tfidf = tfidf_vec.fit_transform(train_df['title'].values.tolist() + test_df['title'].values.tolist())\ntrain_tfidf = tfidf_vec.transform(train_df['title'].values.tolist())\ntest_tfidf = tfidf_vec.transform(test_df['title'].values.tolist())\n#what is \"TfidfVectorizer\" and what is the \"ngram_range\"\n#it converts the text documents to a matrix of tfidf features\n#tfidf features are the method to convert the textual information into the vector space, they are a measure of how important a word in a text is\n#what is ngram_range\n#what is ngram\n#ngram is the number of words in a sequence\n#ngram_range is the lower and upper range of the features to be extracted\n#the ngram_range() gives the upper range and the lower range of the ngram size thst should be considered in the features of the corpus\n#the tfidf_vec.fit() , fits in the corpus of document extracts the features, and then form the vectorised dataset in the vector space\n#the tfidf_vec.transform(), takes the corpus provided as the input and then extract all the features that are already present in the tfidr_vec and ignores all\n# the features of the input that are not already there in the tfidf_vec.transform()","execution_count":41,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"d98582caf4a49d56693e528fd56bca19a8eb2055"},"cell_type":"code","source":"n_comp = 3\nsvd_obj = TruncatedSVD(n_components=n_comp, algorithm='arpack')\n\n#svd acts like the pca for the text features and the N_comp passed as the parameter decides the number of features that each data vector will have","execution_count":42,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1ef6f3761b564f082885345730137fc38983b5c0"},"cell_type":"code","source":"train_tfidf.shape","execution_count":43,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"a232e7b19220c8f9a994ec819d9b51da327e9cc8"},"cell_type":"code","source":"svd_obj.fit(full_tfidf)\ntrain_svd = pd.DataFrame(svd_obj.transform(train_tfidf))","execution_count":44,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"bf566309f43f649960c229ef63e74fdb27fd0728"},"cell_type":"code","source":"svd_obj.transform(train_tfidf)","execution_count":45,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"ba92630e613058e42d88cb45f1c2fe1cda65b461"},"cell_type":"code","source":"\ntest_svd = pd.DataFrame(svd_obj.transform(test_tfidf))","execution_count":46,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"54ebce69706b14a22fd596b308085224121ed868"},"cell_type":"code","source":"train_svd.columns = ['svd_title_'+str(i+1) for i in range(n_comp)]\ntest_svd.columns = ['svd_title_'+str(i+1) for i in range(n_comp)]","execution_count":47,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"fc9d9e241e53640927f71d1bb0f6aa636606088d"},"cell_type":"code","source":"test_svd.columns","execution_count":48,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"534dabdf951ff62227dc995380b44e4bafdecfb8"},"cell_type":"code","source":"test_svd.columns","execution_count":49,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"830c2b5967f126910728e0b213ce6f15d066496e"},"cell_type":"code","source":"train_df.head()","execution_count":50,"outputs":[]},{"metadata":{"_cell_guid":"16f8e6bb-8cfc-478e-973b-56bf50d66228","_kg_hide-input":true,"_uuid":"5493999e6388b8601da5cb707f0876c8d257a24a","collapsed":true,"trusted":true},"cell_type":"code","source":"\n\n### SVD Components ###\n#n_comp = 3\n#svd_obj = TruncatedSVD(n_components=n_comp, algorithm='arpack')\n#what does truncatedSVD do\n#\n#svd_obj.fit(full_tfidf)\n#train_svd = pd.DataFrame(svd_obj.transform(train_tfidf))\n#test_svd = pd.DataFrame(svd_obj.transform(test_tfidf))\n#train_svd.columns = ['svd_title_'+str(i+1) for i in range(n_comp)]\n#test_svd.columns = ['svd_title_'+str(i+1) for i in range(n_comp)]\ntrain_df = pd.concat([train_df, train_svd], axis=1)\ntest_df = pd.concat([test_df, test_svd], axis=1)\ndel full_tfidf, train_tfidf, test_tfidf, train_svd, test_svd","execution_count":51,"outputs":[]},{"metadata":{"_cell_guid":"27d987af-0dbd-4e2d-8703-a1fe69f7f416","_kg_hide-input":true,"_uuid":"270c110e67af84fc019a5c06c5cb63fb3da6a74d","trusted":true},"cell_type":"code","source":"# 1st svd comp #\nplt.figure(figsize=(8,8))\nsns.jointplot(x=train_df[\"svd_title_1\"].values, y=train_df[\"deal_probability\"].values, size=10)\nplt.ylabel('Deal Probability', fontsize=12)\nplt.xlabel('First SVD component on Title', fontsize=12)\nplt.title(\"Deal Probability distribution for First SVD component on title\", fontsize=15)\nplt.show()\n\n# 2nd svd comp #\nplt.figure(figsize=(8,8))\nsns.jointplot(x=train_df[\"svd_title_2\"].values, y=train_df[\"deal_probability\"].values, size=10)\nplt.ylabel('Deal Probability', fontsize=12)\nplt.xlabel('Second SVD component on Title', fontsize=12)\nplt.title(\"Deal Probability distribution for Second SVD component on title\", fontsize=15)\nplt.show()\n\n# 3rd svd comp #\nplt.figure(figsize=(8,8))\nsns.jointplot(x=train_df[\"svd_title_3\"].values, y=train_df[\"deal_probability\"].values, size=10)\nplt.ylabel('Deal Probability', fontsize=12)\nplt.xlabel('Third SVD component on Title', fontsize=12)\nplt.title(\"Deal Probability distribution for Third SVD component on title\", fontsize=15)\nplt.show()","execution_count":52,"outputs":[]},{"metadata":{"_cell_guid":"ee3331ff-c067-4768-a841-3aa44076cd88","_uuid":"c490aef2b274725cd574f01bebf686b1f0034b71"},"cell_type":"markdown","source":"As we can see, the top SVD components capture quite an amount of variation in the data. So this might be helpful features in our modeling process."},{"metadata":{"_cell_guid":"0d7c5e78-c64b-4650-8e57-5a5acd90373f","_uuid":"f11548b173c13491c64259532796039fa3ffe455"},"cell_type":"markdown","source":"**Description:**\n\nLet us first check the number of words in the description column.\n"},{"metadata":{"trusted":true,"_uuid":"d26c6a244b4f1d037d5ab511b69d3c63825f77c5","collapsed":true},"cell_type":"code","source":"train_df[\"description\"].fillna(\"NA\",inplace=True)","execution_count":53,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"2aabecf73f03e746b906e6d7e46a151505b8cc5b"},"cell_type":"code","source":"test_df[\"description\"].fillna(\"NA\", inplace=True)","execution_count":54,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"276e89fe95038693c51edd27d2d39b1882acf754"},"cell_type":"code","source":"train_df[\"desc_nwords\"] = train_df[\"description\"].apply(lambda x: len(x.split()))\ntrain_df[\"desc_nwords\"]","execution_count":55,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6350ba5ec6be72160ad17beac9c7b888f1b14afb"},"cell_type":"code","source":"test_df[\"desc_nwords\"] = test_df[\"description\"].apply(lambda x: len(x.split()))\n\ncnt_srs = train_df['desc_nwords'].value_counts()\ncnt_srs","execution_count":56,"outputs":[]},{"metadata":{"_cell_guid":"d9d04405-8a50-4746-96e9-8fec3576b0f6","_kg_hide-input":true,"_uuid":"01093d9b05aed9c5692e384e15a0525c6f65a840","trusted":true},"cell_type":"code","source":"## Filling missing values ##\ntrain_df[\"description\"].fillna(\"NA\", inplace=True)\ntest_df[\"description\"].fillna(\"NA\", inplace=True)\n\ntrain_df[\"desc_nwords\"] = train_df[\"description\"].apply(lambda x: len(x.split()))\ntest_df[\"desc_nwords\"] = test_df[\"description\"].apply(lambda x: len(x.split()))\n\ncnt_srs = train_df['desc_nwords'].value_counts().head(100)\n\ntrace = go.Bar(\n    x=cnt_srs.index,\n    y=cnt_srs.values,\n    marker=dict(\n        color=\"blue\",\n        #colorscale = 'Blues',\n        reversescale = True\n    ),\n)\n\nlayout = go.Layout(\n    title='Number of words in Description column'\n)\n\ndata = [trace]\nfig = go.Figure(data=data, layout=layout)\npy.iplot(fig, filename=\"desc_nwords\")  ","execution_count":57,"outputs":[]},{"metadata":{"_cell_guid":"2dac9656-fb96-4885-a4c3-619b97bbade0","_uuid":"a5b9b30c9f26b12dff713cec6ae0b4ac95dd284e"},"cell_type":"markdown","source":"Description column has a huge right tail till about 700 words. I have cut the same till top 100 for better visualization.\n\nNow let us create the first 3 SVD components for description column (just like title) and plot the same against deal probability."},{"metadata":{"_cell_guid":"b2ef73f3-2915-4276-8aab-4b96be735727","_kg_hide-input":true,"_uuid":"285b43beab661ec477ddcf198eaebd24944433e1","collapsed":true,"trusted":true},"cell_type":"code","source":"### TFIDF Vectorizer ###\ntfidf_vec = TfidfVectorizer(ngram_range=(1,1), max_features=100000)\nfull_tfidf = tfidf_vec.fit_transform(train_df['description'].values.tolist() + test_df['description'].values.tolist())\ntrain_tfidf = tfidf_vec.transform(train_df['description'].values.tolist())\ntest_tfidf = tfidf_vec.transform(test_df['description'].values.tolist())\n\n### SVD Components ###\nn_comp = 3\nsvd_obj = TruncatedSVD(n_components=n_comp, algorithm='arpack')\nsvd_obj.fit(full_tfidf)\ntrain_svd = pd.DataFrame(svd_obj.transform(train_tfidf))\ntest_svd = pd.DataFrame(svd_obj.transform(test_tfidf))\ntrain_svd.columns = ['svd_desc_'+str(i+1) for i in range(n_comp)]\ntest_svd.columns = ['svd_desc_'+str(i+1) for i in range(n_comp)]\ntrain_df = pd.concat([train_df, train_svd], axis=1)\ntest_df = pd.concat([test_df, test_svd], axis=1)\ndel full_tfidf, train_tfidf, test_tfidf, train_svd, test_svd","execution_count":58,"outputs":[]},{"metadata":{"_cell_guid":"7f53f33f-9247-439c-a84d-3818634d45be","_kg_hide-input":true,"_uuid":"817e8255a4c4773c799d1a3d7b91f5e16af5a933","trusted":true},"cell_type":"code","source":"# 1st svd comp #\nplt.figure(figsize=(8,8))\nsns.jointplot(x=train_df[\"svd_desc_1\"].values, y=train_df[\"deal_probability\"].values, size=10)\nplt.ylabel('Deal Probability', fontsize=12)\nplt.xlabel('First SVD component on Description', fontsize=12)\nplt.title(\"Deal Probability distribution for First SVD component on Description\", fontsize=15)\nplt.show()\n\n# 2nd svd comp #\nplt.figure(figsize=(8,8))\nsns.jointplot(x=train_df[\"svd_desc_2\"].values, y=train_df[\"deal_probability\"].values, size=10)\nplt.ylabel('Deal Probability', fontsize=12)\nplt.xlabel('Second SVD component on Description', fontsize=12)\nplt.title(\"Deal Probability distribution for Second SVD component on title\", fontsize=15)\nplt.show()\n\n# 3rd svd comp #\nplt.figure(figsize=(8,8))\nsns.jointplot(x=train_df[\"svd_desc_3\"].values, y=train_df[\"deal_probability\"].values, size=10)\nplt.ylabel('Deal Probability', fontsize=12)\nplt.xlabel('Second SVD component on Description', fontsize=12)\nplt.title(\"Deal Probability distribution for Third SVD component on Description\", fontsize=15)\nplt.show()","execution_count":59,"outputs":[]},{"metadata":{"_cell_guid":"101e25f2-3dde-4f42-a2bf-1b39872d64a6","_uuid":"5b080dd322618632376658e7c6075f79a7150c3e"},"cell_type":"markdown","source":"Looks like it also captures some good amount of variability. \n\n** Baseline Model:**\n\nNow let us build a baseline model using the given features and newly created features.\n\nIn the following block, we will do :\n 1. Create a new feature for week day\n 2. Label encode the cateforical variables\n 3. Drop the columns that are not needed in the model."},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"b5e13ae6e4325e2458daca087d4f0ade6d532304"},"cell_type":"code","source":"train_y = train_df[\"deal_probability\"].values\ntest_id = test_df[\"item_id\"].values","execution_count":60,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7b38bc9296ab701dfbbd229a0a86d815619b8a6e","collapsed":true},"cell_type":"code","source":"train_df[\"activation_weekday\"] = train_df[\"activation_date\"].dt.weekday\n#train_df[\"activation_weekday\"]\ntest_df[\"activation_weekday\"] = test_df[\"activation_date\"].dt.weekday\n","execution_count":61,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b1acb9a05e2da0bbfae6f60c5ea3fb848866bb8f","collapsed":true},"cell_type":"code","source":"type(train_df[col].values[0])","execution_count":76,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"52ba9cd31b69975ee309d4e882e73e08681d8a85","collapsed":true},"cell_type":"code","source":"np.unique(train_df[col].values).shape","execution_count":75,"outputs":[]},{"metadata":{"_uuid":"66c3ba10cd10e8163c95309711c3d62c3a3264b0"},"cell_type":"raw","source":""},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"73c109f810fe8f8ae1fade7a2295da2b0ef36c65"},"cell_type":"code","source":"cat_vars = [\"region\", \"city\", \"parent_category_name\", \"category_name\", \"user_type\", \"param_1\", \"param_2\", \"param_3\"]\n\n#what does the label encoder fit does\nfor col in cat_vars:\n    lbl = preprocessing.LabelEncoder()\n    lbl.fit(list(train_df[col].values.astype('str')) + list(test_df[col].values.astype('str')))\n    train_df[col] = lbl.transform(list(train_df[col].values.astype('str')))\n    test_df[col] = lbl.transform(list(test_df[col].values.astype('str')))","execution_count":64,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c484e796af93f56fcae063b73c22326937c95d9b"},"cell_type":"code","source":"# Target and ID variables #\n#train_y = train_df[\"deal_probability\"].values\n#test_id = test_df[\"item_id\"].values\n\n# New variable on weekday #\n#train_df[\"activation_weekday\"] = train_df[\"activation_date\"].dt.weekday\n#test_df[\"activation_weekday\"] = test_df[\"activation_date\"].dt.weekday\n\n# Label encode the categorical variables #\ncat_vars = [\"region\", \"city\", \"parent_category_name\", \"category_name\", \"user_type\", \"param_1\", \"param_2\", \"param_3\"]\nfor col in cat_vars:\n    lbl = preprocessing.LabelEncoder()\n    lbl.fit(list(train_df[col].values.astype('str')) + list(test_df[col].values.astype('str')))\n    train_df[col] = lbl.transform(list(train_df[col].values.astype('str')))\n    test_df[col] = lbl.transform(list(test_df[col].values.astype('str')))\n\ncols_to_drop = [\"item_id\", \"user_id\", \"title\", \"description\", \"activation_date\", \"image\"]\ntrain_X = train_df.drop(cols_to_drop + [\"region_en\", \"parent_category_name_en\", \"category_name_en\", \"price_new\", \"deal_probability\"], axis=1)\ntest_X = test_df.drop(cols_to_drop, axis=1)","execution_count":65,"outputs":[]},{"metadata":{"_uuid":"413f7a12938345695f89cc298d3913ef8895f916"},"cell_type":"markdown","source":"Now let us create a custom function to build the LightGBM model."},{"metadata":{"_uuid":"c1749bd659711d53673ce3cc61632257740d3a66"},"cell_type":"markdown","source":"# For Better Accuracy\n\n- Use large max_bin (may be slower)\n\n- Use small learning_rate with large num_iterations\n\n- Use large num_leaves (may cause over-fitting)\n\n- Use bigger training data\n\n- Try dart\n\n# Deal with Over-fitting\n\n- Use small max_bin\n\n \n - Use small num_leaves\n\n - Use min_data_in_leaf and min_sum_hessian_in_leaf\n\n  - Use bagging by set bagging_fraction and bagging_freq\n\n - Use feature sub-sampling by set feature_fraction\n\n - Use bigger training data\n\n - Try lambda_l1, lambda_l2 and min_gain_to_split for regularization\n\n - Try max_depth to avoid growing deep tree"},{"metadata":{"trusted":true,"_uuid":"7eaf4e88adf2b515c2620f9a8806fa347e00a10e"},"cell_type":"code","source":"model.best_iteration","execution_count":66,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d9e7d6deb3cae10ecdfb01058325e1ae240f6510","collapsed":true},"cell_type":"code","source":"def run_lgb(train_X, train_y, val_X, val_y,alpha):\n    params = {\n        \"objective\" : \"regression\",\n        \"metric\" : \"rmse\",\n        \"num_leaves\" : 30,\n        \"learning_rate\" : alpha,\n        \"bagging_fraction\" : 0.7,\n        \"feature_fraction\" : 0.7,\n        \"bagging_frequency\" : 5,\n        \"bagging_seed\" : 2018,\n        \"verbosity\" : -1\n    }\n    \n    lgtrain = lgb.Dataset(train_X, label=train_y)\n    lgval = lgb.Dataset(val_X, label=val_y)\n    evals_result = {}\n    model = lgb.train(params, lgtrain, 1000, valid_sets=[lgval], early_stopping_rounds=10000, verbose_eval=20, evals_result=evals_result)\n    \n    return  model, evals_result","execution_count":70,"outputs":[]},{"metadata":{"_uuid":"1e18f280ab5a2ec2b04c4dc3da368e664d37b190"},"cell_type":"markdown","source":"We will now split the train into development and validation sample and build our models. We will take the last 200K rows as validation sample. "},{"metadata":{"trusted":true,"_uuid":"d84663e6bcdc3bab86885a0c066417a48ba2505a"},"cell_type":"code","source":"train_X.shape","execution_count":68,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"135a8e08158017253adf5a46dcb46fcb0d1a5bbd"},"cell_type":"code","source":"import time\nstart_time = time.time()\n\n\n# Splitting the data for model training#\n#parametertuning, increased the training dataset by 10 times\ndev_X = train_X.iloc[:-200000,:]\nval_X = train_X.iloc[-200000:,:]\ndev_y = train_y[:-200000]\nval_y = train_y[-200000:]\nprint(dev_X.shape, val_X.shape, test_X.shape)\n\n# Training the model #\nlearningList = [0.001,0.003,0.009,0.027,0.081,0.243,0.729,2.178,6.514,19.51]\n#the best in above list is 0.243\ntrain_error = list()\ntest_error = list()\nfor i in learningList:\n    print(\"the learning_rate 0f \",i)\n    model, evals_result = run_lgb(dev_X, dev_y, val_X, val_y, i)\n    pred_train = model.predict(dev_X, num_iteration=model.best_iteration)\n    pred_val= model.predict(val_X,num_iteration = model.best_iteration)\n    train_error.append(metrics.mean_squared_error(dev_y,pred_train))\n    test_error.append(metrics.mean_squared_error(val_y,pred_val))\n    del model, evals_result\n# Making a submission file #\n#pred_test[pred_test>1] = 1\n#pred_test[pred_test<0] = 0\n#sub_df = pd.DataFrame({\"item_id\":test_id})\n#sub_df[\"deal_probability\"] = pred_test\n#sub_df.to_csv(\"baseline_lgb.csv\", index=False)\nplt.plot(learningList,train_error)\nplt.plot(learningList,test_error)\nplt.show()\n#pred_test, model, evals_result = run_lgb(dev_X, dev_y, val_X, val_y, test_X)\n\n# Making a submission file #\n#pred_test[pred_test>1] = 1\n#pred_test[pred_test<0] = 0\n#sub_df = pd.DataFrame({\"item_id\":test_id})\n#sub_df[\"deal_probability\"] = pred_test\n#sub_df.to_csv(\"baseline_lgb.csv\", index=False)\nprint(\"--- %s seconds ---\" % (time.time() - start_time))","execution_count":85,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"41963b341813f5cfb64169223a80afbb8847c5b6"},"cell_type":"code","source":"train_error","execution_count":76,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b047211abe98de34087db1e497caabf112ccd692"},"cell_type":"code","source":"axes = plt.gca()\naxes.set_xlim([min(learningList),max(learningList)])\naxes.set_ylim([min(train_error),max(test_error)])\nplt.plot(learningList,train_error)\nplt.plot(learningList,test_error)\nplt.show()","execution_count":84,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8612b43cfd80b028737a2953845bb01efe076cbf"},"cell_type":"code","source":"type(learningList)","execution_count":83,"outputs":[]},{"metadata":{"_uuid":"3788c1605ef990a0bfe17887b878e8e313bb770c"},"cell_type":"markdown","source":"** Feature Importance : **\n\nNow let us look at the top features from the model."},{"metadata":{"trusted":true,"_uuid":"2c31a6370706f5c0630e565bb32cfdd8e0d1fa9f","collapsed":true},"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(12,18))\nlgb.plot_importance(model, max_num_features=50, height=0.8, ax=ax)\nax.grid(False)\nplt.title(\"LightGBM - Feature Importance\", fontsize=15)\nplt.show()","execution_count":82,"outputs":[]},{"metadata":{"_uuid":"cd521a6ee14c4da42c7504a5acb4d3c5d32063b7"},"cell_type":"markdown","source":"Price seems to be the most important feature followed by image_top_1 and param_1.\n\n** Ways to improve the model:**\n\n 1. Adding more new features \n 2. Tuning the parameters as these are not tuned\n 3. Blending with other models.\n \n** Happy Kaggling.!**"},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"acf1cc55bee8a674e552f858aebb3b5a2f72d58b"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.5","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}