{"cells":[{"metadata":{"_uuid":"420cb069496de95439ff184a52d1cd275af216d9","collapsed":true,"_cell_guid":"35917dd7-cfde-40d9-8c77-117b99c99ecf","trusted":true},"cell_type":"code","source":"# -*- coding: utf-8 -*-","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"57a50c15872fb6261c8b80923b7dfef51fbb0aa8","_cell_guid":"76c0aff5-f279-4ca9-a959-14785f124a59"},"cell_type":"markdown","source":"## 이 모델의 핵심\n\n- activation_weekday(요일) feature 추가\n- LabelEncoder로 카테고리 인코딩\n  \"region\", \"city\", \"parent_category_name\", \"category_name\", \"user_type\", \"param_1\", \"param_2\", \"param_3\"\n- 칼럼 삭제\n  ([\"item_id\", \"user_id\", \"title\", \"description\", \"activation_date\", \"image\")\n- 최종 학습 feature\n    - region\n    - city\n    - parent_category_name\n    - category_name\n    - param_1\n    - param_2\t\n    - param_3\t\n    - price\t\n    - item_seq_number\t\n    - user_type\t\n    - image_top_1\t\n    - title_nwords\t\n    - svd_title_1\t\n    - svd_title_2\t\n    - svd_title_3\t\n    - desc_nwords\t\n    - svd_desc_1\t\n    - svd_desc_2\t\n    - svd_desc_3\t\n    - activation_weekday"},{"metadata":{"_uuid":"29d46ab7178e10c1a5c67f8c00272a915865454f","_cell_guid":"d056880f-54b8-47bc-bc54-787ce8f36545"},"cell_type":"markdown","source":"## LightGBM 설치"},{"metadata":{"_uuid":"93dc53ac7da26a1f1fc191becfaad1825ad1184e","_cell_guid":"84708f16-dff4-406e-9504-11223398b8d0","trusted":true},"cell_type":"code","source":"!pip install lightgbm","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5d98e1d1de61973ace5583444af21f23780c5f8d","_cell_guid":"9be56900-f14f-4f4f-825d-0ad600f9ba8e"},"cell_type":"markdown","source":"## 패키지로드"},{"metadata":{"_uuid":"3197b149cde5d85aa6fb128816a515f436c0e322","collapsed":true,"_cell_guid":"38878483-d3e2-470b-ad40-f21c00a4891c","trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer\nfrom sklearn.decomposition import TruncatedSVD\nfrom sklearn import preprocessing, model_selection, metrics\nimport lightgbm as lgb","execution_count":12,"outputs":[]},{"metadata":{"_uuid":"9fdc2867d7860d44b5ce09054f084d4ebc3f5d6e","_cell_guid":"1e14c86d-4683-4c40-a78c-dd40889551b5"},"cell_type":"markdown","source":"## SNS 색상 설정"},{"metadata":{"_uuid":"4f94853996e7b3ee2baf6da44c4eb8755764ea27","collapsed":true,"_cell_guid":"a433dad5-0d3f-45f7-891d-1ee268bdcbe8","trusted":true},"cell_type":"code","source":"color = sns.color_palette()\n%matplotlib inline","execution_count":13,"outputs":[]},{"metadata":{"_uuid":"96865fa9e30a2507bf3ff7f51a2aad578c8c2aac","_cell_guid":"e151f175-350c-40d4-9159-3b611ecfb183"},"cell_type":"markdown","source":"## plotly 패키지 설치"},{"metadata":{"_uuid":"a45172776c35c5750ceac25801842df1a84424cd","_cell_guid":"dbca85b9-f534-4e42-8567-84369361c514","trusted":true},"cell_type":"code","source":"!pip install plotly","execution_count":14,"outputs":[]},{"metadata":{"_uuid":"475ed662b525f1be3705992940c4689c96451a50","_cell_guid":"b2a35ba6-8a9c-41a9-a87c-d1653efcd2b8"},"cell_type":"markdown","source":"## plotly 로드"},{"metadata":{"_uuid":"9dea2c6ae964a910495e32ef32b694e9a40a9813","_cell_guid":"7c71f485-73ba-45ba-ae4d-fb065405b8ef","trusted":true},"cell_type":"code","source":"import plotly.offline as py\npy.init_notebook_mode(connected=True)\nimport plotly.graph_objs as go\nimport plotly.tools as tls\n\npd.options.mode.chained_assignment = None\npd.options.display.max_columns = 999","execution_count":15,"outputs":[]},{"metadata":{"_uuid":"29bc3aeab6161384641d00f03784d98770064829","_cell_guid":"b45015ba-3162-44f8-ab21-784fc0e8e3c5"},"cell_type":"markdown","source":"## 데이터 로드"},{"metadata":{"_uuid":"908b6e32ce64d228207971af3bd4886923b6398f","_cell_guid":"7a194813-d043-4666-b9ba-91121ddd62ea","trusted":true},"cell_type":"code","source":"train_df = pd.read_csv(\"../input/train.csv\", parse_dates=[\"activation_date\"])\ntest_df = pd.read_csv(\"../input/test.csv\", parse_dates=[\"activation_date\"])\nprint(\"Train file rows and columns are : \", train_df.shape)\nprint(\"Test file rows and columns are : \", test_df.shape)","execution_count":17,"outputs":[]},{"metadata":{"_uuid":"6240d93479b9fb170d885670de62f60a3a9a703e","_cell_guid":"8c32be46-ee56-4e95-96d4-7b33a1c8beb1"},"cell_type":"markdown","source":"## head 출력해보기"},{"metadata":{"_uuid":"984e80155c1fb9b745325567e84c25c54c62e517","_cell_guid":"48bdd947-88cb-4c21-8f73-ca629ebb138f","trusted":true},"cell_type":"code","source":"train_df.head()","execution_count":18,"outputs":[]},{"metadata":{"_uuid":"6116d84b749c508440544e065f44d37ab1675a71","_cell_guid":"2abdf200-d645-42d6-8313-f74dc971ab56"},"cell_type":"markdown","source":"## Deal Probability 히스토그램 출력\n\n아래 그림을 보면 대부분의 deal이 0이다. 즉 거의 deal이 안일어난다고 봐도 될듯 하다.\n"},{"metadata":{"_uuid":"0cd569bb0d78b78b6e6ab4c2fcb137bd0019117e","_cell_guid":"fb4f0048-ef61-4430-b6df-48af031a8d76","trusted":true,"collapsed":true},"cell_type":"code","source":"plt.figure(figsize=(12,8))\nsns.distplot(train_df[\"deal_probability\"].values, bins=100, kde=False)\nplt.xlabel('Deal Probility', fontsize=12)\nplt.title(\"Deal Probability Histogram\", fontsize=14)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"381949f73ffa7cbe91c8f5036abde8c96db1cff8","_cell_guid":"c68fe4da-c6e9-44fb-8eb0-168fd3dd831a"},"cell_type":"markdown","source":"## deal probability Distribution\n\nscatter 차트로 보더라도 약 10만개의 광고는 deal이 0이다.\n"},{"metadata":{"_uuid":"5700bf4e7e9fef0634a1c832ef9c214d9e754530","_cell_guid":"02e461e0-7bb3-43fc-b2c7-5ffc86e14322","trusted":true,"collapsed":true},"cell_type":"code","source":"plt.figure(figsize=(8,6))\nplt.scatter(range(train_df.shape[0]), np.sort(train_df['deal_probability'].values))\nplt.xlabel('index', fontsize=12)\nplt.ylabel('deal probability', fontsize=12)\nplt.title(\"Deal Probability Distribution\", fontsize=14)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7943198af75f3a41598e8807c789e1fa675ea123","collapsed":true,"_cell_guid":"cfd757e2-cc22-437f-80c6-26f7edd88824","trusted":true},"cell_type":"code","source":"from io import StringIO\n\ntemp_data = StringIO(u\"\"\"\nregion,region_en\nСвердловская область, Sverdlovsk oblast\nСамарская область, Samara oblast\nРостовская область, Rostov oblast\nТатарстан, Tatarstan\nВолгоградская область, Volgograd oblast\nНижегородская область, Nizhny Novgorod oblast\nПермский край, Perm Krai\nОренбургская область, Orenburg oblast\nХанты-Мансийский АО, Khanty-Mansi Autonomous Okrug\nТюменская область, Tyumen oblast\nБашкортостан, Bashkortostan\nКраснодарский край, Krasnodar Krai\nНовосибирская область, Novosibirsk oblast\nОмская область, Omsk oblast\nБелгородская область, Belgorod oblast\nЧелябинская область, Chelyabinsk oblast\nВоронежская область, Voronezh oblast\nКемеровская область, Kemerovo oblast\nСаратовская область, Saratov oblast\nВладимирская область, Vladimir oblast\nКалининградская область, Kaliningrad oblast\nКрасноярский край, Krasnoyarsk Krai\nЯрославская область, Yaroslavl oblast\nУдмуртия, Udmurtia\nАлтайский край, Altai Krai\nИркутская область, Irkutsk oblast\nСтавропольский край, Stavropol Krai\nТульская область, Tula oblast\n\"\"\")\n\nregion_df = pd.read_csv(temp_data)\ntrain_df = pd.merge(train_df, region_df, how=\"left\", on=\"region\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f3c09453f4248a83397d63e37ce8630fc1d7216d","_cell_guid":"85b9d427-54c3-4138-97d9-43dec4be8af0","trusted":true,"collapsed":true},"cell_type":"code","source":"!pip install numexpr --upgrade","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a1a33a64659680c6be93a147b3774a8e2edf0861","_cell_guid":"e2bb4499-3409-4293-a283-6c706c9a7f40"},"cell_type":"markdown","source":"## 지역별 분포\n\n아래와 같은 주요 분포를 가진다.\n\n- Krasnodar region - 9.41%\n- Sverdlovsk region - 6.28%\n- Rostov region - 5.99%"},{"metadata":{"_uuid":"78d80c3fe8d2c3ea29055b7988ff3954d49322d8","_cell_guid":"ad383162-4b3a-412b-ac53-7b2eb0faf961","trusted":true,"collapsed":true},"cell_type":"code","source":"temp_series = train_df['region_en'].value_counts()\nlabels = (np.array(temp_series.index))\nsizes = (np.array((temp_series / temp_series.sum())*100))\n\ntrace = go.Pie(labels=labels, values=sizes)\nlayout = go.Layout(\n    title='Region distribution',\n    width=900,\n    height=900,\n)\ndata = [trace]\nfig = go.Figure(data=data, layout=layout)\npy.iplot(fig, filename=\"region\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ff05fa63e7fd77df9ccfd5be935d6516ce7f97fa","_cell_guid":"a4bad5fa-c057-44ab-9be0-2be778a8121a"},"cell_type":"markdown","source":"## 지역별 Deal Probability\n\n아래는 지역별 Deal 확률이다."},{"metadata":{"_uuid":"0eac85eae567371691184d10cdfe22c4cc8479bc","_cell_guid":"c395d8a4-6016-4a1d-b96d-50aa3244bc77","trusted":true,"collapsed":true},"cell_type":"code","source":"plt.figure(figsize=(12,8))\nsns.boxplot(y=\"region_en\", x=\"deal_probability\", data=train_df)\nplt.xlabel('Deal probability', fontsize=12)\nplt.ylabel('Region', fontsize=12)\nplt.title(\"Deal probability by region\")\nplt.xticks(rotation='vertical')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"168368e5780b69fb568692af68093fde38740402","_cell_guid":"c6083364-a302-449d-bf47-dfa6c7f1e6d9"},"cell_type":"markdown","source":"## 상위 20개 도시의 분포\n\n딜이 가장 많은 상위 3개 도시는 아래와 같다.\n\n- Krasnodar\n- Ekaterinburg\n- Novosibirsk"},{"metadata":{"_uuid":"7e43e6cfdbb8dc148d64b04ba06c75633df056ca","_cell_guid":"915e8e37-04fd-451b-9da9-f897655cb97b","trusted":true,"collapsed":true},"cell_type":"code","source":"cnt_srs = train_df['city'].value_counts().head(20)\ntrace = go.Bar(\n    y=cnt_srs.index[::-1],\n    x=cnt_srs.values[::-1],\n    orientation = 'h',\n    marker=dict(\n        color=cnt_srs.values[::-1],\n        colorscale = 'Blues',\n        reversescale = True\n    ),\n)\n\nlayout = dict(\n    title='City distribution of Ads',\n    )\ndata = [trace]\nfig = go.Figure(data=data, layout=layout)\npy.iplot(fig, filename=\"CityAds\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2db85fc451a2ce4e291a0f21dbdb636156fdb864","_cell_guid":"da17ce67-8c9a-4e78-9b32-234b240697bc"},"cell_type":"markdown","source":"## parent category별 분포"},{"metadata":{"_uuid":"7c038147f2fb8bb87297ec2f5fe232e5ee051769","collapsed":true,"_cell_guid":"2e198cbb-9fd7-40d5-9b21-46f3f715181a","trusted":true},"cell_type":"code","source":"temp_data = StringIO(u\"\"\"\nparent_category_name,parent_category_name_en\nЛичные вещи,Personal belongings\nДля дома и дачи,For the home and garden\nБытовая электроника,Consumer electronics\nНедвижимость,Real estate\nХобби и отдых,Hobbies & leisure\nТранспорт,Transport\nУслуги,Services\nЖивотные,Animals\nДля бизнеса,For business\n\"\"\")\n\ntemp_df = pd.read_csv(temp_data)\ntrain_df = pd.merge(train_df, temp_df, on=\"parent_category_name\", how=\"left\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"36435a84ddc125974c35a9a76ac0d15763d492fe","_cell_guid":"76e4474c-6baa-4490-8d26-09c9653f3d45","trusted":true,"collapsed":true},"cell_type":"code","source":"temp_series = train_df['parent_category_name_en'].value_counts()\nlabels = (np.array(temp_series.index))\nsizes = (np.array((temp_series / temp_series.sum())*100))\n\ntrace = go.Pie(labels=labels, values=sizes)\nlayout = go.Layout(\n    title='Parent Category distribution',\n    width=900,\n    height=900,\n)\ndata = [trace]\nfig = go.Figure(data=data, layout=layout)\npy.iplot(fig, filename=\"parentcategory\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"69edf467e08248e98e60fdd5401b78acedd12ff8","_cell_guid":"1f065219-91dd-4bb7-bd21-c9c6fb31c083"},"cell_type":"markdown","source":"## Parent Category별 Deal 분포\n\n다른것 대비 Service 카테고리의 Deal 분포가 상대적으로 조금 높아보인다."},{"metadata":{"_uuid":"53fedd87d768f9d30977c1a34b852ad49789b9b0","_cell_guid":"4012e05f-d244-4ca0-8529-491202fad280","trusted":true,"collapsed":true},"cell_type":"code","source":"plt.figure(figsize=(12,8))\nsns.boxplot(x=\"parent_category_name_en\", y=\"deal_probability\", data=train_df)\nplt.ylabel('Deal probability', fontsize=12)\nplt.xlabel('Parent Category', fontsize=12)\nplt.title(\"Deal probability by parent category\", fontsize=14)\nplt.xticks(rotation='vertical')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ee2072818a9a403c83b9893c61d1c936fb9de052","_cell_guid":"f55635df-502d-49a7-8cad-cf3c97bbbd03"},"cell_type":"markdown","source":"## 카테고리 이름별 광고 갯수\n\n주요한 광고 카테고리는 아래와 같다.\n\n- Clothes, shoes, accessories\n- Children's clothing and footwear\n- Goods for children and toys"},{"metadata":{"_uuid":"ca65208ce49adcc0e303960d2f13bdc63a835ddf","collapsed":true,"_cell_guid":"a372bb40-0dc9-4733-9fc2-fe2dd841e29c","trusted":true},"cell_type":"code","source":"temp_data = StringIO(u\"\"\"\ncategory_name,category_name_en\n\"Одежда, обувь, аксессуары\",\"Clothing, shoes, accessories\"\nДетская одежда и обувь,Children's clothing and shoes\nТовары для детей и игрушки,Children's products and toys\nКвартиры,Apartments\nТелефоны,Phones\nМебель и интерьер,Furniture and interior\nПредложение услуг,Offer services\nАвтомобили,Cars\nРемонт и строительство,Repair and construction\nБытовая техника,Appliances\nТовары для компьютера,Products for computer\n\"Дома, дачи, коттеджи\",\"Houses, villas, cottages\"\nКрасота и здоровье,Health and beauty\nАудио и видео,Audio and video\nСпорт и отдых,Sports and recreation\nКоллекционирование,Collecting\nОборудование для бизнеса,Equipment for business\nЗемельные участки,Land\nЧасы и украшения,Watches and jewelry\nКниги и журналы,Books and magazines\nСобаки,Dogs\n\"Игры, приставки и программы\",\"Games, consoles and software\"\nДругие животные,Other animals\nВелосипеды,Bikes\nНоутбуки,Laptops\nКошки,Cats\nГрузовики и спецтехника,Trucks and buses\nПосуда и товары для кухни,Tableware and goods for kitchen\nРастения,Plants\nПланшеты и электронные книги,Tablets and e-books\nТовары для животных,Pet products\nКомнаты,Room\nФототехника,Photo\nКоммерческая недвижимость,Commercial property\nГаражи и машиноместа,Garages and Parking spaces\nМузыкальные инструменты,Musical instruments\nОргтехника и расходники,Office equipment and consumables\nПтицы,Birds\nПродукты питания,Food\nМотоциклы и мототехника,Motorcycles and bikes\nНастольные компьютеры,Desktop computers\nАквариум,Aquarium\nОхота и рыбалка,Hunting and fishing\nБилеты и путешествия,Tickets and travel\nВодный транспорт,Water transport\nГотовый бизнес,Ready business\nНедвижимость за рубежом,Property abroad\n\"\"\")\n\ntemp_df = pd.read_csv(temp_data)\ntrain_df = pd.merge(train_df, temp_df, on=\"category_name\", how=\"left\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9c06f820781fd7ba0510a62d1a64a41ab6a57a35","_cell_guid":"85f888b2-6388-44b3-9f8b-db009a070c22","trusted":true,"collapsed":true},"cell_type":"code","source":"cnt_srs = train_df['category_name_en'].value_counts()\ntrace = go.Bar(\n    y=cnt_srs.index[::-1],\n    x=cnt_srs.values[::-1],\n    orientation = 'h',\n    marker=dict(\n        color=cnt_srs.values[::-1],\n        colorscale = 'Blues',\n        reversescale = True\n    ),\n)\n\nlayout = dict(\n    title='Category Name of Ads - Count',\n    height=900\n    )\ndata = [trace]\nfig = go.Figure(data=data, layout=layout)\npy.iplot(fig, filename=\"category name\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9cb7d3196fb1ee5de7a3857116b28bb2cdd8ed8f","_cell_guid":"aa78352b-4b90-49a3-a70a-4b88efea0f42"},"cell_type":"markdown","source":"## 사용자 type별 분포"},{"metadata":{"_uuid":"c32c8afd55e687b3890b83db3e1821ba2250499e","_cell_guid":"80cf1204-1ec5-453b-9b41-b6033c0a0fc5","trusted":true,"collapsed":true},"cell_type":"code","source":"temp_series = train_df['user_type'].value_counts()\nlabels = (np.array(temp_series.index))\nsizes = (np.array((temp_series / temp_series.sum())*100))\n\ntrace = go.Pie(labels=labels, values=sizes)\nlayout = go.Layout(\n    title='User Type distribution',\n    width=600,\n    height=600,\n)\ndata = [trace]\nfig = go.Figure(data=data, layout=layout)\npy.iplot(fig, filename=\"usertype\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"697a52debf37bd282b8f28b2a4691df23fb15c5e","_cell_guid":"570e2dd2-805f-4a40-a83f-bcfa89ab91fc"},"cell_type":"markdown","source":"## log(price) 히스토그램의 분포"},{"metadata":{"_uuid":"54c0426c1e0be835ca90e6e238e045d527f97742","_cell_guid":"1949f62e-fe77-46e9-a4c9-7aefc35f9dcc","trusted":true,"collapsed":true},"cell_type":"code","source":"train_df[\"price_new\"] = train_df[\"price\"].values\ntrain_df[\"price_new\"].fillna(np.nanmean(train_df[\"price\"].values), inplace=True)\n\nplt.figure(figsize=(12,8))\nsns.distplot(np.log1p(train_df[\"price_new\"].values), bins=100, kde=False)\nplt.xlabel('Log of price', fontsize=12)\nplt.title(\"Log of Price Histogram\", fontsize=14)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"45803085874a19279e31c5717ae03e39957ffa45","_cell_guid":"0a0761fb-f865-4fc8-8baf-3d89714466af"},"cell_type":"markdown","source":"## activation date의 분포"},{"metadata":{"_uuid":"034ddc0be68fa389b3dd16157e861a8f66429593","scrolled":false,"_cell_guid":"3b2d1445-844f-4f26-98b2-9532a53331a6","trusted":true,"collapsed":true},"cell_type":"code","source":"cnt_srs = train_df['activation_date'].value_counts()\n\ntrace = go.Bar(\n    x=cnt_srs.index,\n    y=cnt_srs.values,\n    marker=dict(\n        color=cnt_srs.values,\n        colorscale = 'Picnic',\n        reversescale = True\n    ),\n)\n\nlayout = go.Layout(\n    title='Activation Dates in Train'\n)\n\ndata = [trace]\nfig = go.Figure(data=data, layout=layout)\npy.iplot(fig, filename=\"ActivationDate\")\n\n# Activation dates in test\ncnt_srs = test_df['activation_date'].value_counts()\n\ntrace = go.Bar(\n    x=cnt_srs.index,\n    y=cnt_srs.values,\n    marker=dict(\n        color=cnt_srs.values,\n        colorscale = 'Picnic',\n        reversescale = True\n    ),\n)\n\nlayout = go.Layout(\n    title='Activation Dates in Test'\n)\n\ndata = [trace]\nfig = go.Figure(data=data, layout=layout)\npy.iplot(fig, filename=\"ActivationDate\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"065b2c67be8ca8936c8d03df5e2ae8cca9047663","_cell_guid":"aac3bded-0dce-4784-91fe-e0174088aa61"},"cell_type":"markdown","source":"## 사용자별 분포"},{"metadata":{"_uuid":"742ac1fe969c68e40a8f2e384af75a4c167cba07","_cell_guid":"4aa0e13f-bfc1-431e-89ae-e1b16b0da61e"},"cell_type":"markdown","source":"## 사용자 모수"},{"metadata":{"_uuid":"78e223c8d364b1c7e2bff3b745d0354f226ab0f1","_cell_guid":"ed44b977-c3f9-40e5-bb33-2ad61cd0ba61","trusted":true,"collapsed":true},"cell_type":"code","source":"!pip install matplotlib_venn","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6579a9809deb5e583cc5eb15023a6caaf1b2f9df","scrolled":true,"_cell_guid":"6b3aad30-cdbb-41a0-b968-6983c803dc1c","trusted":true,"collapsed":true},"cell_type":"code","source":"from matplotlib_venn import venn2\n\nplt.figure(figsize=(10,7))\nvenn2([set(train_df.user_id.unique()), set(test_df.user_id.unique())], set_labels = ('Train set', 'Test set') )\nplt.title(\"Number of users in train and test\", fontsize=15)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"bb2de8d77f88d7f19bfaa7bcf83fc75d7b1f167c","_cell_guid":"5471e17e-9bed-4c88-ba28-312d69fd1011"},"cell_type":"markdown","source":"## 타이틀 갯수"},{"metadata":{"_uuid":"cc1090f89f6c85611cea51277e41372dd89f270b","_cell_guid":"3e7f109d-9682-450a-b318-c12540bf7d4b","trusted":true,"collapsed":true},"cell_type":"code","source":"from matplotlib_venn import venn2\n\nplt.figure(figsize=(10,7))\nvenn2([set(train_df.title.unique()), set(test_df.title.unique())], set_labels = ('Train set', 'Test set') )\nplt.title(\"Number of titles in train and test\", fontsize=15)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2f3b17304c592b1f7efb30a02fbdfef439cd074d","_cell_guid":"e19fa5a7-5cf2-41f6-b7c2-f121073f1b30"},"cell_type":"markdown","source":"## 단어 갯수 분포\n\n대부분의 제목은 1~3개의 단어이고 나머지는 long tail이다."},{"metadata":{"_uuid":"a9d3adcae903346fa55aa49d89156f682fbe43b7","_cell_guid":"d2b0f41d-1944-4906-b4f3-63e6d33656a1","trusted":true,"collapsed":true},"cell_type":"code","source":"train_df[\"title_nwords\"] = train_df[\"title\"].apply(lambda x: len(x.split()))\ntest_df[\"title_nwords\"] = test_df[\"title\"].apply(lambda x: len(x.split()))\n\ncnt_srs = train_df['title_nwords'].value_counts()\n\ntrace = go.Bar(\n    x=cnt_srs.index,\n    y=cnt_srs.values,\n    marker=dict(\n        color=\"blue\",\n        #colorscale = 'Blues',\n        reversescale = True\n    ),\n)\n\nlayout = go.Layout(\n    title='Number of words in title column'\n)\n\ndata = [trace]\nfig = go.Figure(data=data, layout=layout)\npy.iplot(fig, filename=\"title_nwords\")           ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"86cfa710d98220cd05a6fc3f0caece6bfc8958b0","_cell_guid":"8d52e820-8c94-47d9-9031-4f82ed18202f"},"cell_type":"markdown","source":"## Deal 확률 SVD 분해"},{"metadata":{"_uuid":"6a55045b0e0c7bf8a22e6061fb175f8ffac3cd83","collapsed":true,"_cell_guid":"3177803f-ffd4-4bf9-b850-dea460b5fa0d","trusted":true},"cell_type":"code","source":"### TFIDF Vectorizer ###\ntfidf_vec = TfidfVectorizer(ngram_range=(1,1))\nfull_tfidf = tfidf_vec.fit_transform(train_df['title'].values.tolist() + test_df['title'].values.tolist())\ntrain_tfidf = tfidf_vec.transform(train_df['title'].values.tolist())\ntest_tfidf = tfidf_vec.transform(test_df['title'].values.tolist())\n\n### SVD Components ###\nn_comp = 3\nsvd_obj = TruncatedSVD(n_components=n_comp, algorithm='arpack')\nsvd_obj.fit(full_tfidf)\ntrain_svd = pd.DataFrame(svd_obj.transform(train_tfidf))\ntest_svd = pd.DataFrame(svd_obj.transform(test_tfidf))\ntrain_svd.columns = ['svd_title_'+str(i+1) for i in range(n_comp)]\ntest_svd.columns = ['svd_title_'+str(i+1) for i in range(n_comp)]\ntrain_df = pd.concat([train_df, train_svd], axis=1)\ntest_df = pd.concat([test_df, test_svd], axis=1)\ndel full_tfidf, train_tfidf, test_tfidf, train_svd, test_svd","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c44c01f2b7767751497b5a4ee8b62b9c38257758","_cell_guid":"eb91c836-04c0-4ab8-a7e2-c13d24190697","trusted":true,"collapsed":true},"cell_type":"code","source":"# 1st svd comp #\nplt.figure(figsize=(8,8))\nsns.jointplot(x=train_df[\"svd_title_1\"].values, y=train_df[\"deal_probability\"].values, size=10)\nplt.ylabel('Deal Probability', fontsize=12)\nplt.xlabel('First SVD component on Title', fontsize=12)\nplt.title(\"Deal Probability distribution for First SVD component on title\", fontsize=15)\nplt.show()\n\n# 2nd svd comp #\nplt.figure(figsize=(8,8))\nsns.jointplot(x=train_df[\"svd_title_2\"].values, y=train_df[\"deal_probability\"].values, size=10)\nplt.ylabel('Deal Probability', fontsize=12)\nplt.xlabel('Second SVD component on Title', fontsize=12)\nplt.title(\"Deal Probability distribution for Second SVD component on title\", fontsize=15)\nplt.show()\n\n# 3rd svd comp #\nplt.figure(figsize=(8,8))\nsns.jointplot(x=train_df[\"svd_title_3\"].values, y=train_df[\"deal_probability\"].values, size=10)\nplt.ylabel('Deal Probability', fontsize=12)\nplt.xlabel('Third SVD component on Title', fontsize=12)\nplt.title(\"Deal Probability distribution for Third SVD component on title\", fontsize=15)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"eac82556129f0b9505e98a0acfda8f71ad5335a4","_cell_guid":"26452269-cfe5-4717-b9c0-60ac319052f7"},"cell_type":"markdown","source":"## 설명의 단어 갯수 분포"},{"metadata":{"_uuid":"db319f3ff7017554e20d40b9353fcf75be8302e4","_cell_guid":"7025cd4b-e8c8-46e8-a2a8-8516e9cd7066","trusted":true,"collapsed":true},"cell_type":"code","source":"## Filling missing values ##\ntrain_df[\"description\"].fillna(\"NA\", inplace=True)\ntest_df[\"description\"].fillna(\"NA\", inplace=True)\n\ntrain_df[\"desc_nwords\"] = train_df[\"description\"].apply(lambda x: len(x.split()))\ntest_df[\"desc_nwords\"] = test_df[\"description\"].apply(lambda x: len(x.split()))\n\ncnt_srs = train_df['desc_nwords'].value_counts().head(100)\n\ntrace = go.Bar(\n    x=cnt_srs.index,\n    y=cnt_srs.values,\n    marker=dict(\n        color=\"blue\",\n        #colorscale = 'Blues',\n        reversescale = True\n    ),\n)\n\nlayout = go.Layout(\n    title='Number of words in Description column'\n)\n\ndata = [trace]\nfig = go.Figure(data=data, layout=layout)\npy.iplot(fig, filename=\"desc_nwords\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"230f229b61ae9054ccad6e288dac139ac8a9210d","collapsed":true,"_cell_guid":"dbafdfdb-d1d4-42e5-b0e1-669e1908f8b5","trusted":true},"cell_type":"code","source":"### TFIDF Vectorizer ###\ntfidf_vec = TfidfVectorizer(ngram_range=(1,1), max_features=100000)\nfull_tfidf = tfidf_vec.fit_transform(train_df['description'].values.tolist() + test_df['description'].values.tolist())\ntrain_tfidf = tfidf_vec.transform(train_df['description'].values.tolist())\ntest_tfidf = tfidf_vec.transform(test_df['description'].values.tolist())\n\n### SVD Components ###\nn_comp = 3\nsvd_obj = TruncatedSVD(n_components=n_comp, algorithm='arpack')\nsvd_obj.fit(full_tfidf)\ntrain_svd = pd.DataFrame(svd_obj.transform(train_tfidf))\ntest_svd = pd.DataFrame(svd_obj.transform(test_tfidf))\ntrain_svd.columns = ['svd_desc_'+str(i+1) for i in range(n_comp)]\ntest_svd.columns = ['svd_desc_'+str(i+1) for i in range(n_comp)]\ntrain_df = pd.concat([train_df, train_svd], axis=1)\ntest_df = pd.concat([test_df, test_svd], axis=1)\ndel full_tfidf, train_tfidf, test_tfidf, train_svd, test_svd","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"df91c3b428f6fd86c4012fcbc182b0c6e7bd6a93","_cell_guid":"85a4bab8-2f00-404b-8fbb-b9d3053ef143","trusted":true,"collapsed":true},"cell_type":"code","source":"# 1st svd comp #\nplt.figure(figsize=(8,8))\nsns.jointplot(x=train_df[\"svd_desc_1\"].values, y=train_df[\"deal_probability\"].values, size=10)\nplt.ylabel('Deal Probability', fontsize=12)\nplt.xlabel('First SVD component on Description', fontsize=12)\nplt.title(\"Deal Probability distribution for First SVD component on Description\", fontsize=15)\nplt.show()\n\n# 2nd svd comp #\nplt.figure(figsize=(8,8))\nsns.jointplot(x=train_df[\"svd_desc_2\"].values, y=train_df[\"deal_probability\"].values, size=10)\nplt.ylabel('Deal Probability', fontsize=12)\nplt.xlabel('Second SVD component on Description', fontsize=12)\nplt.title(\"Deal Probability distribution for Second SVD component on title\", fontsize=15)\nplt.show()\n\n# 3rd svd comp #\nplt.figure(figsize=(8,8))\nsns.jointplot(x=train_df[\"svd_desc_3\"].values, y=train_df[\"deal_probability\"].values, size=10)\nplt.ylabel('Deal Probability', fontsize=12)\nplt.xlabel('Second SVD component on Description', fontsize=12)\nplt.title(\"Deal Probability distribution for Third SVD component on Description\", fontsize=15)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"47bf45afbf1f9245805febec8755ff02963937b2","_cell_guid":"0455a9d1-a22e-48d3-843a-7e40bda48965"},"cell_type":"markdown","source":"## Feature Engineering\n\n- week day feature만들기\n- 카테고리 feature encoding\n- 모델에 필요없는 칼럼 drop"},{"metadata":{"_uuid":"41a41b8074efec8f5d4bd6554ace638a593dc4a0","_cell_guid":"866d0b33-eaf7-4829-8539-7905fc8cac10","trusted":true,"collapsed":true},"cell_type":"code","source":"# Target and ID variables #\ntrain_y = train_df[\"deal_probability\"].values\ntest_id = test_df[\"item_id\"].values\n\n# New variable on weekday #\ntrain_df[\"activation_weekday\"] = train_df[\"activation_date\"].dt.weekday\ntest_df[\"activation_weekday\"] = test_df[\"activation_date\"].dt.weekday\n\n# Label encode the categorical variables #\ncat_vars = [\"region\", \"city\", \"parent_category_name\", \"category_name\", \"user_type\", \"param_1\", \"param_2\", \"param_3\"]\nfor col in cat_vars:\n    lbl = preprocessing.LabelEncoder()\n    lbl.fit(list(train_df[col].values.astype('str')) + list(test_df[col].values.astype('str')))\n    train_df[col] = lbl.transform(list(train_df[col].values.astype('str')))\n    test_df[col] = lbl.transform(list(test_df[col].values.astype('str')))\n\ncols_to_drop = [\"item_id\", \"user_id\", \"title\", \"description\", \"activation_date\", \"image\"]\ntrain_X = train_df.drop(cols_to_drop + [\"region_en\", \"parent_category_name_en\", \"category_name_en\", \"price_new\", \"deal_probability\"], axis=1)\ntest_X = test_df.drop(cols_to_drop, axis=1)\n\ntrain_X.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e9c17b63d62c3bc0e298ae346de7db254b7f68f8","_cell_guid":"afe9abb7-301f-496f-85e0-43339aa9ab16"},"cell_type":"markdown","source":"## Light GBM모델 만들기"},{"metadata":{"_uuid":"850d95c696c015afd5a7e9edc7489105b4445fc7","collapsed":true,"_cell_guid":"fb5600a6-3dc6-42dc-a5c3-9729b3f0f679","trusted":true},"cell_type":"code","source":"def run_lgb(train_X, train_y, val_X, val_y, test_X):\n    params = {\n        \"objective\" : \"regression\",\n        \"metric\" : \"rmse\",\n        \"num_leaves\" : 30,\n        \"learning_rate\" : 0.1,\n        \"bagging_fraction\" : 0.7,\n        \"feature_fraction\" : 0.7,\n        \"bagging_frequency\" : 5,\n        \"bagging_seed\" : 2018,\n        \"verbosity\" : -1\n    }\n    \n    lgtrain = lgb.Dataset(train_X, label=train_y)\n    lgval = lgb.Dataset(val_X, label=val_y)\n    evals_result = {}\n    model = lgb.train(params, lgtrain, 1000, valid_sets=[lgval], early_stopping_rounds=100, verbose_eval=20, evals_result=evals_result)\n    \n    pred_test_y = model.predict(test_X, num_iteration=model.best_iteration)\n    return pred_test_y, model, evals_result","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0f74f0a5b72ff64eb75718427a3db0d387520182","_cell_guid":"e2a89ac6-326e-42da-bc7d-2b2951a9fb1c"},"cell_type":"markdown","source":"## train/val 데이터 나누기"},{"metadata":{"_uuid":"0ef225e0113b203ff75d6484912e9262f6c250eb","_cell_guid":"f37de11a-c02a-4368-a171-0e1789a863d3","trusted":true,"collapsed":true},"cell_type":"code","source":"# Splitting the data for model training#\ndev_X = train_X.iloc[:-200000,:]\nval_X = train_X.iloc[-200000:,:]\ndev_y = train_y[:-200000]\nval_y = train_y[-200000:]\nprint(dev_X.shape, val_X.shape, test_X.shape)\n\n# Training the model #\npred_test, model, evals_result = run_lgb(dev_X, dev_y, val_X, val_y, test_X)\n\n# Making a submission file #\npred_test[pred_test>1] = 1\npred_test[pred_test<0] = 0\nsub_df = pd.DataFrame({\"item_id\":test_id})\nsub_df[\"deal_probability\"] = pred_test\nsub_df.to_csv(\"baseline_lgb.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ab9fa858f2e85435e42f9e7be3d46996434275b3","_cell_guid":"c4c908ba-7b4c-4c79-811b-9dd778ff4e2a"},"cell_type":"markdown","source":"## Feature Importance 뽑기\n\n가장중요한 Feature는 Price이다. 뒤를 이어 image_top_1과 param_1이 중요함을 알 수 있다."},{"metadata":{"_uuid":"8affb8999eeb9329044535b8126c8724256a0bc5","_cell_guid":"bf0856fb-a8c5-4feb-b42a-18647c1b94d2","trusted":true,"collapsed":true},"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(12,18))\nlgb.plot_importance(model, max_num_features=50, height=0.8, ax=ax)\nax.grid(False)\nplt.title(\"LightGBM - Feature Importance\", fontsize=15)\nplt.show()","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.5","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}