{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-07-27T05:49:23.407467Z","iopub.execute_input":"2022-07-27T05:49:23.407903Z","iopub.status.idle":"2022-07-27T05:49:23.435481Z","shell.execute_reply.started":"2022-07-27T05:49:23.407816Z","shell.execute_reply":"2022-07-27T05:49:23.433922Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> In their fourth Kaggle competition, Avito is challenging you to predict demand for an online advertisement based on its full description (title, description, images, etc.), its context (geographically where it was posted, similar ads already posted) and historical demand for similar ads in similar contexts. With this information, Avito can inform sellers on how to best optimize their listing and provide some indication of how much interest they should realistically expect to receive.","metadata":{}},{"cell_type":"markdown","source":"\n\n- item_id - Ad id.\n- user_id - User id.\n- region - Ad region.\n- city - Ad city.\n- parent_category_name - Top level ad category as classified by Avito’s ad model.\n- category_name - Fine grain ad category as classified by Avito’s ad model.\n- param_1 - Optional parameter from Avito’s ad model.\n- param_2 - Optional parameter from Avito’s ad model.\n- param_3 - Optional parameter from Avito’s ad model.\n- title - Ad title.\n- description - Ad description.\n- price - Ad price.\n- item_seq_number - Ad sequential number for user.\n- activation_date- Date ad was placed.\n- user_type - User type.\n- image - Id code of image. Ties to a jpg file in train_jpg. Not every ad has an image.\n- image_top_1 - Avito’s classification code for the image.\n- deal_probability - The target variable. This is the likelihood that an ad actually sold something. It’s not possible to verify every transaction with certainty, so this column’s value can be any float from zero to one.\n","metadata":{}},{"cell_type":"code","source":"df_train = pd.read_csv(\"../input/avito-demand-prediction/train.csv\")\ndf_test = pd.read_csv(\"../input/avito-demand-prediction/test.csv\")\n\ndf_train.shape, df_test.shape","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:49:23.481668Z","iopub.execute_input":"2022-07-27T05:49:23.481978Z","iopub.status.idle":"2022-07-27T05:50:02.786880Z","shell.execute_reply.started":"2022-07-27T05:49:23.481935Z","shell.execute_reply":"2022-07-27T05:50:02.785935Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train.head()","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:50:02.789093Z","iopub.execute_input":"2022-07-27T05:50:02.789488Z","iopub.status.idle":"2022-07-27T05:50:02.820538Z","shell.execute_reply.started":"2022-07-27T05:50:02.789450Z","shell.execute_reply":"2022-07-27T05:50:02.819468Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train.info()","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:50:02.822278Z","iopub.execute_input":"2022-07-27T05:50:02.822638Z","iopub.status.idle":"2022-07-27T05:50:04.758155Z","shell.execute_reply.started":"2022-07-27T05:50:02.822602Z","shell.execute_reply":"2022-07-27T05:50:04.757032Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Изучение nan'ов","metadata":{}},{"cell_type":"code","source":"is_nan = df_train.isna().sum() / len(df_train) * 100\nprint(\"NaN values in train Dataset\")\nprint(is_nan[is_nan > 0].sort_values(ascending=False))","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:50:04.761114Z","iopub.execute_input":"2022-07-27T05:50:04.761507Z","iopub.status.idle":"2022-07-27T05:50:06.665248Z","shell.execute_reply.started":"2022-07-27T05:50:04.761466Z","shell.execute_reply":"2022-07-27T05:50:06.664147Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Изучение уникальных значений в признаках","metadata":{}},{"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport seaborn as sns\n\n\nplt.figure(figsize=(16, 5))\n\ncols = df_train.columns\nuniques = [len(df_train[col].unique()) for col in cols]\n\nax = sns.barplot(x=cols, y=uniques, palette='hls', log=True)\nax.set(xlabel='Feature', ylabel='log(unique count)', title='Number of unique per feature')\n\n\nfor p, uniq in zip(ax.patches, uniques):\n    ax.text(p.get_x() + p.get_width()/2.,\n            uniq + 10,\n            uniq,\n            ha=\"center\") \n\nax.set_xticklabels(ax.get_xticklabels(), rotation=45);","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:50:06.666854Z","iopub.execute_input":"2022-07-27T05:50:06.667156Z","iopub.status.idle":"2022-07-27T05:50:11.887467Z","shell.execute_reply.started":"2022-07-27T05:50:06.667127Z","shell.execute_reply":"2022-07-27T05:50:11.886020Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Изучение целевого признака","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(8, 5))\n\nax = sns.distplot(df_train[\"deal_probability\"].values, bins=50, kde=False)\nax.set_xlabel('Deal Probility', fontsize=15)\nax.set_ylabel('Deal Probility', fontsize=15)\nax.set_title(\"Deal Probability Histogram\", fontsize=20);","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:50:11.890562Z","iopub.execute_input":"2022-07-27T05:50:11.891104Z","iopub.status.idle":"2022-07-27T05:50:12.370831Z","shell.execute_reply.started":"2022-07-27T05:50:11.891060Z","shell.execute_reply":"2022-07-27T05:50:12.369575Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train[\"deal_prob_cat\"] = pd.cut(df_train.deal_probability, bins=10)\ndf_train.head()","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:50:12.375725Z","iopub.execute_input":"2022-07-27T05:50:12.376734Z","iopub.status.idle":"2022-07-27T05:50:12.463648Z","shell.execute_reply.started":"2022-07-27T05:50:12.376688Z","shell.execute_reply":"2022-07-27T05:50:12.462662Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"prob_cat_percent = df_train[\"deal_prob_cat\"].value_counts(normalize=True)\n\nplt.figure(figsize=(12,5))\ng = sns.barplot(x=prob_cat_percent.index, y=prob_cat_percent.values)\ng.set_xlabel('Deal Probability Categorical',fontsize=16)\ng.set_ylabel('% of frequency', fontsize=16)\ng.set_title('Deal Probability Categorical', fontsize=20)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:50:12.465001Z","iopub.execute_input":"2022-07-27T05:50:12.465931Z","iopub.status.idle":"2022-07-27T05:50:12.700832Z","shell.execute_reply.started":"2022-07-27T05:50:12.465890Z","shell.execute_reply":"2022-07-27T05:50:12.699886Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Изучение признаков","metadata":{}},{"cell_type":"markdown","source":"**Регион и город**","metadata":{}},{"cell_type":"code","source":"df_train.groupby(['region', 'city']).count()['user_id']","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:50:12.702347Z","iopub.execute_input":"2022-07-27T05:50:12.702688Z","iopub.status.idle":"2022-07-27T05:50:15.020953Z","shell.execute_reply.started":"2022-07-27T05:50:12.702653Z","shell.execute_reply":"2022-07-27T05:50:15.020020Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"city_ads = df_train.groupby('city').agg(\n    {'deal_probability': ['mean', 'count']}\n).reset_index().sort_values([('deal_probability', 'mean')], ascending=False).reset_index(drop=True)\n\ncity_ads","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:50:15.025293Z","iopub.execute_input":"2022-07-27T05:50:15.025569Z","iopub.status.idle":"2022-07-27T05:50:15.197532Z","shell.execute_reply.started":"2022-07-27T05:50:15.025544Z","shell.execute_reply":"2022-07-27T05:50:15.196390Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(f\"There are {len(df_train.city.unique())} cities in total.\")\nprint(f\"There are {city_ads[city_ads['deal_probability']['count'] > 100].shape[0]} cities with more that 100 ads.\")\nprint(f\"There are {city_ads[city_ads['deal_probability']['count'] > 1000].shape[0]} cities with more that 1000 ads.\")\nprint(f\"There are {city_ads[city_ads['deal_probability']['count'] > 10000].shape[0]} cities with more that 10000 ads.\")","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:50:15.199501Z","iopub.execute_input":"2022-07-27T05:50:15.199901Z","iopub.status.idle":"2022-07-27T05:50:15.339169Z","shell.execute_reply.started":"2022-07-27T05:50:15.199865Z","shell.execute_reply":"2022-07-27T05:50:15.338028Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Регион**","metadata":{}},{"cell_type":"code","source":"df_train['region'].value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:50:15.340696Z","iopub.execute_input":"2022-07-27T05:50:15.341321Z","iopub.status.idle":"2022-07-27T05:50:15.554875Z","shell.execute_reply.started":"2022-07-27T05:50:15.341283Z","shell.execute_reply":"2022-07-27T05:50:15.553877Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data = df_train['region'].value_counts().iloc[:10]\ndata","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:50:15.556287Z","iopub.execute_input":"2022-07-27T05:50:15.556853Z","iopub.status.idle":"2022-07-27T05:50:15.772279Z","shell.execute_reply.started":"2022-07-27T05:50:15.556813Z","shell.execute_reply":"2022-07-27T05:50:15.771110Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(16,5))\n\nax = sns.barplot(x=data.index, y=data.values)\nax.set_xlabel('Most popular Region', fontsize=16)\nax.set_ylabel('Count', fontsize=16)\nax.set_title('Region Count', fontsize=20)\nplt.xticks(rotation=40);","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:50:15.773766Z","iopub.execute_input":"2022-07-27T05:50:15.774260Z","iopub.status.idle":"2022-07-27T05:50:16.021954Z","shell.execute_reply.started":"2022-07-27T05:50:15.774221Z","shell.execute_reply":"2022-07-27T05:50:16.020898Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Город**","metadata":{}},{"cell_type":"code","source":"df_train['city'].value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:50:16.023393Z","iopub.execute_input":"2022-07-27T05:50:16.024499Z","iopub.status.idle":"2022-07-27T05:50:16.243037Z","shell.execute_reply.started":"2022-07-27T05:50:16.024459Z","shell.execute_reply":"2022-07-27T05:50:16.241840Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data = df_train['city'].value_counts().iloc[:10]\ndata","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:50:16.244623Z","iopub.execute_input":"2022-07-27T05:50:16.245040Z","iopub.status.idle":"2022-07-27T05:50:16.463935Z","shell.execute_reply.started":"2022-07-27T05:50:16.245000Z","shell.execute_reply":"2022-07-27T05:50:16.462689Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(16,5))\n\nax = sns.barplot(x=data.index, y=data.values)\nax.set_xlabel('Most popular City', fontsize=16)\nax.set_ylabel('Count', fontsize=16)\nax.set_title('City Count', fontsize=20)\nplt.xticks(rotation=40);","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:50:16.465986Z","iopub.execute_input":"2022-07-27T05:50:16.466409Z","iopub.status.idle":"2022-07-27T05:50:16.699246Z","shell.execute_reply.started":"2022-07-27T05:50:16.466370Z","shell.execute_reply":"2022-07-27T05:50:16.698283Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Категория**","metadata":{}},{"cell_type":"code","source":"df_train.groupby(['parent_category_name', 'category_name']).count()['user_id']","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:50:16.700593Z","iopub.execute_input":"2022-07-27T05:50:16.703085Z","iopub.status.idle":"2022-07-27T05:50:19.001818Z","shell.execute_reply.started":"2022-07-27T05:50:16.703043Z","shell.execute_reply":"2022-07-27T05:50:19.000838Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data = df_train['parent_category_name'].value_counts().iloc[:10]\ndata","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:50:19.003468Z","iopub.execute_input":"2022-07-27T05:50:19.004130Z","iopub.status.idle":"2022-07-27T05:50:19.211454Z","shell.execute_reply.started":"2022-07-27T05:50:19.004091Z","shell.execute_reply":"2022-07-27T05:50:19.210377Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(16,5))\n\nax = sns.barplot(x=data.index, y=data.values)\nax.set_xlabel('Most popular parent_category_name', fontsize=16)\nax.set_ylabel('Count', fontsize=16)\nax.set_title('parent_category_name Count', fontsize=20)\nplt.xticks(rotation=40);","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:50:19.213100Z","iopub.execute_input":"2022-07-27T05:50:19.213612Z","iopub.status.idle":"2022-07-27T05:50:19.452123Z","shell.execute_reply.started":"2022-07-27T05:50:19.213573Z","shell.execute_reply":"2022-07-27T05:50:19.450893Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data = df_train['category_name'].value_counts().iloc[:10]\ndata","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:50:19.453831Z","iopub.execute_input":"2022-07-27T05:50:19.454187Z","iopub.status.idle":"2022-07-27T05:50:19.662707Z","shell.execute_reply.started":"2022-07-27T05:50:19.454151Z","shell.execute_reply":"2022-07-27T05:50:19.661653Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(16,5))\n\nax = sns.barplot(x=data.index, y=data.values)\nax.set_xlabel('Most popular category_name', fontsize=16)\nax.set_ylabel('Count', fontsize=16)\nax.set_title('category_name Count', fontsize=20)\nplt.xticks(rotation=40);","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:50:19.664103Z","iopub.execute_input":"2022-07-27T05:50:19.664836Z","iopub.status.idle":"2022-07-27T05:50:19.903846Z","shell.execute_reply.started":"2022-07-27T05:50:19.664792Z","shell.execute_reply":"2022-07-27T05:50:19.902903Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**param**","metadata":{}},{"cell_type":"code","source":"df_train['param_1'].value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:50:19.906065Z","iopub.execute_input":"2022-07-27T05:50:19.906664Z","iopub.status.idle":"2022-07-27T05:50:20.113628Z","shell.execute_reply.started":"2022-07-27T05:50:19.906624Z","shell.execute_reply":"2022-07-27T05:50:20.112660Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data = df_train['param_1'].value_counts().iloc[:10]\ndata","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:50:20.114904Z","iopub.execute_input":"2022-07-27T05:50:20.115695Z","iopub.status.idle":"2022-07-27T05:50:20.322919Z","shell.execute_reply.started":"2022-07-27T05:50:20.115653Z","shell.execute_reply":"2022-07-27T05:50:20.321843Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(16,5))\n\nax = sns.barplot(x=data.index, y=data.values)\nax.set_xlabel('Most popular param_1', fontsize=16)\nax.set_ylabel('Count', fontsize=16)\nax.set_title('param_1 Count', fontsize=20)\nplt.xticks(rotation=40);","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:50:20.324577Z","iopub.execute_input":"2022-07-27T05:50:20.324938Z","iopub.status.idle":"2022-07-27T05:50:20.549813Z","shell.execute_reply.started":"2022-07-27T05:50:20.324902Z","shell.execute_reply":"2022-07-27T05:50:20.548858Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data = df_train['param_2'].value_counts().iloc[:10]\ndata","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:50:20.551289Z","iopub.execute_input":"2022-07-27T05:50:20.552271Z","iopub.status.idle":"2022-07-27T05:50:20.698421Z","shell.execute_reply.started":"2022-07-27T05:50:20.552230Z","shell.execute_reply":"2022-07-27T05:50:20.697389Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(16,5))\n\nax = sns.barplot(x=data.index, y=data.values)\nax.set_xlabel('Most popular param_2', fontsize=16)\nax.set_ylabel('Count', fontsize=16)\nax.set_title('param_2 Count', fontsize=20)\nplt.xticks(rotation=40);","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:50:20.699723Z","iopub.execute_input":"2022-07-27T05:50:20.705745Z","iopub.status.idle":"2022-07-27T05:50:21.065921Z","shell.execute_reply.started":"2022-07-27T05:50:20.704960Z","shell.execute_reply":"2022-07-27T05:50:21.064941Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data = df_train['param_3'].value_counts().iloc[:10]\ndata","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:50:21.073704Z","iopub.execute_input":"2022-07-27T05:50:21.075213Z","iopub.status.idle":"2022-07-27T05:50:21.202137Z","shell.execute_reply.started":"2022-07-27T05:50:21.075170Z","shell.execute_reply":"2022-07-27T05:50:21.201051Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(16,5))\n\nax = sns.barplot(x=data.index, y=data.values)\nax.set_xlabel('Most popular param_3', fontsize=16)\nax.set_ylabel('Count', fontsize=16)\nax.set_title('param_3 Count', fontsize=20)\nplt.xticks(rotation=40);","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:50:21.204627Z","iopub.execute_input":"2022-07-27T05:50:21.204911Z","iopub.status.idle":"2022-07-27T05:50:21.434930Z","shell.execute_reply.started":"2022-07-27T05:50:21.204886Z","shell.execute_reply":"2022-07-27T05:50:21.433927Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**param1 + param2 + param3**","metadata":{}},{"cell_type":"code","source":"df_train['params'] = df_train['param_1'].fillna('') + ' ' + df_train['param_2'].fillna('') + ' ' + df_train['param_3'].fillna('')\ndf_train['params'] = df_train['params'].str.strip()\ndf_train['params'].value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:50:21.436386Z","iopub.execute_input":"2022-07-27T05:50:21.438230Z","iopub.status.idle":"2022-07-27T05:50:24.038705Z","shell.execute_reply.started":"2022-07-27T05:50:21.438188Z","shell.execute_reply":"2022-07-27T05:50:24.037716Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from nltk.util import ngrams\nfrom collections import Counter\n\n\ntext = ' '.join(df_train['params'].values)\ntext = [i for i in ngrams(text.lower().split(), 3)]\nprint('Common trigrams.')\nCounter(text).most_common(40)","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:50:24.040228Z","iopub.execute_input":"2022-07-27T05:50:24.040818Z","iopub.status.idle":"2022-07-27T05:50:30.467518Z","shell.execute_reply.started":"2022-07-27T05:50:24.040781Z","shell.execute_reply":"2022-07-27T05:50:30.466462Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del text","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:50:30.469320Z","iopub.execute_input":"2022-07-27T05:50:30.470025Z","iopub.status.idle":"2022-07-27T05:50:30.771130Z","shell.execute_reply.started":"2022-07-27T05:50:30.469986Z","shell.execute_reply":"2022-07-27T05:50:30.769874Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**title**","metadata":{}},{"cell_type":"code","source":"df_train['title'].value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:50:30.772682Z","iopub.execute_input":"2022-07-27T05:50:30.773085Z","iopub.status.idle":"2022-07-27T05:50:31.827701Z","shell.execute_reply.started":"2022-07-27T05:50:30.773047Z","shell.execute_reply":"2022-07-27T05:50:31.826559Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data = df_train['title'].value_counts().iloc[:10]\ndata","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:50:31.830299Z","iopub.execute_input":"2022-07-27T05:50:31.831033Z","iopub.status.idle":"2022-07-27T05:50:32.803919Z","shell.execute_reply.started":"2022-07-27T05:50:31.830991Z","shell.execute_reply":"2022-07-27T05:50:32.802932Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(16,5))\n\nax = sns.barplot(x=data.index, y=data.values)\nax.set_xlabel('Most popular title', fontsize=16)\nax.set_ylabel('Count', fontsize=16)\nax.set_title('title Count', fontsize=20)\nplt.xticks(rotation=40);","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:50:32.805833Z","iopub.execute_input":"2022-07-27T05:50:32.806413Z","iopub.status.idle":"2022-07-27T05:50:33.074170Z","shell.execute_reply.started":"2022-07-27T05:50:32.806344Z","shell.execute_reply":"2022-07-27T05:50:33.072952Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**user_type**","metadata":{}},{"cell_type":"code","source":"pie_data = df_train['user_type'].value_counts()\npie_data","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:50:33.078684Z","iopub.execute_input":"2022-07-27T05:50:33.081607Z","iopub.status.idle":"2022-07-27T05:50:33.386530Z","shell.execute_reply.started":"2022-07-27T05:50:33.081565Z","shell.execute_reply":"2022-07-27T05:50:33.383035Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.set_style(\"darkgrid\")\nf, ax = plt.subplots(figsize=(10,5))\n\ncolors = sns.color_palette('pastel')\nax.pie(pie_data.values, labels=pie_data.index, colors=colors)\nax.set_facecolor('lightgrey')\nax.set_xlabel('Distribution of user_type')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:50:33.390947Z","iopub.execute_input":"2022-07-27T05:50:33.394077Z","iopub.status.idle":"2022-07-27T05:50:33.546264Z","shell.execute_reply.started":"2022-07-27T05:50:33.394023Z","shell.execute_reply":"2022-07-27T05:50:33.545247Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**item_seq_number**","metadata":{}},{"cell_type":"code","source":"df_train['item_seq_number'].value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:50:33.550979Z","iopub.execute_input":"2022-07-27T05:50:33.558141Z","iopub.status.idle":"2022-07-27T05:50:33.615639Z","shell.execute_reply.started":"2022-07-27T05:50:33.558091Z","shell.execute_reply":"2022-07-27T05:50:33.614686Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train['item_seq_number'].hist();","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:50:33.620994Z","iopub.execute_input":"2022-07-27T05:50:33.625278Z","iopub.status.idle":"2022-07-27T05:50:34.221627Z","shell.execute_reply.started":"2022-07-27T05:50:33.625200Z","shell.execute_reply":"2022-07-27T05:50:34.220681Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.scatter(df_train.item_seq_number, df_train.deal_probability, label='item_seq_number vs deal_probability');\nplt.xlabel('item_seq_number');\nplt.ylabel('deal_probability');","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:50:34.226292Z","iopub.execute_input":"2022-07-27T05:50:34.228797Z","iopub.status.idle":"2022-07-27T05:50:36.868398Z","shell.execute_reply.started":"2022-07-27T05:50:34.228749Z","shell.execute_reply":"2022-07-27T05:50:36.867262Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**price**","metadata":{}},{"cell_type":"code","source":"pd.set_option('display.float_format', lambda x: f'{x:.3f}')","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:50:36.869999Z","iopub.execute_input":"2022-07-27T05:50:36.870380Z","iopub.status.idle":"2022-07-27T05:50:36.876005Z","shell.execute_reply.started":"2022-07-27T05:50:36.870341Z","shell.execute_reply":"2022-07-27T05:50:36.875019Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train['price'].describe()","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:50:36.877531Z","iopub.execute_input":"2022-07-27T05:50:36.878156Z","iopub.status.idle":"2022-07-27T05:50:36.964733Z","shell.execute_reply.started":"2022-07-27T05:50:36.878117Z","shell.execute_reply":"2022-07-27T05:50:36.963582Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train[df_train['price'] > 100_000_000]","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:50:36.966500Z","iopub.execute_input":"2022-07-27T05:50:36.966883Z","iopub.status.idle":"2022-07-27T05:50:37.759627Z","shell.execute_reply.started":"2022-07-27T05:50:36.966847Z","shell.execute_reply":"2022-07-27T05:50:37.758576Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(12,5))\n\ng = sns.histplot(df_train['price'].dropna(), bins=50)\ng.set_xlabel('Price', fontsize=15)\ng.set_ylabel('Probility', fontsize=15)\ng.set_title(\"Price Histogram\", fontsize=20);","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:50:37.761242Z","iopub.execute_input":"2022-07-27T05:50:37.761817Z","iopub.status.idle":"2022-07-27T05:50:39.061861Z","shell.execute_reply.started":"2022-07-27T05:50:37.761767Z","shell.execute_reply":"2022-07-27T05:50:39.060891Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train['price_log'] = np.log(df_train['price'] + 1)\n\nplt.figure(figsize=(12,5))\n\ng = sns.histplot(df_train['price_log'].dropna(), bins=50)\ng.set_xlabel('Price Log', fontsize=15)\ng.set_ylabel('Probility', fontsize=15)\ng.set_title(\"Price Histogram\", fontsize=20);","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:50:39.063548Z","iopub.execute_input":"2022-07-27T05:50:39.063933Z","iopub.status.idle":"2022-07-27T05:50:40.356818Z","shell.execute_reply.started":"2022-07-27T05:50:39.063894Z","shell.execute_reply":"2022-07-27T05:50:40.355847Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"np.exp(5), np.exp(10)","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:50:40.358449Z","iopub.execute_input":"2022-07-27T05:50:40.359110Z","iopub.status.idle":"2022-07-27T05:50:40.366042Z","shell.execute_reply.started":"2022-07-27T05:50:40.359071Z","shell.execute_reply":"2022-07-27T05:50:40.364824Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Взаимное распределение","metadata":{}},{"cell_type":"markdown","source":"**Price**","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(12,14))\n\nplt.subplot(3,1,1)\ng = sns.countplot(x='parent_category_name', data=df_train)\ng.set_xlabel('User Type', fontsize=16)\ng.set_ylabel('Count', fontsize=16)\ng.set_title('Category of Ad', fontsize=20)\ng.set_xticklabels(g.get_xticklabels(), rotation=45)\n\nplt.subplot(3,1,2)\ng1 = sns.boxplot(x='parent_category_name',y='deal_probability', data=df_train)\ng1.set_xlabel(\"Category's Name\", fontsize=16)\ng1.set_ylabel('Deal Probability', fontsize=16)\ng1.set_title('Category of Ad', fontsize=20)\ng1.set_xticklabels(g.get_xticklabels(), rotation=45)\n\nplt.subplot(3,1,3)\ng2 = sns.boxplot(x='parent_category_name', y='price_log', data=df_train)\ng2.set_xlabel(\"Category's Name\", fontsize=16)\ng2.set_ylabel('Price Log', fontsize=16)\ng2.set_title('Category of Ad', fontsize=20)\ng2.set_xticklabels(g1.get_xticklabels(), rotation=45)\n\nplt.subplots_adjust(hspace=0.7, top=1.2)","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:50:40.367814Z","iopub.execute_input":"2022-07-27T05:50:40.368593Z","iopub.status.idle":"2022-07-27T05:50:45.592637Z","shell.execute_reply.started":"2022-07-27T05:50:40.368551Z","shell.execute_reply":"2022-07-27T05:50:45.591688Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cm = sns.light_palette(\"green\", as_cmap=True)\npd.crosstab(df_train['parent_category_name'],\n            df_train['deal_prob_cat']).style.background_gradient(cmap=cm)","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:50:45.593741Z","iopub.execute_input":"2022-07-27T05:50:45.594120Z","iopub.status.idle":"2022-07-27T05:50:46.014750Z","shell.execute_reply.started":"2022-07-27T05:50:45.594082Z","shell.execute_reply":"2022-07-27T05:50:46.013658Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cm = sns.light_palette(\"green\", as_cmap=True)\npd.crosstab(df_train['parent_category_name'],\n            df_train['deal_prob_cat'],\n            normalize='index').style.background_gradient(cmap=cm)","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:50:46.017125Z","iopub.execute_input":"2022-07-27T05:50:46.017773Z","iopub.status.idle":"2022-07-27T05:50:46.381904Z","shell.execute_reply.started":"2022-07-27T05:50:46.017721Z","shell.execute_reply":"2022-07-27T05:50:46.380908Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train['price_log'] = np.log(df_train['price'] + 1)\n\nplt.figure(figsize=(12,5))\n\ng = sns.boxplot(x='deal_prob_cat', y='price_log', data=df_train)\ng.set_xlabel('The Deal Probability Categorical Dist', fontsize=16)\ng.set_ylabel('Price Log Dist', fontsize=16)\ng.set_title('Looking the Price Log of each deal_prob_cat', fontsize=20);","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:50:46.383493Z","iopub.execute_input":"2022-07-27T05:50:46.384514Z","iopub.status.idle":"2022-07-27T05:50:46.990977Z","shell.execute_reply.started":"2022-07-27T05:50:46.384474Z","shell.execute_reply":"2022-07-27T05:50:46.989916Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Regions**","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(16,20))\nplt.subplot(3,1,1)\ng = sns.countplot(x='region', data=df_train)\ng.set_xlabel('Ad Regions', fontsize=16)\ng.set_ylabel('Count', fontsize=16)\ng.set_title('Ad Regions Count', fontsize=20)\ng.set_xticklabels(g.get_xticklabels(), rotation=70)\n\nplt.subplot(3,1,2)\ng1 = sns.boxplot(x='region', y='deal_probability',data=df_train)\ng1.set_xlabel('Ad Regions', fontsize=16)\ng1.set_ylabel('Deal Probability', fontsize=16)\ng1.set_title('Ad Regions Deal Prob Distribuition', fontsize=20)\ng1.set_xticklabels(g1.get_xticklabels(), rotation=70)\n\nplt.subplot(3,1,3)\ng2 = sns.boxplot(x='region', y='price_log',data=df_train)\ng2.set_xlabel('Ad Regions', fontsize=16)\ng2.set_ylabel('Price Log Distribuition', fontsize=16)\ng2.set_title('Ad Regions Price Distribuition', fontsize=20)\ng2.set_xticklabels(g2.get_xticklabels(), rotation=70)\n\nplt.subplots_adjust(hspace=0.7, top=1.2);","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:50:46.992596Z","iopub.execute_input":"2022-07-27T05:50:46.992998Z","iopub.status.idle":"2022-07-27T05:50:53.165578Z","shell.execute_reply.started":"2022-07-27T05:50:46.992945Z","shell.execute_reply":"2022-07-27T05:50:53.164032Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cm = sns.light_palette(\"green\", as_cmap=True)\npd.crosstab(df_train['region'],\n            df_train['deal_prob_cat']).style.background_gradient(cmap=cm)","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:50:53.167080Z","iopub.execute_input":"2022-07-27T05:50:53.168315Z","iopub.status.idle":"2022-07-27T05:50:53.589151Z","shell.execute_reply.started":"2022-07-27T05:50:53.168262Z","shell.execute_reply":"2022-07-27T05:50:53.588030Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cm = sns.light_palette(\"green\", as_cmap=True)\npd.crosstab(df_train['region'],\n            df_train['deal_prob_cat'],\n            normalize='index').style.background_gradient(cmap=cm)","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:50:53.590726Z","iopub.execute_input":"2022-07-27T05:50:53.591641Z","iopub.status.idle":"2022-07-27T05:50:54.014286Z","shell.execute_reply.started":"2022-07-27T05:50:53.591599Z","shell.execute_reply":"2022-07-27T05:50:54.013135Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**param**","metadata":{}},{"cell_type":"code","source":"params = df_train.param_1.value_counts().head(20)","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:50:54.016013Z","iopub.execute_input":"2022-07-27T05:50:54.016408Z","iopub.status.idle":"2022-07-27T05:50:54.229351Z","shell.execute_reply.started":"2022-07-27T05:50:54.016370Z","shell.execute_reply":"2022-07-27T05:50:54.228153Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(16,5))\n\ng = sns.barplot(x=params.index, y=params.values)\ng.set_xlabel(\"Params\", fontsize=15)\ng.set_ylabel(\"Count\", fontsize=15)\ng.set_title(\"Most Frequent params in Ads\", fontsize=20)\ng.set_xticklabels(g.get_xticklabels(), rotation=70)\n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:50:54.232380Z","iopub.execute_input":"2022-07-27T05:50:54.232724Z","iopub.status.idle":"2022-07-27T05:50:54.638862Z","shell.execute_reply.started":"2022-07-27T05:50:54.232668Z","shell.execute_reply":"2022-07-27T05:50:54.637883Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"The top five Ad params in %\")\nprint(round((params / len(df_train) * 100).head(n=5),2))","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:50:54.640204Z","iopub.execute_input":"2022-07-27T05:50:54.641132Z","iopub.status.idle":"2022-07-27T05:50:54.656356Z","shell.execute_reply.started":"2022-07-27T05:50:54.641090Z","shell.execute_reply":"2022-07-27T05:50:54.655099Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"subset_param = df_train[df_train.param_1.isin(params.index)]\nsubset_param","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:50:54.661525Z","iopub.execute_input":"2022-07-27T05:50:54.663824Z","iopub.status.idle":"2022-07-27T05:50:56.351948Z","shell.execute_reply.started":"2022-07-27T05:50:54.663786Z","shell.execute_reply":"2022-07-27T05:50:56.350956Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(16,12))\nplt.subplot(2,1,1)\ng = sns.boxplot(x='param_1', y='price_log', data=subset_param)\ng.set_xlabel(\"\", fontsize=15)\ng.set_ylabel(\"Price Dist(log)\", fontsize=15)\ng.set_title(\"Price of TOP 20 params_1\", fontsize=20)\ng.set_xticklabels(g.get_xticklabels(),rotation=90)\n\nplt.subplot(2,1,2)\ng1 = sns.boxplot(x='param_1', y='deal_probability', data=subset_param)\ng1.set_xlabel(\"TOP 20 Params\", fontsize=15)\ng1.set_ylabel(\"Deal Probability\", fontsize=15)\ng1.set_title(\"Deal Probability of TOP 20 params_1\", fontsize=20)\ng1.set_xticklabels(g.get_xticklabels(), rotation=90)\n\nplt.subplots_adjust(hspace=0.9, top=1.2);","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:50:56.353659Z","iopub.execute_input":"2022-07-27T05:50:56.354044Z","iopub.status.idle":"2022-07-27T05:50:59.158803Z","shell.execute_reply.started":"2022-07-27T05:50:56.354006Z","shell.execute_reply":"2022-07-27T05:50:59.157605Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**user_type**","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(15, 8))\nsns.boxplot(x=\"parent_category_name\", y=\"price_log\", hue=\"user_type\",  data=df_train)\nplt.title(\"Price by parent category and user type\")\nplt.xticks(rotation=70);","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:50:59.160567Z","iopub.execute_input":"2022-07-27T05:50:59.160933Z","iopub.status.idle":"2022-07-27T05:51:02.230786Z","shell.execute_reply.started":"2022-07-27T05:50:59.160899Z","shell.execute_reply":"2022-07-27T05:51:02.229916Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(15, 8))\nsns.boxplot(x=\"parent_category_name\", y=\"deal_probability\", hue=\"user_type\",  data=df_train)\nplt.title(\"Price by parent category and user type\")\nplt.xticks(rotation=70);","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:51:02.232718Z","iopub.execute_input":"2022-07-27T05:51:02.233119Z","iopub.status.idle":"2022-07-27T05:51:05.628508Z","shell.execute_reply.started":"2022-07-27T05:51:02.233080Z","shell.execute_reply":"2022-07-27T05:51:05.627417Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**title**","metadata":{}},{"cell_type":"code","source":"title_freq = df_train.title.value_counts()[:15]\n\nplt.figure(figsize=(16,12))\n\nplt.subplot(2,1,1)\ng = sns.boxplot(x='title', y='price_log', \n                data=df_train[df_train.title.isin(title_freq.index.values)])\ng.set_xlabel(\"\", fontsize=15)\ng.set_ylabel(\"Price Log\", fontsize=15)\ng.set_title(\"TOP 35 titles by Price_log\", fontsize=20)\ng.set_xticklabels(g.get_xticklabels(), rotation=70)\n\nplt.subplot(2,1,2)\ng1 = sns.boxplot(x='title', y='deal_probability', \n                data=df_train[df_train.title.isin(title_freq.index.values)])\ng1.set_xlabel(\"TOP 15 Titles\", fontsize=15)\ng1.set_ylabel(\"Deal Probability\", fontsize=15)\ng1.set_title(\"TOP 15 titles by Deal Probability\", fontsize=20)\ng1.set_xticklabels(g1.get_xticklabels(), rotation=70)\n\nplt.subplots_adjust(hspace=0.7, top=1.2)\n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:51:05.629873Z","iopub.execute_input":"2022-07-27T05:51:05.630842Z","iopub.status.idle":"2022-07-27T05:51:08.474885Z","shell.execute_reply.started":"2022-07-27T05:51:05.630796Z","shell.execute_reply":"2022-07-27T05:51:08.473896Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cm = sns.light_palette(\"green\", as_cmap=True)\npd.crosstab(df_train[df_train.title.isin(title_freq.index.values)]['title'], \n            df_train[df_train.title.isin(title_freq.index.values)]['deal_prob_cat'],\n            normalize='index').style.background_gradient(cmap=cm)","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:51:08.476602Z","iopub.execute_input":"2022-07-27T05:51:08.477253Z","iopub.status.idle":"2022-07-27T05:51:08.892709Z","shell.execute_reply.started":"2022-07-27T05:51:08.477210Z","shell.execute_reply":"2022-07-27T05:51:08.891704Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from nltk.corpus import stopwords","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:51:08.894084Z","iopub.execute_input":"2022-07-27T05:51:08.895060Z","iopub.status.idle":"2022-07-27T05:51:08.900476Z","shell.execute_reply.started":"2022-07-27T05:51:08.895018Z","shell.execute_reply":"2022-07-27T05:51:08.899418Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"stopWords = set(stopwords.words('russian'))","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:51:08.902384Z","iopub.execute_input":"2022-07-27T05:51:08.902762Z","iopub.status.idle":"2022-07-27T05:51:08.916079Z","shell.execute_reply.started":"2022-07-27T05:51:08.902723Z","shell.execute_reply":"2022-07-27T05:51:08.915135Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Wall time: 43.4 s\n","metadata":{}},{"cell_type":"code","source":"%%time\nfrom wordcloud import WordCloud\n\nwordcloud = WordCloud(background_color='white',\n                      stopwords=stopWords,\n                      max_words=500,\n                      max_font_size=200,\n                      width=1000, height=800,\n                      random_state=42,\n                     ).generate(\" \".join(df_train['title'].astype(str)))\n\nfig = plt.figure(figsize=(12,14))\nplt.imshow(wordcloud)\nplt.title(\"WORD CLOUD - TITLE\", fontsize=25)\nplt.axis('off');","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:51:08.929538Z","iopub.execute_input":"2022-07-27T05:51:08.930710Z","iopub.status.idle":"2022-07-27T05:51:45.757600Z","shell.execute_reply.started":"2022-07-27T05:51:08.930669Z","shell.execute_reply":"2022-07-27T05:51:45.756715Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del wordcloud","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:51:45.759218Z","iopub.execute_input":"2022-07-27T05:51:45.760173Z","iopub.status.idle":"2022-07-27T05:51:45.765934Z","shell.execute_reply.started":"2022-07-27T05:51:45.760123Z","shell.execute_reply":"2022-07-27T05:51:45.765096Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**description**","metadata":{}},{"cell_type":"code","source":"df_train['description'].iloc[2]","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:51:45.767134Z","iopub.execute_input":"2022-07-27T05:51:45.767939Z","iopub.status.idle":"2022-07-27T05:51:45.777237Z","shell.execute_reply.started":"2022-07-27T05:51:45.767903Z","shell.execute_reply":"2022-07-27T05:51:45.776133Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train['description'] = df_train['description'].apply(lambda x: str(x).replace('/\\n', ' '))","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:51:45.778618Z","iopub.execute_input":"2022-07-27T05:51:45.779593Z","iopub.status.idle":"2022-07-27T05:51:47.152492Z","shell.execute_reply.started":"2022-07-27T05:51:45.779553Z","shell.execute_reply":"2022-07-27T05:51:47.151354Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Wall time: 55.7 s\n","metadata":{}},{"cell_type":"code","source":"%%time\nfrom nltk.util import ngrams\nfrom collections import Counter\n\ntext = ' '.join(df_train['description'].values)\ntext = [i for i in ngrams(text.lower().split(), 3)]\nprint('Common trigrams.')\nCounter(text).most_common(40)","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:52:27.872020Z","iopub.execute_input":"2022-07-27T05:52:27.873152Z","iopub.status.idle":"2022-07-27T05:53:23.548092Z","shell.execute_reply.started":"2022-07-27T05:52:27.873116Z","shell.execute_reply":"2022-07-27T05:53:23.547012Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del text","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:53:23.550902Z","iopub.execute_input":"2022-07-27T05:53:23.551231Z","iopub.status.idle":"2022-07-27T05:53:25.504725Z","shell.execute_reply.started":"2022-07-27T05:53:23.551203Z","shell.execute_reply":"2022-07-27T05:53:25.503635Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Модель","metadata":{}},{"cell_type":"code","source":"df_train = pd.read_csv(\"../input/avito-demand-prediction/train.csv\")\ndf_test = pd.read_csv(\"../input/avito-demand-prediction/test.csv\")\n\ndf_train.shape, df_test.shape","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from nltk.corpus import stopwords\n\nstopWords = set(stopwords.words('russian'))","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:54:07.612730Z","iopub.execute_input":"2022-07-27T05:54:07.613521Z","iopub.status.idle":"2022-07-27T05:54:07.620555Z","shell.execute_reply.started":"2022-07-27T05:54:07.613479Z","shell.execute_reply":"2022-07-27T05:54:07.618390Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<img src='https://drive.google.com/uc?export=view&id=14nvoZ71VqMfYBY6ff_7_Btx-tVwL_4lB' width=800>","metadata":{}},{"cell_type":"code","source":"%%time\nfrom sklearn.feature_extraction.text import TfidfVectorizer\nvectorizer = TfidfVectorizer(stop_words=stopWords, max_features=2000)\nvectorizer.fit(df_train['title'])\n\ntrain_title = vectorizer.transform(df_train['title'])\ntest_title = vectorizer.transform(df_test['title'])","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:54:37.506976Z","iopub.execute_input":"2022-07-27T05:54:37.507342Z","iopub.status.idle":"2022-07-27T05:55:07.075489Z","shell.execute_reply.started":"2022-07-27T05:54:37.507311Z","shell.execute_reply":"2022-07-27T05:55:07.073581Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_title","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:55:07.077729Z","iopub.execute_input":"2022-07-27T05:55:07.079249Z","iopub.status.idle":"2022-07-27T05:55:07.086357Z","shell.execute_reply.started":"2022-07-27T05:55:07.079204Z","shell.execute_reply":"2022-07-27T05:55:07.085261Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_title_df = pd.DataFrame.sparse.from_spmatrix(train_title, columns=vectorizer.get_feature_names_out())\ntrain_title_df","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:55:07.088167Z","iopub.execute_input":"2022-07-27T05:55:07.088555Z","iopub.status.idle":"2022-07-27T05:55:07.274732Z","shell.execute_reply.started":"2022-07-27T05:55:07.088517Z","shell.execute_reply":"2022-07-27T05:55:07.273747Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_title_df = pd.DataFrame.sparse.from_spmatrix(test_title, columns=vectorizer.get_feature_names_out())\ntest_title_df","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:55:07.277568Z","iopub.execute_input":"2022-07-27T05:55:07.278241Z","iopub.status.idle":"2022-07-27T05:55:07.393926Z","shell.execute_reply.started":"2022-07-27T05:55:07.278198Z","shell.execute_reply":"2022-07-27T05:55:07.392991Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del train_title, test_title","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:55:07.395245Z","iopub.execute_input":"2022-07-27T05:55:07.396154Z","iopub.status.idle":"2022-07-27T05:55:07.401486Z","shell.execute_reply.started":"2022-07-27T05:55:07.396112Z","shell.execute_reply":"2022-07-27T05:55:07.399740Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def rmse(predictions, targets):\n    return np.sqrt(((predictions - targets) ** 2).mean())","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:55:07.403042Z","iopub.execute_input":"2022-07-27T05:55:07.403808Z","iopub.status.idle":"2022-07-27T05:55:07.409700Z","shell.execute_reply.started":"2022-07-27T05:55:07.403767Z","shell.execute_reply":"2022-07-27T05:55:07.408623Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train.isna().sum()","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:55:07.411391Z","iopub.execute_input":"2022-07-27T05:55:07.412303Z","iopub.status.idle":"2022-07-27T05:55:09.691092Z","shell.execute_reply.started":"2022-07-27T05:55:07.412239Z","shell.execute_reply":"2022-07-27T05:55:09.690048Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_test.isna().sum()","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:55:09.693052Z","iopub.execute_input":"2022-07-27T05:55:09.693736Z","iopub.status.idle":"2022-07-27T05:55:10.362010Z","shell.execute_reply.started":"2022-07-27T05:55:09.693697Z","shell.execute_reply":"2022-07-27T05:55:10.360935Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train['price'] = df_train['price'].fillna(df_train['price'].mean())\ndf_test['price'] = df_test['price'].fillna(df_train['price'].mean())\n\ndf_test.isna().sum()","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:55:10.363304Z","iopub.execute_input":"2022-07-27T05:55:10.363674Z","iopub.status.idle":"2022-07-27T05:55:11.041018Z","shell.execute_reply.started":"2022-07-27T05:55:10.363638Z","shell.execute_reply":"2022-07-27T05:55:11.039940Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for col in ['param_1', 'param_2', 'param_3', 'image_top_1', 'title', 'description']:\n    df_train[col] = df_train[col].fillna('')\n    df_test[col] = df_test[col].fillna('')\n    \ndf_test.isna().sum()","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:55:17.786474Z","iopub.execute_input":"2022-07-27T05:55:17.786839Z","iopub.status.idle":"2022-07-27T05:55:20.127571Z","shell.execute_reply.started":"2022-07-27T05:55:17.786810Z","shell.execute_reply":"2022-07-27T05:55:20.126489Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train['image_top_1'] = df_train['image_top_1'].astype('str')\ndf_test['image_top_1'] = df_test['image_top_1'].astype('str')    ","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:55:23.296787Z","iopub.execute_input":"2022-07-27T05:55:23.297159Z","iopub.status.idle":"2022-07-27T05:55:24.551173Z","shell.execute_reply.started":"2022-07-27T05:55:23.297130Z","shell.execute_reply":"2022-07-27T05:55:24.550057Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cat_features = ['region', 'city', 'parent_category_name', 'category_name', 'param_1', 'param_2', 'param_3', 'user_type', 'image_top_1']\ntext_features = ['title', 'description']\n\ndf_train.drop(['image', 'item_id', 'user_id', 'activation_date'], axis=1, inplace=True)\ndf_test.drop(['image', 'item_id', 'user_id', 'activation_date'], axis=1, inplace=True)","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:55:27.205279Z","iopub.execute_input":"2022-07-27T05:55:27.205740Z","iopub.status.idle":"2022-07-27T05:55:28.523751Z","shell.execute_reply.started":"2022-07-27T05:55:27.205694Z","shell.execute_reply":"2022-07-27T05:55:28.522683Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del_cols = ['deal_prob_cat', 'params', 'price_log']\n\nfor col in del_cols:\n    if col in df_train.columns:\n        df_train = df_train.drop(columns=col)\n    if col in df_test.columns:\n        df_test = df_test.drop(columns=col)\n\ndf_train.head()","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:58:05.884841Z","iopub.execute_input":"2022-07-27T05:58:05.887280Z","iopub.status.idle":"2022-07-27T05:58:06.681884Z","shell.execute_reply.started":"2022-07-27T05:58:05.887238Z","shell.execute_reply":"2022-07-27T05:58:06.680761Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from scipy.sparse import hstack, csr_matrix\nfrom sklearn.model_selection import train_test_split\n\n\nX = df_train.drop(columns=['deal_probability', 'title', 'description'])\nX_test = df_test.drop(columns=['title', 'description'])\n\ny = df_train['deal_probability']","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:58:11.853441Z","iopub.execute_input":"2022-07-27T05:58:11.854481Z","iopub.status.idle":"2022-07-27T05:58:12.003617Z","shell.execute_reply.started":"2022-07-27T05:58:11.854437Z","shell.execute_reply":"2022-07-27T05:58:12.002528Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X.dtypes","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:58:13.199673Z","iopub.execute_input":"2022-07-27T05:58:13.200669Z","iopub.status.idle":"2022-07-27T05:58:13.210024Z","shell.execute_reply.started":"2022-07-27T05:58:13.200629Z","shell.execute_reply":"2022-07-27T05:58:13.208523Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.model_selection import StratifiedKFold\n\n\nspliter = StratifiedKFold(n_splits=5, shuffle=True,\n                          random_state=3)\n\n_y = (df_train.deal_probability.round(2)*100).astype(int)\n\nFOLD_LIST = list(spliter.split(_y, _y))","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:58:16.051055Z","iopub.execute_input":"2022-07-27T05:58:16.051787Z","iopub.status.idle":"2022-07-27T05:58:16.490010Z","shell.execute_reply.started":"2022-07-27T05:58:16.051732Z","shell.execute_reply":"2022-07-27T05:58:16.488948Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"FOLD_LIST","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:58:16.492377Z","iopub.execute_input":"2022-07-27T05:58:16.492828Z","iopub.status.idle":"2022-07-27T05:58:16.501871Z","shell.execute_reply.started":"2022-07-27T05:58:16.492784Z","shell.execute_reply":"2022-07-27T05:58:16.500892Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from catboost import CatBoostRegressor, Pool\nfrom tqdm import tqdm_notebook\n\n\n_models = []\n\noof_predictions = np.zeros(shape=[X.shape[0]])\n\nfor fold_id, (train_idx, val_idx) in tqdm_notebook(enumerate(FOLD_LIST)):\n    \n    X_train, Y_train = X.loc[train_idx], y.loc[train_idx]\n    X_val, Y_val = X.loc[val_idx], y.loc[val_idx]\n    \n    train_dataset = Pool(X_train, Y_train,\n                         cat_features=cat_features)\n    \n    eval_dataset = Pool(X_val, Y_val,\n                        cat_features=cat_features)\n    \n    model = CatBoostRegressor(\n        learning_rate=0.1, iterations=1000, eval_metric='RMSE',\n        metric_period=50, early_stopping_rounds=20, task_type=\"GPU\",\n    )\n    model.fit(train_dataset, eval_set=eval_dataset)\n    \n    _models.append(model)\n    preds = model.predict(X_val)\n    oof_predictions[val_idx] += preds\n    print('fold_id:', fold_id, rmse(Y_val, preds))","metadata":{"execution":{"iopub.status.busy":"2022-07-27T05:58:17.137103Z","iopub.execute_input":"2022-07-27T05:58:17.137481Z","iopub.status.idle":"2022-07-27T06:09:15.526923Z","shell.execute_reply.started":"2022-07-27T05:58:17.137451Z","shell.execute_reply":"2022-07-27T06:09:15.525815Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"oof_predictions","metadata":{"execution":{"iopub.status.busy":"2022-07-27T06:09:34.209757Z","iopub.execute_input":"2022-07-27T06:09:34.210146Z","iopub.status.idle":"2022-07-27T06:09:34.218246Z","shell.execute_reply.started":"2022-07-27T06:09:34.210116Z","shell.execute_reply":"2022-07-27T06:09:34.217041Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"oof_predictions /= len(FOLD_LIST)\noof_predictions","metadata":{"execution":{"iopub.status.busy":"2022-07-27T06:09:35.553434Z","iopub.execute_input":"2022-07-27T06:09:35.553820Z","iopub.status.idle":"2022-07-27T06:09:35.563755Z","shell.execute_reply.started":"2022-07-27T06:09:35.553789Z","shell.execute_reply":"2022-07-27T06:09:35.562274Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"rmse(oof_predictions, y)","metadata":{"execution":{"iopub.status.busy":"2022-07-27T06:09:36.481291Z","iopub.execute_input":"2022-07-27T06:09:36.481913Z","iopub.status.idle":"2022-07-27T06:09:36.529116Z","shell.execute_reply.started":"2022-07-27T06:09:36.481877Z","shell.execute_reply":"2022-07-27T06:09:36.527960Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**+ tfidf features**","metadata":{}},{"cell_type":"markdown","source":"```\nCatBoostError: catboost/libs/data/features_layout.cpp:109: All feature names should be different, but 'price' used more than once.\n```\n","metadata":{}},{"cell_type":"code","source":"df_train.rename(columns={'price': 'ad_price'}, inplace=True)\ndf_test.rename(columns={'price': 'ad_price'}, inplace=True)","metadata":{"execution":{"iopub.status.busy":"2022-07-27T06:09:52.793084Z","iopub.execute_input":"2022-07-27T06:09:52.794116Z","iopub.status.idle":"2022-07-27T06:09:52.801331Z","shell.execute_reply.started":"2022-07-27T06:09:52.794060Z","shell.execute_reply":"2022-07-27T06:09:52.800241Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train_tfidf_title = pd.concat([df_train, train_title_df], axis=1)\ndf_train_tfidf_title.shape","metadata":{"execution":{"iopub.status.busy":"2022-07-27T06:09:56.746535Z","iopub.execute_input":"2022-07-27T06:09:56.746921Z","iopub.status.idle":"2022-07-27T06:09:57.057772Z","shell.execute_reply.started":"2022-07-27T06:09:56.746891Z","shell.execute_reply":"2022-07-27T06:09:57.056649Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del train_title_df","metadata":{"execution":{"iopub.status.busy":"2022-07-27T06:09:58.207418Z","iopub.execute_input":"2022-07-27T06:09:58.209944Z","iopub.status.idle":"2022-07-27T06:09:58.215206Z","shell.execute_reply.started":"2022-07-27T06:09:58.209904Z","shell.execute_reply":"2022-07-27T06:09:58.214201Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_test_tfidf_title = pd.concat([df_test, test_title_df], axis=1)\ndf_test_tfidf_title.shape","metadata":{"execution":{"iopub.status.busy":"2022-07-27T06:09:59.954251Z","iopub.execute_input":"2022-07-27T06:09:59.954702Z","iopub.status.idle":"2022-07-27T06:10:00.283351Z","shell.execute_reply.started":"2022-07-27T06:09:59.954663Z","shell.execute_reply":"2022-07-27T06:10:00.277604Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del test_title_df","metadata":{"execution":{"iopub.status.busy":"2022-07-27T06:10:01.316402Z","iopub.execute_input":"2022-07-27T06:10:01.316765Z","iopub.status.idle":"2022-07-27T06:10:01.321580Z","shell.execute_reply.started":"2022-07-27T06:10:01.316735Z","shell.execute_reply":"2022-07-27T06:10:01.320451Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X = df_train_tfidf_title.drop(columns=['deal_probability', 'title', 'description'])\nX_test = df_test_tfidf_title.drop(columns=['title', 'description'])\n\ny = df_train_tfidf_title['deal_probability']","metadata":{"execution":{"iopub.status.busy":"2022-07-27T06:10:02.605741Z","iopub.execute_input":"2022-07-27T06:10:02.606442Z","iopub.status.idle":"2022-07-27T06:10:03.636011Z","shell.execute_reply.started":"2022-07-27T06:10:02.606403Z","shell.execute_reply":"2022-07-27T06:10:03.635002Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del df_train_tfidf_title, df_test_tfidf_title","metadata":{"execution":{"iopub.status.busy":"2022-07-27T06:10:03.976569Z","iopub.execute_input":"2022-07-27T06:10:03.977025Z","iopub.status.idle":"2022-07-27T06:10:04.077330Z","shell.execute_reply.started":"2022-07-27T06:10:03.976983Z","shell.execute_reply":"2022-07-27T06:10:04.076198Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from catboost import CatBoostRegressor, Pool\nfrom tqdm import tqdm_notebook\n\n\n_models_tfidf = []\n\noof_predictions = np.zeros(shape=[X.shape[0]])\n\nfor fold_id, (train_idx, val_idx) in tqdm_notebook(enumerate(FOLD_LIST)):\n    \n    X_train, Y_train = X.loc[train_idx], y.loc[train_idx]\n    X_val, Y_val = X.loc[val_idx], y.loc[val_idx]\n    \n    train_dataset = Pool(X_train, Y_train,\n                         cat_features=cat_features)\n    \n    eval_dataset = Pool(X_val, Y_val,\n                        cat_features=cat_features)\n    \n    model = CatBoostRegressor(\n        learning_rate=0.1, iterations=1000, eval_metric='RMSE',\n        metric_period=50, early_stopping_rounds=20, task_type=\"GPU\",\n    )\n    model.fit(train_dataset, eval_set=eval_dataset)\n    \n    _models_tfidf.append(model)\n    preds = model.predict(X_val)\n    oof_predictions[val_idx] += preds\n    print('fold_id:', fold_id, rmse(Y_val, preds))","metadata":{"execution":{"iopub.status.busy":"2022-07-27T06:10:06.279729Z","iopub.execute_input":"2022-07-27T06:10:06.280111Z","iopub.status.idle":"2022-07-27T06:32:50.889155Z","shell.execute_reply.started":"2022-07-27T06:10:06.280079Z","shell.execute_reply":"2022-07-27T06:32:50.888036Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"oof_predictions /= len(FOLD_LIST)\noof_predictions","metadata":{"execution":{"iopub.status.busy":"2022-07-27T06:32:57.157176Z","iopub.execute_input":"2022-07-27T06:32:57.157806Z","iopub.status.idle":"2022-07-27T06:32:57.167221Z","shell.execute_reply.started":"2022-07-27T06:32:57.157772Z","shell.execute_reply":"2022-07-27T06:32:57.165797Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"rmse(oof_predictions, y)","metadata":{"execution":{"iopub.status.busy":"2022-07-27T06:32:58.368164Z","iopub.execute_input":"2022-07-27T06:32:58.368524Z","iopub.status.idle":"2022-07-27T06:32:58.415742Z","shell.execute_reply.started":"2022-07-27T06:32:58.368495Z","shell.execute_reply":"2022-07-27T06:32:58.414840Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**predictions**","metadata":{}},{"cell_type":"code","source":"pred = np.zeros(shape=[X_test.shape[0]])\n\nfor model in tqdm_notebook(_models):\n# for model in tqdm_notebook(_models_tfidf):\n    preds = model.predict(X_test)\n    pred += preds","metadata":{"execution":{"iopub.status.busy":"2022-07-27T06:33:02.502831Z","iopub.execute_input":"2022-07-27T06:33:02.503592Z","iopub.status.idle":"2022-07-27T06:33:52.822807Z","shell.execute_reply.started":"2022-07-27T06:33:02.503551Z","shell.execute_reply":"2022-07-27T06:33:52.821767Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pred /= len(_models)\n# pred /= len(_models_tfidf)\n\npred","metadata":{"execution":{"iopub.status.busy":"2022-07-27T06:33:52.824773Z","iopub.execute_input":"2022-07-27T06:33:52.825735Z","iopub.status.idle":"2022-07-27T06:33:52.834200Z","shell.execute_reply.started":"2022-07-27T06:33:52.825694Z","shell.execute_reply":"2022-07-27T06:33:52.832873Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_sub = pd.read_csv('../input/avito-demand-prediction/sample_submission.csv')\nsample_sub","metadata":{"execution":{"iopub.status.busy":"2022-07-27T06:33:52.835932Z","iopub.execute_input":"2022-07-27T06:33:52.836510Z","iopub.status.idle":"2022-07-27T06:33:53.162185Z","shell.execute_reply.started":"2022-07-27T06:33:52.836467Z","shell.execute_reply":"2022-07-27T06:33:53.161176Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pred = model.predict(X_test)\n\nsample_sub['deal_probability'] = np.clip(pred, 0, 1)\nsample_sub.to_csv('sub.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2022-07-27T06:33:53.164693Z","iopub.execute_input":"2022-07-27T06:33:53.165099Z","iopub.status.idle":"2022-07-27T06:34:04.495035Z","shell.execute_reply.started":"2022-07-27T06:33:53.165060Z","shell.execute_reply":"2022-07-27T06:34:04.494016Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_sub","metadata":{"execution":{"iopub.status.busy":"2022-07-27T06:34:04.496455Z","iopub.execute_input":"2022-07-27T06:34:04.497105Z","iopub.status.idle":"2022-07-27T06:34:04.512242Z","shell.execute_reply.started":"2022-07-27T06:34:04.497064Z","shell.execute_reply":"2022-07-27T06:34:04.510946Z"},"trusted":true},"execution_count":null,"outputs":[]}]}