{"cells":[{"metadata":{"_uuid":"a0fe11e3b3d607d0a2a61e10b073406df899fd8b"},"cell_type":"markdown","source":"# Avito Demand Prediction Challenge\n"},{"metadata":{"collapsed":true,"trusted":true,"_uuid":"18971ff77c48bfd341b0146f92ab3454d18e5074"},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\n\npd.set_option(\"display.max_columns\", 500)","execution_count":2,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2dd5ae01d86f44633adc658932f08709c89afabb"},"cell_type":"code","source":"data = pd.read_csv(\"../input/train.csv\")\ndata.head()","execution_count":3,"outputs":[]},{"metadata":{"_uuid":"bd180bf2efa4174c4ce76b54b14adba751b3a49a"},"cell_type":"markdown","source":"In the dataset there are such feature columns:\n\n* `item_id` — Ad id.\n* `user_id` — User id.\n* `region` — Ad region.\n* `city` — Ad city.\n* `parent_category_name` — Top level ad category as classified by Avito's ad model.\n* `category_name` — Fine grain ad category as classified by Avito's ad model.\n* `param_1` — Optional parameter from Avito's ad model.\n* `param_2` — Optional parameter from Avito's ad model.\n* `param_3` — Optional parameter from Avito's ad model.\n* `title` — Ad title.\n* `description` — Ad description.\n* `price` — Ad price.\n* `item_seq_number` — Ad sequential number for user.\n* `activation_date`— Date ad was placed.\n* `user_type` — User type.\n* `image` — Id code of image. Ties to a jpg file in train_jpg. Not every ad has an image.\n* `image_top_1` — Avito's classification code for the image.\n* `deal_probability` — The target variable. This is the likelihood that an ad actually sold something. It's not possible to verify every transaction with certainty, so this column's value can be any float from zero to one.\n\n\nWe will remove the following from them:\n* `item_id`,` user_id` is useless information for us,\n* `city` - seems too small division,\n* `title` - we have more meaningful texts,\n* param_1, param_2, param_3 are optional and not always present parameters,\n* `activation_date`,` item_seq_number` - also looks useless,\n* `image`,` image_top_1` - we will concentrate on text and categorical variables, not images."},{"metadata":{"collapsed":true,"trusted":true,"_uuid":"19aa269a506bc1fd79ec2456411726f0ee62ed79"},"cell_type":"code","source":"cols_to_drop = [\"item_id\", \"user_id\", \"city\", \"param_1\", \"param_2\", \"param_3\", \"title\",\n    \"activation_date\", \"item_seq_number\", \"image\", \"image_top_1\"]\ndata = data.drop(labels=cols_to_drop, axis=1)","execution_count":4,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4910e224b3d32c63f9830757f645490edf1ce021"},"cell_type":"code","source":"data.head()","execution_count":5,"outputs":[]},{"metadata":{"_uuid":"ad57ef27fdf1baa8f613a694f3dcaa47476923e4"},"cell_type":"markdown","source":"We categorize the categorical attributes (we transform it into a numerical form):"},{"metadata":{"collapsed":true,"trusted":true,"_uuid":"7fa23d65fcd1ec8ae07d71200af1df5996c14065"},"cell_type":"code","source":"from sklearn.preprocessing import LabelEncoder\n\ndef label_encoding(data):\n    temp  = LabelEncoder()\n    temp.fit(data)\n    data = temp.transform(data)\n    return data","execution_count":6,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f5571341ce8f0060d262d20c925bdb576d8e2bb7"},"cell_type":"code","source":"# parent_category = LabelEncoder()\n# parent_category.fit(data[\"parent_category_name\"])\n# data[\"parent_category_name\"] = parent_category.transform(data[\"parent_category_name\"])\ndata[\"parent_category_name\"]  = label_encoding(data[\"parent_category_name\"])\ndata[\"parent_category_name\"].head()","execution_count":7,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3430335876c4392b3f075be608e8fa82225f7d0f"},"cell_type":"code","source":"# category = LabelEncoder()\n# category.fit(data[\"category_name\"])\n# data[\"category_name\"] = category.transform(data[\"category_name\"])\ndata[\"category_name\"] = label_encoding(data[\"category_name\"])\ndata[\"category_name\"].head()","execution_count":8,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3feb8b7a9b7b4018a3ee578eb13d774a407927a2"},"cell_type":"code","source":"# user_type = LabelEncoder()\n# user_type.fit(data[\"user_type\"])\n# data[\"user_type\"] = user_type.transform(data[\"user_type\"])\ndata[\"user_type\"] = label_encoding(data[\"user_type\"])\ndata[\"user_type\"].head()","execution_count":9,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f765a925dde1f0b08efe87214e273d56070c4921"},"cell_type":"code","source":"# region = LabelEncoder()\n# region.fit(data[\"region\"])\n# data[\"region\"] = region.transform(data[\"region\"])\ndata[\"region\"] = label_encoding(data[\"region\"])\ndata[\"region\"].head()","execution_count":10,"outputs":[]},{"metadata":{"collapsed":true,"trusted":true,"_uuid":"7e08e80bc1a33e076b65135b1b51d8fb8ea4bc67"},"cell_type":"code","source":"data = data.dropna()","execution_count":11,"outputs":[]},{"metadata":{"_uuid":"1a16888f0f482ae1e0461690a0aaad897d3f46e1"},"cell_type":"markdown","source":"So it looks like now:"},{"metadata":{"trusted":true,"_uuid":"692e2b525794b5af41760f8ea7cc49390025a090"},"cell_type":"code","source":"data.head()","execution_count":12,"outputs":[]},{"metadata":{"_uuid":"41dcff6999c83cd71b87a8a0c8784fbf73f3f5d1"},"cell_type":"markdown","source":"### 1.2. Preprocessing: texts"},{"metadata":{"_uuid":"054af98d3936dcfdae3a98efe68296e3f64188cf"},"cell_type":"markdown","source":"Experimental way it was found out that some ads are empty. That about them did not break anything, run `fillna ()`."},{"metadata":{"collapsed":true,"trusted":true,"_uuid":"a424d0a8038f16f5deeb99f02a18c5dcd0ba595b"},"cell_type":"code","source":"data[\"description\"].fillna(\"\", inplace=True)","execution_count":13,"outputs":[]},{"metadata":{"_uuid":"eb421ed2c07b5123e64a0aeaba0bebc92eb392ff"},"cell_type":"markdown","source":"Texts first bring to the lower case, and then calculate the number of tokens and bring everything to the lemmas. On the way, we'll take away the stop words. This can be done as follows (This process takes a very long time, so preprocessing is limited when uploading data):\n\n```python\nfrom nltk import word_tokenize\nfrom pymystem3 import Mystem\nfrom nltk.corpus import stopwords\n\nmystem = Mystem()\n\n\ndef count_words(text):\n    try:\n        len_words = len(word_tokenize(text))\n    except:\n        len_words = 0\n    return len_words\n\ndef do_lemmas(text):\n    try:\n        stops = stopwords.words(\"russian\")\n        lemmas = [lemma for lemma in mystem.lemmatize(text) if lemma not in stops]\n        return lemmas\n    except:\n        return \"\"\n        \ndata_new[\"word_count\"] = data_new[\"description\"].apply(count_words)\ndata_new[\"lemmas\"] = data_new[\"description\"].apply(do_lemmas)\n```"},{"metadata":{"_uuid":"044679fa1464950de270bcdd3c8db9712a9b3aff"},"cell_type":"markdown","source":"We use the casual_tokenize module from the nltk library to tokenize the text.\n\nCleaning the text from punctuation marks with the built-in Python function .isalpha ()"},{"metadata":{"collapsed":true,"trusted":true,"_uuid":"cf5c562172cc7bfd3b787db55480446732735783"},"cell_type":"code","source":"from nltk import casual_tokenize","execution_count":14,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"0620a12b7269b50b8372dfd92f2feae819232c94"},"cell_type":"code","source":"def tokenize(text):\n    tokens = casual_tokenize(str(text))\n    clean_stuff = [word.lower() for word in tokens if word.isalpha()]\n    line = \" \".join(clean_stuff)\n    return line","execution_count":15,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1110de0f7585cba51f1a2c9c009634d05493b122"},"cell_type":"code","source":"%%time\ndata[\"description\"] = data[\"description\"].apply(tokenize)","execution_count":16,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"434cd826c156ab872114adcde14ca96c4b1e21b1"},"cell_type":"code","source":"data[\"description\"].head()","execution_count":17,"outputs":[]},{"metadata":{"_uuid":"87a0663f03593a8127af26429cd1dec81a5c5837"},"cell_type":"markdown","source":"### 1.3. TF-IDF texts + branch features"},{"metadata":{"_uuid":"9bfc001a63533079f4bc7e699696919480076368"},"cell_type":"markdown","source":"First, we select the target variable and all the categorical and quantitative features:"},{"metadata":{"trusted":true,"_uuid":"9d40da762e7e66073247ffaa02ca91b57fa485d8","collapsed":true},"cell_type":"code","source":"cat_num_cols = [\"region\", \"parent_category_name\", \"category_name\", \"price\", \"user_type\"]\nX_cat_num = data[cat_num_cols].values","execution_count":18,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"61d74811c48f45fa5989590e3e7cebc75afe36bf","collapsed":true},"cell_type":"code","source":"y = data[\"deal_probability\"].values","execution_count":19,"outputs":[]},{"metadata":{"_uuid":"a28ddb9bf2ed68579d07e18a8f70de44b7c1ef9f"},"cell_type":"markdown","source":"For a stable solution to the problem with texts, using TF-IDF + SVM, you need to use PCA (the method of the main components, which you can learn more about: https://habr.com/post/304214/), or limit the size of the vectors using the TF-IDF method to get rid of data that does not significantly affect the result.\n\nApply TF-IDF:"},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"c9113100a5e694ed89d009c3506188b1e58d8422"},"cell_type":"code","source":"from sklearn.feature_extraction.text import TfidfVectorizer\nfrom nltk.corpus import stopwords","execution_count":20,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"cd0df7893257ffa4baf913df52339e06c102f390"},"cell_type":"code","source":"tfidf = TfidfVectorizer(\n    sublinear_tf=True,\n    strip_accents=\"unicode\",\n    analyzer=\"word\",\n    token_pattern=r\"\\w{1,}\",\n    stop_words=stopwords.words(\"russian\"),\n    max_features=10000\n)","execution_count":21,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"415e3b3751ce3e8f33648522b1fd209c5b59813e"},"cell_type":"code","source":"%%time\ntfidf.fit(data[\"description\"].values)\nX_texts = tfidf.transform(data[\"description\"].values)","execution_count":22,"outputs":[]},{"metadata":{"_uuid":"6b5bb3b56c486416399117139d246790958acaeb"},"cell_type":"markdown","source":"## 2. Integrating the features"},{"metadata":{"_uuid":"aa63b007add529382fcc0c5d23105ea6c274a9bf"},"cell_type":"markdown","source":"At this stage, it is necessary to connect all the categories we received. signs with the result of the work of TF-IDF."},{"metadata":{"collapsed":true,"trusted":true,"_uuid":"630bb9e2af76695e214096d82e3c5300ee9e9a07"},"cell_type":"code","source":"from scipy.sparse import hstack","execution_count":23,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"188fd50b1b0c5ea8362735e46dce9329dd30e1d5"},"cell_type":"code","source":"X = hstack((X_texts, X_cat_num))","execution_count":24,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9772d1e0aafc775f86c1ad230863f46709d63910"},"cell_type":"code","source":"X.shape","execution_count":25,"outputs":[]},{"metadata":{"_uuid":"4d28bbdda3907f20f7dddf3963ae748f53404ae9"},"cell_type":"markdown","source":"We keep it just in case:"},{"metadata":{"collapsed":true,"trusted":true,"_uuid":"2eae89b93ed20bd47b204c9013c4ba32429069bc"},"cell_type":"code","source":"import os\nfrom sklearn.externals import joblib","execution_count":26,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b7638edecef23300079acd428ff24394648bcd66"},"cell_type":"code","source":"try:\n    os.mkdir(\"./models\")\nexcept:\n    pass\njoblib.dump(X, \"./models/X.pkl\")\njoblib.dump(y, \"./models/y.pkl\")\njoblib.dump(tfidf, \"./models/tfidf.pkl\")","execution_count":27,"outputs":[]},{"metadata":{"_uuid":"c4b357d5b6c4c6da06e99f10733fb7009195bcd9"},"cell_type":"markdown","source":"## 3. Any different algorithms"},{"metadata":{"_uuid":"6a4019e06b6d168c968b5f7d770821f00f9624eb"},"cell_type":"markdown","source":"## 3.0. Preparation"},{"metadata":{"_uuid":"851d6353abe69a1c37da5a01e874de8476cfb884"},"cell_type":"markdown","source":"For the beginning ** we will break the sample ** into the training and test:"},{"metadata":{"collapsed":true,"trusted":true,"_uuid":"8ebfa851d20d38448d34f0282861209b4070d901"},"cell_type":"code","source":"from sklearn.model_selection import train_test_split","execution_count":28,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7deabe63c9e362f39f77bd236b73d00e2e1a4877","collapsed":true},"cell_type":"code","source":"X_train, X_test, y_train, y_test = train_test_split(X, y)","execution_count":29,"outputs":[]},{"metadata":{"_uuid":"5a468832f29620afbab9568702b68adf1c54b77b"},"cell_type":"markdown","source":"** Create an error function ** RMSE (root of mean squared error) to evaluate by it:"},{"metadata":{"collapsed":true,"trusted":true,"_uuid":"93ae58de1273bbbcd7ceb2f838956e8af976bfb8"},"cell_type":"code","source":"from sklearn.metrics import mean_squared_error, make_scorer\nfrom math import sqrt","execution_count":30,"outputs":[]},{"metadata":{"collapsed":true,"trusted":true,"_uuid":"d76d191248ed517231773ff54c6b07528f6dc5e6"},"cell_type":"code","source":"def rmse_func(y_calc, y_test):\n    rms = sqrt(mean_squared_error(y_actual, y_predicted))\n    return rms\n\nrmse = make_scorer(rmse_func, greater_is_better=False)","execution_count":31,"outputs":[]},{"metadata":{"_uuid":"79cf2528b9bbd066ced42a0564e9cd13f52b614b"},"cell_type":"markdown","source":"### 3.1. SVM\n\nThe classical state of the art solution for text-based tasks is SVM on the RBF core."},{"metadata":{"collapsed":true,"trusted":true,"_uuid":"f3292a5e58a236cb2f500cff281fdd20dfc9f7bc"},"cell_type":"code","source":"from sklearn.model_selection import GridSearchCV\nfrom sklearn.svm import SVR","execution_count":32,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0ea77521c7f4118768f1ac960ebba3c4c0a9d8a2"},"cell_type":"code","source":"%env JOBLIB_TEMP_FOLDER=/tmp","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f9ed224045f650526d9c9dcde4eec5913392a843"},"cell_type":"code","source":"params = {\"C\": np.arange(1, 100, 2)}\nsvr = GridSearchCV(\n    SVR(),\n    param_grid=params,\n    scoring=rmse,\n    cv=5,\n    verbose=1,\n    n_jobs=-1\n)\nsvr.fit(X_train, y_train)","execution_count":null,"outputs":[]},{"metadata":{"collapsed":true,"trusted":true,"_uuid":"66d038ce81edb84fb3a6a6b1ff127ad8eed9d77d"},"cell_type":"code","source":"print(\"SVR results:\\n\\t- best params: {}\\n\\t- best score: {}\".format(svr.best_params_, svr.best_score_))","execution_count":null,"outputs":[]},{"metadata":{"collapsed":true,"trusted":true,"_uuid":"fd4e60b070a4d72063d646416f7ddf1c8e5663bc"},"cell_type":"code","source":"result = svr.predict(y)\nids = data['item_id']\nids['deal_probability'] = result\nids.to_csv(\"submit2.csv\",index=True,header=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"2d5edb2573877b5c3b204b8f5df0a3139601f8c7"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}