{"cells":[{"metadata":{"_cell_guid":"5f9b1649-d4ff-4870-8edb-d3f69a4a4835","_uuid":"30bca7702ffa3d745e6ed724e8e96a7948490035"},"cell_type":"markdown","source":"

Impor tLibraries

"},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","collapsed":true,"scrolled":true,"trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nfrom transliterate import translit, get_available_language_codes\n","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"e8871bba-24f5-4d67-b1a0-c020fdb2e1c1","_uuid":"4229eed20c603a61250b79c3980b4a1002df1aa0"},"cell_type":"markdown","source":"

Data Preparation

\n\n

For Avito Demand Prediction challenge, this function is to translate the descriptive columns with Russian characters into English equivalents using transliterate library

"},{"metadata":{"_cell_guid":"b415f21d-ef81-4135-9206-c32fde87098b","_uuid":"622d8fa2daf5303066a85399b6fdb195d38c67b0","collapsed":true,"trusted":true},"cell_type":"code","source":"###### Function to input file name with Attributes in Russian and Translate them into Eglish for Train and Test data file format ####\ndef translate_data(input_file,output_file):\n full_input_file = \"../input/\"+input_file\n file_data = pd.read_csv(full_input_file)\n \n ##### Translate Russian language columns #####\n region = (file_data['region']).apply(translit, 'ru', reversed=True)\n city = (file_data['city']).apply(translit, 'ru', reversed=True)\n parent_category_name = (file_data['parent_category_name']).apply(translit, 'ru', reversed=True)\n category_name = (file_data['category_name']).apply(translit, 'ru', reversed=True)\n\n param_1 = file_data['param_1'].apply(lambda row:row if pd.isnull(row) else translit(row, 'ru', reversed=True))\n param_2 = file_data['param_2'].apply(lambda row:row if pd.isnull(row) else translit(row, 'ru', reversed=True))\n param_3 = file_data['param_3'].apply(lambda row:row if pd.isnull(row) else translit(row, 'ru', reversed=True))\n title = file_data['title'].apply(lambda row:row if pd.isnull(row) else translit(row, 'ru', reversed=True))\n description = file_data['description'].apply(lambda row:row if pd.isnull(row) else translit(row, 'ru', reversed=True))\n \n ### Create translated data frame #######\n file_data_translated = file_data\n file_data_translated['region'] = region\n file_data_translated['city'] = city\n file_data_translated['parent_category_name'] = parent_category_name\n file_data_translated['category_name'] = category_name\n file_data_translated['param_1'] = param_1\n file_data_translated['param_2'] = param_2\n file_data_translated['param_3'] = param_3\n file_data_translated['title'] = title\n file_data_translated['description'] = description\n \n ##### Export the translated data frame into output file #####\n return file_data_translated\n #file_data_translated = pd.read_csv(\"train_translated.csv\")\n #file_data_translated.head(n=5)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"003102af-9a3e-400c-89fc-0aae5aa9f691","_uuid":"7961f66574fcca115b31c19e68746cd9ee53bdfa","collapsed":true,"trusted":true},"cell_type":"code","source":"#### Function Calls to translate Russian language description columns into English equivalents for train.csv and test.csv #####\ntest_translated = translate_data(\"../input/test.csv\",\"test_translated.csv\")\ntrain_translated = translate_data(\"../input/train.csv\",\"train_translated.csv\")\n","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"34fe81b3-bb2f-47f3-a3ac-e43f82ea97f6","_uuid":"c383a3bafa9518fe3e8bb7da74a49f98bac34b58"},"cell_type":"markdown","source":"

Data Analysis

\n\nDigging into the data to extract some inner insights of the data"},{"metadata":{"_cell_guid":"6a209ff4-0004-4145-a921-19cf4eb80769","_uuid":"8193e41973afa879909b30d61629e4bd848f0539"},"cell_type":"markdown","source":"

Feature Engineering

\n\nThe columns such as City, Region, Categories, Params, Title, Description has descriptive data in String format. So using feature engineering to classify the features of the Columns like City Region, Category. For the other fields like Title, Params, description we need a different kind of approach."},{"metadata":{"_cell_guid":"a94549d4-1c34-43e6-8470-19380c494e59","_uuid":"4c2df29b1c94dbdd2b546d1ec93f64ea14d8d477","collapsed":true,"trusted":true},"cell_type":"code","source":"#### This function is for peforming Feature Engineering for the translated train.csv and test.csv files and export transformed data into output files ####\ndef transform_feature_data(file_data,output_file):\n #full_input_file = \"../input/\"+input_file\n #file_data = pd.read_csv(full_input_file)\n \n file_data_transformed_feature = file_data\n sliced_data_for_feature_engg = file_data.iloc[:, [2,3,4,5,6,7,8,13,14]]\n sliced_data_for_feature_engg = sliced_data_for_feature_engg.apply(lambda s: s.map({k:i for i,k in enumerate(s.unique())}))\n file_data_transformed_feature.iloc[:, [2,3,4,5,6,7,8,13,14]] = sliced_data_for_feature_engg\n file_data_transformed_feature['description'] = [len(str(x).split()) if x is not None else 0 for x in file_data_transformed_feature['description']]\n file_data_transformed_feature['title'] = [len(str(x).split()) if x is not None else 0 for x in file_data_transformed_feature['title']]\n\n #file_data_transformed_feature.head(5)\n\n ##### Export the translated data frame into output file #####\n file_data_transformed_feature.to_csv(output_file) ","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"5f1d782c-b1c4-4fa4-96e2-18bf986e540a","_uuid":"7fb12eae90c826b81414d34016bb48171e43b19b","collapsed":true,"trusted":true},"cell_type":"code","source":"#### Function Calls for feature engineering in tranlated train.csv and test.csv and create output transformed file #####\ntransform_feature_data(test_translated,\"test_transformed.csv\")\ntransform_feature_data(train_translated,\"train_transformed.csv\")","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.5","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}