{"cells":[{"metadata":{"_uuid":"81ee30f8423c9d1b090741f8ca09b719ab3cd499"},"cell_type":"markdown","source":"# Kernel references \nhttps://www.kaggle.com/bminixhofer/aggregated-features-lightgbm/output\n\nAdding new features from Benjamin's great Kernel\n- Number of ads per user \n- Average number of active days of all Ad's put up per user \n- Average number of times all Ad's were made active per user "},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","collapsed":true,"trusted":true},"cell_type":"code","source":"import pandas as pd \nimport numpy as np \nimport time \nimport gc ","execution_count":1,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true,"collapsed":true},"cell_type":"code","source":"np.random.seed(42)\n\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import roc_auc_score\nfrom sklearn.model_selection import StratifiedKFold\n\nfrom keras.models import Model\nfrom keras.layers import Input, Dropout, Dense, Embedding, SpatialDropout1D, concatenate\nfrom keras.layers import GRU, Bidirectional, GlobalAveragePooling1D, GlobalMaxPooling1D, BatchNormalization, Conv1D, MaxPooling1D, Flatten\nfrom keras.preprocessing.sequence import pad_sequences\nfrom keras.preprocessing import text, sequence\nfrom keras.callbacks import Callback\nfrom keras import backend as K\nfrom keras.models import Model\n\nfrom keras import optimizers\n\nfrom sklearn.preprocessing import LabelBinarizer\nfrom sklearn.preprocessing import LabelEncoder, MinMaxScaler, StandardScaler\n\nimport warnings\nwarnings.filterwarnings('ignore')\n\nimport os\nos.environ['OMP_NUM_THREADS'] = '4'\n\nimport threading\nimport multiprocessing\nfrom multiprocessing import Pool, cpu_count\nfrom contextlib import closing\ncores = 4\n\nfrom keras import backend as K\nfrom keras.optimizers import RMSprop, Adam\nfrom keras.callbacks import ModelCheckpoint, EarlyStopping\n\n### rmse loss for keras\ndef root_mean_squared_error(y_true, y_pred):\n    return K.sqrt(K.mean(K.square(y_true - y_pred))) ","execution_count":2,"outputs":[]},{"metadata":{"_uuid":"812cbb082e0c3dbe7b13df36e38f0133a7a6f08e"},"cell_type":"markdown","source":"#  HANDLING VARIABLES \n              \n*  Text variable -  title_description ( combination of title and description)\n  Using title and description fields separately gives a small boost to local CV and LB score but takes twice as long to train\n\n*  Categorical variables - region, city, category_name, parent_category_name, image_code_1, param_1, param123\n   param_1, param_2, param_3 variables are combined since param_2 and param_3 have close to 50% NaN values. \n\n*  Continous variables - price, item_seq_number \n\n*  Continuous variables based on train_active -  avg_days_up_user, avg_times_up_user, n_user_items\n\n    These features are from Benjamin's great Kernel \n    https://www.kaggle.com/bminixhofer/aggregated-features-lightgbm/output\n\nThe Following features are uploaded as a data set in a csv aggregated_features generated from Benjamin's kernel mentioned above. \n\n- Number of ads per user \n- Average number of active days of all Ad's put up per user \n- Average number of times all Ad's were made active per user \n\n\n\n   "},{"metadata":{"_uuid":"a396b2750e4c8596b9876e7e91dc67e25fdfabd6"},"cell_type":"markdown","source":"# STEP BY STEP EXPLANATION \n# PSEUDO CODE FOR WORK FLOW \n\nStep 1 :  Preprocess input data. Generate the target array \n\nStep 2:  Feature engineering with  tokens for word sequences and label encoding for categorical variables \n\nStep 3:  Transform train using word tokens and label encoders\n\nStep 4:   Generate word embedding file ( In this case Fast Text is used )\n\nStep 5:  Initialize the RNN model with learning rates, epochs, batch size and any necessary parameters \n\nStep 6:  Run a KFold on training data and the target variable. For each fold call the prediction function to \n             generate the output 'deal_probability'. \n             \nStep 7:  Combine the 'deal_probability' values in any number of ways \n            Simple Average,  Log Average, Output with least RMSE etc to generate final output "},{"metadata":{"_uuid":"b2ccff16ab970c228efbd83e52bfcce854cf7260"},"cell_type":"markdown","source":"# FUNCTION - preprocess_dataset\n \n*  Handling Missing Values\n*  Casting categorical variables into type \"category\"\n*  Combine 3 param variables into single feature\n\n# FUNCTION - keras_fit \n\n* Combine title and description into a single column and delete title and description \n\nTOKENIZATION OF TEXT \n\n*    Tokenize the sentences in 'title_description'. Maximum number of words in vocabulary is taken as 200,000.\n*    The number was chosen based on a quick look at the corresponding word_counts(). \n*    Words with index greater than 200,000 appear only less than 4 times. \n   \nLABEL ENCODING OF CATEGORICAL VARIABLES - \n\n*  sklearn label encoding is used \n*  The label encoding is done for train and test values to ensure that no new labels are encountered during the prediction \n   phase on test\n*  A new Label Encoder is created for every categorical field that is to be label encoded \n*  The output of this function will generate the transformed train data frame along with the word tokenizer and label encoder labels \n\nLOG TRANSFORMATION OF CONTINUOUS VARIABLES - \n\n* np.log1p is applied on the continuous values (log1p to avoid log of zero issues)\n* Log transformation is used on price, item_seq_number\n* Log transformation is used on the following values of avg_days_up_user, avg_times_up_user, n_user_items\n\nOUTPUT OF FUNCTION \n\n*  train data frame after necessary feature engineering \n*  Tokenizer for word features and label encoders for each of the categorical variables  \n\n# FUNCTION  keras_train_transform \n\n* Apply the tokenizer and label encoders to the vectorized train data frame \n\nLabel Encoding - Assigns a unique label name to each category type . This is the transformation process. \nText sequences - Each word in a sentence is assigned a unique label name. The seq_title_description field is generated \n                              which is a list of all the indexes for the words in the sentence. If a word index is greater than the limit \n                              provided by us ( 200,000 in this case) the word is not included in the sequence. \n\n* Input - train data frame with raw variables \n* Output - train data frame with transformed variables ready for training \n\n# FUNCTION  keras_train_transform \n\n* Apply the tokenizer and label encoders to the vectorized test data frame \n* Input - test data frame with raw variables \n* Output - test data frame with transformed variables ready for prediction \n   \n # FUNCTION  get_keras_data\n \n* It is easy to pass an input to the RNN fit call in Keras when Training and Validation data sets are passed as Dictionaries \n* Converts the transformed train data frame into a dictionary \n*  The function also takes the list of word indexes and pads it with zeros to a pre determined length \n* Each key in the dictionary will be the corresponding column name \n   "},{"metadata":{"_uuid":"fe99b725c4f55ef4cf471e685107536cd428b98d","_cell_guid":"6d1a6395-fb0c-4f97-9d07-46e3f764f389","collapsed":true,"trusted":true},"cell_type":"code","source":"def preprocess_dataset(dataset):\n    \n    t1 = time.time()\n    print(\"Filling Missing Values.....\")\n    \n    dataset['price'] = dataset['price'].fillna(0).astype('float32')\n    dataset['param_1'].fillna(value='missing', inplace=True)\n    dataset['param_2'].fillna(value='missing', inplace=True)\n    dataset['param_3'].fillna(value='missing', inplace=True)\n    \n    dataset['param_1'] = dataset['param_1'].astype(str)\n    dataset['param_2'] = dataset['param_2'].astype(str)\n    dataset['param_3'] = dataset['param_3'].astype(str)\n    \n    print(\"Casting data types to type Category.......\")\n    dataset['category_name'] = dataset['category_name'].astype('category')\n    dataset['parent_category_name'] = dataset['parent_category_name'].astype('category')\n    dataset['region'] = dataset['region'].astype('category')\n    dataset['city'] = dataset['city'].astype('category')\n    \n    dataset['image_top_1'] = dataset['image_top_1'].fillna('missing')\n    dataset['image_code'] = dataset['image_top_1'].astype('str')\n    del dataset['image_top_1']\n    gc.collect()\n\n    #dataset['week'] = pd.to_datetime(dataset['activation_date']).dt.week.astype('uint8')\n    #dataset['day'] = pd.to_datetime(dataset['activation_date']).dt.day.astype('uint8')\n    #dataset['wday'] = pd.to_datetime(dataset['activation_date']).dt.dayofweek.astype('uint8')\n    del dataset['activation_date']\n    gc.collect()\n    \n    print(\"Creating New Feature.....\")\n    dataset['param123'] = (dataset['param_1']+'_'+dataset['param_2']+'_'+dataset['param_3']).astype(str)\n    del dataset['param_2'], dataset['param_3']\n    gc.collect()\n        \n    print(\"PreProcessing Function completed.\")\n    \n    return dataset\n\ndef keras_fit(train):\n    \n    t1 = time.time()\n    train['title_description']= (train['title']+\" \"+train['description']).astype(str)\n    del train['description'], train['title']\n    gc.collect()\n    \n    print(\"Start Tokenization.....\")\n    tokenizer = text.Tokenizer(num_words = max_words_title_description)\n    all_text = np.hstack([train['title_description'].str.lower()])\n    tokenizer.fit_on_texts(all_text)\n    del all_text\n    gc.collect()\n    \n    print(\"Loading Test for Label Encoding on Train + Test\")\n    use_cols_test = ['region', 'city', 'parent_category_name', 'category_name', 'param_1', 'param_2', 'param_3', 'image_top_1', 'activation_date']\n    test = pd.read_csv(\"../input/avito-demand-prediction/test.csv\", usecols = use_cols_test)\n    \n    test['image_top_1'] = test['image_top_1'].fillna('missing')\n    test['image_code'] = test['image_top_1'].astype('str')\n    del test['image_top_1']\n    gc.collect()\n    \n    #test['week'] = pd.to_datetime(test['activation_date']).dt.week.astype('uint8')\n    #test['day'] = pd.to_datetime(test['activation_date']).dt.day.astype('uint8')\n    #test['wday'] = pd.to_datetime(test['activation_date']).dt.dayofweek.astype('uint8')\n    del test['activation_date']\n    gc.collect()\n    \n    test['param_1'].fillna(value='missing', inplace=True)\n    test['param_1'] = test['param_1'].astype(str)\n    test['param_2'].fillna(value='missing', inplace=True)\n    test['param_2'] = test['param_2'].astype(str)\n    test['param_3'].fillna(value='missing', inplace=True)\n    test['param_3'] = test['param_3'].astype(str)\n\n    print(\"Creating New Feature.....\")\n    test['param123'] = (test['param_1']+'_'+test['param_2']+'_'+test['param_3']).astype(str)\n    del test['param_2'], test['param_3']\n    gc.collect()\n    \n    ntrain = train.shape[0]\n    DF = pd.concat([train, test], axis = 0)\n    del train, test\n    gc.collect()\n    print(DF.shape)\n    \n    print(\"Start Label Encoding process....\")\n    le_region = LabelEncoder()\n    le_region.fit(DF.region)\n    \n    le_city = LabelEncoder()\n    le_city.fit(DF.city)\n    \n    le_category_name = LabelEncoder()\n    le_category_name.fit(DF.category_name)\n    \n    le_parent_category_name = LabelEncoder()\n    le_parent_category_name.fit(DF.parent_category_name)\n    \n    le_param_1 = LabelEncoder()\n    le_param_1.fit(DF.param_1)\n    \n    le_param123 = LabelEncoder()\n    le_param123.fit(DF.param123)\n    \n    le_image_code = LabelEncoder()\n    le_image_code.fit(DF.image_code)\n    \n    #le_week = LabelEncoder()\n    #le_week.fit(DF.week)\n    #le_day = LabelEncoder()\n    #le_day.fit(DF.day)\n    #le_wday = LabelEncoder()\n    #le_wday.fit(DF.wday)\n    \n    train = DF[0:ntrain]\n    del DF \n    gc.collect()\n    \n    train['price'] = np.log1p(train['price'])\n    train['avg_days_up_user'] = np.log1p(train['avg_days_up_user'])\n    train['avg_times_up_user'] = np.log1p(train['avg_times_up_user'])\n    train['n_user_items'] = np.log1p(train['n_user_items'])\n    train['item_seq_number'] = np.log(train['item_seq_number'])\n    print(\"Fit on Train Function completed.\")\n    \n    return train, tokenizer, le_region, le_city, le_category_name, le_parent_category_name, le_param_1, le_param123, le_image_code\n\ndef keras_train_transform(dataset):\n    \n    t1 = time.time()\n    \n    dataset['seq_title_description']= tokenizer.texts_to_sequences(dataset.title_description.str.lower())\n    print(\"Transform done for test\")\n    print(\"Time taken for Sequence Tokens is\"+str(time.time()-t1))\n    del train['title_description']\n    gc.collect()\n\n    dataset['region'] = le_region.transform(dataset['region'])\n    dataset['city'] = le_city.transform(dataset['city'])\n    dataset['category_name'] = le_category_name.transform(dataset['category_name'])\n    dataset['parent_category_name'] = le_parent_category_name.transform(dataset['parent_category_name'])\n    dataset['param_1'] = le_param_1.transform(dataset['param_1'])\n    dataset['param123'] = le_param123.transform(dataset['param123'])\n    #dataset['day'] = le_day.transform(dataset['day'])\n    #dataset['week'] = le_week.transform(dataset['week'])\n    #dataset['wday'] = le_wday.transform(dataset['wday'])\n    dataset['image_code'] = le_image_code.transform(dataset['image_code'])\n    \n    print(\"Transform on test function completed.\")\n    \n    return dataset\n    \ndef keras_test_transform(dataset):\n    \n    t1 = time.time()\n    dataset['title_description']= (dataset['title']+\" \"+dataset['description']).astype(str)\n    del dataset['description'], dataset['title']\n    gc.collect()\n    \n    dataset['seq_title_description']= tokenizer.texts_to_sequences(dataset.title_description.str.lower())\n    print(\"Transform done for test\")\n    print(\"Time taken for Sequence Tokens is\"+str(time.time()-t1))\n    \n    del dataset['title_description']\n    gc.collect()\n\n    dataset['region'] = le_region.transform(dataset['region'])\n    dataset['city'] = le_city.transform(dataset['city'])\n    dataset['category_name'] = le_category_name.transform(dataset['category_name'])\n    dataset['parent_category_name'] = le_parent_category_name.transform(dataset['parent_category_name'])\n    dataset['param_1'] = le_param_1.transform(dataset['param_1'])\n    dataset['param123'] = le_param123.transform(dataset['param123'])\n    #dataset['day'] = le_day.transform(dataset['day'])\n    #dataset['week'] = le_week.transform(dataset['week'])\n    #dataset['wday'] = le_wday.transform(dataset['wday'])\n    dataset['image_code'] = le_image_code.transform(dataset['image_code'])\n    \n    dataset['price'] = np.log1p(dataset['price'])\n    dataset['item_seq_number'] = np.log(dataset['item_seq_number'])\n    dataset['avg_days_up_user'] = np.log1p(dataset['avg_days_up_user'])\n    dataset['avg_times_up_user'] = np.log1p(dataset['avg_times_up_user'])\n    dataset['n_user_items'] = np.log1p(dataset['n_user_items'])\n    \n    print(\"Transform on test function completed.\")\n    \n    return dataset\n    \ndef get_keras_data(dataset):\n    X = {\n        'seq_title_description': pad_sequences(dataset.seq_title_description, maxlen=max_seq_title_description_length)\n        ,'region': np.array(dataset.region)\n        ,'city': np.array(dataset.city)\n        ,'category_name': np.array(dataset.category_name)\n        ,'parent_category_name': np.array(dataset.parent_category_name)\n        ,'param_1': np.array(dataset.param_1)\n        ,'param123': np.array(dataset.param123)\n        ,'image_code':np.array(dataset.image_code)\n        ,'avg_ad_days':np.array(dataset.avg_days_up_user )\n        ,'avg_ad_times':np.array(dataset.avg_times_up_user)\n        ,'n_user_items':np.array(dataset.n_user_items)\n        ,'price': np.array(dataset[[\"price\"]])\n        ,'item_seq_number': np.array(dataset[[\"item_seq_number\"]])\n    }\n    \n    print(\"Data ready for Vectorization\")\n    \n    return X\n","execution_count":3,"outputs":[]},{"metadata":{"_uuid":"1847057fa535a6fdddaf1c1cc05245734c0f0ae6","_cell_guid":"4ec9e489-bc16-40e7-b40f-b8b00e6c1d6f","trusted":true,"collapsed":true},"cell_type":"code","source":"# Loading Train data - No Params, No Image data \ndtypes_train = {\n                'price': 'float32',\n                'deal probability': 'float32',\n                'item_seq_number': 'uint32'\n}\n\n# No user_id\nuse_cols = ['item_id', 'user_id', 'image_top_1', 'region', 'city', 'parent_category_name', 'category_name', 'param_1', 'param_2', 'param_3', 'title', 'description', 'price', 'item_seq_number', 'activation_date', 'deal_probability']\ntrain = pd.read_csv(\"../input/avito-demand-prediction/train.csv\", parse_dates=[\"activation_date\"], usecols = use_cols, dtype = dtypes_train)\n\ntrain_features = pd.read_csv('../input/avito-additional-features/aggregated_features.csv')\ntrain = train.merge(train_features, on = ['user_id'], how = 'left')\ndel train_features\ngc.collect()\n\ntrain['avg_days_up_user'] = train['avg_days_up_user'].fillna(0).astype('uint32')\ntrain['avg_times_up_user'] = train['avg_times_up_user'].fillna(0).astype('uint32')\ntrain['n_user_items'] = train['n_user_items'].fillna(0).astype('uint32')\n\ny_train = np.array(train['deal_probability'])\n\ndel train['deal_probability']\ngc.collect()\n\nmax_seq_title_description_length = 100\nmax_words_title_description = 200000\n\ntrain = preprocess_dataset(train)\ntrain, tokenizer, le_region, le_city, le_category_name, le_parent_category_name, le_param_1, le_param123, le_image_code = keras_fit(train)\ntrain = keras_train_transform(train)\nprint(\"Tokenization done and TRAIN READY FOR Validation splitting\")\n\n# Calculation of max values for Categorical fields \n\nmax_region = np.max(train.region.max())+2\nmax_city= np.max(train.city.max())+2\nmax_category_name = np.max(train.category_name.max())+2\nmax_parent_category_name = np.max(train.parent_category_name.max())+2\nmax_param_1 = np.max(train.param_1.max())+2\nmax_param123 = np.max(train.param123.max())+2\n#max_week = np.max(train.week.max())+2\n#max_day = np.max(train.day.max())+2\n#max_wday = np.max(train.wday.max())+2\nmax_image_code = np.max(train.image_code.max())+2\n\n\ndel train['item_id'], train['user_id']\ngc.collect()\n","execution_count":4,"outputs":[]},{"metadata":{"_uuid":"b54eb26f15fb052fb774f5e755654536b1569ff3"},"cell_type":"markdown","source":"# EMBEDDING FILE - FASTTEXT \n\n* A 300 Dimension Fast Text Vector is used \n* Calculating the vocab_size from tokenizer is very important. The values are used in the Embedding in the RNNmodel. \n* For every word available in the data set the corresponding 300 D vector is added to the embedding matrix \n    - The embedding matrix is of size vocab_size, embedding size which in this case is 300. \n    - The idea is that the index of a word in the vocabulary also becomes the row number for that word in the matrix \n    - This way we can easily retrieve the word vector of any word \n* For words not available in the data set the vector value is assigned as zero. \n* One can also try replacing the zero vectors with average of available vectors or a random vector using np.random\n* We keep track of the number of words in the dataset for which word vectors are available \n\nHOW TO USE THESE EMBEDDINGS \n\nThe embedding layer can be used in two ways \n\nTrainable = True :  The initial word vector values are trained further in the RNN training stage. Takes a very long time \nTrainable = False:  The parameters for title_description are not trained and are kept at the same value through all the epochs of the RNN training process. "},{"metadata":{"_uuid":"323f100f2ab9ac36f586eba48ccec7a2f0a784f2","_cell_guid":"8b07a2cb-4208-4a18-98c7-e60a717b5cd9","trusted":true,"collapsed":true},"cell_type":"code","source":"# EMBEDDINGS COMBINATION \n# FASTTEXT\n\nEMBEDDING_DIM1 = 300\nEMBEDDING_FILE1 = '../input/fasttest-common-crawl-russian/cc.ru.300.vec'\ndef get_coefs(word, *arr): return word, np.asarray(arr, dtype='float32')\nembeddings_index1 = dict(get_coefs(*o.rstrip().rsplit(' ')) for o in open(EMBEDDING_FILE1))\n\nvocab_size = len(tokenizer.word_index)+2\nEMBEDDING_DIM1 = 300# this is from the pretrained vectors\nembedding_matrix1 = np.zeros((vocab_size, EMBEDDING_DIM1))\nprint(embedding_matrix1.shape)\n# Creating Embedding matrix \nc = 0 \nc1 = 0 \nw_Y = []\nw_No = []\nfor word, i in tokenizer.word_index.items():\n    if word in embeddings_index1:\n        c +=1\n        embedding_vector = embeddings_index1[word]\n        w_Y.append(word)\n    else:\n        embedding_vector = None\n        w_No.append(word)\n        c1 +=1\n    if embedding_vector is not None:    \n        embedding_matrix1[i] = embedding_vector\n\nprint(c,c1, len(w_No), len(w_Y))\nprint(embedding_matrix1.shape)\ndel embeddings_index1\ngc.collect()\n\nprint(\" FAST TEXT DONE\")\n","execution_count":5,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b878b98bb6bbb1ee3aacde385726772e3c0a892e","collapsed":true},"cell_type":"code","source":"print(vocab_size)","execution_count":6,"outputs":[]},{"metadata":{"_uuid":"375bc5a815c4a0766c383fc5abce285feb590855"},"cell_type":"markdown","source":"# GENERATE RNN MODEL \n\n* Initialize the inputs for all the variables. The word sequences for title_description have a length of 100 and other variables have a length of 1 \n* An embedding layer is generated for each of the Categorical and Text variables \n* NO Embeddings are required for continous variable like price \n* A Recurrent Neural network of 50 GRU units is applied to the embedding of title_description \n* The embeddings of the categorical variables are Flattened using the Flatten() command \n*  The GRU, Flatten values are concatenated with the continuous values and treated as the main layer \n*  This main layer is then passed through 2 Dense layers of 512 layers and 64 layers \n  \nWHAT CAN BE TUNED \n\n* Architecture of the network itself.  Bidirectional GRU's with Batch Normalization is worth a shot. \n* Learning rate \n* Number of Dense layers and the number of hidden units in each layer \n* Dropout values \n* The optimizer function - Adam is currently used. Other options are available \n* Number of GRU units \n\n\n\n"},{"metadata":{"_uuid":"e74c68e6105ce951262e9f50b7da365660e6c59e","_cell_guid":"c0a3837a-e7ca-4618-a672-ba40210d005f","collapsed":true,"trusted":true},"cell_type":"code","source":"def RNN_model():\n\n    #Inputs\n    seq_title_description = Input(shape=[100], name=\"seq_title_description\")\n    region = Input(shape=[1], name=\"region\")\n    city = Input(shape=[1], name=\"city\")\n    category_name = Input(shape=[1], name=\"category_name\")\n    parent_category_name = Input(shape=[1], name=\"parent_category_name\")\n    param_1 = Input(shape=[1], name=\"param_1\")\n    param123 = Input(shape=[1], name=\"param123\")\n    image_code = Input(shape=[1], name=\"image_code\")\n    price = Input(shape=[1], name=\"price\")\n    item_seq_number = Input(shape = [1], name = 'item_seq_number')\n    avg_ad_days = Input(shape=[1], name=\"avg_ad_days\")\n    avg_ad_times = Input(shape=[1], name=\"avg_ad_times\")\n    n_user_items = Input(shape=[1], name=\"n_user_items\")\n    \n    #Embeddings layers\n\n    emb_seq_title_description = Embedding(vocab_size, EMBEDDING_DIM1, weights = [embedding_matrix1], trainable = False)(seq_title_description)\n    emb_region = Embedding(vocab_size, 10)(region)\n    emb_city = Embedding(vocab_size, 10)(city)\n    emb_category_name = Embedding(vocab_size, 10)(category_name)\n    emb_parent_category_name = Embedding(vocab_size, 10)(parent_category_name)\n    emb_param_1 = Embedding(vocab_size, 10)(param_1)\n    emb_param123 = Embedding(vocab_size, 10)(param123)\n    emb_image_code = Embedding(vocab_size, 10)(image_code)\n\n    rnn_layer1 = GRU(50) (emb_seq_title_description)\n    \n    #main layer\n    main_l = concatenate([\n          rnn_layer1\n        , Flatten() (emb_region)\n        , Flatten() (emb_city)\n        , Flatten() (emb_category_name)\n        , Flatten() (emb_parent_category_name)\n        , Flatten() (emb_param_1)\n        , Flatten() (emb_param123)\n        , Flatten() (emb_image_code)\n        , avg_ad_days\n        , avg_ad_times\n        , n_user_items\n        , price\n        , item_seq_number\n    ])\n    \n    main_l = Dropout(0.1)(Dense(512,activation='relu') (main_l))\n    main_l = Dropout(0.1)(Dense(64,activation='relu') (main_l))\n    \n    #output\n    output = Dense(1,activation=\"sigmoid\") (main_l)\n    \n    #model\n    model = Model([seq_title_description, region, city, category_name, parent_category_name, param_1, param123, price, item_seq_number, image_code, avg_ad_days, avg_ad_times, n_user_items], output)\n    model.compile(optimizer = 'adam',\n                  loss= root_mean_squared_error,\n                  metrics = [root_mean_squared_error])\n    return model\n\ndef rmse(y, y_pred):\n\n    Rsum = np.sum((y - y_pred)**2)\n    n = y.shape[0]\n    RMSE = np.sqrt(Rsum/n)\n    return RMSE \n\ndef eval_model(model, X_test1):\n    val_preds = model.predict(X_test1)\n    y_pred = val_preds[:, 0]\n    \n    y_true = np.array(y_test1)\n    \n    yt = pd.DataFrame(y_true)\n    yp = pd.DataFrame(y_pred)\n    \n    print(yt.isnull().any())\n    print(yp.isnull().any())\n    \n    v_rmse = rmse(y_true, y_pred)\n    print(\" RMSE for VALIDATION SET: \"+str(v_rmse))\n    return v_rmse\n\nexp_decay = lambda init, fin, steps: (init/fin)**(1/(steps-1)) - 1","execution_count":7,"outputs":[]},{"metadata":{"_uuid":"3a7ff77f674ce78eb2183bfa85e875c0b440ff5e"},"cell_type":"markdown","source":"# PREDICTING THE OUTPUT \n\n* A function is defined with parameter as the trained modelRNN\n* Test data is loaded in chunks to AVOID MEMORY OVERLOAD \n* Test data is preprocessed \n* Transformations are applied using Keras word token, label encoders and the train_active based features \n* Generate dictionary for Test \n* Predict the output "},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"dc5ded2811f4059b015dc87851758e8b563448de"},"cell_type":"code","source":"def predictions(model):\n    import time\n    t1 = time.time()\n    def load_test():\n        for df in pd.read_csv('../input/avito-demand-prediction/test.csv', chunksize= 250000):\n            yield df\n\n    item_ids = np.array([], dtype=np.int32)\n    preds= np.array([], dtype=np.float32)\n\n    i = 0 \n    \n    for df in load_test():\n    \n        i +=1\n        print(df.dtypes)\n        item_id = df['item_id']\n        print(\" Chunk number is \"+str(i))\n    \n        test = preprocess_dataset(df)\n    \n        train_features = pd.read_csv('../input/avito-additional-features/aggregated_features.csv')\n        test = test.merge(train_features, on = ['user_id'], how = 'left')\n        del train_features\n        gc.collect()\n    \n        print(test.dtypes)\n        \n        test['avg_days_up_user'] = test['avg_days_up_user'].fillna(0).astype('uint32')\n        test['avg_times_up_user'] = test['avg_times_up_user'].fillna(0).astype('uint32')\n        test['n_user_items'] = test['n_user_items'].fillna(0).astype('uint32')\n        test = keras_test_transform(test)\n        del df\n        gc.collect()\n    \n        print(test.dtypes)\n    \n        X_test = get_keras_data(test)\n        del test \n        gc.collect()\n    \n        Batch_Size = 512*3\n        preds1 = modelRNN.predict(X_test, batch_size = Batch_Size, verbose = 1)\n        print(preds1.shape)\n        del X_test\n        gc.collect()\n        print(\"RNN Prediction is done\")\n\n        preds1 = preds1.reshape(-1,1)\n        #print(predsl.shape)\n        preds1 = np.clip(preds1, 0, 1)\n        print(preds1.shape)\n        item_ids = np.append(item_ids, item_id)\n        print(item_ids.shape)\n        preds = np.append(preds, preds1)\n        print(preds.shape)\n        \n    print(\"All chunks done\")\n    t2 = time.time()\n    print(\"Total time for Parallel Batch Prediction is \"+str(t2-t1))\n    return preds ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"49a76eb235db7a9f193883afbb2721e4da29bbb8"},"cell_type":"markdown","source":"# PROCESSING FOR KFOLD \n\n* K Fold accepts only arrays as inputs \n* The train data frame is converted into array to generate the indexes for KFOLD \n* The array is then used to reconstruct a data frame with which a dictionary is generated \n* Dictionary is the best input type for model.fit in Keras "},{"metadata":{"trusted":true,"_uuid":"fe6ae6ae86c45274d91b0915afe4434dce009a86","collapsed":true},"cell_type":"code","source":"train1 = np.array(train.values)\ndel train\ngc.collect()\n\ndef get_data_frame(dataset):\n    \n    DF = pd.DataFrame()\n    \n    DF['avg_days_up_user'] = np.array(dataset[:,0])\n    DF['avg_times_up_user'] = np.array(dataset[:,1])\n    DF['category_name'] = np.array(dataset[:,2])\n    DF['city'] = np.array(dataset[:,3])\n    DF['image_code'] = np.array(dataset[:,4])\n    DF['item_seq_number'] = np.array(dataset[:,5])\n    DF['n_user_items'] = np.array(dataset[:,6])\n    DF['param123'] = np.array(dataset[:,7])\n    DF['param_1'] = np.array(dataset[:,8])\n    DF['parent_category_name'] = np.array(dataset[:,9])\n    DF['price'] = np.array(dataset[:,10])\n    DF['region'] = np.array(dataset[:,11])\n    DF['seq_title_description'] = np.array(dataset[:,12])\n    \n    return DF ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2484fc26cc0ecc6db58690d2b99c1feff53ccb95","_cell_guid":"c24f4ac2-e862-42b1-82ae-8d7ba4bfc73f","trusted":true,"collapsed":true},"cell_type":"code","source":"from sklearn.model_selection import train_test_split\nfrom sklearn.model_selection import KFold\nimport time \nskf = KFold(n_splits = 3)\nKfold_preds_final = []\nk = 0\nRMSE = []\n\nfor train_idx, test_idx in skf.split(train1, y_train):\n    \n    print(\"Number of Folds..\"+str(k+1))\n    \n    # Initialize a new Model for Current FOLD \n    epochs = 1\n    batch_size = 512 * 3\n    steps = (int(train1.shape[0]/batch_size))*epochs\n    lr_init, lr_fin = 0.009, 0.0045\n    lr_decay = exp_decay(lr_init, lr_fin, steps)\n    modelRNN = RNN_model()\n    K.set_value(modelRNN.optimizer.lr, lr_init)\n    K.set_value(modelRNN.optimizer.decay, lr_decay)\n\n    #K Fold Split \n    \n    X_train1, X_test1 = train1[train_idx], train1[test_idx]\n    print(X_train1.shape, X_test1.shape)\n    y_train1, y_test1 = y_train[train_idx], y_train[test_idx]\n    print(y_train1.shape, y_test1.shape)\n    gc.collect()\n    \n    print(type(X_train1))\n    print(X_train1.shape)\n    print(type(X_train1[:,12]))\n    \n    X_train_final = get_data_frame(X_train1)\n    X_test_final = get_data_frame(X_test1)\n    \n    del X_train1, X_test1\n    gc.collect()\n    \n    X_train_f = get_keras_data(X_train_final)\n    X_test_f = get_keras_data(X_test_final)\n    \n    del X_train_final, X_test_final\n    gc.collect()\n\n    # Fit the NN Model \n    for i in range(3):\n        hist = modelRNN.fit(X_train_f, y_train1, batch_size=batch_size+(batch_size*(2*i)), epochs=epochs, validation_data=(X_test_f, y_test1), verbose=1)\n\n    del X_train_f\n    gc.collect()\n\n    # Print RMSE for Validation set for Kth Fold \n    v_rmse = eval_model(modelRNN, X_test_f)\n    RMSE.append(v_rmse)\n    \n    del X_test_f\n    del y_train1, y_test1\n    gc.collect()\n    \n    # Predict test set for Kth Fold \n    preds = predictions(modelRNN)\n    del modelRNN \n    gc.collect()\n\n    print(\"Predictions done for Fold \"+str(k))\n    print(preds.shape)\n    Kfold_preds_final.append(preds)\n    del preds\n    gc.collect()\n    print(\"Number of folds completed....\"+str(len(Kfold_preds_final)))\n    print(Kfold_preds_final[k][0:10])\n\nprint(\"All Folds completed\"+str(k+1))   \nprint(\"RNN FOLD MODEL Done\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2aff2cd91c603e7d2e0f281835806fe591635b02"},"cell_type":"markdown","source":"# SELECTING KFOLD OUTPUT \n\n*  Average of all Kfold predictions is catpured in pred_final1 \n*    The KFOLD run with least RMSE score is identified and the corresponding output is taken as pred_final2 \n*    2 Separate output files are generated for comparison "},{"metadata":{"trusted":true,"_uuid":"31e33368c69a455ba099e341aad1ac8bf6975ab1","collapsed":true},"cell_type":"code","source":"pred_final1 = np.average(Kfold_preds_final, axis =0) # Average of all K Folds\nprint(pred_final1.shape)\n\nmin_value = min(RMSE)\nRMSE_idx = RMSE.index(min_value)\nprint(RMSE_idx)\npred_final2 = Kfold_preds_final[RMSE_idx]\nprint(pred_final2.shape)\n\n#del Kfold_preds_final, train1\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"18a13c16b16ed8eed472abb512a26b1474332e79","collapsed":true},"cell_type":"code","source":"pred_final1[0:5]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5e52defb29957170cd83926035df484b36f098d4","collapsed":true},"cell_type":"code","source":"pred_final2[0:5]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"22eb2aa29ced6806f75b62a677c60a2449577c7d","_cell_guid":"e6833179-8039-44d9-867b-553c2d32dea3","trusted":true,"collapsed":true},"cell_type":"code","source":"test_cols = ['item_id']\ntest = pd.read_csv('../input/avito-demand-prediction/test.csv', usecols = test_cols)\n\n# using Average of KFOLD preds \n\nsubmission1 = pd.DataFrame( columns = ['item_id', 'deal_probability'])\n\nsubmission1['item_id'] = test['item_id']\nsubmission1['deal_probability'] = pred_final1\n\nprint(\"Check Submission NOW!!!!!!!!@\")\nsubmission1.to_csv(\"Avito_Shanth_RNN_AVERAGE.csv\", index=False)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"607052770f9d7670e586f7b9043638f9d96f56d4","collapsed":true},"cell_type":"code","source":"# Using KFOLD preds with Minimum value \nsubmission2 = pd.DataFrame( columns = ['item_id', 'deal_probability'])\n\nsubmission2['item_id'] = test['item_id']\nsubmission2['deal_probability'] = pred_final2\n\nprint(\"Check Submission NOW!!!!!!!!@\")\nsubmission2.to_csv(\"Avito_Shanth_RNN_MIN.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"99af851c68c6a60e25ea5ceea1ba61eaa491fa14"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.5","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}