{"cells":[{"metadata":{"_uuid":"1f9c7b896ad9b3f9ee75e3451b128b949bdc39c3"},"cell_type":"markdown","source":"## Using self-trained embeddings from train_active on the description\n\nThis kernel shows how to use the Word2Vec model created in [this kernel](https://www.kaggle.com/christofhenkel/using-train-active-for-training-word-embeddings) on the description. To compare the performance with [the pre-trained embedding model](https://www.kaggle.com/christofhenkel/fasttext-starter-description-only) we use exactly the same model structure."},{"metadata":{"_cell_guid":"cd154f8d-298b-4c43-b244-abe4a3b4b72e","_uuid":"cbad1e4cb69e82f118cf32da7880edbc90098a31","trusted":true},"cell_type":"code","source":"import pandas as pd\nfrom gensim.models import word2vec\nfrom keras.preprocessing import text, sequence\nimport numpy as np\nfrom tqdm import tqdm\nfrom keras.layers import Input, SpatialDropout1D,Dropout, GlobalAveragePooling1D, CuDNNGRU, Bidirectional, Dense, Embedding\nfrom keras.models import Model\nfrom keras.optimizers import Adam\nfrom keras.callbacks import ModelCheckpoint\nimport keras.backend as K\nimport numpy as np\nfrom sklearn import metrics\nfrom sklearn.model_selection import train_test_split\nimport os\n\n\nEMBEDDING = '../input/using-train-active-for-training-word-embeddings/avito.w2v'\nTRAIN_CSV = '../input/avito-demand-prediction/train.csv'\nTEST_CSV = '../input/avito-demand-prediction/test.csv'\n","execution_count":1,"outputs":[]},{"metadata":{"_cell_guid":"1c996a20-0524-4689-b62a-245f7940a67a","_uuid":"965cc2439216eda704f32e677f947a03977ad43d","trusted":true},"cell_type":"code","source":"max_features = 100000\nmaxlen = 100\nembed_size = 100\ntrain = pd.read_csv(TRAIN_CSV, index_col = 0)\nlabels = train[['deal_probability']].copy()\ntrain = train[['description']].copy()\n\ntokenizer = text.Tokenizer(num_words=max_features)\n\nprint('fitting tokenizer...',end='')\ntrain['description'] = train['description'].astype(str)\ntokenizer.fit_on_texts(list(train['description'].fillna('NA').values))\nprint('done.')","execution_count":2,"outputs":[]},{"metadata":{"_cell_guid":"0cde366b-2bfc-4217-ab3d-2c265b3a1319","_uuid":"1a401fc2a67f34413fbf8f7bc26725a3558056c6","trusted":true},"cell_type":"code","source":"model = word2vec.Word2Vec.load(EMBEDDING)\nword_index = tokenizer.word_index\nnb_words = min(max_features, len(word_index))\nembedding_matrix = np.zeros((nb_words, embed_size))\nfor word, i in word_index.items():\n    if i >= max_features: continue\n    try:\n        embedding_vector = model[word]\n    except KeyError:\n        embedding_vector = None\n    if embedding_vector is not None: embedding_matrix[i] = embedding_vector","execution_count":3,"outputs":[]},{"metadata":{"_cell_guid":"eba90731-c158-4ff7-89cd-7f9078d94d89","_uuid":"dd1642767ad516f431ac59f28ab0919bb34cc6ec","trusted":true},"cell_type":"code","source":"X_train, X_valid, y_train, y_valid = train_test_split(train['description'].values, labels['deal_probability'].values, test_size = 0.1, random_state = 23)\n\nprint('convert to sequences...',end='')\nX_train = tokenizer.texts_to_sequences(X_train)\nX_valid = tokenizer.texts_to_sequences(X_valid)\nprint('done.')\nprint('padding...',end='')\nX_train = sequence.pad_sequences(X_train, maxlen=maxlen)\nX_valid = sequence.pad_sequences(X_valid, maxlen=maxlen)\nprint('done.')\n\ndel train","execution_count":4,"outputs":[]},{"metadata":{"_uuid":"59e0f0861d480e994e250509f3443ed57b7474c9"},"cell_type":"markdown","source":"Lets define the model and illustrate the architecture"},{"metadata":{"_cell_guid":"caa1e9b6-21cc-469d-9093-e55e7bcc041c","_uuid":"6f04c12c882dc77d549b4e8b3536fde6bd750319","trusted":true,"_kg_hide-input":false,"_kg_hide-output":false},"cell_type":"code","source":"def root_mean_squared_error(y_true, y_pred):\n    return K.sqrt(K.mean(K.square(y_pred - y_true)))\n\ndef build_model():\n    inp = Input(shape = (maxlen, ))\n    emb = Embedding(nb_words, embed_size, weights = [embedding_matrix],\n                    input_length = maxlen, trainable = False)(inp)\n    main = SpatialDropout1D(0.2)(emb)\n    main = Bidirectional(CuDNNGRU(32,return_sequences = True))(main)\n    main = GlobalAveragePooling1D()(main)\n    main = Dropout(0.2)(main)\n    out = Dense(1, activation = \"sigmoid\")(main)\n\n    model = Model(inputs = inp, outputs = out)\n\n    model.compile(optimizer = Adam(lr=0.001), loss = 'mean_squared_error',\n                  metrics =[root_mean_squared_error])\n    \n    return model\n\nmodel = build_model()\nmodel.summary()","execution_count":7,"outputs":[]},{"metadata":{"_uuid":"fcb3ffa301fac8fe3faf380aa48c4fd09647bb14"},"cell_type":"markdown","source":"Lets train our model for four epochs and save the best epoch."},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"EPOCHS = 4\nfile_path = \"model.hdf5\"\n\ncheck_point = ModelCheckpoint(file_path, monitor = \"val_loss\", mode = \"min\", save_best_only = True, verbose = 1)\nhistory = model.fit(X_train, y_train, batch_size = 256, epochs = EPOCHS, validation_data = (X_valid, y_valid),\n                verbose = 1, callbacks = [check_point])","execution_count":8,"outputs":[]},{"metadata":{"_cell_guid":"382825ef-434e-4f9e-8a63-4af894ced750","_uuid":"2d91fb63f6b08a3b488bc8517088d4a8c923519c","trusted":true,"_kg_hide-input":false,"_kg_hide-output":false},"cell_type":"code","source":"model.load_weights(file_path)\nprediction = model.predict(X_valid)\nprint('RMSE:', np.sqrt(metrics.mean_squared_error(y_valid, prediction)))","execution_count":9,"outputs":[]},{"metadata":{"_uuid":"cb596e1c0df5ef051c7f8ac3700fd5aa08ff793c"},"cell_type":"markdown","source":"Thats some improvement compared to using  [the pre-trained embedding model](https://www.kaggle.com/christofhenkel/fasttext-starter-description-only) which scored 0.2370. Additionally since the embeddings here are trained also on param_1, param_2, param_3 and title which have much more out of vocabulary words when using Fasttext. Hence self-trained embeddings are clearly performing better."},{"metadata":{"_uuid":"5e743df07753b5b8094739842736f26288fc6381"},"cell_type":"markdown","source":"Ok, now we are ready to do a submission and compare the LB score with [the pre-trained embedding model](https://www.kaggle.com/christofhenkel/fasttext-starter-description-only)."},{"metadata":{"_cell_guid":"04a70f03-f676-4081-b7a6-ef0dae1c048b","collapsed":true,"_uuid":"d79d956c4f47381479a397069fe06f0f5a7c49b0","trusted":false},"cell_type":"code","source":"test = pd.read_csv(TEST_CSV, index_col = 0)\ntest = test[['description']].copy()\n\ntest['description'] = test['description'].astype(str)\nX_test = test['description'].values\nX_test = tokenizer.texts_to_sequences(X_test)\n\nprint('padding')\nX_test = sequence.pad_sequences(X_test, maxlen=maxlen)\nprediction = model.predict(X_test,batch_size = 128, verbose = 1)\n\nsample_submission = pd.read_csv('../input/avito-demand-prediction/sample_submission.csv', index_col = 0)\nsubmission = sample_submission.copy()\nsubmission['deal_probability'] = prediction\nsubmission.to_csv('submission.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"dfa1edc4a6531d84ba560d3cc1838e15b53177c9"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"}},"nbformat":4,"nbformat_minor":1}