{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Отборочное задание №2 на стажировку в команду Core ML VK к Семёну Полякову: \nЦель — научиться вычислять эмбеддинги текстов.\nНеобходимо сделать модель машинного обучения, вычисляющую эмбеддинги текста (векторное представление фиксированной длины), используя готовые наработки либо обучив её самостоятельно. Вид векторного представления может быть произвольным, но каждый из эмбеддингов не должен содержать более 1000 чисел.\n\nПосле чего нужно по датасету вопросов Quora Question Pairs (https://www.kaggle.com/c/quora-question-pairs/) обучить модель, возвращающую вероятность похожести пар вопросов только по паре их эмбеддингов. Модель должна выдавать на тестовой выборке в публичном лидерборде kaggle logloss не больше 0,5. \n\nВ качестве результата необходимо предоставить код обучения моделей и формирования итогового файла с результатами, код вычисления эмбеддингов, а также указать ссылку на свой профиль в kaggle. Кроме logloss будет учитываться сложность и эффективность представленного решения.\n\n## Решает задачу:\nЛисоветин Никита Валерьевич ([github](https://github.com/turing228), [vk](https://vk.com/nikitalisovetin))"},{"metadata":{},"cell_type":"markdown","source":"# What did I read before?\nhttps://www.nkj.ru/open/36052/\n\nhttps://habr.com/ru/company/ods/blog/329410/\n\nhttps://habr.com/ru/company/ods/blog/328372/\n\nhttps://stats.stackexchange.com/questions/153531/what-is-batch-size-in-neural-network\n\n\n# What do I want to implement? Usefull links\n\nhttps://keras.io/optimizers/\n\nhttps://www.pyimagesearch.com/2019/02/04/keras-multiple-inputs-and-mixed-data/\n\nhttps://www.kaggle.com/colinmorris/embedding-layers\n\nhttps://www.kaggle.com/sudalairajkumar/a-look-at-different-embeddings\n\nhttps://www.kaggle.com/shujian/blend-of-lstm-and-cnn-with-4-embeddings-1200d\n\nhttps://www.kaggle.com/christofhenkel/how-to-preprocessing-when-using-embeddings\n\nhttps://www.kaggle.com/shujian/different-embeddings-with-attention-fork-fork\n\nhttps://www.kaggle.com/sbongo/do-pretrained-embeddings-give-you-the-extra-edge\n\nhttps://www.kaggle.com/rajmehra03/a-detailed-explanation-of-keras-embedding-layer\n\nhttps://www.kaggle.com/antmarakis/bidirectional-gru-model\n\nhttps://www.kaggle.com/anokas/data-analysis-xgboost-starter-0-35460-lb\n\nhttps://www.kaggle.com/arathee2/random-forest-vs-xgboost-vs-deep-neural-network"},{"metadata":{},"cell_type":"markdown","source":"# What have I done?\n\n1. Попробовал вариант со своим эмбеддингом\n2. Сравнил с предобученными эмбеддингами\n3. Научился обучать Keras с несколькими input'ами\n4. Выбрал наилучший оптимизатор решения\n5. Сделал препроцессинг данных, но результаты сильно ухудшились, поэтому убрал его\n6. Проверил одну версию решения, которая работала 2 часа. С учетом сильного переобучения (в два-три раза в сравнении с двумя эпохами), результат получится 0.7 logloss"},{"metadata":{"_uuid":"01f361ddc47e0b386595316fe3d7f4dabbd260db"},"cell_type":"markdown","source":"**Notebook Objective:**\n\nObjective of the notebook is to look at the different pretrained embeddings provided in the dataset and to see how they are useful in the model building process. \n\nFirst let us import the necessary modules and read the input data."},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":true},"cell_type":"code","source":"import os\nimport time\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nfrom tqdm import tqdm\nimport math\nfrom sklearn.model_selection import train_test_split\nfrom sklearn import metrics\n\nfrom keras.preprocessing.text import Tokenizer\nfrom keras.preprocessing.sequence import pad_sequences\nfrom keras.layers import Dense, Input, LSTM, Embedding, Dropout, Activation, CuDNNGRU, Conv1D\nfrom keras.layers import Bidirectional, GlobalMaxPool1D\nfrom keras.models import Model\nfrom keras import initializers, regularizers, constraints, optimizers, layers","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"train_df = pd.read_csv(\"../input/quora-question-pairs/train.csv\")\ntest_df = pd.read_csv(\"../input/quora-question-pairs/test.csv\")\nprint(\"Train shape : \",train_df.shape)\nprint(\"Test shape : \",test_df.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## split to train and val\ndf_train, df_val = train_test_split(train_df, test_size=0.0001, random_state=2019)\ndf_test = test_df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\nimport re\nimport csv\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n# import codecs\n\nfrom string import punctuation\nfrom collections import defaultdict\n# from tqdm import tqdm\n\nfrom sklearn.preprocessing import StandardScaler\n\nfrom nltk.corpus import stopwords\nfrom nltk.stem import SnowballStemmer\n\nfrom keras.preprocessing.text import Tokenizer\nfrom keras.preprocessing.sequence import pad_sequences\nfrom keras.layers import Dense, Input, Embedding, Dropout, Activation, LSTM, Lambda\nfrom keras.layers.merge import concatenate\nfrom keras.models import Model\nfrom keras.layers.normalization import BatchNormalization\nfrom keras.callbacks import EarlyStopping, ModelCheckpoint\n# from keras.layers.convolutional import Conv1D\nfrom keras.layers.pooling import GlobalAveragePooling1D\nimport keras.backend as K\n\n\nMax_Sequence_Length = 60\nMax_Num_Words = 200000 # There are about 201000 unique words in training dataset, 200000 is enough for tokenization\nEmbedding_Dim = 300\nValidation_Split_Ratio = 0.1\nEMBEDDING_FILE = '../input/quora-insincere-questions-classification/embeddings/glove.840B.300d/glove.840B.300d.txt'\n\nact_f = 'relu'\nre_weight = True # whether to re-weight classes to fit the 17.4% share in test set\n\nembeddings_index = {}\nf = open(EMBEDDING_FILE, encoding='utf-8')\n\n# for line in tqdm(f):\nfor line in f:\n    values = line.split()\n    # word = values[0]\n    word = ''.join(values[:-300])   \n    # coefs = np.asarray(values[1:], dtype='float32')\n    coefs = np.asarray(values[-300:], dtype='float32')\n    embeddings_index[word] = coefs\nf.close()\n\nNum_Lstm = np.random.randint(175, 275)\nNum_Dense = np.random.randint(100, 150)\nRate_Drop_Lstm = 0.15 + np.random.rand() * 0.25\nRate_Drop_Dense = 0.15 + np.random.rand() * 0.25\nprint('Processing text dataset')\n\ndef text_to_wordlist(text, remove_stopwords=False, stem_words=False):\n    # Clean the text, with the option to remove stopwords and to stem words.\n    \n    # Convert words to lower case and split them\n    text = text.lower().split()\n\n    # Optionally, remove stop words\n    if remove_stopwords:\n        stop_words = set(stopwords.words(\"english\"))\n        text = [w for w in text if not w in stop_words]\n    \n    text = \" \".join(text)\n    \n    # Remove punctuation from text\n    # text = \"\".join([c for c in text if c not in punctuation])\n\n    # Clean the text\n    text = re.sub(r\"[^A-Za-z0-9^,!.\\/'+-=]\", \" \", text)\n    text = re.sub(r\"what's\", \"what is \", text)\n    text = re.sub(r\"\\'s\", \" \", text)\n    text = re.sub(r\"\\'ve\", \" have \", text)\n    text = re.sub(r\"can't\", \"cannot \", text)\n    text = re.sub(r\"n't\", \" not \", text)\n    text = re.sub(r\"i'm\", \"i am \", text)\n    text = re.sub(r\"\\'re\", \" are \", text)\n    text = re.sub(r\"\\'d\", \" would \", text)\n    text = re.sub(r\"\\'ll\", \" will \", text)\n    text = re.sub(r\",\", \" \", text)\n    text = re.sub(r\"\\.\", \" \", text)\n    text = re.sub(r\"!\", \" ! \", text)\n    text = re.sub(r\"\\/\", \" \", text)\n    text = re.sub(r\"\\^\", \" ^ \", text)\n    text = re.sub(r\"\\+\", \" + \", text)\n    text = re.sub(r\"\\-\", \" - \", text)\n    text = re.sub(r\"\\=\", \" = \", text)\n    text = re.sub(r\"'\", \" \", text)\n    text = re.sub(r\":\", \" : \", text)\n    text = re.sub(r\"(\\d+)(k)\", r\"\\g<1>000\", text)\n    text = re.sub(r\" e g \", \" eg \", text)\n    text = re.sub(r\" b g \", \" bg \", text)\n    text = re.sub(r\" u s \", \" american \", text)\n    # text = re.sub(r\"\\0s\", \"0\", text) # It doesn't make sense to me\n    text = re.sub(r\" 9 11 \", \"911\", text)\n    text = re.sub(r\"e - mail\", \"email\", text)\n    text = re.sub(r\"j k\", \"jk\", text)\n    text = re.sub(r\"\\s{2,}\", \" \", text)\n    \n    # Optionally, shorten words to their stems\n    if stem_words:\n        text = text.split()\n        stemmer = SnowballStemmer('english')\n        stemmed_words = [stemmer.stem(word) for word in text]\n        text = \" \".join(stemmed_words)\n    \n    # Return a list of words\n    return(text)\n\n# load data and process with text_to_wordlist\ntrain_texts_1 = [] \ntrain_texts_2 = []\ntrain_labels = []\n\n#df_train = pd.read_csv(Train_Data_File, encoding='utf-8')\n# df_train = df_train.sample(5000) # train data sample to test code\ndf_train = df_train.fillna('empty')\ntrain_q1 = df_train.question1.values\ntrain_q2 = df_train.question2.values\ntrain_labels = df_train.is_duplicate.values\n\nfor text in train_q1:\n    train_texts_1.append(text_to_wordlist(text, remove_stopwords=False, stem_words=False))\n    \nfor text in train_q2:\n    train_texts_2.append(text_to_wordlist(text, remove_stopwords=False, stem_words=False))\n\n'''\nwith open(Train_Data_File, encoding='utf-8') as f:\n    reader = csv.reader(f, delimiter=',')\n    header = next(reader) # Skip header row\n    for values in reader:\n        train_texts_1.append(text_to_wordlist(values[3], remove_stopwords=False, stem_words=False))\n        train_texts_2.append(text_to_wordlist(values[4], remove_stopwords=False, stem_words=False))\n        train_labels.append(int(values[5]))\n'''\nprint('{} texts are found in train.csv'.format(len(train_texts_1)))\n\ntest_texts_1 = []\ntest_texts_2 = []\ntest_ids = []\n\n#df_test = pd.read_csv(Test_Data_File, encoding='utf-8')\n# df_test = df_test.sample(5000) # test data sample to test code\ndf_test = df_test.fillna('empty')\ntest_q1 = df_test.question1.values\ntest_q2 = df_test.question2.values\ntest_ids = df_test.test_id.values\n\n'''\nwith open(Test_Data_File, encoding='utf-8') as f:\n    reader = csv.reader(f, delimiter=',')\n    header = next(reader)\n    for values in reader:\n        test_texts_1.append(text_to_wordlist(values[1], remove_stopwords=False, stem_words=False))\n        test_texts_2.append(text_to_wordlist(values[2], remove_stopwords=False, stem_words=False))\n        test_ids.append(values[0])\n'''\n\nfor text in test_q1:\n    test_texts_1.append(text_to_wordlist(text, remove_stopwords=False, stem_words=False))\n    \nfor text in test_q2:\n    test_texts_2.append(text_to_wordlist(text, remove_stopwords=False, stem_words=False))\n    \nprint('{} texts are found in test.csv'.format(len(test_texts_1)))\n\n# Tokenize words in all sentences\ntokenizer = Tokenizer(num_words=Max_Num_Words)\ntokenizer.fit_on_texts(train_texts_1 + train_texts_2 + test_texts_1 + test_texts_2)\n\ntrain_sequences_1 = tokenizer.texts_to_sequences(train_texts_1)\ntrain_sequences_2 = tokenizer.texts_to_sequences(train_texts_2)\ntest_sequences_1 = tokenizer.texts_to_sequences(test_texts_1)\ntest_sequences_2 = tokenizer.texts_to_sequences(test_texts_2)\n\nword_index = tokenizer.word_index\nprint('{} unique tokens are found'.format(len(word_index)))\n\n# pad all train with Max_Sequence_Length\ntrain_data_1 = pad_sequences(train_sequences_1, maxlen=Max_Sequence_Length)\ntrain_data_2 = pad_sequences(train_sequences_2, maxlen=Max_Sequence_Length)\n# train_labels = np.array(train_labels)\nprint('Shape of train data tensor:', train_data_1.shape)\nprint('Shape of train labels tensor:', train_labels.shape)\n\n# pad all test with Max_Sequence_Length\ntest_data_1 = pad_sequences(test_sequences_1, maxlen=Max_Sequence_Length)\ntest_data_2 = pad_sequences(test_sequences_2, maxlen=Max_Sequence_Length)\n# test_ids = np.array(test_ids)\nprint('Shape of test data tensor:', test_data_2.shape)\nprint('Shape of test ids tensor:', test_ids.shape)\n\n# leaky features\n\nquestions = pd.concat([df_train[['question1', 'question2']], \\\n        df_test[['question1', 'question2']]], axis=0).reset_index(drop='index')\nq_dict = defaultdict(set)\nfor i in range(questions.shape[0]):\n        q_dict[questions.question1[i]].add(questions.question2[i])\n        q_dict[questions.question2[i]].add(questions.question1[i])\n\ndef q1_freq(row):\n    return(len(q_dict[row['question1']]))\n    \ndef q2_freq(row):\n    return(len(q_dict[row['question2']]))\n    \ndef q1_q2_intersect(row):\n    return(len(set(q_dict[row['question1']]).intersection(set(q_dict[row['question2']]))))\n\ndf_train['q1_q2_intersect'] = df_train.apply(q1_q2_intersect, axis=1, raw=True)\ndf_train['q1_freq'] = df_train.apply(q1_freq, axis=1, raw=True)\ndf_train['q2_freq'] = df_train.apply(q2_freq, axis=1, raw=True)\n\ndf_test['q1_q2_intersect'] = df_test.apply(q1_q2_intersect, axis=1, raw=True)\ndf_test['q1_freq'] = df_test.apply(q1_freq, axis=1, raw=True)\ndf_test['q2_freq'] = df_test.apply(q2_freq, axis=1, raw=True)\n\nleaks = df_train[['q1_q2_intersect', 'q1_freq', 'q2_freq']]\ntest_leaks = df_test[['q1_q2_intersect', 'q1_freq', 'q2_freq']]\n\nss = StandardScaler()\nss.fit(np.vstack((leaks, test_leaks)))\nleaks = ss.transform(leaks)\ntest_leaks = ss.transform(test_leaks)\n\n# Create embedding matrix for embedding layer\nprint('Preparing embedding matrix')\n\nnum_words = min(Max_Num_Words, len(word_index))+1\n\nembedding_matrix = np.zeros((num_words, Embedding_Dim))\nfor word, i in word_index.items():\n    embedding_vector = embeddings_index.get(word)\n    if embedding_vector is not None:\n        embedding_matrix[i] = embedding_vector\nprint('Null word embeddings: '.format(np.sum(np.sum(embedding_matrix, axis=1) == 0)))\n\n# Train Validation split\n# np.random.seed(2019)\nperm = np.random.permutation(len(train_data_1))\nidx_train = perm[:int(len(train_data_1)*(1-Validation_Split_Ratio))]\nidx_val = perm[int(len(train_data_1)*(1-Validation_Split_Ratio)):]\n\ndata_1_train = np.vstack((train_data_1[idx_train], train_data_2[idx_train]))\ndata_2_train = np.vstack((train_data_2[idx_train], train_data_1[idx_train]))\nleaks_train = np.vstack((leaks[idx_train], leaks[idx_train]))\nlabels_train = np.concatenate((train_labels[idx_train], train_labels[idx_train]))\n\ndata_1_val = np.vstack((train_data_1[idx_val], train_data_2[idx_val]))\ndata_2_val = np.vstack((train_data_2[idx_val], train_data_1[idx_val]))\nleaks_val = np.vstack((leaks[idx_val], leaks[idx_val]))\nlabels_val = np.concatenate((train_labels[idx_val], train_labels[idx_val]))\n\nweight_val = np.ones(len(labels_val))\nif re_weight:\n    weight_val *= 0.471544715\n    weight_val[labels_val==0] = 1.309033281","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"maxlen = Max_Sequence_Length\nmax_features = num_words\ndropout = 0.3\nepochs = 4\n\n# Each instance will consist of two inputs: a single question1, and a single question2\nquestion1_input = Input(shape=(maxlen,), name='question1')\nquestion2_input = Input(shape=(maxlen,), name='question2')\nquestion1_embedded = Embedding(max_features, Embedding_Dim, weights=[embedding_matrix], name='question1_embedded')(question1_input)\nquestion2_embedded = Embedding(max_features, Embedding_Dim, weights=[embedding_matrix], name='question2_embedded')(question2_input)\n\n# the first branch operates on the first input\nx = Bidirectional(CuDNNGRU(64, return_sequences=True))(question1_embedded)\nx = GlobalMaxPool1D()(x)\nx = Dense(16, activation=\"relu\")(x)\nx = Dropout(dropout)(x)\nx = Dense(1, activation=\"sigmoid\")(x)\nx = Model(inputs=question1_input, outputs=x)\n\n# the second branch opreates on the second input\ny = Bidirectional(CuDNNGRU(64, return_sequences=True))(question2_embedded)\ny = GlobalMaxPool1D()(y)\ny = Dense(16, activation=\"relu\")(y)\ny = Dropout(dropout)(y)\ny = Dense(1, activation=\"sigmoid\")(y)\ny = Model(inputs=question2_input, outputs=y)\n\n# combine the output of the two branches\ncombined = layers.concatenate([x.output, y.output])\n \n# apply a FC layer and then a regression prediction on the\n# combined outputs\nz = Dense(16, activation=\"relu\")(combined)\nz = Dropout(dropout)(z)\nz = Dense(1, activation=\"sigmoid\")(z)\n\nmodel = Model(\n    inputs = [question1_input, question2_input],\n    outputs = z,\n)\n\nmodel.compile(loss='binary_crossentropy', optimizer='Nadam', metrics=['accuracy'])\n\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model.fit([train_data_1, train_data_2], train_labels, batch_size=1024, epochs=epochs, validation_data=([data_1_val, data_2_val], labels_val))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pred_glove_val_y = model.predict([data_1_val, data_2_val], batch_size=1024, verbose=1)\nprint(\"log_loss score is {0}\".format(metrics.log_loss(labels_val, pred_glove_val_y)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pred_glove_test_y = model.predict([test_data_1, test_data_2], batch_size=2048, verbose=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pred_test_y = pred_glove_test_y\nout_df = pd.DataFrame({\"test_id\":test_df[\"test_id\"].values})\nout_df['is_duplicate'] = pred_test_y\nout_df.to_csv(\"submission.csv\", index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.6"}},"nbformat":4,"nbformat_minor":1}