{"cells":[{"metadata":{"_uuid":"c70fb99772fff3a1cf6dfb45f5d671623d5589a3"},"cell_type":"markdown","source":"
**Quaro虚假问题识别**
\n\n对于给定的问题，判断是否包含**政治敏感，色情，反动，种族歧视**等问题"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"markdown","source":"# 1. 数据探索和清洗\n\n- 文本分类任务首先要了解分为几类，各个类别之间的比例\n- 数据清洗：填充缺失值；针对中英文特别标点符号的处理；大小写的转换；数字的处理"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"import os \nimport time\nimport numpy as np\nimport pandas as pd\nfrom tqdm import tqdm\nprint(os.listdir(\"../input\"))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7f8aaa8160e120512c3e30dd7f4604d606111ee8"},"cell_type":"code","source":"# 读取需要处理的数据\ntrain_df = pd.read_csv(\"../input/train.csv\")\ntest_df = pd.read_csv(\"../input/test.csv\")\nprint(\"Train shape : \", train_df.shape)\nprint(\"Test shape : \", test_df.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c0e14880510b21a792dcdabdb8971e75d2e5dcbc"},"cell_type":"code","source":"train_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"01e3fd55c591c9b7fadce89e1377b1da710df5e3"},"cell_type":"code","source":"train_df.target.value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"07ddfeed812463257586bba8af02f01293e04b6b"},"cell_type":"code","source":"train_df.target.value_counts() / train_df.target.count()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c364368bef77d5e64346f7810c6260a572437ae7"},"cell_type":"markdown","source":"可以看出正负样本比例严重失调，负样本（不是虚假问题）的比例有接近94%，而正样本（虚假问题）的比例达到6%，所以处理时需要考虑到样本失调的问题。"},{"metadata":{"trusted":true,"_uuid":"53d6828d5930e103681e29c7abeb73f353610ef1"},"cell_type":"code","source":"# 查看虚假问题的前5例是什么类型\ntrain_df.loc[train_df['target']==1, 'question_text'][:5].tolist()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2cdaad60c9684b85cb85d7f58e5ded44ee2f0f76"},"cell_type":"code","source":"# 查看问题中是否存在缺失值\ntrain_df[\"question_text\"].isnull().sum().sum()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"2493149ef63da1a7b839afa95d5ba763fb164462"},"cell_type":"code","source":"test_df[\"question_text\"].isnull().sum().sum()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"61a5227ee7d978ed3380919975a144f24acf6987"},"cell_type":"markdown","source":"说明训练集和测试集中不存在缺失值"},{"metadata":{"trusted":true,"_uuid":"1dcb332ba23fea2ccf15722034338d7bcc9636c6"},"cell_type":"code","source":"# 如果存在缺失值，需要对缺失值进行处理\ntrain_df['question_text'].fillna(\"__na__\", inplace=True)\ntest_df['question_text'].fillna(\"__na__\", inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cb0abf17e084e86ec21c0a808984fa08fc0b9d7f"},"cell_type":"markdown","source":"## 数据清洗的内容\n\n- 处理重复的标点符号：`! ? .`，比如将`!!`转换成`multiExclamation`，将`??`转换成`multiQuestion`，将`..`转换成`multiStop`\n\n- **将标点符号和单词用空格隔开，便于分词**\n\n- 处理英文缩写\n\n如：What's the scariest thing that ever happened to anyone? -> What is the scariest thing that ever happened to anyone? \n\n- 去除停用词\n\n- 词形还原：将过去式，过去分词，现在进行时等还原成动词原形"},{"metadata":{"trusted":true,"_uuid":"1d4a508fc9e90f6e7e95efbcd51cb3d508b2ad12"},"cell_type":"code","source":"# 导入英文数据清洗时常用的库\nimport re\nimport nltk\nfrom nltk import tokenize\nfrom nltk.corpus import stopwords\nstoplist = stopwords.words('english')\n# 对单词的形式进行转换\nfrom nltk.stem import WordNetLemmatizer\nwnl = WordNetLemmatizer()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9b6966fcc6b0a852635e9b7b6e72cdbb8a737c71"},"cell_type":"code","source":"# 添加需要去除的标点符号集\npuncts = [',', '.', '\"', ':', ')', '(', '-', '!', '?', '|', ';', \"'\", '$', '&', '/', \n '[', ']', '>', '%', '=', '#', '*', '+', '\\\\', '•', '~', '@', '£', '·', '_', \n '{', '}', '©', '^', '®', '`', '<', '→', '°', '€', '™', '›', '♥', '←', '×', \n '§', '″', '′', 'Â', '█', '½', 'à', '…', '“', '★', '”', '–', '●', 'â', '►', \n '−', '¢', '²', '¬', '░', '¶', '↑', '±', '¿', '▾', '═', '¦', '║', '―', '¥',\n '▓', '—', '‹', '─', '▒', '：', '¼', '⊕', '▼', '▪', '†', '■', '’', '▀', '¨', \n '▄', '♫', '☆', 'é', '¯', '♦', '¤', '▲', 'è', '¸', '¾', 'Ã', '⋅', '‘', '∞', \n '∙', '）', '↓', '、', '│', '（', '»', '，', '♪', '╩', '╚', '³', '・', '╦', '╣', \n '╔', '╗', '▬', '❤', 'ï', 'Ø', '¹', '≤', '‡', '√', ]\n\n# 定义一些常见的缩写\ncontraction_patterns = [ (r'won\\'t', 'will not'), (r'can\\'t', 'cannot'), (r'i\\'m', 'i am'),\n (r'ain\\'t', 'is not'), (r'(\\w+)\\'ll', '\\g<1> will'), \n (r'(\\w+)n\\'t', '\\g<1> not'),(r'(\\w+)\\'ve', '\\g<1> have'), \n (r'(\\w+)\\'s', '\\g<1> is'), (r'(\\w+)\\'re', '\\g<1> are'), \n (r'(\\w+)\\'d', '\\g<1> would'), (r'&', 'and'), (r'dammit', 'damn it'),\n (r'dont', 'do not'), (r'wont', 'will not') ]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"63566c0f2824f00e93272813245bf7c5ff86acc1"},"cell_type":"code","source":"# 定义清洗文本的函数\ndef clean_text(text):\n # 去除对分类没什么作用的数字\n text = re.sub('[0-9]+', '', text)\n # 对重复出现的标点进行替换\n text = re.sub(r'(\\!)\\1+', 'multiExclamation', text)\n text = re.sub(r'(\\?)\\1+', 'multiQuestion', text)\n text = re.sub(r'(\\.)\\1+', 'multiStop', text)\n \n # 在标点前后加空格\n for punct in puncts:\n text = text.replace(punct, f' {punct} ')\n # 对缩写进行替换\n patterns = [(re.compile(regex), repl) for (regex, repl) in contraction_patterns]\n for (pattern, repl) in patterns:\n (text, count) = re.subn(pattern, repl, text)\n \n # 对文本段进行分词操作\n text_split = tokenize.word_tokenize(text)\n text = [word for word in text_split if word not in stoplist]\n text = [wnl.lemmatize(word) for word in text]\n \n return \" \".join(text)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6de1754cdbaacb1df1d2100767749d11380918bc"},"cell_type":"code","source":"train_df['question_text'] = train_df['question_text'].apply(lambda x: clean_text(x))\ntest_df['question_text'] = test_df['question_text'].apply(lambda x: clean_text(x))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9bec4b45d0da22f59edc801efd4eadfa489507af"},"cell_type":"code","source":"# 添加计算处理之后一段话长度的列\ntrain_df['text_len'] = train_df['question_text'].apply(lambda x: len(x.split()))\ntest_df['text_len'] = test_df['question_text'].apply(lambda x: len(x.split()))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"438b7c0f88cb7868f8ce863fb71202a97bbda065"},"cell_type":"code","source":"train_df['text_len'].describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"de78b764698f502ac916909384c5caf06cde2997"},"cell_type":"code","source":"train_df['text_len'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"153178a9f8282b5f9f3fc6a521fc5a656487bb89"},"cell_type":"code","source":"test_df['text_len'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ee6c21a9c99d1e9439483ce935154e4527c74bdb"},"cell_type":"code","source":"# 将训练集数据划分为训练集和验证集\nfrom sklearn.model_selection import train_test_split\n# 保留5%的数据集作为验证集\ntrain_df, val_df = train_test_split(train_df, test_size=0.05, random_state=2018)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"48003b1aa7e0261b4449346abc5b9a504ec1f65e"},"cell_type":"code","source":"train_X = train_df['question_text'].values\nval_X = val_df['question_text'].values\ntest_X = test_df['question_text'].values","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"02288d6abc647c116f3023b682cb329737c26ccd"},"cell_type":"markdown","source":"## 将文本数据序列化，数字化，以适合深度学习模型训练"},{"metadata":{"trusted":true,"_uuid":"dcfbe4e79efcbc0103653396c0972bead822fda9"},"cell_type":"code","source":"from keras.preprocessing.text import Tokenizer\nfrom keras.preprocessing.sequence import pad_sequences","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a2e4c6793f15d98b6ae05c5a19ed863658394ab6"},"cell_type":"code","source":"# 设置一些参数\nembed_size = 300 # 设置词向量的长度\nmax_features = 95000 # 设置保留频度为前95000个的单词\nmaxlen = 70 # 设置问题保留的最大单词长度（超过70个的只取前70）","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"427591a986e0a94e40fbb4baaee82b8f2b9c3419"},"cell_type":"code","source":"train_X.shape, val_X.shape, test_X.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5e55b181fe58dc976134e29fd55f4ef099871463"},"cell_type":"code","source":"# 设置词典的长度，保留出现频度在前max_features个的单词\ntokenizer = Tokenizer(num_words = max_features)\n# 将训练集，验证集和测试集的句子结合\ntext = np.concatenate((np.concatenate((train_X,val_X)),test_X))\n# 基于已有的单词建立词典\ntokenizer.fit_on_texts(list(text))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"daa4275501fdd93438d1c3d20e358259b945ada6"},"cell_type":"code","source":"# 基于建立的词典将句子中的每个单词序列化为在词典中对应的下标\ntrain_X = tokenizer.texts_to_sequences(train_X)\nval_X = tokenizer.texts_to_sequences(val_X)\ntest_X = tokenizer.texts_to_sequences(test_X)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"522658159f75946d5ff2c6057bd88110fa4815c4"},"cell_type":"markdown","source":"经过上面这一步之后，原始的一维数组转换成了二维的列表，其中的每一个元素表示原数据中的每一条问题，将每一个问题转换为其中单词在词典中对应的索引"},{"metadata":{"trusted":true,"_uuid":"413c618c9f2434bacc1388ca70d4410efd6cf0bf"},"cell_type":"code","source":"# 对上面得到的二维列表进行对齐，将长度大于70的在后面截断，对长度小于70的在后面补0\ntrain_X = pad_sequences(train_X, maxlen=maxlen)\nval_X = pad_sequences(val_X, maxlen=maxlen)\ntest_X = pad_sequences(test_X, maxlen=maxlen)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a08217b9f4d87cb5a415a3f2d318a15befe173d2"},"cell_type":"markdown","source":"## 打乱数据\n"},{"metadata":{"trusted":true,"_uuid":"c6769c57a3858c81b344d663db4410d610208a04"},"cell_type":"code","source":"# 获取target的值\ntrain_y = train_df['target'].values\nval_y = val_df['target'].values\n\n# 打乱数据\nnp.random.seed(2018)\ntrain_idx = np.random.permutation(len(train_y))\nval_idx = np.random.permutation(len(val_y))\n\n# 获取打乱之后的训练集特征，训练集目标以及验证集特征，验证集目标\ntrain_X = train_X[train_idx]\nval_X = val_X[val_idx]\ntrain_y = train_y[train_idx]\nval_y = val_y[val_idx]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"008c41e4aca3cd0331bf4d06e32c48ff49691824"},"cell_type":"markdown","source":"# 加载词向量\n\n我们使用提供的预训练的词向量，官方给出了4种，我们选择其中的两种，对每个词我们取两者词向量的均值作为我们最终的词向量"},{"metadata":{"trusted":true,"_uuid":"b42936eb01a9e26879e600a898807afeb2a2ccc2"},"cell_type":"code","source":"print(os.listdir(\"../input/embeddings\"))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e5f7886645ea83808d5a075fc15d7a8ecc1d9729"},"cell_type":"code","source":"# 对embedding文件进行处理的函数\n# 对于文件中的每一行，通过空格分割\n# 返回的第一个元素的单词，第二个元素是对应的词向量\ndef get_coefs(word, *arr):\n return word, np.asarray(arr, dtype='float32')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"eb9c9d09d970f0dbc49f6880a72932c5fe77a6ba"},"cell_type":"code","source":"# 加载glove模型的词向量\ndef load_glove(word_index=None):\n EMBEDDING_FILE = \"../input/embeddings/glove.840B.300d/glove.840B.300d.txt\"\n embeddings_index = dict(get_coefs(*o.split(\" \")) for o in open(EMBEDDING_FILE))\n \n # 将所有的词向量进行堆叠\n all_embs = np.stack(embeddings_index.values())\n # 设置随机生成词向量时的均值和方差\n emb_mean, emb_std = -0.005838499, 0.48782197\n # 获取词向量的长度\n embed_size = all_embs.shape[1]\n \n # 获取之前词典中单词的总数量\n nb_words = min(max_features, len(word_index))\n # 随机生成词向量矩阵\n embedding_matrix = np.random.normal(emb_mean, emb_std, (nb_words, embed_size))\n # 对于字典中存在的单词，将对应的词向量替换为已有的词向量\n for word, i in word_index.items():\n if i>= max_features: \n continue\n embedding_vector = embeddings_index.get(word)\n if embedding_vector is not None:\n embedding_matrix[i] = embedding_vector\n return embedding_matrix ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"346d862dd0c9bce84298f155e1f7fe8eb3edf1b5"},"cell_type":"code","source":"# 加载para模型的词向量\ndef load_para(word_index=None):\n EMBEDDING_FILE = \"../input/embeddings/paragram_300_sl999/paragram_300_sl999.txt\"\n embeddings_index = dict(get_coefs(*o.split(\" \")) for o in open(EMBEDDING_FILE,\n encoding='utf8',\n errors='ignore')\n if len(o)>100)\n \n # 将所有的词向量进行堆叠\n all_embs = np.stack(embeddings_index.values())\n # 设置随机生成词向量时的均值和方差\n emb_mean, emb_std = -0.0053247833, 0.49346462\n # 获取词向量的长度\n embed_size = all_embs.shape[1]\n \n # 获取之前词典中单词的总数量\n nb_words = min(max_features, len(word_index))\n # 随机生成词向量矩阵\n embedding_matrix = np.random.normal(emb_mean, emb_std, (nb_words, embed_size))\n # 对于字典中存在的单词，将对应的词向量替换为已有的词向量\n for word, i in word_index.items():\n if i>= max_features: \n continue\n embedding_vector = embeddings_index.get(word)\n if embedding_vector is not None:\n embedding_matrix[i] = embedding_vector\n return embedding_matrix ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0399383531e09aac7c4310a8666bd6c7ed6a241e"},"cell_type":"code","source":"# 获取之前生成字典的键值对\n# 键对应单词，值对应索引\nword_index = tokenizer.word_index\nembedding_matrix_1 = load_glove(word_index)\nembedding_matrix_3 = load_para(word_index)\n# 将由两个不同的语料库得到的词向量平均\nembedding_matrix = np.mean([embedding_matrix_1, embedding_matrix_3], axis=0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"13f0a897e4d0847ed25e66d99c5796881afd2f5f"},"cell_type":"code","source":"# 对删除的垃圾进行回收\nimport gc\ndel embedding_matrix_1, embedding_matrix_3\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"17a0c2922c60c7be40d5ce8697c4f4843a4f966b"},"cell_type":"markdown","source":"# 初级篇——使用CNN/LSTM/CNN+LSTM做文本分类"},{"metadata":{"trusted":true,"_uuid":"348b27a8ebab37c03c10568213e8a56077a05e32"},"cell_type":"code","source":"from keras.layers import Dense, Input, CuDNNLSTM, Embedding, Dropout, Activation, CuDNNGRU\nfrom keras.layers import Conv1D, Bidirectional, GlobalMaxPool1D, GlobalMaxPooling1D\nfrom keras.layers import GlobalAveragePooling1D\nfrom keras.layers import Conv2D, MaxPool2D, MaxPooling1D, concatenate\nfrom keras.layers import Reshape, Flatten, Concatenate, Dropout, SpatialDropout1D\nfrom keras.optimizers import Adam\nfrom keras.models import Model\nfrom keras import backend as K\nfrom keras.engine.topology import Layer\nfrom keras import initializers, regularizers, constraints, optimizers, layers\nfrom keras.callbacks import *","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"633414f34ac629c19fd0b215f1fe8ed1ca528814"},"cell_type":"code","source":"from sklearn.metrics import f1_score\n\n# 定义训练预测函数\ndef train_pred(model, epochs=2):\n # 对于每一个epoch\n for e in range(epochs):\n model.fit(train_X, train_y, batch_size=512, epochs=1, \n validation_data=(val_X, val_y))\n # 对于验证集结果进行预测\n pred_val_y = model.predict([val_X], batch_size=1024, verbose=0)\n search_result = threshold_search(val_y, pred_val_y)\n print(search_result)\n pred_test_y = model.predict([test_X], batch_size=1024, verbose=0)\n return pred_val_y, pred_test_y\n\n# 寻找预测概率的最佳分割阈值\ndef threshold_search(y_true, y_proba):\n best_threshold = 0\n best_score = 0\n for threshold in [i*0.001 for i in range(250, 450)]:\n # 计算每一个阈值的f1得分\n score = f1_score(y_true=y_true, y_pred=y_proba > threshold)\n if score > best_score:\n best_threshold = threshold\n best_score = score\n search_result = {'threshold': best_threshold, 'f1': best_score}\n return search_result","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"76d94a48cd5446819d8a0d5f7b72d4f99fc69d31"},"cell_type":"code","source":"# 定义CNN模型\ndef model_cnn(embedding_matrix):\n # 定义滤波器大小的搜索值\n filter_sizes = [1,2,3,5]\n num_filters = 36\n \n inp = Input(shape=(maxlen,))\n # Embedding的第一个参数是词汇量的大小，第二个参数是词向量的大小\n x = Embedding(max_features, embed_size, weights=[embedding_matrix])(inp)\n # 将输入转换成CNN2D需要的形式，长、宽以及通道数\n x = Reshape((maxlen,embed_size, 1))(x)\n \n maxpool_pool = []\n # 使用不同大小的滤波器\n for i in range(len(filter_sizes)):\n # 卷积层\n conv = Conv2D(num_filters, kernel_size=(filter_sizes[i], embed_size),\n kernel_initializer='he_normal', activation='elu')(x)\n # 池化层，将不同滤波器得到的卷积层结果转化为相同大小\n maxpool_pool.append(MaxPool2D(pool_size=(maxlen-filter_sizes[i]+1, 1))(conv))\n \n # 将不同卷积核得到的结果进行连接\n z = Concatenate(axis=1)(maxpool_pool)\n # 展开\n z = Flatten()(z)\n z = Dropout(0.1)(z)\n \n # 全连接\n outp = Dense(1, activation='sigmoid')(z)\n \n model = Model(inputs=inp, outputs=outp)\n model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])\n return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e9f1c426e7eff39fb33fde8b22c7f6aea1afd322"},"cell_type":"code","source":"# 由于使用的标准是准确率而不是f1度量，所以得到的结果不是很好\npred_val_y1, pred_test_y1 = train_pred(model_cnn(embedding_matrix), epochs=2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"16be8d8f50f59b9852c14bcfe29ff110394bd890"},"cell_type":"code","source":"# 定义f1度量\ndef f1(y_true, y_pred):\n # 先计算召回率\n def recall(y_true, y_pred):\n true_positives = K.sum(K.round(K.clip(y_true, 0, 1)))\n possible_positives = K.sum(K.round(K.clip(y_true, 0, 1)))\n recall = true_positives / (possible_positives + K.epsilon())\n return recall\n \n # 再计算准确率\n def precision(y_true, y_pred):\n # 真正例\n true_positives = K.sum(K.round(K.clip(y_true*y_pred, 0, 1)))\n # 预测正例数\n predicted_positives = K.sum(K.round(K.clip(y_pred, 0, 1)))\n # 计算准确率\n precision = true_positives / (predicted_positives + K.epsilon())\n return precision\n \n precision = precision(y_true, y_pred)\n recall = recall(y_true, y_pred)\n return 2*((precision*recall)/(precision+recall+K.epsilon()))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8cdac7a9738bac9d46c24359c3cf1dd57072231e"},"cell_type":"code","source":"# 定义LSTM模型\ndef model_lstm(embedding_matrix):\n inp = Input(shape=(maxlen,))\n x = Embedding(max_features, embed_size, weights=[embedding_matrix], trainable=False)(inp)\n x = SpatialDropout1D(0.1)(x)\n x = Bidirectional(CuDNNLSTM(50, return_sequences=False))(x)\n x = Dense(16, activation='relu')(x)\n x = Dropout(0.1)(x)\n outp = Dense(1, activation='sigmoid')(x)\n \n model = Model(inputs=inp, outputs=outp)\n model.compile(loss='binary_crossentropy', optimizer='adam', metrics=[f1])\n return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a15d6201771365348ed7caf68dae141a81dd0622"},"cell_type":"code","source":"# 计算得到LSTM预测的结果\npred_val_y2, pred_test_y2 = train_pred(model_lstm(embedding_matrix), epochs=4)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ff744d4c5a11ada1e310cbb57a851af8c243768c"},"cell_type":"code","source":"# 定义CNN和LSTM结合的网络\ndef model_cnn_lstm(embedding_matrix):\n inp = Input(shape=(maxlen,))\n x = Embedding(max_features, embed_size, weights=[embedding_matrix], trainable=False)(inp)\n x = SpatialDropout1D(0.1)(x)\n # 卷积\n x = Conv1D(64, 3, activation='relu')(x)\n # 池化\n x = Bidirectional(CuDNNLSTM(50, return_sequences=False))(x)\n x = Dense(16, activation='relu')(x)\n x = Dropout(0.1)(x)\n outp = Dense(1, activation='sigmoid')(x)\n \n model = Model(inputs=inp, outputs=outp)\n model.compile(loss='binary_crossentropy', optimizer='adam', metrics=[f1])\n \n return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e69ce70f7e715538c6e21068a86da9a07d9eebe9"},"cell_type":"code","source":"# 结合CNN和LSTM得到的预测结果\npred_val_y3, pred_test_y3 = train_pred(model_cnn_lstm(embedding_matrix), epochs=4)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fa133d506a5f686c34a0daf7dbb8f2b000bf9bdb"},"cell_type":"markdown","source":"# 进阶篇——为模型添加魔法\n\n**使用Attention\\Capsule等网络增强吗模型的特征提取能力，使用CLR学习率策略让模型更快收敛并获得更好的性能**"},{"metadata":{"_uuid":"e77f7cf033db1a4c1b1d8728cefefa4bbc6aaee7"},"cell_type":"markdown","source":"## Attention\n注意力机制的直觉是：人类视觉通过快速扫描全局图像，获得需要重点关注的目标区域，也就是一般所说的**注意力焦点**，而后对这一区域投入更多注意力资源，以获取所需要关注目标的细节信息，而抑制其他无用信息。这也是人类利用有限的注意力资源从大量信息中快速筛选出高价值信息的手段，是人类在长期进化中形成的一种生存机制，人类视觉注意力机制极大地提高了视觉信息处理的效率与准确性。\n$$Attention(Query, Source) = \\sum_{i=1}^{L_x} Similarity(Query, key_i) * Value_i$$\n\n- **soft attention**\n- **self attention**\n- **multi-head Attention**"},{"metadata":{"trusted":true,"_uuid":"40a623c3c8799a290218dfad5c2a0a51af6d45ec"},"cell_type":"code","source":"class Attention(Layer):\n def __init__(self, step_dim,\n W_regularizer=None, b_regularizer=None,\n W_constraint=None, b_constraint=None,\n bias=True, **kwargs):\n self.supports_masking = True\n self.init = initializers.get('glorot_uniform')\n\n self.W_regularizer = regularizers.get(W_regularizer)\n self.b_regularizer = regularizers.get(b_regularizer)\n\n self.W_constraint = constraints.get(W_constraint)\n self.b_constraint = constraints.get(b_constraint)\n\n self.bias = bias\n self.step_dim = step_dim\n self.features_dim = 0\n super(Attention, self).__init__(**kwargs)\n\n def build(self, input_shape):\n assert len(input_shape) == 3\n\n self.W = self.add_weight((input_shape[-1],),\n initializer=self.init,\n name='{}_W'.format(self.name),\n regularizer=self.W_regularizer,\n constraint=self.W_constraint)\n self.features_dim = input_shape[-1]\n\n if self.bias:\n self.b = self.add_weight((input_shape[1],),\n initializer='zero',\n name='{}_b'.format(self.name),\n regularizer=self.b_regularizer,\n constraint=self.b_constraint)\n else:\n self.b = None\n\n self.built = True\n\n def compute_mask(self, input, input_mask=None):\n return None\n\n def call(self, x, mask=None):\n features_dim = self.features_dim\n step_dim = self.step_dim\n\n eij = K.reshape(K.dot(K.reshape(x, (-1, features_dim)),\n K.reshape(self.W, (features_dim, 1))), (-1, step_dim))\n\n if self.bias:\n eij += self.b\n\n eij = K.tanh(eij)\n\n a = K.exp(eij)\n\n if mask is not None:\n a *= K.cast(mask, K.floatx())\n\n a /= K.cast(K.sum(a, axis=1, keepdims=True) + K.epsilon(), K.floatx())\n\n a = K.expand_dims(a)\n weighted_input = x * a\n return K.sum(weighted_input, axis=1)\n\n def compute_output_shape(self, input_shape):\n return input_shape[0], self.features_dim","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6ef7bdb71971d0bb9911cbb8e06198d592957957"},"cell_type":"code","source":"def model_lstm_atten(embedding_matrix):\n \n inp = Input(shape=(maxlen,))\n x = Embedding(max_features, embed_size, weights=[embedding_matrix], trainable=False)(inp)\n x = SpatialDropout1D(0.1)(x)\n x = Bidirectional(CuDNNLSTM(40, return_sequences=True))(x)\n y = Bidirectional(CuDNNGRU(40, return_sequences=True))(x)\n \n atten_1 = Attention(maxlen)(x)\n atten_2 = Attention(maxlen)(y)\n avg_pool = GlobalAveragePooling1D()(y)\n max_pool = GlobalMaxPooling1D()(y)\n \n conc = concatenate([atten_1, atten_2, avg_pool, max_pool])\n conc = Dense(16, activation=\"relu\")(conc)\n conc = Dropout(0.1)(conc)\n outp = Dense(1, activation=\"sigmoid\")(conc) \n\n model = Model(inputs=inp, outputs=outp)\n model.compile(loss='binary_crossentropy', optimizer='adam', metrics=[f1])\n \n return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0e6f4f4626de7a9f33aa16fa7901dd91d98e8d72"},"cell_type":"code","source":"# 将lstm和attention结合\npred_val_y4, pred_test_y4 = train_pred(model_lstm_atten(embedding_matrix), epochs=5)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d52bd594925e6856703df465178050b85537b3fd"},"cell_type":"markdown","source":"## Capsule胶囊网络\n\n所谓胶囊，就是一个向量，它可包含任意个值，每个值代表了当前需要识别物体的（比如图片）的一个特征。相比传统的CNN卷积层的每个值都是某一块区域和卷积核完成卷积操作，即线性加权求和的结果，它只有一个值，是标量。而胶囊网络的每一个值都是向量，也就是说，**这个向量不仅可表示物体的特征，还可以包括物体的方向、状态等**。"},{"metadata":{"trusted":true,"_uuid":"898aa705454a5bc52a76b871f37601d7a0df8c92"},"cell_type":"code","source":"def squash(x, axis=-1):\n # s_squared_norm is really small\n # s_squared_norm = K.sum(K.square(x), axis, keepdims=True) + K.epsilon()\n # scale = K.sqrt(s_squared_norm)/ (0.5 + s_squared_norm)\n # return scale * x\n s_squared_norm = K.sum(K.square(x), axis, keepdims=True)\n scale = K.sqrt(s_squared_norm + K.epsilon())\n return x / scale\n\nclass Capsule(Layer):\n def __init__(self, num_capsule, dim_capsule, routings=3, kernel_size=(9, 1), share_weights=True,\n activation='default', **kwargs):\n super(Capsule, self).__init__(**kwargs)\n self.num_capsule = num_capsule\n self.dim_capsule = dim_capsule\n self.routings = routings\n self.kernel_size = kernel_size\n self.share_weights = share_weights\n if activation == 'default':\n self.activation = squash\n else:\n self.activation = Activation(activation)\n\n def build(self, input_shape):\n super(Capsule, self).build(input_shape)\n input_dim_capsule = input_shape[-1]\n if self.share_weights:\n self.W = self.add_weight(name='capsule_kernel',\n shape=(1, input_dim_capsule,\n self.num_capsule * self.dim_capsule),\n # shape=self.kernel_size,\n initializer='glorot_uniform',\n trainable=True)\n else:\n input_num_capsule = input_shape[-2]\n self.W = self.add_weight(name='capsule_kernel',\n shape=(input_num_capsule,\n input_dim_capsule,\n self.num_capsule * self.dim_capsule),\n initializer='glorot_uniform',\n trainable=True)\n\n def call(self, u_vecs):\n if self.share_weights:\n u_hat_vecs = K.conv1d(u_vecs, self.W)\n else:\n u_hat_vecs = K.local_conv1d(u_vecs, self.W, [1], [1])\n\n batch_size = K.shape(u_vecs)[0]\n input_num_capsule = K.shape(u_vecs)[1]\n u_hat_vecs = K.reshape(u_hat_vecs, (batch_size, input_num_capsule,\n self.num_capsule, self.dim_capsule))\n u_hat_vecs = K.permute_dimensions(u_hat_vecs, (0, 2, 1, 3))\n # final u_hat_vecs.shape = [None, num_capsule, input_num_capsule, dim_capsule]\n\n b = K.zeros_like(u_hat_vecs[:, :, :, 0]) # shape = [None, num_capsule, input_num_capsule]\n for i in range(self.routings):\n b = K.permute_dimensions(b, (0, 2, 1)) # shape = [None, input_num_capsule, num_capsule]\n c = K.softmax(b)\n c = K.permute_dimensions(c, (0, 2, 1))\n b = K.permute_dimensions(b, (0, 2, 1))\n outputs = self.activation(tf.keras.backend.batch_dot(c, u_hat_vecs, [2, 2]))\n if i < self.routings - 1:\n b = tf.keras.backend.batch_dot(outputs, u_hat_vecs, [2, 3])\n\n return outputs\n\n def compute_output_shape(self, input_shape):\n return (None, self.num_capsule, self.dim_capsule)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0b78916ca44e292f2ed47f7491db568e90399081"},"cell_type":"code","source":"\nimport tensorflow as tf\nfrom keras.layers import BatchNormalization\ndef model_capsule(embedding_matrix):\n K.clear_session()\n inp = Input(shape=(maxlen,))\n x = Embedding(max_features, embed_size, weights=[embedding_matrix], trainable=False)(inp)\n x = SpatialDropout1D(rate=0.2)(x)\n x = Bidirectional(CuDNNGRU(100, return_sequences=True, \n kernel_initializer=initializers.glorot_normal(seed=12300),\n recurrent_initializer=initializers.orthogonal(gain=1.0, seed=10000)))(x)\n\n x = Capsule(num_capsule=10, dim_capsule=10, routings=4, share_weights=True)(x)\n x = Flatten()(x)\n\n x = Dense(100, activation=\"relu\", kernel_initializer=initializers.glorot_normal(seed=12300))(x)\n x = Dropout(0.12)(x)\n x = BatchNormalization()(x)\n\n x = Dense(1, activation=\"sigmoid\")(x)\n model = Model(inputs=inp, outputs=x)\n model.compile(loss='binary_crossentropy', optimizer=Adam(),)\n return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"91ef6b3a84d0876af0047b60c418ab0ffd03dbfd"},"cell_type":"code","source":"pred_val_y5, pred_test_y5 = train_pred(model_capsule(embedding_matrix), epochs=5)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0b3412edb17f57453b8d9bba086a9a91e6fa11aa"},"cell_type":"markdown","source":"## CLR学习策略"},{"metadata":{"trusted":true,"_uuid":"27ae02d138298297c8bdb97f689869f6a811ba16"},"cell_type":"code","source":"\nclass CyclicLR(Callback):\n def __init__(self, base_lr=0.001, max_lr=0.006, step_size=2000., mode='triangular',\n gamma=1., scale_fn=None, scale_mode='cycle'):\n super(CyclicLR, self).__init__()\n\n self.base_lr = base_lr\n self.max_lr = max_lr\n self.step_size = step_size\n self.mode = mode\n self.gamma = gamma\n if scale_fn == None:\n if self.mode == 'triangular':\n self.scale_fn = lambda x: 1.\n self.scale_mode = 'cycle'\n elif self.mode == 'triangular2':\n self.scale_fn = lambda x: 1/(2.**(x-1))\n self.scale_mode = 'cycle'\n elif self.mode == 'exp_range':\n self.scale_fn = lambda x: gamma**(x)\n self.scale_mode = 'iterations'\n else:\n self.scale_fn = scale_fn\n self.scale_mode = scale_mode\n self.clr_iterations = 0.\n self.trn_iterations = 0.\n self.history = {}\n\n self._reset()\n\n def _reset(self, new_base_lr=None, new_max_lr=None,\n new_step_size=None):\n \"\"\"Resets cycle iterations.\n Optional boundary/step size adjustment.\n \"\"\"\n if new_base_lr != None:\n self.base_lr = new_base_lr\n if new_max_lr != None:\n self.max_lr = new_max_lr\n if new_step_size != None:\n self.step_size = new_step_size\n self.clr_iterations = 0.\n \n def clr(self):\n cycle = np.floor(1+self.clr_iterations/(2*self.step_size))\n x = np.abs(self.clr_iterations/self.step_size - 2*cycle + 1)\n if self.scale_mode == 'cycle':\n return self.base_lr + (self.max_lr-self.base_lr)*np.maximum(0, (1-x))*self.scale_fn(cycle)\n else:\n return self.base_lr + (self.max_lr-self.base_lr)*np.maximum(0, (1-x))*self.scale_fn(self.clr_iterations)\n\n def on_train_begin(self, logs={}):\n logs = logs or {}\n\n if self.clr_iterations == 0:\n K.set_value(self.model.optimizer.lr, self.base_lr)\n else:\n K.set_value(self.model.optimizer.lr, self.clr()) \n \n def on_batch_end(self, epoch, logs=None):\n \n logs = logs or {}\n self.trn_iterations += 1\n self.clr_iterations += 1\n\n self.history.setdefault('lr', []).append(K.get_value(self.model.optimizer.lr))\n self.history.setdefault('iterations', []).append(self.trn_iterations)\n\n for k, v in logs.items():\n self.history.setdefault(k, []).append(v)\n \n K.set_value(self.model.optimizer.lr, self.clr())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"57a12d39a618b9307f4b90678c5b8a6b6db307e8"},"cell_type":"code","source":"\nclr = CyclicLR(base_lr=0.001, max_lr=0.002,\n step_size=300., mode='exp_range',\n gamma=0.99994)\ndef train_pred2(model, epochs=2):\n for e in range(epochs):\n model.fit(train_X, train_y, batch_size=512, epochs=1, validation_data=(val_X, val_y), callbacks=[clr])\n pred_val_y = model.predict([val_X], batch_size=1024, verbose=0)\n search_result = threshold_search(val_y, pred_val_y)\n print(search_result)\n pred_test_y = model.predict([test_X], batch_size=1024, verbose=0)\n return pred_val_y, pred_test_y","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"135aac022c3533ae6828f523458fde184bff0803"},"cell_type":"code","source":"\ndef model_3gru_atten(embedding_matrix):\n inp = Input(shape=(maxlen,))\n x = Embedding(max_features, embed_size, weights=[embedding_matrix], trainable=False)(inp)\n x = Bidirectional(CuDNNGRU(128, return_sequences=True))(x)\n x = Bidirectional(CuDNNGRU(100, return_sequences=True))(x)\n x = Bidirectional(CuDNNGRU(64, return_sequences=True))(x)\n x = Attention(maxlen)(x)\n x = Dense(1, activation=\"sigmoid\")(x)\n model = Model(inputs=inp, outputs=x)\n model.compile(loss='binary_crossentropy', optimizer='adam', metrics=[f1])\n \n return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6a28e7f6d570776a127e4ebf7768e7f950dde8bc"},"cell_type":"code","source":"pred_val_y6, pred_test_y6 = train_pred2(model_3gru_atten(embedding_matrix), epochs=3)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"724b960dbe433896e53152e2bdc0cb53ac867bde"},"cell_type":"code","source":"pred_val_y7, pred_test_y7 = train_pred2(model_lstm_atten(embedding_matrix), epochs=4)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"208df78f45b4f9ee6178c9045f981fe5a063f0ac"},"cell_type":"markdown","source":"# 融合\n\n- 不同模型使用一样的词向量，同一个模型使用不同的词向量（多模型）\n- 简单加权融合\n- stacking\n- 其他"},{"metadata":{"trusted":true,"_uuid":"07560b2b7b594eaf9ccfc5b70ac2ddc7c3e9e6e6"},"cell_type":"code","source":"# 这里进行简单加权融合\npred_val_y = 0.09*pred_val_y1 + 0.15*pred_val_y2 + 0.08*pred_val_y3 + 0.22*pred_val_y4 + \\\n 0.18*pred_val_y5 + 0.16*pred_val_y7 + 0.12*pred_val_y7\nsearch_result = threshold_search(val_y, pred_val_y)\nprint(search_result)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"12f6d3b52337b2436ca58cae206bac96107f674f"},"cell_type":"code","source":"pred_test_y = 0.09*pred_test_y1 + 0.15*pred_test_y2 + 0.08*pred_test_y3 + 0.22*pred_test_y4 + \\\n 0.18*pred_test_y5 + 0.16*pred_test_y6 + 0.12*pred_test_y7\npred_test_y = (pred_test_y > search_result['threshold']).astype(int)\nsub = pd.read_csv('../input/sample_submission.csv')\nsub['prediction'] = pred_test_y\nsub.to_csv(\"submission.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"093d526eba7f818da2be4e4523e5040682761f51"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}