{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Toxic Comment classification"},{"metadata":{},"cell_type":"markdown","source":"**This is my first Kaggle commit. All that I've learned about text classification is by going through the kernels of this competition:\nQuora Insincere Questions Classification. Notable among those kernels were:**\n\n**https://www.kaggle.com/shujian/blend-of-lstm-and-cnn-with-4-embeddings-1200d**\n\n**https://www.kaggle.com/suicaokhoailang/beating-the-baseline-with-one-weird-trick-0-691**\n\n**https://www.kaggle.com/danofer/different-embeddings-with-attention-fork**\n\n**https://www.kaggle.com/sudalairajkumar/a-look-at-different-embeddings****\n\n**I would like to thank the creators of these kernels for sharing their knowledge with the community.**"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nimport os\nprint(os.listdir(\"../input\"))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"toxic_dir = \"../input/jigsaw-toxic-comment-classification-challenge\"\nquora_dir = \"../input/quora-insincere-questions-classification\"\n\nprint(os.listdir(toxic_dir))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport gc","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Loading the data"},{"metadata":{"trusted":true},"cell_type":"code","source":"train = pd.read_csv(toxic_dir + \"/train.csv\")\ntest_df = pd.read_csv(toxic_dir + \"/test.csv\")\n\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(train.shape)\ntrain.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.model_selection import train_test_split\n\n## split to train and val\ntrain_df, val_df = train_test_split(train, test_size=0.1, random_state=2019)\ntrain_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"list_classes = [\"toxic\", \"severe_toxic\", \"obscene\", \"threat\", \"insult\", \"identity_hate\"]\ny_train = train_df[list_classes].values\ny_val = val_df[list_classes].values\nprint(y_train.shape,y_val.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Standard text processing"},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras.preprocessing.text import Tokenizer\nfrom keras.preprocessing.sequence import pad_sequences\n\n## fill up the missing values\ntrain_X = train_df[\"comment_text\"].fillna(\"_na_\").values\nval_X = val_df[\"comment_text\"].fillna(\"_na_\").values\ntest_X = test_df[\"comment_text\"].fillna(\"_na_\").values\n\nmaxlen = 50\nmax_features = 50000\n\n## Tokenize the sentences\ntokenizer = Tokenizer(num_words=max_features)\ntokenizer.fit_on_texts(list(train_X))\ntrain_X = tokenizer.texts_to_sequences(train_X)\nval_X = tokenizer.texts_to_sequences(val_X)\ntest_X = tokenizer.texts_to_sequences(test_X)\n\n## Pad the sentences \ntrain_X = pad_sequences(train_X, maxlen=maxlen)\nval_X = pad_sequences(val_X, maxlen=maxlen)\ntest_X = pad_sequences(test_X, maxlen=maxlen)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_coefs(word,*arr): \n     return word, np.asarray(arr, dtype='float32')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"embeddings_dir = quora_dir + \"/embeddings\"\nos.listdir(embeddings_dir)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Creating embedding matrix\n\nEmbeddings generally represent geometrical encodings of words based on how frequently appear together in a text corpus. Various implementations of word embeddings described below differs in the way as how they are constructed. A very good article on word embeddings is available here:\n\n**https://www.kaggle.com/sbongo/do-pretrained-embeddings-give-you-the-extra-edge**"},{"metadata":{},"cell_type":"markdown","source":"#### 1. Extracting GLoVE embeddings\n\nThe main idea behind it is that you train a model on the context on each word, so similar words will have similar numerical representations.\n\nGLoVE learns by constructing a co-occurrence matrix (words X context) that basically count how frequently a word appears in a context. Since it's going to be a gigantic matrix, we factorize this matrix to achieve a lower-dimension representation. There's a lot of details that goes in GLo[](http://)VE but that's the rough idea."},{"metadata":{"trusted":true},"cell_type":"code","source":"EMBEDDING_FILE_1 = embeddings_dir + '/glove.840B.300d/glove.840B.300d.txt'\nembeddings_index = dict(get_coefs(*o.split(\" \")) for o in open(EMBEDDING_FILE_1))\n\nall_embs = np.stack(embeddings_index.values())\nemb_mean,emb_std = all_embs.mean(), all_embs.std()\nembed_size = all_embs.shape[1]\n\nword_index = tokenizer.word_index\nnb_words = min(max_features, len(word_index))\n\n# Creating the embedding matrix with GLoVE\nembedding_matrix_1 = np.random.normal(emb_mean, emb_std, (nb_words, embed_size))\n\nfor word, i in word_index.items():\n    if i >= max_features: continue\n    embedding_vector = embeddings_index.get(word)\n    if embedding_vector is not None: embedding_matrix_1[i] = embedding_vector\n\ndel embeddings_index; gc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(embedding_matrix_1.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### 2. Using FastText embeddings\n\nFastText is quite different from GLoVE. While GLoVE treats each word as the smallest unit to train on, FastText uses n-gram characters as the smallest unit. For example, the word vector ,\"apple\", could be broken down into separate word vectors units as \"ap\",\"app\",\"ple\". The biggest benefit of using FastText is that it generate better word embeddings for rare words, or even words not seen during training because the n-gram character vectors are shared with other words. This is something that GLoVE cannot achieve"},{"metadata":{"trusted":true},"cell_type":"code","source":"EMBEDDING_FILE_2 = embeddings_dir + '/wiki-news-300d-1M/wiki-news-300d-1M.vec'\nembeddings_index = dict(get_coefs(*o.split(\" \")) for o in open(EMBEDDING_FILE_2) if len(o)>100)\n\nall_embs = np.stack(embeddings_index.values())\nemb_mean,emb_std = all_embs.mean(), all_embs.std()\nembed_size = all_embs.shape[1]\n\nword_index = tokenizer.word_index\nnb_words = min(max_features, len(word_index))\n\n# Creating the embedding matrix with wikinews embeddings\nembedding_matrix_2 = np.random.normal(emb_mean, emb_std, (nb_words, embed_size))\n\nfor word, i in word_index.items():\n    if i >= max_features: continue\n    embedding_vector = embeddings_index.get(word)\n    if embedding_vector is not None: embedding_matrix_2[i] = embedding_vector\n        \ndel embeddings_index; gc.collect() ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(embedding_matrix_2.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### 3. Concatenate them"},{"metadata":{"trusted":true},"cell_type":"code","source":"embedding_matrix = np.concatenate((embedding_matrix_1, embedding_matrix_2), axis=1)  \n\ndel embedding_matrix_1, embedding_matrix_2\ngc.collect()\n\nprint(\"Shape of embedding matrix: \",np.shape(embedding_matrix))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Training models"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Importing necessary libraries\nfrom keras.layers import Dense, Input, CuDNNLSTM, LSTM, Embedding, Dropout, Activation, CuDNNGRU, Conv1D\nfrom keras.layers import Bidirectional, GlobalMaxPool1D, GlobalMaxPooling1D, GlobalAveragePooling1D\nfrom keras.layers import Input, Embedding, Dense, Conv2D, MaxPool2D, concatenate\nfrom keras.layers import Reshape, Flatten, Concatenate, Dropout, SpatialDropout1D\nfrom keras.optimizers import Adam\nfrom keras.models import Model\nfrom keras import backend as K\nfrom keras.engine.topology import Layer\nfrom keras import initializers, regularizers, constraints, optimizers, layers","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"spatial_dropout = 0.4\n\ninp = Input(shape=(maxlen,))\n\nx = Embedding(max_features, embed_size * 2, weights=[embedding_matrix])(inp)\nx = SpatialDropout1D(spatial_dropout)(x)\n#x = Bidirectional(LSTM(128, dropout=0.4, recurrent_dropout=0.4, activation='relu', return_sequences=True))(x)\nx = Bidirectional(CuDNNLSTM(128, return_sequences=True))(x)\n\navg_pool = GlobalAveragePooling1D()(x)\nmax_pool = GlobalMaxPooling1D()(x)\nconc = concatenate([avg_pool, max_pool])\n\nx = Dense(16, activation=\"relu\")(conc)\nx = Dropout(0.2)(x)\nx = Dense(6, activation=\"sigmoid\")(x)\nmodel = Model(inputs=inp, outputs=x)\n\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model.compile(loss='binary_crossentropy',\n                  optimizer='adam',\n                  metrics=['accuracy'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"batch_size = 512\nepochs = 5\n\nhist = model.fit(train_X, y_train, batch_size=batch_size, epochs=epochs, validation_data=(val_X, y_val))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y_test = model.predict([test_X], batch_size=1024, verbose=1)\nsample_submission = pd.read_csv(toxic_dir + \"/sample_submission.csv\")\nsample_submission[list_classes] = y_test\nsample_submission.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}