{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Bert Model for Classification Toxic Comments","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Transformer\nTransformer is a transduction architecture that relies on the attention-mechanism (which is a concept that can help to improve the NLP models performance) transforming between the input and the output with the help of two parts Encoder and Decoder. It was proposed in the paper Attention is all you need by google in 2017 to show how the transformers generalizes well with different models’ tasks. The Transformer can also allow for more parallelization and can reach a new state of the art in translation quality after being trained for less time.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Bert\nBert stands for Bidirectional Encoder Representations from Transformers. It’s google new techniques for NLP pre-training languages representation. Which means now machine learning communities can use Bert models that have been training already on a large amount of words, for NLP models to do wide variety of tasks such as Question Answering tasks, Named Entity Recognition (NER), and Classification like sentiment analysis.\nYou will see that Bert also trained on the Encoder stacks in the transformers to use the same attention mechanism. But why is it called bidirectional?\nBecause the transformers encoder reads the entire sequence of the words at once which is the opposite to the directional models that read the input sequentially for left to the right or from the right to the left. The bidirectional method will help the model to learn and understand the meaning and the intention of the word based on its surrounding. Since we will use it for toxic classification, we will explain only the Bert steps for classification tasks only.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"The input of Bert is a special input supplied with [CLS] token stand for classification. As in the Transformers the Bert will take a sequence of words (vector) as an input which keeps feed up from the first encoder layer up to the last layer in the stack. Each layer in the stack will apply the self-attention method to the sequence after that it will pass to the feed forward network to deliver the next encoder layer.\nThe output of Bert model contains the vector of a size (hidden size) and the first position in the output is the [CLS] token. Now this output can be used as an input to our classifier neural network for classification the toxicity of the words. In the research paper they achieve a great result by using only a single layer neural network as the classifier.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"> # Notebook content:\n\n1. Import Libraries\n2. Run Bert Model on TPU\n3. Functions and Variables<br>\n    3.1 Function for Encoding the comment<br>\n    3.2 Function for Neural Network model<br>\n4. Preprocessing\n    4.1 Import Datasets<br>\n    4.2 tokenaizer <br>\n    4.3 Encode The Comments<br>\n    4.4 Prepare tensorflow dataset for modeling<br>\n5. Machine Learning<br>\n    5.1 Training The Model, Tuning Hyper-Parameters<br>\n    5.2 Testing The Model\n\n    \n    ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# 1. Import Libraries","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from numpy.random import seed\nseed(40)\nimport tensorflow as tf\ntf.random.set_seed(40)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np \nimport pandas as pd \nfrom keras.models import Sequential\nfrom keras.layers.core import Dense, Activation, Dropout\nfrom keras.layers.embeddings import Embedding\nfrom keras.layers.normalization import BatchNormalization\nfrom keras.utils import np_utils\nfrom sklearn import preprocessing, decomposition, model_selection, metrics, pipeline\nfrom keras.preprocessing import sequence, text\nfrom keras.callbacks import EarlyStopping\n\nimport os\nfrom tensorflow.keras.layers import Dense, Input\nfrom tensorflow.keras.optimizers import Adam\nfrom tensorflow.keras.models import Model\nfrom tensorflow.keras.callbacks import ModelCheckpoint\nfrom kaggle_datasets import KaggleDatasets\nimport transformers\nfrom transformers import TFAutoModel, AutoTokenizer\nfrom tokenizers import BertWordPieceTokenizer, Tokenizer, models, pre_tokenizers, decoders, processors\nfrom plotly import graph_objs as go\nimport plotly.express as px\nimport plotly.figure_factory as ff\nfrom tqdm.notebook import tqdm\n\n\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n%matplotlib inline\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 2. Run Bert Model on TPU","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Detect hardware, return appropriate distribution strategy\ntry:\n    # TPU detection. No parameters necessary if TPU_NAME environment variable is\n    # set: this is always the case on Kaggle.\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n    print('Running on TPU ', tpu.master())\nexcept ValueError:\n    tpu = None\n\nif tpu:\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n    # Default distribution strategy in Tensorflow. Works on CPU and single GPU.\n    strategy = tf.distribute.get_strategy()\n\nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 3. Functions and Variables","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## 3.1 Function for Encoding the comment","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def regular_encode(texts, tokenizer, maxlen=512):\n        # encode the word to vector of integer\n\n    encode_dictionary = tokenizer.batch_encode_plus(texts, return_attention_masks=False, return_token_type_ids=False,\n    pad_to_max_length=True,max_length=maxlen)\n    \n    return np.array(encode_dictionary['input_ids'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 3.2 Function for Neural Network Model","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def build_model(transformer, max_len=512):\n\n#Input: for define input layer\n#shape is vector with 512-dimensional vectors\n    input_word_ids = Input(shape=(max_len,), dtype=tf.int32, name=\"input_word_ids\")\n    sequence_output = transformer(input_word_ids)[0]\n# to get the vector\n    cls_token = sequence_output[:, 0, :]\n# define output layer\n    out = Dense(1, activation='sigmoid')(cls_token)\n# initiate the model with inputs and outputs\n    model = Model(inputs=input_word_ids, outputs=out)\n# get the learning rate adam(1e-5) and the metrica\n    model.compile(Adam(lr=1e-5), loss='binary_crossentropy', metrics=[tf.keras.metrics.AUC()])\n    \n    return model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 4. Preprocessing","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"\n# define variables for modeling use\nEPOCHS = 3 #number of epochs in model\nBATCH_SIZE = 16 * strategy.num_replicas_in_sync # the batch size in each epoch (128)\nMAX_LEN = 192\n\n# distilbert pre-trained model is faster than the bert base model, but it give lower accuracy than the bert base\n#MODEL ='distilbert-base-multilingual-cased'\n\nMODEL='bert-base-multilingual-cased'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#API to build highly flexible and efficient TensorFlow input pipelines.\nAUTO = tf.data.experimental.AUTOTUNE","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 4.1 Import Datasets\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train1 = pd.read_csv(\"/kaggle/input/jigsaw-multilingual-toxic-comment-classification/jigsaw-toxic-comment-train.csv\")\n\nvalid = pd.read_csv('/kaggle/input/jigsaw-multilingual-toxic-comment-classification/validation.csv')\ntest = pd.read_csv('/kaggle/input/jigsaw-multilingual-toxic-comment-classification/test.csv')\nsub = pd.read_csv('/kaggle/input/jigsaw-multilingual-toxic-comment-classification/sample_submission.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 4.2 tokenaizer ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"#use the pre-trained model bert as a tokenizer \n#bert tokenizer has vocabulary for emoji. this is the reason we don't need to remove emoji from \n#datasets, for more details see the (EDA & data cleaning) notebook\n\ntokenizer = AutoTokenizer.from_pretrained(MODEL)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 4.3 Encode The Comments","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time \n#call the function regular encode on for all the 3 dataset to convert each words after the tokenizer\n#into a vector\n#x_train,x_test, and x_validation will have the comment text column only,(in test called \"content\")\nx_train = regular_encode(train1.comment_text.values, tokenizer, maxlen=MAX_LEN)\nx_valid = regular_encode(valid.comment_text.values, tokenizer, maxlen=MAX_LEN)\nx_test = regular_encode(test.content.values, tokenizer, maxlen=MAX_LEN)\n\n#y_train,y_valid will have te target column \"toxic\"\ny_train = train1.toxic.values\ny_valid = valid.toxic.values","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 4.4 Prepare Tensorflow Dataset For Modeling","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Create and prepare a source dataset from your input data to fit the model in the next step.\n# Apply dataset transformations to preprocess the data.\n# Iterate over the dataset and process the elements.\n\ntrain_dataset = (\n    tf.data.Dataset # create dataset\n    .from_tensor_slices((x_train, y_train)) # Once you have a dataset, you can apply transformations \n    .repeat()\n    .shuffle(2048,seed=40) # Combines consecutive elements of this dataset into batches.\n    .batch(BATCH_SIZE)\n    .prefetch(AUTO)) #This allows later elements to be prepared while the current element is being processed (pipline).\n\n\nvalid_dataset = (tf.data.Dataset.from_tensor_slices((x_valid, y_valid)).batch(BATCH_SIZE)\n    .cache().prefetch(AUTO))\n\ntest_dataset = (tf.data.Dataset.from_tensor_slices(x_test).batch(BATCH_SIZE))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 5. Machine Learning","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Bert models\n### Bert-base-multilingual-cased\n- Train on:12-layer, 768-hidden, 12-heads110M parameters.\n- Train on 104 languages with Wikipedies\n\n### Distilbert-base-multilingual-cased\n- Train on:6-layer, 768-hidden, 12-heads, 134M parameters\n- Train on 104 languages with Wikipedies","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n# in the TPU\nwith strategy.scope():\n    #take the encoder results of bert from transformers and use it as an input in the NN model\n    transformer_layer = TFAutoModel.from_pretrained(MODEL)\n    model = build_model(transformer_layer, max_len=MAX_LEN)\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 5.1 Training The Model, Tuning Hyper-Parameters","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"#train the model\n# training the data and tune our model with the results of the metrics we get from the validation dataset\nn_steps = x_train.shape[0] // BATCH_SIZE\ntrain_history = model.fit(train_dataset, steps_per_epoch=n_steps, validation_data=valid_dataset,\n                epochs=EPOCHS)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 5.2 Testing The Mode","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"#test the model on validation\nn_steps = x_valid.shape[0] // BATCH_SIZE\ntrain_history_2 = model.fit(valid_dataset.repeat(), steps_per_epoch=n_steps,epochs=EPOCHS*2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(train_history.history.keys())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.plot(train_history.history['loss'])\nplt.plot(train_history.history['val_loss'])\nplt.title('model AUC')\nplt.ylabel('AUC')\nplt.xlabel('epoch')\nplt.legend(['train', 'validation'], loc='upper left')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#predict and submit\n# sub['toxic'] = model.predict(test_dataset, verbose=1)\n# sub.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"raw","source":"","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# sub.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}