{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport tensorflow as tf\nimport tensorflow_hub as hub\nfrom tqdm import tqdm\nfrom tqdm import tqdm_notebook\nfrom sklearn.metrics import auc\nfrom sklearn.metrics import classification_report\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n\nfrom transformers import AutoTokenizer,BertTokenizer,TFBertModel,TFOpenAIGPTModel,OpenAIGPTTokenizer,DistilBertTokenizer, TFDistilBertModel,XLMTokenizer, TFXLMModel\nfrom transformers import TFAutoModel, AutoTokenizer\nfrom kaggle_datasets import KaggleDatasets\nfrom sklearn.metrics import roc_curve,confusion_matrix,auc\nfrom tokenizers import Tokenizer, models, pre_tokenizers, decoders, processors\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\nimport matplotlib as mpl\n\n\nfrom sklearn.feature_extraction.text import CountVectorizer\nfrom sklearn.metrics import confusion_matrix, accuracy_score\nfrom sklearn.model_selection import train_test_split\n\nfrom tensorflow.keras.preprocessing.text import Tokenizer\nfrom tensorflow.keras.preprocessing.sequence import pad_sequences\nfrom tensorflow.keras.models import Sequential\nfrom tensorflow.keras.layers import *\nfrom tensorflow.keras.initializers import Constant\nimport re\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n for filename in filenames:\n print(os.path.join(dirname, filename))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# **Main Purpose of the NoteBook**\n The purpose of this notebook is really simple, I was curious as how different Transformers Architecture will perform over the given data. This notebook is really simple to understand, there is no fancy EDA or any data exploration. **The sole purpose of this notebook is to just compare different Transformers Architecture. I will be judging the Model performance over the accuracy, AUC,recall and precision**\n\n NoteBook also can be used as a introduction to as how tensorflow keras can be integrated with hugging face. \nI have tried to make this notebook quite user friendly and easy to understand so that every one can understand the basic of hugging face \n\n![](https://huggingface.co/front/thumbnails/models.png)\n\nThe following Transformers Architecture have been tested in the notebook\n\n### 1. BERT\n### 2. OpenAIGPT\n### 3. Transformer XL\n### 4. XLM\n### 5. XLM-Roberta-Large\n\n\n![](https://media-exp1.licdn.com/dms/image/C5112AQHdCkQOA8sjlQ/article-cover_image-shrink_600_2000/0?e=1596067200&v=beta&t=aXNCLbGrTOsjxeWZkzx5ksTNThTUFVaEBPl1_vsf6K8)\n \n However I also later on added some classical LSTM models with and without attention mechanims in order to see its performance on the dataset. The same metric system will be used to judge the performance of LSTMs,The following two type of LSTM models have been used \n \n \n \n### 6.LSTM with Glovec Embedding\n### 7.LSTM with Attention Mechanism\n \nI also have given useful links for each of the model used so that you can further expand your knowledge\n \n \nThis is my first notebook and I have tried to put into a lot of effort. Please upvote, it will really encourage me to make better and more helpful notebooks in future \n\n \n The notebook takes much time to run therefore I cannot run the entire notebook in one go. Therefore, I have ran notebook sequentailly and attached the output of each model performance in the markdown. If you want to check any model just run the relevant section while also executing the functions which are common to all models. Again, the purpose of notebook is to show how without any substaintial hyperparameter tuning different transformers and attention mechanism LSTMs perform on the task","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Credits:\n I have used some help from other notebooks, therefore I would like to give the credit to desire individuals:\n
    \n
  • 1. [xhlulu](https://www.kaggle.com/xhlulu/jigsaw-tpu-distilbert-with-huggingface-and-keras)
    • \n
  • 2. [tanulsingh077](https://www.kaggle.com/tanulsingh077/deep-learning-for-nlp-zero-to-transformers-bert/notebook)
    • \n
\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## TPU Distribution Strategy for Efficent Model Training","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Detect hardware, return appropriate distribution strategy\ntry:\n # TPU detection. No parameters necessary if TPU_NAME environment variable is\n # set: this is always the case on Kaggle.\n tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n print('Running on TPU ', tpu.master())\nexcept ValueError:\n tpu = None\n\nif tpu:\n tf.config.experimental_connect_to_cluster(tpu)\n tf.tpu.experimental.initialize_tpu_system(tpu)\n strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n # Default distribution strategy in Tensorflow. Works on CPU and single GPU.\n strategy = tf.distribute.get_strategy()\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"train1 = pd.read_csv(\"/kaggle/input/jigsaw-multilingual-toxic-comment-classification/jigsaw-toxic-comment-train.csv\")\ntrain2 = pd.read_csv(\"/kaggle/input/jigsaw-multilingual-toxic-comment-classification/jigsaw-unintended-bias-train.csv\")\ntrain2.toxic = train2.toxic.round().astype(int)\n\nvalid_raw = pd.read_csv('/kaggle/input/jigsaw-multilingual-toxic-comment-classification/validation.csv')\ntest_raw = pd.read_csv('/kaggle/input/jigsaw-multilingual-toxic-comment-classification/test.csv')\nsub = pd.read_csv('/kaggle/input/jigsaw-multilingual-toxic-comment-classification/sample_submission.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Combine train1 with a subset of train2\ntrain_raw = pd.concat([\n train1[['comment_text', 'toxic']],\n train2[['comment_text', 'toxic']].query('toxic==1'),\n train2[['comment_text', 'toxic']].query('toxic==0').sample(n=100000, random_state=0)\n])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"valid_raw['toxic'].value_counts().plot(kind='bar')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_raw['toxic'].value_counts().plot(kind='bar')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"neg, pos = np.bincount(train_raw['toxic'])\ntotal = neg + pos\nprint('Examples:\\n Total: {}\\n Positive: {} ({:.2f}% of total)\\n'.format(\n total, pos, 100 * pos / total))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 1. BERT","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"![BERT](https://www.researchgate.net/profile/Wei_Shi102/publication/336995759/figure/fig1/AS:824051559825408@1573480610178/The-architecture-from-BERT-Devlin-et-al-2019-for-fine-tuning-of-implicit-discourse.ppm)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Usefull Links About BERT\n The following links are quite useful if you to further your knowledge about BERT\n\n
  • [illustrated-bert](http://jalammar.github.io/illustrated-bert/)
    • \n
  • [a-visual-guide-to-using-bert-for-the-first-time](http://jalammar.github.io/a-visual-guide-to-using-bert-for-the-first-time/)
    • \n
  • [Chris Mccromick AI](https://www.youtube.com/watch?v=_eSGWNqKeeY&t=1s)
    • \n
  • [Chris Mccromick AI-2](https://www.youtube.com/watch?v=l8ZYCvgGu0o)
    • \n
  • [CS224n Video](https://www.youtube.com/watch?v=S-CspeZ8FHc)\n
  • [Orginal Paper](https://arxiv.org/abs/1810.04805)
    • \n    \n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# First load the real tokenizer\ntokenizer = BertTokenizer.from_pretrained('bert-base-uncased')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Hyperparameters-Transformers","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"BATCH_SIZE = 16 * strategy.num_replicas_in_sync\nEPOCHS=2\nLEARNING_RATE=1e-5\nearly_stopping=early_stopping = tf.keras.callbacks.EarlyStopping(\n monitor='val_loss', \n verbose=1,\n patience=10,\n mode='max',\n restore_best_weights=True)\nAUTO = tf.data.experimental.AUTOTUNE\nGCS_DS_PATH = KaggleDatasets().get_gcs_path()\nmax_seq_length = 192\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\ndef single_encoding_function(text,tokenizer,name='BERT'):\n input_ids=[]\n if name=='BERT':\n tokenizer.pad_token ='[PAD]'\n elif name=='OPENAIGPT2':\n tokenizer.pad_token=''\n elif name=='Transformer XL':\n print(tokenizer.eos_token)\n tokenizer.pad_token= tokenizer.eos_token\n elif name=='DistilBert':\n tokenizer.pad_token='[PAD]'\n \n for sentence in tqdm(text):\n encoded=tokenizer.encode(sentence,max_length=max_seq_length,pad_to_max_length=True)\n input_ids.append(encoded)\n return input_ids\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train=np.array(single_encoding_function(train_raw['comment_text'].values.tolist(),tokenizer,name=\"BERT\"))\ny_train=np.array(train_raw['toxic'])\nX_valid=np.array(single_encoding_function(valid_raw['comment_text'].values.tolist(),tokenizer,name=\"BERT\"))\ny_valid=np.array(valid_raw['toxic'])\nX_test=np.array(single_encoding_function(test_raw['content'].values.tolist(),tokenizer,name=\"BERT\"))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"steps_per_epoch = X_train.shape[0] // BATCH_SIZE","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Making Tensor Data Pipeline","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def make_data():\n train = (\n tf.data.Dataset\n .from_tensor_slices((X_train, y_train))\n .repeat()\n .shuffle(2048)\n .batch(BATCH_SIZE)\n .prefetch(AUTO))\n\n valid = (\n tf.data.Dataset\n .from_tensor_slices((X_valid, y_valid))\n .batch(BATCH_SIZE)\n .cache()\n .prefetch(AUTO)\n )\n\n test = (\n tf.data.Dataset\n .from_tensor_slices(X_test)\n .batch(BATCH_SIZE)\n )\n return train,valid,test","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train,valid,test=make_data()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Making TF Model for Different Transformers","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def build_model(transformer_layer,max_len=max_seq_length):\n input_word_ids = tf.keras.layers.Input(shape=(max_len,), dtype=tf.int32, name=\"input_word_ids\")\n sequence_output = transformer_layer(input_word_ids)[0]\n \n cls_token = sequence_output[:, 0, :]\n out = tf.keras.layers.Dense(1, activation='sigmoid')(cls_token)\n \n model = tf.keras.Model(inputs=input_word_ids, outputs=out)\n \n \n return model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Functions to plot Metrics and loss to Compare Different Transformers\n\n![](https://miro.medium.com/fit/c/1838/551/1*aPYAckB1ZtfcqoY8wZ915w.jpeg)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Note:\n These are four function whic will help me to compare the performance of different models. The first two functions being used will plot loss and other choosen metrics over the number of epochs\n\nThe last two function will plot confusion matrix and roc for the models respectively","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Useful links to understand Choosen Metrics:\n These are the link which can help you to understand the metrics which I have used to compare the performance of different models\n\n
  • [Confusion Matrix](https://www.coursera.org/lecture/python-machine-learning/confusion-matrices-basic-evaluation-metrics-90kLk)
    • \n
  • [ROC,Precision,Recall,AUC](https://www.coursera.org/lecture/python-machine-learning/precision-recall-and-roc-curves-8v6DL)
    • \n    \n ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"mpl.rcParams['figure.figsize'] = (12, 10)\ncolors = plt.rcParams['axes.prop_cycle'].by_key()['color']\n\ndef plot_loss(history):\n# Use a log scale to show the wide range of values.\n plt.semilogy(history.epoch, history.history['loss'],\n color='red', label='Train Loss')\n plt.semilogy(history.epoch, history.history['val_loss'],\n color='green', label='Val Loss',\n linestyle=\"--\")\n plt.xlabel('Epoch')\n plt.ylabel('Loss')\n \n plt.legend()\n \n \ndef plot_metrics(history):\n metrics = ['loss', 'auc', 'precision', 'recall']\n for n, metric in enumerate(metrics):\n name = metric.replace(\"_\",\" \").capitalize()\n plt.subplot(2,2,n+1)\n plt.plot(history.epoch, history.history[metric], color=colors[0], label='Train')\n plt.plot(history.epoch, history.history['val_'+metric],\n color=colors[0], linestyle=\"--\", label='Val')\n plt.xlabel('Epoch')\n plt.ylabel(name)\n if metric == 'loss':\n plt.ylim([0, plt.ylim()[1]])\n elif metric == 'auc':\n plt.ylim([0.8,1])\n else:\n plt.ylim([0,1])\n\n plt.legend()\n\ndef plot_cm(y_true, y_pred, title):\n ''''\n input y_true-Ground Truth Labels\n y_pred-Predicted Value of Model\n title-What Title to give to the confusion matrix\n \n Draws a Confusion Matrix for better understanding of how the model is working\n \n return None\n \n '''\n \n figsize=(10,10)\n cm = confusion_matrix(y_true, y_pred, labels=np.unique(y_true))\n cm_sum = np.sum(cm, axis=1, keepdims=True)\n cm_perc = cm / cm_sum.astype(float) * 100\n annot = np.empty_like(cm).astype(str)\n nrows, ncols = cm.shape\n for i in range(nrows):\n for j in range(ncols):\n c = cm[i, j]\n p = cm_perc[i, j]\n if i == j:\n s = cm_sum[i]\n annot[i, j] = '%.1f%%\\n%d/%d' % (p, c, s)\n elif c == 0:\n annot[i, j] = ''\n else:\n annot[i, j] = '%.1f%%\\n%d' % (p, c)\n cm = pd.DataFrame(cm, index=np.unique(y_true), columns=np.unique(y_true))\n cm.index.name = 'Actual'\n cm.columns.name = 'Predicted'\n fig, ax = plt.subplots(figsize=figsize)\n plt.title(title)\n sns.heatmap(cm, cmap= \"YlGnBu\", annot=annot, fmt='', ax=ax)\n\ndef roc_curve_plot(fpr,tpr,roc_auc):\n plt.figure()\n lw = 2\n plt.plot(fpr, tpr, color='darkorange',\n lw=lw, label='ROC curve (area = %0.2f)' %roc_auc)\n plt.plot([0, 1], [0, 1], color='navy', lw=lw, linestyle='--')\n plt.xlim([0.0, 1.0])\n plt.ylim([0.0, 1.05])\n plt.xlabel('False Positive Rate')\n plt.ylabel('True Positive Rate')\n plt.title('Receiver operating characteristic example')\n plt.legend(loc=\"lower right\")\n plt.show()\n\n ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Model Compilation Under TPUs","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\ndef compile_model(name):\n with strategy.scope():\n METRICS = [\n tf.keras.metrics.TruePositives(name='tp'),\n tf.keras.metrics.FalsePositives(name='fp'),\n tf.keras.metrics.TrueNegatives(name='tn'),\n tf.keras.metrics.FalseNegatives(name='fn'), \n tf.keras.metrics.BinaryAccuracy(name='accuracy'),\n tf.keras.metrics.Precision(name='precision'),\n tf.keras.metrics.Recall(name='recall'),\n tf.keras.metrics.AUC(name='auc')]\n if name=='bert-base-uncased':\n transformer_layer = (\n TFBertModel.from_pretrained(name)\n )\n elif name=='openai-gpt':\n transformer_layer = (\n TFOpenAIGPTModel.from_pretrained(name)\n )\n elif name=='distilbert-base-cased':\n transformer_layer = (\n TFDistilBertModel.from_pretrained(name)\n )\n elif name=='xlm-mlm-en-2048':\n transformer_layer = (\n TFBertModel.from_pretrained(name)\n )\n elif name=='jplu/tf-xlm-roberta-large':\n transformer_layer = (\n TFAutoModel.from_pretrained(name)\n )\n model = build_model(transformer_layer, max_len=max_seq_length)\n model.compile(optimizer=tf.keras.optimizers.Adam(\n learning_rate=LEARNING_RATE), loss='binary_crossentropy', metrics=METRICS)\n return model\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"steps_per_epoch=X_train.shape[0]//BATCH_SIZE\n\nmodel=compile_model('bert-base-uncased')\nprint(model.summary())\nhistory=model.fit(\n train,steps_per_epoch=steps_per_epoch,\n epochs=EPOCHS,callbacks=[early_stopping], validation_data=valid\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## BERT Performance-Loss","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_loss(history)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Bert Performance-Metrics","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_metrics(history)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## BERT Performance-Confusion Matrix","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"y_predict=model.predict(valid, verbose=1)\ny_predict[ y_predict> 0.5] = 1\ny_predict[y_predict <= 0.5] = 0\nplot_cm(y_valid, y_predict, 'BERT-Confusion Matrix')\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## BERT Performance-ROC Curve","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"y_predict_prob=model.predict(valid, verbose=1)\nfpr, tpr, _ = roc_curve(y_valid,y_predict_prob)\nroc_auc = auc(fpr, tpr)\nroc_curve_plot(fpr,tpr,roc_auc)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# 2. OpenAIGPT","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"![](http://www.topbots.com/wp-content/uploads/2019/04/cover_GPT_web-1280x640.jpg)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Usefull Links About OPENAI-GPT\n The following links are quite useful if you to further your knowledge about OPENAI-GPT(Please note most of these resources are related to open-ai GPT-2 but are sufficent to understand to orginal open-ai-gpt as well\n\n
  • [illustrated-gpt2](http://jalammar.github.io/illustrated-gpt2/)
    • \n
  • [openai-gpt-language-modeling](https://towardsdatascience.com/openai-gpt-language-modeling-on-gutenberg-with-tensorflow-keras-876f9f324b6c)
    • \n
  • [better-language-models](https://openai.com/blog/better-language-models/)
    • \n
  • [Orginal Paper](https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf)
    • \n    ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# # First load the real tokenizer\ntokenizer = OpenAIGPTTokenizer.from_pretrained('openai-gpt')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train=np.array(single_encoding_function(train_raw['comment_text'],tokenizer,'OPENAIGPT2'))\ny_train=np.array(train_raw['toxic'])\nX_valid=np.array(single_encoding_function(valid_raw['comment_text'],tokenizer,'OPENAIGPT2'))\ny_valid=np.array(valid_raw['toxic'])\nX_test=np.array(single_encoding_function(test_raw['content'],tokenizer,'OPENAIGPT2'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"steps_per_epoch = X_train.shape[0] // BATCH_SIZE","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train,valid,test=make_data()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\nmodel=compile_model('openai-gpt')\nprint(model.summary())\n\nhistory=model.fit(\n train,steps_per_epoch=steps_per_epoch,\n epochs=EPOCHS,callbacks=[early_stopping], validation_data=valid\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## OpenAIGPT Performance-Loss","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_loss(history)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## OpenAIGPT Performance-Metrics","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_metrics(history)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## OpenAIGPT-Confusion Matrix","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"y_predict=model.predict(valid, verbose=1)\ny_predict[ y_predict> 0.5] = 1\ny_predict[y_predict <= 0.5] = 0\nplot_cm(y_valid, y_predict, 'OpenAIGPT-Confusion Matrix')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## OpenAIGPT-ROC","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"y_predict_prob=model.predict(valid, verbose=1)\nfpr, tpr, _ = roc_curve(y_valid,y_predict_prob)\nroc_auc = auc(fpr, tpr)\nroc_curve_plot(fpr,tpr,roc_auc)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 3-DistilBert ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"![](https://jalammar.github.io/images/distilBERT/bert-distilbert-tutorial-sentence-embedding.png)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Usefull Links About DistilBERT\n The following links are quite useful if you to further your knowledge about Transformer-XL\n\n
  • [distilbert](https://medium.com/huggingface/distilbert-8cf3380435b5)
    • \n
  • [a-visual-guide-to-using-dsitillbert-for-the-first-time](http://jalammar.github.io/a-visual-guide-to-using-bert-for-the-first-time/)
    • \n
  • [Orginal Paper](https://arxiv.org/pdf/1910.01108.pdf)
    • \n    ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# # First load the real tokenizer\ntokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-cased')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train=np.array(single_encoding_function(train_raw['comment_text'],tokenizer,'DistilBert'))\ny_train=np.array(train_raw['toxic'])\nX_valid=np.array(single_encoding_function(valid_raw['comment_text'],tokenizer,'DistilBert'))\ny_valid=np.array(valid_raw['toxic'])\nX_test=np.array(single_encoding_function(test_raw['content'],tokenizer,'DistilBert'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train,valid,test=make_data()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"steps_per_epoch = X_train.shape[0] // BATCH_SIZE","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\nmodel=compile_model('distilbert-base-cased')\nprint(model.summary())\n\nhistory=model.fit(\n train,steps_per_epoch=steps_per_epoch,\n epochs=EPOCHS,callbacks=[early_stopping], validation_data=valid\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## DistilBert Performance-Loss","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_loss(history)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## DistilBert XL Performance-Metrics","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_metrics(history)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## DistilBert Performance-Confusion Matrix","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"y_predict=model.predict(valid, verbose=1)\ny_predict[ y_predict> 0.5] = 1\ny_predict[y_predict <= 0.5] = 0\nplot_cm(y_valid, y_predict, 'Transformer XL Performance-Confusion Matrix')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## DistilBert Performace-ROC","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"y_predict_prob=model.predict(valid, verbose=1)\nfpr, tpr, _ = roc_curve(y_valid,y_predict_prob)\nroc_auc = auc(fpr, tpr)\nroc_curve_plot(fpr,tpr,roc_auc)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 4-XLM","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"![](https://i.pinimg.com/564x/88/66/9b/88669b6455bde00c747fd9535e7ee98e.jpg)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Usefull Links About XLM\n The following links are quite useful if you to further your knowledge about XLM\n\n
  • [xlm-enhancing-bert-for-cross-lingual-language-model](https://towardsdatascience.com/xlm-enhancing-bert-for-cross-lingual-language-model-5aeed9e6f14b#:~:text=Background,(or%20sub%2Dwords).)
    • \n
  • [GitHub XLM](https://github.com/facebookresearch/XLM)
    • \n
  • [a-deep-dive-into-multilingual-nlp-models](https://peltarion.com/blog/data-science/a-deep-dive-into-multilingual-nlp-models)
    • \n
  • [Orginal Paper](a-deep-dive-into-multilingual-nlp-models)
    • \n    ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# # First load the real tokenizer\ntokenizer = XLMTokenizer.from_pretrained('xlm-mlm-en-2048')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train=np.array(single_encoding_function(train_raw['comment_text'],tokenizer,'XLM'))\ny_train=np.array(train_raw['toxic'])\nX_valid=np.array(single_encoding_function(valid_raw['comment_text'],tokenizer,'XLM'))\ny_valid=np.array(valid_raw['toxic'])\nX_test=np.array(single_encoding_function(test_raw['content'],tokenizer,'XLM'))\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train,valid,test=make_data()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\nmodel=compile_model('xlm-mlm-en-2048')\nprint(model.summary())\n\nhistory=model.fit(\n train,steps_per_epoch=steps_per_epoch,\n epochs=EPOCHS,callbacks=[early_stopping], validation_data=valid\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## XLM Performance-Loss","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_loss(history)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## XLM Performance-Metrics","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_metrics(history)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## XLM-Confusion Matrix","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"y_predict=model.predict(valid, verbose=1)\ny_predict[ y_predict> 0.5] = 1\ny_predict[y_predict <= 0.5] = 0\nplot_cm(y_valid, y_predict, 'XLM-Confusion Matrix')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## XLM-ROC Curve","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"y_predict_prob=model.predict(valid, verbose=1)\nfpr, tpr, _ = roc_curve(y_valid,y_predict_prob)\nroc_auc = auc(fpr, tpr)\nroc_curve_plot(fpr,tpr,roc_auc)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 5-XLM-Roberta-large -the Choosen one :D","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Usefull Links About XLM-Roberta-Large\n The following links are quite useful if you to further your knowledge about XLM-Roberta-Large\n\n
  • [xlm-roberta-the-multilingual-alternative](https://towardsdatascience.com/xlm-roberta-the-multilingual-alternative-for-non-english-nlp-cf0b889ccbbf)
    • \n
  • [Github-XLMR](https://github.com/pytorch/fairseq/tree/master/examples/xlmr)
    • \n
  • [bert-roberta-distilbert-xlnet-one-us](https://www.kdnuggets.com/2019/09/bert-roberta-distilbert-xlnet-one-use.html)
    • \n
  • [Explaining Roberta Technique](https://arxiv.org/pdf/1907.11692.pdf)
    • \n
  • [FaceBook AI post](https://ai.facebook.com/blog/-xlm-r-state-of-the-art-cross-lingual-understanding-through-self-supervision/)\n
  • [Orginal Paper](https://arxiv.org/pdf/1911.02116.pdf)
    • \n    ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Youtube Video-Explaining XLM-Roberta-Large!!!-Really Useful","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from IPython.display import YouTubeVideo\n\nYouTubeVideo(\"Ot6A3UFY72c\", width=800, height=300)\n\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def regular_encode(texts, tokenizer, maxlen=512):\n enc_di = tokenizer.batch_encode_plus(\n texts, \n return_attention_masks=False, \n return_token_type_ids=False,\n pad_to_max_length=True,\n max_length=maxlen\n )\n \n return np.array(enc_di['input_ids'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tokenizer = AutoTokenizer.from_pretrained('jplu/tf-xlm-roberta-large')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time \n\nX_train = regular_encode(train_raw.comment_text.values, tokenizer, maxlen=max_seq_length)\nX_valid = regular_encode(valid_raw.comment_text.values, tokenizer, maxlen=max_seq_length)\nX_test = regular_encode(test_raw.content.values, tokenizer, maxlen=max_seq_length)\n\ny_train = train_raw.toxic.values\ny_valid = valid_raw.toxic.values","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"steps_per_epoch = X_train.shape[0] // BATCH_SIZE","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train,valid,test=make_data()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\nfinal_model=compile_model('jplu/tf-xlm-roberta-large')\nprint(final_model.summary())\n\nhistory=final_model.fit(\n train,steps_per_epoch=steps_per_epoch,\n epochs=EPOCHS,callbacks=[early_stopping], validation_data=valid\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## XLM-Roberta-large-Loss","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_loss(history)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## XLM-Roberta-large-Metrics","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_metrics(history)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## XLM-Roberta-large-Confusion Matrix","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"y_predict=final_model.predict(valid, verbose=1)\ny_predict[ y_predict> 0.5] = 1\ny_predict[y_predict <= 0.5] = 0\nplot_cm(y_valid, y_predict, 'XLM-Roberta-Confusion Matrix')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## XLM-Roberta-large-ROC Curve","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"y_predict_prob=final_model.predict(valid, verbose=1)\nfpr, tpr, _ = roc_curve(y_valid,y_predict_prob)\nroc_auc = auc(fpr, tpr)\nroc_curve_plot(fpr,tpr,roc_auc)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"steps_per_epoch = X_valid.shape[0] // BATCH_SIZE\ntrain_history_2 = final_model.fit(\n valid.repeat(),\n steps_per_epoch=steps_per_epoch,\n epochs=EPOCHS\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 6-**Glovec Embedding LSTM**","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"![](https://user-images.githubusercontent.com/10358317/43802664-22c08c8a-9a9f-11e8-83e1-fea4bf334f6e.png)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Usefull Links About LSTM\n The following links are quite useful if you to further your knowledge about LSTMs\n\n
  • [Understanding-LSTMs](https://colah.github.io/posts/2015-08-Understanding-LSTMs/)
    • \n
  • [illustrated-guide-to-lstms-and-gru-s-a-step-by-step-explanation](https://towardsdatascience.com/illustrated-guide-to-lstms-and-gru-s-a-step-by-step-explanation-44e9eb85bf21)
    • \n
  • [Andrew Ng Explaining LSTM](https://www.coursera.org/lecture/nlp-sequence-models/long-short-term-memory-lstm-KXoay)
    • \n
  • [Orginal Paper](https://www.bioinf.jku.at/publications/older/2604.pdf)
    • \n    \n    \n\n The following links are quite useful if you want to understand about Glovec \n\n
  • [CS224n Notes](http://web.stanford.edu/class/cs224n/readings/cs224n-2019-notes02-wordvecs2.pdf)
    • \n
  • [Anrew Ng Explaining Glovec](https://www.coursera.org/lecture/nlp-sequence-models/glove-word-vectors-IxDTG)
    • \n
  • [CS22n Video](https://www.youtube.com/watch?v=ASn7ExxLZws&t=1s)
    • \n
  • [Orginal Paper](http://nlp.stanford.edu/pubs/glove.pdf)
    • \n    \n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Hyperparameter for LSTMs ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"max_seq_length = 512\nembedding_dim=300\nBATCH_SIZE = 32\nEPOCHS=1\nLEARNING_RATE=1e-5\nearly_stopping=early_stopping = tf.keras.callbacks.EarlyStopping(\n monitor='val_loss', \n verbose=1,\n patience=2,\n mode='max',\n restore_best_weights=True)\nAUTO = tf.data.experimental.AUTOTUNE","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## DataPipeline for LSTMs","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"\ntokenizer = Tokenizer(split=' ', oov_token='', filters=' ')\ntokenizer.fit_on_texts(train_raw['comment_text'].values)\n\n# this takes our sentences and replaces each word with an integer\nX_train = tokenizer.texts_to_sequences(train_raw['comment_text'].values)\nX_train=np.array(pad_sequences(X_train, max_seq_length))\ny_train=np.array(train_raw['toxic'])\n\nX_valid = tokenizer.texts_to_sequences(valid_raw['comment_text'].values)\nX_valid = np.array(pad_sequences(X_valid, max_seq_length))\ny_valid=np.array(valid_raw['toxic'])\n\nX_test = tokenizer.texts_to_sequences(test_raw['content'].values)\nX_test = np.array(pad_sequences(X_test, max_seq_length))\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('The X_train,X_valid,X_test shape respectively is {}-{}-{}'.format(X_train.shape,X_valid.shape,X_test.shape))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train,valid,test=make_data()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Glovec Embedding","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"!wget http://nlp.stanford.edu/data/glove.6B.zip\n!unzip glove*.zip","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"embeddings_index = {}\n\nf = open(os.path.join(os.getcwd(), 'glove.6B.{}d.txt'.format(str(embedding_dim))))\nfor line in f:\n values = line.split()\n word = values[0]\n coefs = np.asarray(values[1:], dtype='float32')\n embeddings_index[word] = coefs\nf.close()\n\nprint('Found %s word vectors.' % len(embeddings_index))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# first create a matrix of zeros, this is our embedding matrix\nword_index = tokenizer.word_index\nnum_words=len(word_index)+1\nembedding_matrix = np.zeros((num_words, embedding_dim))\n\n# for each word in out tokenizer lets try to find that work in our w2v model\nfor word, i in tqdm(word_index.items()):\n embedding_vector = embeddings_index.get(word)\n if embedding_vector is not None:\n # we found the word - add that words vector to the matrix\n embedding_matrix[i] = embedding_vector\n else:\n # doesn't exist, assign a random vector\n embedding_matrix[i] = np.random.randn(embedding_dim)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"steps_per_epoch=X_train.shape[0]//BATCH_SIZE\nwith strategy.scope():\n model = Sequential()\n\n model.add(Embedding(num_words,\n embedding_dim,\n embeddings_initializer=Constant(embedding_matrix),\n input_length=max_seq_length,\n trainable=True))\n model.add(SpatialDropout1D(0.2))\n model.add(Bidirectional(LSTM(64, return_sequences=True)))\n model.add(Bidirectional(LSTM(32)))\n model.add(Dropout(0.25))\n model.add(Dense(units=1, activation='sigmoid'))\n\n METRICS = [\n tf.keras.metrics.TruePositives(name='tp'),\n tf.keras.metrics.FalsePositives(name='fp'),\n tf.keras.metrics.TrueNegatives(name='tn'),\n tf.keras.metrics.FalseNegatives(name='fn'), \n tf.keras.metrics.BinaryAccuracy(name='accuracy'),\n tf.keras.metrics.Precision(name='precision'),\n tf.keras.metrics.Recall(name='recall'),\n tf.keras.metrics.AUC(name='auc')]\n model.compile(loss ='binary_crossentropy', optimizer=tf.keras.optimizers.Adam(\n learning_rate=LEARNING_RATE),metrics =METRICS)\n\nhistory=model.fit(\ntrain,steps_per_epoch=steps_per_epoch,epochs=EPOCHS,callbacks=[early_stopping], validation_data=valid)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## LSTM with Glovec Embedding Loss","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_loss(history)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## LSTM with Glovec Embedding-Metrics","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_metrics(history)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## LSTM with Glovec Embedding-Confusion Matrix","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"y_predict=model.predict(valid, verbose=1)\ny_predict[y_predict> 0.5] = 1\ny_predict[y_predict <= 0.5] = 0\nplot_cm(y_valid, y_predict, 'LSTM with Glovec-Confusion Matrix')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## LSTM with Glovec Embedding-ROC","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"y_predict_prob=model.predict(valid, verbose=1)\nfpr, tpr, _ = roc_curve(y_valid,y_predict_prob)\nroc_auc = auc(fpr, tpr)\nroc_curve_plot(fpr,tpr,roc_auc)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 7-**LSTM with attention Mechanism**","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"![](https://www.mdpi.com/sensors/sensors-19-00861/article_deploy/html/images/sensors-19-00861-g004.png)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Usefull Links About LSTM with Attention\n The following links are quite useful if you to further your knowledge about LSTMs\n\n
  • [comprehensive-guide-attention-mechanism-deep-learning](https://www.analyticsvidhya.com/blog/2019/11/comprehensive-guide-attention-mechanism-deep-learning/)
    • \n
  • [attention-long-short-term-memory-recurrent-neural-networks](https://machinelearningmastery.com/attention-long-short-term-memory-recurrent-neural-networks/)
    • \n
  • [Andrew Ng Explaining LSTM with Attention](https://www.youtube.com/watch?v=SysgYptB198)
    • \n
  • [Orginal Paper](https://www.aclweb.org/anthology/D16-1058.pdf)
    • \n    ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"with strategy.scope():\n # Define an input sequence and process it.\n inputs = Input(shape=(max_seq_length,))\n embedding=Embedding(num_words,\n embedding_dim,\n embeddings_initializer=Constant(embedding_matrix),\n input_length=max_seq_length,\n trainable=True)(inputs)\n \n\n # Apply dropout to prevent overfitting\n embedded_inputs = Dropout(0.2)(embedding)\n \n # Apply Bidirectional LSTM over embedded inputs\n lstm_outs =Bidirectional(\n LSTM(300, return_sequences=True)\n )(embedded_inputs)\n \n # Apply dropout to LSTM outputs to prevent overfitting\n lstm_outs = Dropout(0.2)(lstm_outs)\n \n # Attention Mechanism - Generate attention vectors\n attention_vector = TimeDistributed(Dense(1))(lstm_outs)\n attention_vector = Reshape((X_train.shape[1],))(attention_vector)\n attention_vector = Activation('softmax', name='attention_vec')(attention_vector)\n attention_output = Dot(axes=1)([lstm_outs, attention_vector])\n \n # Last layer: fully connected with softmax activation\n fc = Dense(300, activation='relu')(attention_output)\n output = Dense(1, activation='sigmoid')(fc)\n \n model=tf.keras.Model(inputs,output)\n \n \n METRICS = [\n tf.keras.metrics.TruePositives(name='tp'),\n tf.keras.metrics.FalsePositives(name='fp'),\n tf.keras.metrics.TrueNegatives(name='tn'),\n tf.keras.metrics.FalseNegatives(name='fn'), \n tf.keras.metrics.BinaryAccuracy(name='accuracy'),\n tf.keras.metrics.Precision(name='precision'),\n tf.keras.metrics.Recall(name='recall'),\n tf.keras.metrics.AUC(name='auc')]\n model.compile(loss ='binary_crossentropy', optimizer=tf.keras.optimizers.Adam(\n learning_rate=LEARNING_RATE),metrics =METRICS)\n print(model.summary())\n\nhistory=model.fit(\ntrain,steps_per_epoch=steps_per_epoch,epochs=EPOCHS,callbacks=[early_stopping], validation_data=valid)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## LSTM with Attention Mechanism -Loss","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_loss(history)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## LSTM with Attention Mechanism-Metrics","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_metrics(history)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## LSTM with Attention Mechanism-Confusion Matrix","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"y_predict=model.predict(valid, verbose=1)\ny_predict[ y_predict> 0.5] = 1\ny_predict[y_predict <= 0.5] = 0\nplot_cm(y_valid, y_predict, 'LSTM with Attention Mechanism-Confusion Matrix')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## LSTM with Attention Mechanism-ROC","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fy_predict_prob=model.predict(valid, verbose=1)\nfpr, tpr, _ = roc_curve(y_valid,y_predict_prob)\nroc_auc = auc(fpr, tpr)\nroc_curve_plot(fpr,tpr,roc_auc)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Final Submission","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"sub['toxic'] = final_model.predict(test, verbose=1)\nsub.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}