{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## What are TPUs?\nThe Tensor Processing Unit (TPU) is a custom integrated chip, designed specifically to accelerate the process of training machine learning models. \n\n## TPUs for free at Kaggle\n**You can use up to 30 hours per week of TPUs and up to 9h at a time in a single session.**\n**For more info you can visit [here](https://www.kaggle.com/docs/tpu).**\n\n## Why do we need TFRecord format?\nThe TFRecord format is tensorflow's custom data format which is simple for storing a sequence of binary records. The advantages of using TFRecords are amazingly more efficient storage, fast I/O, self-contained files, etc. The main advantage of TPUs are faster I/O which results in faster model training.\n\nFor understanding the basics of TFRecords, please visit Ryan Holbrook notebook: [TFRecords Basics](https://www.kaggle.com/ryanholbrook/tfrecords-basics).\n\n**In this notebook you will learn how to convert text dataset into TFRecord format.**\n\n## Useful resources which helped me:¶\n- https://www.tensorflow.org/tutorials/load_data/tfrecord\n- https://www.kaggle.com/mgornergoogle/five-flowers-with-keras-and-xception-on-tpu\n- https://towardsdatascience.com/a-practical-guide-to-tfrecords-584536bc786c\n- https://www.kaggle.com/omkargangan/commonlit-readability-competition\n- https://cloud.google.com/blog/products/ai-machine-learning/what-makes-tpus-fine-tuned-for-deep-learning\n- https://pub.towardsai.net/writing-tfrecord-files-the-right-way-7c3cee3d7b12","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2021-10-13T17:17:10.043669Z","iopub.execute_input":"2021-10-13T17:17:10.043978Z","iopub.status.idle":"2021-10-13T17:17:10.051696Z","shell.execute_reply.started":"2021-10-13T17:17:10.043946Z","shell.execute_reply":"2021-10-13T17:17:10.050764Z"}}},{"cell_type":"markdown","source":"# Imports","metadata":{}},{"cell_type":"code","source":"import re\nimport numpy as np \nimport pandas as pd \nimport tensorflow as tf\nfrom sklearn.model_selection import train_test_split\nfrom tensorflow.keras.preprocessing.text import Tokenizer\nfrom tensorflow.keras.preprocessing.sequence import pad_sequences\nfrom nltk.corpus import stopwords\nfrom string import punctuation\nfrom nltk.tokenize import word_tokenize\nfrom tensorflow.keras.preprocessing import sequence\nfrom sklearn.feature_extraction.text import TfidfVectorizer\nfrom sklearn.feature_extraction.text import CountVectorizer\nfrom nltk.stem import WordNetLemmatizer\nfrom tensorflow.keras.utils import to_categorical","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Load the data","metadata":{}},{"cell_type":"code","source":"df = pd.read_csv('../input/jigsaw-multilingual-toxic-comment-classification/jigsaw-toxic-comment-train.csv')\ndf = df.drop(labels = ['id','severe_toxic','obscene','threat','insult','identity_hate'], axis=1)\ndf.columns = ['comment','toxic']","metadata":{"execution":{"iopub.status.busy":"2021-10-13T17:06:08.420918Z","iopub.execute_input":"2021-10-13T17:06:08.421193Z","iopub.status.idle":"2021-10-13T17:06:11.38178Z","shell.execute_reply.started":"2021-10-13T17:06:08.421164Z","shell.execute_reply":"2021-10-13T17:06:11.380967Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Clean the data","metadata":{}},{"cell_type":"code","source":"def clean_document(doc):\n    doc = doc.lower()\n    # tokenization\n    tokens = word_tokenize(doc)\n    stop = stopwords.words('english')\n    bad_tokens = stop + list(punctuation)\n    clean_tokens = [t for t in tokens if t.lower() not in bad_tokens]\n    # lemmatization\n    lemma = WordNetLemmatizer()\n    clean_tokens = [lemma.lemmatize(t) for t in clean_tokens]\n    return ' '.join(clean_tokens)","metadata":{"execution":{"iopub.status.busy":"2021-10-13T17:06:20.947373Z","iopub.execute_input":"2021-10-13T17:06:20.947662Z","iopub.status.idle":"2021-10-13T17:06:20.953561Z","shell.execute_reply.started":"2021-10-13T17:06:20.947634Z","shell.execute_reply":"2021-10-13T17:06:20.952908Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.comment = df.comment.apply(clean_document)","metadata":{"execution":{"iopub.status.busy":"2021-10-13T17:07:14.503008Z","iopub.execute_input":"2021-10-13T17:07:14.503295Z","iopub.status.idle":"2021-10-13T17:13:34.652615Z","shell.execute_reply.started":"2021-10-13T17:07:14.503266Z","shell.execute_reply":"2021-10-13T17:13:34.651693Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X = df['comment']\ny = df['toxic']","metadata":{"execution":{"iopub.status.busy":"2021-10-13T17:16:58.736532Z","iopub.execute_input":"2021-10-13T17:16:58.736843Z","iopub.status.idle":"2021-10-13T17:16:58.741129Z","shell.execute_reply.started":"2021-10-13T17:16:58.736811Z","shell.execute_reply":"2021-10-13T17:16:58.740263Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# One-hot encoding the labels","metadata":{}},{"cell_type":"code","source":"y = to_categorical(y, num_classes=2)","metadata":{"execution":{"iopub.status.busy":"2021-10-13T17:17:13.267258Z","iopub.execute_input":"2021-10-13T17:17:13.267561Z","iopub.status.idle":"2021-10-13T17:17:13.276483Z","shell.execute_reply.started":"2021-10-13T17:17:13.267529Z","shell.execute_reply":"2021-10-13T17:17:13.275618Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Divide into test-train","metadata":{}},{"cell_type":"code","source":"from sklearn.model_selection import train_test_split\nX_train, X_test, y_train, y_test = train_test_split(X,y,test_size=0.1, random_state=1)","metadata":{"execution":{"iopub.status.busy":"2021-10-13T17:17:15.417434Z","iopub.execute_input":"2021-10-13T17:17:15.418018Z","iopub.status.idle":"2021-10-13T17:17:15.477342Z","shell.execute_reply.started":"2021-10-13T17:17:15.41797Z","shell.execute_reply":"2021-10-13T17:17:15.476461Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Data Processing","metadata":{}},{"cell_type":"code","source":"tokenizer = Tokenizer()\ntokenizer.fit_on_texts(X_train)","metadata":{"execution":{"iopub.status.busy":"2021-10-13T17:17:17.141573Z","iopub.execute_input":"2021-10-13T17:17:17.141833Z","iopub.status.idle":"2021-10-13T17:17:28.334738Z","shell.execute_reply.started":"2021-10-13T17:17:17.141806Z","shell.execute_reply":"2021-10-13T17:17:28.333815Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"vocabulary = tokenizer.index_word\nvocab_len = len(vocabulary)\nvocab_len","metadata":{"execution":{"iopub.status.busy":"2021-10-13T17:17:28.336231Z","iopub.execute_input":"2021-10-13T17:17:28.336569Z","iopub.status.idle":"2021-10-13T17:17:28.352287Z","shell.execute_reply.started":"2021-10-13T17:17:28.33654Z","shell.execute_reply":"2021-10-13T17:17:28.351685Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_sequence = tokenizer.texts_to_sequences(X_train)","metadata":{"execution":{"iopub.status.busy":"2021-10-13T17:17:28.353346Z","iopub.execute_input":"2021-10-13T17:17:28.353975Z","iopub.status.idle":"2021-10-13T17:17:36.690298Z","shell.execute_reply.started":"2021-10-13T17:17:28.353938Z","shell.execute_reply":"2021-10-13T17:17:36.689443Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"doc_len = []\nfor doc in train_sequence:\n    doc_len.append(len(doc))\nmax(doc_len)","metadata":{"execution":{"iopub.status.busy":"2021-10-13T17:17:36.692112Z","iopub.execute_input":"2021-10-13T17:17:36.692433Z","iopub.status.idle":"2021-10-13T17:17:36.754042Z","shell.execute_reply.started":"2021-10-13T17:17:36.692399Z","shell.execute_reply":"2021-10-13T17:17:36.753139Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"np.quantile(doc_len, 0.99)\n","metadata":{"execution":{"iopub.status.busy":"2021-10-13T17:17:36.755216Z","iopub.execute_input":"2021-10-13T17:17:36.755781Z","iopub.status.idle":"2021-10-13T17:17:36.805886Z","shell.execute_reply.started":"2021-10-13T17:17:36.755742Z","shell.execute_reply":"2021-10-13T17:17:36.804781Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"max_len = 347","metadata":{"execution":{"iopub.status.busy":"2021-10-13T17:17:36.807291Z","iopub.execute_input":"2021-10-13T17:17:36.808166Z","iopub.status.idle":"2021-10-13T17:17:36.813133Z","shell.execute_reply.started":"2021-10-13T17:17:36.808118Z","shell.execute_reply":"2021-10-13T17:17:36.812458Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_sequence_matrix = sequence.pad_sequences(train_sequence, maxlen= max_len)\ntest_sequence = tokenizer.texts_to_sequences(X_test)\ntest_sequence_matrix = sequence.pad_sequences(test_sequence, maxlen= max_len)","metadata":{"execution":{"iopub.status.busy":"2021-10-13T17:17:36.814273Z","iopub.execute_input":"2021-10-13T17:17:36.815141Z","iopub.status.idle":"2021-10-13T17:17:40.885331Z","shell.execute_reply.started":"2021-10-13T17:17:36.815092Z","shell.execute_reply":"2021-10-13T17:17:40.884432Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"len(train_sequence_matrix)","metadata":{"execution":{"iopub.status.busy":"2021-10-13T17:37:40.713923Z","iopub.execute_input":"2021-10-13T17:37:40.714727Z","iopub.status.idle":"2021-10-13T17:37:40.720393Z","shell.execute_reply.started":"2021-10-13T17:37:40.714688Z","shell.execute_reply":"2021-10-13T17:37:40.719641Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(train_sequence_matrix.shape)\nprint(y_train.shape)","metadata":{"execution":{"iopub.status.busy":"2021-10-13T17:17:40.886655Z","iopub.execute_input":"2021-10-13T17:17:40.887246Z","iopub.status.idle":"2021-10-13T17:17:40.892728Z","shell.execute_reply.started":"2021-10-13T17:17:40.887193Z","shell.execute_reply":"2021-10-13T17:17:40.891893Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_sequence_matrix = train_sequence_matrix.reshape(-1,347,1)\ntrain_sequence_matrix.shape","metadata":{"execution":{"iopub.status.busy":"2021-10-13T17:32:06.810102Z","iopub.execute_input":"2021-10-13T17:32:06.810506Z","iopub.status.idle":"2021-10-13T17:32:06.818324Z","shell.execute_reply.started":"2021-10-13T17:32:06.810469Z","shell.execute_reply":"2021-10-13T17:32:06.817319Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_sequence_matrix = train_sequence_matrix.reshape(-1,347,1)\ntrain_sequence_matrix.shape","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Feature Creation functions\n\nThe following functions can be used to convert a value to a type compatible which takes a scalar input values and returns a tf.train.Feature.","metadata":{}},{"cell_type":"code","source":"def _bytes_feature(value):\n    if isinstance(value, type(tf.constant(0))): # if value ist tensor\n        value = value.numpy() # get value of tensor\n    return tf.train.Feature(bytes_list=tf.train.BytesList(value=[value]))\n\ndef _int64_feature(value):\n    return tf.train.Feature(int64_list=tf.train.Int64List(value=[value]))\n\ndef _float_feature(value):\n    return tf.train.Feature(float_list=tf.train.FloatList(value=[value]))\n\ndef serialize_array(array):\n    array = tf.io.serialize_tensor(array)\n    return array","metadata":{"execution":{"iopub.status.busy":"2021-10-13T17:17:57.108338Z","iopub.execute_input":"2021-10-13T17:17:57.108755Z","iopub.status.idle":"2021-10-13T17:17:57.116503Z","shell.execute_reply.started":"2021-10-13T17:17:57.108724Z","shell.execute_reply":"2021-10-13T17:17:57.115552Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Serializing and Writing \n\nNow, we'll create a dictionary to store the actual image, height, width and depth of the image and the label where we first serialize the array and then convert it to a bytes_feature.  All these `key:value` mappings make up the features for one Example.\n","metadata":{}},{"cell_type":"code","source":"def parse_text_data(text, label):\n    data = {\n        'text' : _bytes_feature(serialize_array(text)),\n        'label' : _bytes_feature(serialize_array(label))\n    }\n    out = tf.train.Example(features=tf.train.Features(feature=data))\n    return out","metadata":{"execution":{"iopub.status.busy":"2021-10-13T17:18:12.074193Z","iopub.execute_input":"2021-10-13T17:18:12.07465Z","iopub.status.idle":"2021-10-13T17:18:12.080144Z","shell.execute_reply.started":"2021-10-13T17:18:12.074618Z","shell.execute_reply":"2021-10-13T17:18:12.079016Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def write_text_to_tfr(text_data, label, filename:str=\"text\"):\n    filename= filename+\".tfrecords\"\n    writer = tf.io.TFRecordWriter(filename)\n    count = 0\n    for index in range(len(text_data)):\n        current_text = text_data[index] \n        current_label = label[index]\n        out = parse_text_data(text=current_text, label=current_label)\n        writer.write(out.SerializeToString())\n        count += 1\n    writer.close()\n    print(f\"Wrote {count} elements to TFRecord\")\n    return count","metadata":{"execution":{"iopub.status.busy":"2021-10-13T17:18:20.205823Z","iopub.execute_input":"2021-10-13T17:18:20.206116Z","iopub.status.idle":"2021-10-13T17:18:20.213263Z","shell.execute_reply.started":"2021-10-13T17:18:20.206087Z","shell.execute_reply":"2021-10-13T17:18:20.212323Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"write_text_to_tfr(text_data=train_sequence_matrix, label=y_train, filename=\"jigsaw_toxic_comment_train\")","metadata":{"execution":{"iopub.status.busy":"2021-10-13T17:19:09.310054Z","iopub.execute_input":"2021-10-13T17:19:09.310336Z","iopub.status.idle":"2021-10-13T17:19:38.360603Z","shell.execute_reply.started":"2021-10-13T17:19:09.310308Z","shell.execute_reply":"2021-10-13T17:19:38.359651Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"write_text_to_tfr(text_data=test_sequence_matrix, label=y_test, filename=\"jigsaw_toxic_comment_test\")","metadata":{"execution":{"iopub.status.busy":"2021-10-13T17:19:38.362341Z","iopub.execute_input":"2021-10-13T17:19:38.363343Z","iopub.status.idle":"2021-10-13T17:19:41.554963Z","shell.execute_reply.started":"2021-10-13T17:19:38.363293Z","shell.execute_reply":"2021-10-13T17:19:41.554033Z"},"trusted":true},"execution_count":null,"outputs":[]}]}