{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Transfer Learning for NLP with TensorFlow Hub\n\n*Author: Eda AYDIN*","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"}},{"cell_type":"markdown","source":"# A. Project Objectives\n\nWe're going to focus on three learning objectives:\n\n1. Use pre-trained NLP text embeddings models from [TensorFlow Hub](https://tfhub.dev/)\n2. Perform transfer learning to fine-tune models on real-world text data\n3. Visualize model performance metrics with [TensorBoard](https://www.tensorflow.org/tensorboard)\n\nBy the time we complete this project, you will be able to use pre-trained NLP text embedding models from TensorFlow Hub, perform transfer learning to fine-tune models on real-world data, build and evaluate multiple models for text classification with TensorFlow, and visualize model performance metrics with Tensorboard.\n\n**Prerequisities:** In order to successfully complete this project, we should be competent in the Python programming language, be familiar with deep learning for Natural Language Processing (NLP), and have trained models with TensorFlow or and its Keras API.\n\n","metadata":{}},{"cell_type":"markdown","source":"# B. Project Structure","metadata":{}},{"cell_type":"markdown","source":"# Task 1: Introduction to the Project\n\n[TensorFlow Hub](https://tfhub.dev/) is a repository of pre-trained TensorFlow models.\n\nIn this project, we will use pre-trained models from TensorFlow Hub with [`tf.keras`](https://www.tensorflow.org/api_docs/python/tf/keras) for text classification. Transfer learning makes it possible to save training resources and to achieve good model generalization even when training on a small dataset. In this project, we will demonstrate this by training with several different TF-Hub modules\n","metadata":{}},{"cell_type":"markdown","source":"# Task 2:  Setup your TensorFlow and Colab Runtime","metadata":{}},{"cell_type":"code","source":"!nvidia-smi","metadata":{"execution":{"iopub.status.busy":"2023-03-12T18:51:07.305398Z","iopub.execute_input":"2023-03-12T18:51:07.305751Z","iopub.status.idle":"2023-03-12T18:51:08.294418Z","shell.execute_reply.started":"2023-03-12T18:51:07.305719Z","shell.execute_reply":"2023-03-12T18:51:08.293259Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\n\nimport tensorflow as tf\nimport tensorflow_hub as hub\nimport tensorflow_datasets as tfds\n\nimport matplotlib.pyplot as plt\nplt.rcParams['figure.figsize'] = (12, 8)\nfrom  IPython import display\n\nimport pathlib\nimport shutil\nimport tempfile\n\n!pip install -q git+https://github.com/tensorflow/docs\n\nimport tensorflow_docs as tfdocs\nimport tensorflow_docs.modeling\nimport tensorflow_docs.plots\n\nprint(\"Version: \", tf.__version__)\nprint(\"Hub version: \", hub.__version__)\nprint(\"GPU is\", \"available\" if tf.config.list_physical_devices('GPU') else \"NOT AVAILABLE\")\n\nlogdir = pathlib.Path(tempfile.mkdtemp())/\"tensorboard_logs\"\nshutil.rmtree(logdir, ignore_errors=True)","metadata":{"execution":{"iopub.status.busy":"2023-03-12T18:51:08.296649Z","iopub.execute_input":"2023-03-12T18:51:08.296958Z","iopub.status.idle":"2023-03-12T18:51:32.452513Z","shell.execute_reply.started":"2023-03-12T18:51:08.296925Z","shell.execute_reply":"2023-03-12T18:51:32.451048Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Task 3: Load the Quora Insincere Questions Dataset\n\n### Data and General Description\n\nIn this project we will predicting whether a question asked on Quora is sincere or not.\n\nAn insincere question is defined as a question intended to make a statement rather than look for helpful answers. Some characteristics that can signify that a question is insincere:\n\n* Has a non-neutral tone\n    * Has an exaggerated tone to underscore a point about a group of people\n    * Is rhetorical and meant to imply a statement about a group of people\n* Is disparaging or inflammatory\n    * Suggests a discriminatory idea against a protected class of people, or seeks confirmation of a stereotype\n    * Makes disparaging attacks/insults against a specific person or group of people\n    * Based on an outlandish premise about a group of people\n    * Disparages against a characteristic that is not fixable and not measurable\n* Isn't grounded in reality\n    * Based on false information, or contains absurd assumptions\n* Uses sexual content (incest, bestiality, pedophilia) for shock value, and not to seek genuine answers\n\nThe training data includes the question that was asked, and whether it was identified as insincere `(target = 1)`. The ground-truth labels contain some amount of noise: they are not guaranteed to be perfect.\n\nNote that the distribution of questions in the dataset should not be taken to be representative of the distribution of questions asked on Quora. This is, in part, because of the combination of sampling procedures and sanitization measures that have been applied to the final dataset.\n\n### File descriptions\n\n* train.csv - the training set\n* test.csv - the test set\n* sample_submission.csv - A sample submission in the correct format\n* enbeddings/ - (see below)\n\n### Data fields\n\n* qid - unique question identifier\n* question_text - Quora question text\n* target - a question labeled \"insincere\" has a value of 1, otherwise 0\n\nThis is a Kernels-only competition. The files in this Data section are downloadable for reference in Stage 1. Stage 2 files will only be available in Kernels and not available for download.\n\n### What will be available in the 2nd stage of the competition?\n\nIn the second stage of the competition, we will re-run your selected Kernels. The following files will be swapped with new data:\n\n* `test.csv` - This will be swapped with the complete public and private test dataset. This file will have ~56k rows in stage 1 and ~376k rows in stage 2. The public leaderboard data remains the same for both versions. The file name will be the same (both test.csv) to ensure that your code will run.\n* `sample_submission.csv` - similar to test.csv, this will be changed from ~56k in stage 1 to ~376k rows in stage 2 . The file name will remain the same.\n\n### Embeddings\n\nExternal data sources are not allowed for this competition. We are, though, providing a number of word embeddings along with the dataset that can be used in the models. These are as follows:\n\n* GoogleNews-vectors-negative300 - https://code.google.com/archive/p/word2vec/\n* glove.840B.300d - https://nlp.stanford.edu/projects/glove/\n* paragram_300_sl999 - https://cogcomp.org/page/resource_view/106\n* wiki-news-300d-1M - https://fasttext.cc/docs/en/english-vectors.html","metadata":{}},{"cell_type":"code","source":"\"\"\"\n# Decompress and read the data into a pandas DataFrame without \ndf = pd.read_csv(\"https://archive.org/download/fine-tune-bert-tensorflow-train.csv/train.csv.zip\",compression = 'zip', low_memory = False)\ndf.shape\n\"\"\"","metadata":{"execution":{"iopub.status.busy":"2023-03-12T18:51:32.454854Z","iopub.execute_input":"2023-03-12T18:51:32.456013Z","iopub.status.idle":"2023-03-12T18:51:32.464243Z","shell.execute_reply.started":"2023-03-12T18:51:32.455968Z","shell.execute_reply":"2023-03-12T18:51:32.463110Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df = pd.read_csv(\"/kaggle/input/quora-insincere-questions-classification/train.csv\")\ntest_df = pd.read_csv(\"/kaggle/input/quora-insincere-questions-classification/test.csv\")","metadata":{"execution":{"iopub.status.busy":"2023-03-12T18:51:32.467850Z","iopub.execute_input":"2023-03-12T18:51:32.468666Z","iopub.status.idle":"2023-03-12T18:51:37.227183Z","shell.execute_reply.started":"2023-03-12T18:51:32.468624Z","shell.execute_reply":"2023-03-12T18:51:37.226027Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.head()","metadata":{"execution":{"iopub.status.busy":"2023-03-12T18:51:37.232418Z","iopub.execute_input":"2023-03-12T18:51:37.233275Z","iopub.status.idle":"2023-03-12T18:51:37.256325Z","shell.execute_reply.started":"2023-03-12T18:51:37.233226Z","shell.execute_reply":"2023-03-12T18:51:37.255306Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_df.head()","metadata":{"execution":{"iopub.status.busy":"2023-03-12T18:51:37.260500Z","iopub.execute_input":"2023-03-12T18:51:37.263413Z","iopub.status.idle":"2023-03-12T18:51:37.278991Z","shell.execute_reply.started":"2023-03-12T18:51:37.263377Z","shell.execute_reply":"2023-03-12T18:51:37.278086Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.shape","metadata":{"execution":{"iopub.status.busy":"2023-03-12T18:51:37.283397Z","iopub.execute_input":"2023-03-12T18:51:37.284190Z","iopub.status.idle":"2023-03-12T18:51:37.295592Z","shell.execute_reply.started":"2023-03-12T18:51:37.284125Z","shell.execute_reply":"2023-03-12T18:51:37.294276Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.isnull().sum()","metadata":{"execution":{"iopub.status.busy":"2023-03-12T18:51:37.300309Z","iopub.execute_input":"2023-03-12T18:51:37.301231Z","iopub.status.idle":"2023-03-12T18:51:37.505471Z","shell.execute_reply.started":"2023-03-12T18:51:37.301195Z","shell.execute_reply":"2023-03-12T18:51:37.498044Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Count the number of occurrences of each target value\ntarget_counts = df['target'].value_counts()\n\n# Create a list with the target labels\ntarget_labels = ['Sincere', 'Insincere']\n\n# Create a list with the target percentages\ntarget_percentages = [target_counts[0] / len(df), target_counts[1] / len(df)]\n\n# Create the pie plot\nfig, ax = plt.subplots(figsize= (6,6))\nax.pie(target_percentages,\n       labels=target_labels,\n       autopct='%1.1f%%',\n       wedgeprops={'linewidth': 3.0, 'edgecolor': 'white'},\n       textprops={\"size\":\"x-large\"})\nax.set_title(\"Target Distribution\", fontsize=18)\nplt.axis('equal')\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2023-03-12T18:51:37.509536Z","iopub.execute_input":"2023-03-12T18:51:37.512124Z","iopub.status.idle":"2023-03-12T18:51:37.721617Z","shell.execute_reply.started":"2023-03-12T18:51:37.512085Z","shell.execute_reply":"2023-03-12T18:51:37.720381Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Split the data","metadata":{}},{"cell_type":"code","source":"from sklearn.model_selection import train_test_split\n\ntrain_df, remaning = train_test_split(df,\n                                     random_state = 42,\n                                     train_size = 0.01,\n                                     stratify = df.target.values)\nvalid_df, _ = train_test_split(remaning,\n                            random_state = 42,\n                            train_size = 0.001,\n                            stratify = remaning.target.values)\ntrain_df.shape, valid_df.shape","metadata":{"execution":{"iopub.status.busy":"2023-03-12T18:51:37.730434Z","iopub.execute_input":"2023-03-12T18:51:37.730987Z","iopub.status.idle":"2023-03-12T18:51:39.081703Z","shell.execute_reply.started":"2023-03-12T18:51:37.730934Z","shell.execute_reply":"2023-03-12T18:51:39.080462Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.target.head(15).values","metadata":{"execution":{"iopub.status.busy":"2023-03-12T18:51:39.083503Z","iopub.execute_input":"2023-03-12T18:51:39.084184Z","iopub.status.idle":"2023-03-12T18:51:39.093131Z","shell.execute_reply.started":"2023-03-12T18:51:39.084144Z","shell.execute_reply":"2023-03-12T18:51:39.092078Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.question_text.head(15).values","metadata":{"execution":{"iopub.status.busy":"2023-03-12T18:51:39.094852Z","iopub.execute_input":"2023-03-12T18:51:39.095710Z","iopub.status.idle":"2023-03-12T18:51:39.105190Z","shell.execute_reply.started":"2023-03-12T18:51:39.095672Z","shell.execute_reply":"2023-03-12T18:51:39.103647Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Task 4: TensorFlow Hub for Natural Language Processing  ","metadata":{}},{"cell_type":"markdown","source":"Our text data consits of questions and corresponding labels.\n\nYou can think of a question vector as a distributed representation of a question, and is computed for every question in the training set. The question vector along with the output label is then used to train the statistical classification model. \n\nThe intuition is that the question vector captures the semantics of the question and, as a result, can be effectively used for classification. \n\nTo obtain question vectors, we have two alternatives that have been used for several text classification problems in NLP: \n* word-based representations and \n* context-based representations","metadata":{}},{"cell_type":"markdown","source":"#### Word-based Representations\n\n- A **word-based representation** of a question combines word embeddings of the content words in the question. We can use the average of the word embeddings of content words in the question. Average of word embeddings have been used for different NLP tasks.\n- Examples of pre-trained embeddings include:\n  - **Word2Vec**: These are pre-trained embeddings of words learned from a large text corpora. Word2Vec has been pre-trained on a corpus of news articles with  300 million tokens, resulting in 300-dimensional vectors.\n  - **GloVe**: has been pre-trained on a corpus of tweets with 27 billion tokens, resulting in 200-dimensional vectors.\n","metadata":{}},{"cell_type":"markdown","source":"#### Context-based Representations\n\n- **Context-based representations** may use language models to generate vectors of sentences. So, instead of learning vectors for individual words in the sentence, they compute a vector for sentences on the whole, by taking into account the order of words and the set of co-occurring words.\n- Examples of deep contextualised vectors include:\n  - **Embeddings from Language Models (ELMo)**: uses character-based word representations and bidirectional LSTMs. The pre-trained model computes a contextualised vector of 1024 dimensions. ELMo is available on Tensorflow Hub.\n  - **Universal Sentence Encoder (USE)**: The encoder uses a Transformer  architecture that uses attention mechanism to incorporate information about the order and the collection of words. The pre-trained model of USE that returns a vector of 512 dimensions is also available on Tensorflow Hub.\n  - **Neural-Net Language Model (NNLM)**: The model simultaneously learns representations of words and probability functions for word sequences, allowing it to capture semantics of a sentence. We will use a  pretrained  models available on Tensorflow Hub, that are trained on the English Google News 200B corpus, and computes a vector of 128 dimensions for the larger model and 50 dimensions for the smaller model.\n","metadata":{}},{"cell_type":"markdown","source":"Tensorflow Hub provides a number of [modules](https://tfhub.dev/s?module-type=text-embedding&tf-version=tf2&q=tf2) to convert sentences into embeddings such as Universal sentence ecoders, NNLM, BERT and Wikiwords.","metadata":{}},{"cell_type":"markdown","source":"Transfer learning makes it possible to save training resources and to achieve good model generalization even when training on a small dataset. In this project, we will demonstrate this by training with several different TF-Hub modules.","metadata":{}},{"cell_type":"markdown","source":"# Tasks 5 & 6: Define Function to Build and Compile Models","metadata":{}},{"cell_type":"code","source":"def train_and_evaluate_model(module_url, embed_size, name, trainable=False):\n    # Define a KerasLayer that loads the pre-trained module at the given module_url,\n    # with input_shape=[], output_shape=[embed_size], dtype=tf.string, and trainable=trainable\n    hub_layer = hub.KerasLayer(module_url, input_shape = [], output_shape = [embed_size], dtype= tf.string,\n                              trainable = trainable)\n    \n    # Define a sequential model with hub_layer as the first layer,\n    # followed by two dense layers with 256 and 64 units, respectively,\n    # and a final dense layer with 1 unit and sigmoid activation\n    model = tf.keras.models.Sequential([\n        hub_layer,\n        tf.keras.layers.Dense(256, activation = \"relu\"),\n        tf.keras.layers.Dense(64, activation = \"relu\"),\n        tf.keras.layers.Dense(1, activation=\"sigmoid\")\n    ])\n    \n    # Compile the model with Adam optimizer with learning_rate=0.0001,\n    # binary cross-entropy loss, and binary accuracy as the evaluation metric\n    model.compile(optimizer = tf.keras.optimizers.Adam(learning_rate = 0.0001),\n                 loss = tf.losses.BinaryCrossentropy(),\n                 metrics = [tf.metrics.BinaryAccuracy(name=\"accuracy\")])\n    \n    # Train the model on the training data, using validation data for early stopping\n    # and callbacks for logging and monitoring training progress\n    history = model.fit(train_df[\"question_text\"], train_df[\"target\"],\n                       epochs = 100,\n                       batch_size = 32,\n                       validation_data = (valid_df[\"question_text\"], valid_df[\"target\"]),\n                       callbacks = [tfdocs.modeling.EpochDots(),\n                                    tf.keras.callbacks.EarlyStopping(monitor=\"val_loss\", patience=2, mode=\"min\"),\n                                    tf.keras.callbacks.TensorBoard(logdir/name)],\n                       verbose = 0)\n    \n    # Return the training history\n    return history\n","metadata":{"execution":{"iopub.status.busy":"2023-03-12T18:51:39.106672Z","iopub.execute_input":"2023-03-12T18:51:39.107199Z","iopub.status.idle":"2023-03-12T18:51:39.118379Z","shell.execute_reply.started":"2023-03-12T18:51:39.107122Z","shell.execute_reply":"2023-03-12T18:51:39.117392Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Task 7:  Train Various Text Classification Models","metadata":{}},{"cell_type":"code","source":"# Define the available module URLs and embedding sizes as a dictionary\nmodule_urls = {\n    \"gnews-swivel-20dim\": {\"url\": \"https://tfhub.dev/google/tf2-preview/gnews-swivel-20dim/1\", \"embed_size\": 20},\n    \"nnlm-en-dim50\": {\"url\": \"https://tfhub.dev/google/tf2-preview/nnlm-en-dim50/1\", \"embed_size\": 50},\n    \"nnlm-en-dim128\": {\"url\": \"https://tfhub.dev/google/tf2-preview/nnlm-en-dim128/1\", \"embed_size\": 128},\n}\n\n# Create an empty dictionary to store the model histories\nhistories = {}\n\n# Iterate through the module URLs and train and evaluate the models\nfor name, values in module_urls.items():\n    url = values[\"url\"]\n    embed_size = values[\"embed_size\"]\n    history = train_and_evaluate_model(url, embed_size=embed_size, name=name)\n    histories[name] = history\n","metadata":{"execution":{"iopub.status.busy":"2023-03-12T18:51:39.120408Z","iopub.execute_input":"2023-03-12T18:51:39.120798Z","iopub.status.idle":"2023-03-12T18:53:06.025644Z","shell.execute_reply.started":"2023-03-12T18:51:39.120759Z","shell.execute_reply":"2023-03-12T18:53:06.024655Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Task 8: Compare Accuracy and Loss Curves","metadata":{}},{"cell_type":"code","source":"plt.rcParams['figure.figsize'] = (12, 8)\nplotter = tfdocs.plots.HistoryPlotter(metric = 'accuracy')\nplotter.plot(histories)\nplt.xlabel(\"Epochs\")\nplt.legend(bbox_to_anchor=(1.0, 1.0), loc='upper left')\nplt.title(\"Accuracy Curves for Models\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-03-12T18:53:06.027376Z","iopub.execute_input":"2023-03-12T18:53:06.027663Z","iopub.status.idle":"2023-03-12T18:53:11.530250Z","shell.execute_reply.started":"2023-03-12T18:53:06.027635Z","shell.execute_reply":"2023-03-12T18:53:11.529076Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plotter = tfdocs.plots.HistoryPlotter(metric = 'loss')\nplotter.plot(histories)\nplt.xlabel(\"Epochs\")\nplt.legend(bbox_to_anchor=(1.0, 1.0), loc='upper left')\nplt.title(\"Loss Curves for Models\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-03-12T18:53:11.531762Z","iopub.execute_input":"2023-03-12T18:53:11.532557Z","iopub.status.idle":"2023-03-12T18:53:11.860785Z","shell.execute_reply.started":"2023-03-12T18:53:11.532513Z","shell.execute_reply":"2023-03-12T18:53:11.859794Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Task 9: Fine-tune Model from TF Hub","metadata":{}},{"cell_type":"code","source":"# Define the available module URLs and embedding sizes as a dictionary\nmodule_urls = {\n    \"gnews-swivel-20dim\": {\"url\": \"https://tfhub.dev/google/tf2-preview/gnews-swivel-20dim/1\", \"embed_size\": 20},\n    \"nnlm-en-dim50\": {\"url\": \"https://tfhub.dev/google/tf2-preview/nnlm-en-dim50/1\", \"embed_size\": 50},\n    \"nnlm-en-dim128\": {\"url\": \"https://tfhub.dev/google/tf2-preview/nnlm-en-dim128/1\", \"embed_size\": 128},\n    \"gnews-swivel-20dim-finetuned\": {\"url\": \"https://tfhub.dev/google/tf2-preview/gnews-swivel-20dim/1\", \"embed_size\": 20}\n}\n# Create an empty dictionary to store the model histories\nhistories = {}\n\n# Iterate through the module URLs and train and evaluate the models\nfor name, values in module_urls.items():\n    url = values[\"url\"]\n    embed_size = values[\"embed_size\"]\n    trainable = False if \"fine_tuned\" not in name else True\n    history = train_and_evaluate_model(url, embed_size=embed_size, name=name, trainable= trainable)\n    histories[name] = history","metadata":{"execution":{"iopub.status.busy":"2023-03-12T18:53:11.863148Z","iopub.execute_input":"2023-03-12T18:53:11.863814Z","iopub.status.idle":"2023-03-12T18:55:28.295167Z","shell.execute_reply.started":"2023-03-12T18:53:11.863773Z","shell.execute_reply":"2023-03-12T18:55:28.293918Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.rcParams['figure.figsize'] = (12, 8)\nplotter = tfdocs.plots.HistoryPlotter(metric = 'accuracy')\nplotter.plot(histories)\nplt.xlabel(\"Epochs\")\nplt.legend(bbox_to_anchor=(1.0, 1.0), loc='upper left')\nplt.title(\"Accuracy Curves for Models\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-03-12T18:57:18.821102Z","iopub.execute_input":"2023-03-12T18:57:18.821936Z","iopub.status.idle":"2023-03-12T18:57:20.947030Z","shell.execute_reply.started":"2023-03-12T18:57:18.821895Z","shell.execute_reply":"2023-03-12T18:57:20.946042Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plotter = tfdocs.plots.HistoryPlotter(metric = 'loss')\nplotter.plot(histories)\nplt.xlabel(\"Epochs\")\nplt.legend(bbox_to_anchor=(1.0, 1.0), loc='upper left')\nplt.title(\"Loss Curves for Models\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-03-12T18:57:28.098950Z","iopub.execute_input":"2023-03-12T18:57:28.100039Z","iopub.status.idle":"2023-03-12T18:57:28.417949Z","shell.execute_reply.started":"2023-03-12T18:57:28.099993Z","shell.execute_reply":"2023-03-12T18:57:28.416977Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Task 10: Train Bigger Models and Visualize Metrics with TensorBoard","metadata":{}},{"cell_type":"code","source":"# Define the available module URLs and embedding sizes as a dictionary\nmodule_urls = {\n    \"universal-sentence-encoder\": {\"url\": \"https://tfhub.dev/google/universal-sentence-encoder/4\", \"embed_size\": 512},\n    \"universal-sentence-encoder-large\": {\"url\": \"https://tfhub.dev/google/universal-sentence-encoder-large/5\", \"embed_size\": 512}\n}\n\n# Create an empty dictionary to store the model histories\nhistories = {}\n\n# Iterate through the module URLs and train and evaluate the models\nfor name, values in module_urls.items():\n    url = values[\"url\"]\n    embed_size = values[\"embed_size\"]\n    history = train_and_evaluate_model(url, embed_size=embed_size, name=name)\n    histories[name] = history","metadata":{"execution":{"iopub.status.busy":"2023-03-12T18:57:30.577525Z","iopub.execute_input":"2023-03-12T18:57:30.577887Z","iopub.status.idle":"2023-03-12T19:04:34.663885Z","shell.execute_reply.started":"2023-03-12T18:57:30.577852Z","shell.execute_reply":"2023-03-12T19:04:34.662820Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.rcParams['figure.figsize'] = (12, 8)\nplotter = tfdocs.plots.HistoryPlotter(metric = 'accuracy')\nplotter.plot(histories)\nplt.xlabel(\"Epochs\")\nplt.legend(bbox_to_anchor=(1.0, 1.0), loc='upper left')\nplt.title(\"Accuracy Curves for Models\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-03-12T19:04:34.666940Z","iopub.execute_input":"2023-03-12T19:04:34.667375Z","iopub.status.idle":"2023-03-12T19:05:28.192281Z","shell.execute_reply.started":"2023-03-12T19:04:34.667333Z","shell.execute_reply":"2023-03-12T19:05:28.191091Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plotter = tfdocs.plots.HistoryPlotter(metric = 'loss')\nplotter.plot(histories)\nplt.xlabel(\"Epochs\")\nplt.legend(bbox_to_anchor=(1.0, 1.0), loc='upper left')\nplt.title(\"Loss Curves for Models\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-03-12T19:05:28.193850Z","iopub.execute_input":"2023-03-12T19:05:28.194217Z","iopub.status.idle":"2023-03-12T19:05:28.491954Z","shell.execute_reply.started":"2023-03-12T19:05:28.194179Z","shell.execute_reply":"2023-03-12T19:05:28.490992Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%load_ext tensorboard\n%tensorboard --logdir {logdir}","metadata":{"execution":{"iopub.status.busy":"2023-03-12T19:49:04.531859Z","iopub.execute_input":"2023-03-12T19:49:04.532469Z","iopub.status.idle":"2023-03-12T19:49:17.106091Z","shell.execute_reply.started":"2023-03-12T19:49:04.532425Z","shell.execute_reply":"2023-03-12T19:49:17.104710Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Resources\n\n* [Quora Insincere Questions Classification Dataset](https://www.kaggle.com/c/quora-insincere-questions-classification/data)\n* [Pie Charts with Labels in Matplotlib](https://www.pythoncharts.com/matplotlib/pie-chart-matplotlib/)","metadata":{}}]}