{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 5GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# About this notebook\nI uses this notebook to learn NLP myself. Here, I does copy a lot of other notebooks (with citations :D) to get fast understanding in the field.\n\nI adds explanation and re-arrange stuff in order that beginners like me can easily follow.\n\nIf you find it useful for yourself, please upvote for me and the orgiginal notebook also.\n\nThank you so much for reading!","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"The contents include\n1. Understanding the data set and requirements: Discuss what we have been given and what we need to do.\n2. Data Preprocessing: Prepare data that are ready to feed to any NN models\n3. NN models: Discuss what NN models, machine learning techniques we are going to explore\n4. Trainings: Training our models and discuss the results\n5. Finish up: Transfer the results to submission.csv","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# 1. Understanding the Data Set and Requirements\n\nJigsaw Multilingual Toxic Comment Classification: Use TPUs to identify toxicity comments across multiple languages.\n\n<span style=\"color:red\"> **Simplest explanation**</span>: Given a comment in a language => classify that comment is toxic or non-toxic.\n\nThe data set includes these files and the files have 4 columns\n\n<span style=\"color:red\"> **Files**</span>\n* **jigsaw-toxic-comment-train.csv** - data from the 1st competition. https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge\n\nThe dataset is made up of English comments from Wikipedia’s talk page edits.\n\n* **jigsaw-unintended-bias-train.csv** - data from the 2nf competition. https://www.kaggle.com/c/jigsaw-unintended-bias-in-toxicity-classification\n\nThis is an expanded version of the Civil Comments dataset with a range of additional labels.\n\n\n* **validation.csv** - comments from Wikipedia talk pages in different non-English languages.\n* **test.csv** - comments from Wikipedia talk pages in different non-English languages.\n\n\n* **jigsaw-toxic-comment-train-processed-seqlen128.csv** - training data preprocessed for **BERT**\n* **jigsaw-unintended-bias-train-processed-seqlen128.csv** - training data preprocessed for **BERT**\n* **validation-processed-seqlen128.csv** - validation data preprocessed for **BERT**\n* **test-processed-seqlen128.csv** - test data preprocessed for **BERT**\n\n\n* **sample_submission.csv** - a sample submission file in the correct format\n\n\n<span style=\"color:red\"> **Columns**</span>\n* **id** - identifier within each file.\n* **comment_text** - the text of the comment to be classified.\n* **lang** - the language of the comment.\n* **toxic** - whether or not the comment is classified as toxic (1 vs 0). (Does not exist in test.csv.)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Explanation of data set files","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"**jigsaw-toxic-comment-train.csv** have **223549** rows of **comment_text** is classified into 6 levels of toxicity","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"DIR_INPUT= '/kaggle/input/jigsaw-multilingual-toxic-comment-classification'\ntrain_df1 = pd.read_csv(DIR_INPUT + '/jigsaw-toxic-comment-train.csv')\ntrain_df1.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(train_df1.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**jigsaw-unintended-bias-train.csv** have **1902194** rows of **comment_text** and 43 columns of information about comments","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df2 = pd.read_csv(DIR_INPUT + '/jigsaw-unintended-bias-train.csv')\ntrain_df2.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(train_df2.shape)\nprint(train_df2.columns)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**validation.csv** have **8000** rows of **comment_text** and 2 columns of languages and toxic categories","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"validation_df = pd.read_csv(DIR_INPUT + '/validation.csv')\nvalidation_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(validation_df.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Requirements","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"**test.csv** have **63812** rows of **comment_text** in multi-languages and 1 columns of language types.\n\n<span style=\"color:red\"> **We need to classify these comments as toxic or non-toxic, and then put our prediction to the sample_submission file**</span>\n\n<span style=\"color:red\"> **We can use all above files as our trainning/validation data set**</span>","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df = pd.read_csv(DIR_INPUT + '/test.csv')\ntest_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(test_df.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**sample_submission.csv** have **63812** rows of **id** and **toxic** classification.\n\n<span style=\"color:red\"> **We need to replace the value of the toxic columns with our prediction with respect to the id in test.csv files..**</span>","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_submission= pd.read_csv(DIR_INPUT + '/sample_submission.csv')\nsample_submission.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(sample_submission.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 2. Data Preprocessing\nHere, we will prepare our data to be ready for feeding any NN models.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}