{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"---\n## [Jigsaw Rate Severity of Toxic Comments][1]\n---\n\n**Comments**: Thanks to previous great Notebooks for data preprocessing.\n\n1. [☣️ Jigsaw - Incredibly Simple Naive Bayes [0.768]][2]\n2. [AutoNLP for toxic ratings ;)][3]\n3. [Regression Ensemble LB=0.78][4]\n4. [Jigsaw Ensemble [0.86]][5]\n\n\n[1]: https://www.kaggle.com/c/jigsaw-toxic-severity-rating/overview\n[2]: https://www.kaggle.com/julian3833/jigsaw-incredibly-simple-naive-bayes-0-768\n[3]: https://www.kaggle.com/abhishek/autonlp-for-toxic-ratings\n[4]: https://www.kaggle.com/ekaterinadranitsyna/regression-ensemble-lb-0-78/notebook\n[5]: https://www.kaggle.com/andrej0marinchenko/jigsaw-ensemble-0-86","metadata":{}},{"cell_type":"markdown","source":"# 0. Settings","metadata":{}},{"cell_type":"code","source":"# Import dependencies \nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt \n%matplotlib inline\n\nimport os\nimport pathlib\nimport gc\nimport sys\nimport re\nimport math \nimport random\nimport time \nimport tqdm \nfrom tqdm import tqdm \n\nimport warnings\nwarnings.filterwarnings('ignore')\n\nfrom sklearn.model_selection import KFold \nfrom sklearn.model_selection import StratifiedKFold \n\nimport tensorflow as tf\nimport tensorflow.keras as keras\nfrom tensorflow.keras.layers.experimental import preprocessing\n\nimport transformers \nimport datasets \n\nprint('import done!')","metadata":{"execution":{"iopub.status.busy":"2022-01-04T03:35:03.395201Z","iopub.execute_input":"2022-01-04T03:35:03.395942Z","iopub.status.idle":"2022-01-04T03:35:03.407362Z","shell.execute_reply.started":"2022-01-04T03:35:03.395897Z","shell.execute_reply":"2022-01-04T03:35:03.406556Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# global config\nconfig = {\n    'model_path': '../input/roberta-base-211212',\n    'tokenizer_path': '../input/roberta-base-tokenizer-211212',\n    'batch_size': 8,\n    'n_folds': 30,\n    'nontoxic_n_factor': 0.4,\n    'num_words': 3,\n    'under_over_ratio_factor': 1.0,\n    'clipping_score': 10.0,\n    'upsampling_threshold': 4.0,\n    'capped': False,\n}\n\nAUTOTUNE = tf.data.experimental.AUTOTUNE\n\n# For reproducible results    \ndef seed_all(s):\n    random.seed(s)\n    np.random.seed(s)\n    tf.random.set_seed(s)\n    os.environ['TF_CUDNN_DETERMINISTIC'] = '1'\n    os.environ['PYTHONHASHSEED'] = str(s) \n    print('Seeds setted!')\nglobal_seed = 42\nseed_all(global_seed)","metadata":{"execution":{"iopub.status.busy":"2022-01-04T03:35:04.377708Z","iopub.execute_input":"2022-01-04T03:35:04.378432Z","iopub.status.idle":"2022-01-04T03:35:04.386317Z","shell.execute_reply.started":"2022-01-04T03:35:04.378392Z","shell.execute_reply":"2022-01-04T03:35:04.385548Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 1. Data Preprocessing","metadata":{}},{"cell_type":"markdown","source":"### 1.1 Create train data\n\nFor training data, I used [Toxic Comment Classification Challenge][1] dataset.\n\n[1]: https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge/data","metadata":{}},{"cell_type":"code","source":"# Extract classified text samples and clean the texts.\ndf = pd.read_csv('../input/jigsaw-toxic-comment-classification-challenge/train.csv')\n\ndf['toxic_label'] = (df[['toxic', 'severe_toxic', 'obscene', 'threat', 'insult', 'identity_hate']].sum(axis=1) > 0 ).astype(int)\n\ncategories = df.loc[:, 'toxic':'identity_hate'].sum()\nplt.title('Category Frequency')\nplt.bar(categories.index, categories.values)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-04T03:35:08.215445Z","iopub.execute_input":"2022-01-04T03:35:08.215709Z","iopub.status.idle":"2022-01-04T03:35:09.284525Z","shell.execute_reply.started":"2022-01-04T03:35:08.215680Z","shell.execute_reply":"2022-01-04T03:35:09.283755Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In the previous competition the task was to perform multi-class classification. Text sample could be labeled with one or several categories or not labeled with any. Non-toxic comments represent the majority of text samples, while toxic comments are a minority class and extremely toxic comments are more rare than plain toxic.\n\nIn this competition we have to score texts based on the level of toxicity. To get a toxicity score from the previous data we can use the following approaches:\n- Adjust the values in the DataFrame according to extremety of the category (for example, \"toxic\" and \"severe toxic\" should have different score) and then sum up per row values.","metadata":{}},{"cell_type":"code","source":"# Multiplication factors for categories.\ncat_mtpl = {'toxic': 1.0, 'severe_toxic': 2.5, 'obscene': 1.0,\n            'threat': 2.0, 'insult': 1.5, 'identity_hate': 2.0}\n\nfor category in cat_mtpl:\n    df[category] = df[category] * cat_mtpl[category]\n\ndf['score'] = df.loc[:, 'toxic':'identity_hate'].sum(axis=1)\n#df['score'] = df['score'] / df['score'].max()\n\nbins = math.ceil(df['score'].max())\n\nplt.hist(df['score'], bins=bins)\nplt.title('Scores Distribution: Adjusted Sum')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-04T03:36:02.707704Z","iopub.execute_input":"2022-01-04T03:36:02.708053Z","iopub.status.idle":"2022-01-04T03:36:02.944663Z","shell.execute_reply.started":"2022-01-04T03:36:02.708015Z","shell.execute_reply":"2022-01-04T03:36:02.943768Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 1.2 Downsampling\nThe dataset is very unbalanced. Here we downsample the majority class.","metadata":{}},{"cell_type":"code","source":"df['toxic_label'].value_counts(normalize=True)","metadata":{"execution":{"iopub.status.busy":"2022-01-04T03:36:08.850261Z","iopub.execute_input":"2022-01-04T03:36:08.850515Z","iopub.status.idle":"2022-01-04T03:36:08.859925Z","shell.execute_reply.started":"2022-01-04T03:36:08.850486Z","shell.execute_reply":"2022-01-04T03:36:08.858599Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"factor = config['nontoxic_n_factor']\nn_samples_toxic = (df['toxic_label'] == 1).sum()\nn_samples_toxic = round(n_samples_toxic * factor)\n\ndf_untoxic_undersample = df[df['toxic_label'] == 0].sample(n=n_samples_toxic, random_state=global_seed)\ndf_toxic = df.query('toxic_label == 1')\ntrain_df = pd.concat([df_untoxic_undersample, df_toxic]).reset_index(drop=True)\ntrain_df['toxic_label'].value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-01-04T03:36:09.718030Z","iopub.execute_input":"2022-01-04T03:36:09.718683Z","iopub.status.idle":"2022-01-04T03:36:09.771803Z","shell.execute_reply.started":"2022-01-04T03:36:09.718643Z","shell.execute_reply":"2022-01-04T03:36:09.771116Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(f'Mean toxicity score: {train_df[\"score\"].mean()}\\n'\n      f'Standard deviation: {train_df[\"score\"].std()}')\n\nplt.hist(train_df['score'], bins=bins)\nplt.title('Scores Distribution: Adjusted Sum')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-04T03:36:10.558209Z","iopub.execute_input":"2022-01-04T03:36:10.558757Z","iopub.status.idle":"2022-01-04T03:36:10.756323Z","shell.execute_reply.started":"2022-01-04T03:36:10.558703Z","shell.execute_reply":"2022-01-04T03:36:10.755541Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 1.3 Text Cleaning","metadata":{}},{"cell_type":"code","source":"from bs4 import BeautifulSoup\ndef text_cleaning(text: str) -> str:\n    \"\"\"Function cleans text removing special characters,\n    extra spaces, embedded URL links, HTML tags and emojis.\n    Code source: https://www.kaggle.com/manabendrarout/pytorch-roberta-ranking-baseline-jrstc-infer\n    :param text: Original text\n    :return: Preprocessed text\n    \"\"\"\n    template = re.compile(r'https?://\\S+|www\\.\\S+')  # website links\n    text = template.sub(r'', text)\n\n    soup = BeautifulSoup(text, 'lxml')  # HTML tags\n    only_text = soup.get_text()\n    text = only_text\n\n    emoji_pattern = re.compile(\"[\"\n                               u\"\\U0001F600-\\U0001F64F\"  # emoticons\n                               u\"\\U0001F300-\\U0001F5FF\"  # symbols & pictographs\n                               u\"\\U0001F680-\\U0001F6FF\"  # transport & map symbols\n                               u\"\\U0001F1E0-\\U0001F1FF\"  # flags (iOS)\n                               u\"\\U00002702-\\U000027B0\"\n                               u\"\\U000024C2-\\U0001F251\"\n                               \"]+\", flags=re.UNICODE)\n    text = emoji_pattern.sub(r'', text)\n\n    text = re.sub(r\"[^a-zA-Z\\d]\", \" \", text)  # special characters\n    text = re.sub(' +', ' ', text)  # extra spaces\n    # Replace repeating characters more than 3 times to length of 3\n    text = re.sub(r'([*!?\\'])\\1\\1{2,}', r'\\1\\1\\1', text)    \n    # Add space around repeating characters\n    text = re.sub(r'([*!?\\']+)', r' \\1 ', text)    \n    # patterns with repeating characters \n    text = re.sub(r'([a-zA-Z])\\1{2,}\\b', r'\\1\\1', text)\n    text = re.sub(r'([a-zA-Z])\\1\\1{2,}\\B', r'\\1\\1\\1', text)\n    text = re.sub(r'[ ]{2,}', ' ', text)\n    text = text.strip()  # spaces at the beginning and at the end of string\n\n    return text\n\ntrain_df['comment_text'] = train_df['comment_text'].apply(text_cleaning)\nprint('cleaning done!')","metadata":{"execution":{"iopub.status.busy":"2022-01-04T03:36:12.926375Z","iopub.execute_input":"2022-01-04T03:36:12.926630Z","iopub.status.idle":"2022-01-04T03:36:21.836889Z","shell.execute_reply.started":"2022-01-04T03:36:12.926601Z","shell.execute_reply":"2022-01-04T03:36:21.835361Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import nltk\nfrom nltk.corpus import stopwords\nstop = stopwords.words('english')\n\ndef text_cleaning_2(data, col):\n    \n    data[col] = data[col].str.replace('https?://\\S+|www\\.\\S+', ' social medium ')      \n        \n    data[col] = data[col].str.lower()\n    data[col] = data[col].str.replace(\"4\", \"a\") \n    data[col] = data[col].str.replace(\"2\", \"l\")\n    data[col] = data[col].str.replace(\"5\", \"s\") \n    data[col] = data[col].str.replace(\"1\", \"i\") \n    data[col] = data[col].str.replace(\"!\", \"i\") \n    data[col] = data[col].str.replace(\"|\", \"i\") \n    data[col] = data[col].str.replace(\"0\", \"o\") \n    data[col] = data[col].str.replace(\"l3\", \"b\") \n    data[col] = data[col].str.replace(\"7\", \"t\") \n    data[col] = data[col].str.replace(\"7\", \"+\") \n    data[col] = data[col].str.replace(\"8\", \"ate\") \n    data[col] = data[col].str.replace(\"3\", \"e\") \n    data[col] = data[col].str.replace(\"9\", \"g\")\n    data[col] = data[col].str.replace(\"6\", \"g\")\n    data[col] = data[col].str.replace(\"@\", \"a\")\n    data[col] = data[col].str.replace(\"$\", \"s\")\n    data[col] = data[col].str.replace(\"#ofc\", \" of fuckin course \")\n    data[col] = data[col].str.replace(\"fggt\", \" faggot \")\n    data[col] = data[col].str.replace(\"your\", \" your \")\n    data[col] = data[col].str.replace(\"self\", \" self \")\n    data[col] = data[col].str.replace(\"cuntbag\", \" cunt bag \")\n    data[col] = data[col].str.replace(\"fartchina\", \" fart china \")    \n    data[col] = data[col].str.replace(\"youi\", \" you i \")\n    data[col] = data[col].str.replace(\"cunti\", \" cunt i \")\n    data[col] = data[col].str.replace(\"sucki\", \" suck i \")\n    data[col] = data[col].str.replace(\"pagedelete\", \" page delete \")\n    data[col] = data[col].str.replace(\"cuntsi\", \" cuntsi \")\n    data[col] = data[col].str.replace(\"i'm\", \" i am \")\n    data[col] = data[col].str.replace(\"offuck\", \" of fuck \")\n    data[col] = data[col].str.replace(\"centraliststupid\", \" central ist stupid \")\n    data[col] = data[col].str.replace(\"hitleri\", \" hitler i \")\n    data[col] = data[col].str.replace(\"i've\", \" i have \")\n    data[col] = data[col].str.replace(\"i'll\", \" sick \")\n    data[col] = data[col].str.replace(\"fuck\", \" fuck \")\n    data[col] = data[col].str.replace(\"f u c k\", \" fuck \")\n    data[col] = data[col].str.replace(\"shit\", \" shit \")\n    data[col] = data[col].str.replace(\"bunksteve\", \" bunk steve \")\n    data[col] = data[col].str.replace('wikipedia', ' social medium ')\n    data[col] = data[col].str.replace(\"faggot\", \" faggot \")\n    data[col] = data[col].str.replace(\"delanoy\", \" delanoy \")\n    data[col] = data[col].str.replace(\"jewish\", \" jewish \")\n    data[col] = data[col].str.replace(\"sexsex\", \" sex \")\n    data[col] = data[col].str.replace(\"allii\", \" all ii \")\n    data[col] = data[col].str.replace(\"i'd\", \" i had \")\n    data[col] = data[col].str.replace(\"'s\", \" is \")\n    data[col] = data[col].str.replace(\"youbollocks\", \" you bollocks \")\n    data[col] = data[col].str.replace(\"dick\", \" dick \")\n    data[col] = data[col].str.replace(\"cuntsi\", \" cuntsi \")\n    data[col] = data[col].str.replace(\"mothjer\", \" mother \")\n    data[col] = data[col].str.replace(\"cuntfranks\", \" cunt \")\n    data[col] = data[col].str.replace(\"ullmann\", \" jewish \")\n    data[col] = data[col].str.replace(\"mr.\", \" mister \")\n    data[col] = data[col].str.replace(\"aidsaids\", \" aids \")\n    data[col] = data[col].str.replace(\"njgw\", \" nigger \")\n    data[col] = data[col].str.replace(\"wiki\", \" social medium \")\n    data[col] = data[col].str.replace(\"administrator\", \" admin \")\n    data[col] = data[col].str.replace(\"gamaliel\", \" jewish \")\n    data[col] = data[col].str.replace(\"rvv\", \" vanadalism \")\n    data[col] = data[col].str.replace(\"admins\", \" admin \")\n    data[col] = data[col].str.replace(\"pensnsnniensnsn\", \" penis \")\n    data[col] = data[col].str.replace(\"pneis\", \" penis \")\n    data[col] = data[col].str.replace(\"pennnis\", \" penis \")\n    data[col] = data[col].str.replace(\"pov.\", \" point of view \")\n    data[col] = data[col].str.replace(\"vandalising\", \" vandalism \")\n    data[col] = data[col].str.replace(\"cock\", \" dick \")\n    data[col] = data[col].str.replace(\"asshole\", \" asshole \")\n    data[col] = data[col].str.replace(\"youi\", \" you \")\n    data[col] = data[col].str.replace(\"afd\", \" all fucking day \")\n    data[col] = data[col].str.replace(\"sockpuppets\", \" sockpuppetry \")\n    data[col] = data[col].str.replace(\"iiprick\", \" iprick \")\n    data[col] = data[col].str.replace(\"penisi\", \" penis \")\n    data[col] = data[col].str.replace(\"warrior\", \" warrior \")\n    data[col] = data[col].str.replace(\"loil\", \" laughing out insanely loud \")\n    data[col] = data[col].str.replace(\"vandalise\", \" vanadalism \")\n    data[col] = data[col].str.replace(\"helli\", \" helli \")\n    data[col] = data[col].str.replace(\"lunchablesi\", \" lunchablesi \")\n    data[col] = data[col].str.replace(\"special\", \" special \")\n    data[col] = data[col].str.replace(\"ilol\", \" i lol \")\n    data[col] = data[col].str.replace(r'\\b[uU]\\b', 'you')\n    data[col] = data[col].str.replace(r\"what's\", \"what is \")\n    data[col] = data[col].str.replace(r\"\\'s\", \" is \")\n    data[col] = data[col].str.replace(r\"\\'ve\", \" have \")\n    data[col] = data[col].str.replace(r\"can't\", \"cannot \")\n    data[col] = data[col].str.replace(r\"n't\", \" not \")\n    data[col] = data[col].str.replace(r\"i'm\", \"i am \")\n    data[col] = data[col].str.replace(r\"\\'re\", \" are \")\n    data[col] = data[col].str.replace(r\"\\'d\", \" would \")\n    data[col] = data[col].str.replace(r\"\\'ll\", \" will \")\n    data[col] = data[col].str.replace(r\"\\'scuse\", \" excuse \")\n    data[col] = data[col].str.replace('\\s+', ' ')  # will remove more than one whitespace character\n#     text = re.sub(r'\\b([^\\W\\d_]+)(\\s+\\1)+\\b', r'\\1', re.sub(r'\\W+', ' ', text).strip(), flags=re.I)  # remove repeating words coming immediately one after another\n    data[col] = data[col].str.replace(r'(.)\\1+', r'\\1\\1') # 2 or more characters are replaced by 2 characters\n#     text = re.sub(r'((\\b\\w+\\b.{1,2}\\w+\\b)+).+\\1', r'\\1', text, flags = re.I)\n    data[col] = data[col].str.replace(\"[:|♣|'|§|♠|*|/|?|=|%|&|-|#|•|~|^|>|<|►|_]\", '')\n    \n    \n    data[col] = data[col].str.replace(r\"what's\", \"what is \")    \n    data[col] = data[col].str.replace(r\"\\'ve\", \" have \")\n    data[col] = data[col].str.replace(r\"can't\", \"cannot \")\n    data[col] = data[col].str.replace(r\"n't\", \" not \")\n    data[col] = data[col].str.replace(r\"i'm\", \"i am \")\n    data[col] = data[col].str.replace(r\"\\'re\", \" are \")\n    data[col] = data[col].str.replace(r\"\\'d\", \" would \")\n    data[col] = data[col].str.replace(r\"\\'ll\", \" will \")\n    data[col] = data[col].str.replace(r\"\\'scuse\", \" excuse \")\n    data[col] = data[col].str.replace(r\"\\'s\", \" \")\n\n    # Clean some punctutations\n    data[col] = data[col].str.replace('\\n', ' \\n ')\n    data[col] = data[col].str.replace(r'([a-zA-Z]+)([/!?.])([a-zA-Z]+)',r'\\1 \\2 \\3')\n    # Replace repeating characters more than 3 times to length of 3\n    data[col] = data[col].str.replace(r'([*!?\\'])\\1\\1{2,}',r'\\1\\1\\1')    \n    # Add space around repeating characters\n    data[col] = data[col].str.replace(r'([*!?\\']+)',r' \\1 ')    \n    # patterns with repeating characters \n    data[col] = data[col].str.replace(r'([a-zA-Z])\\1{2,}\\b',r'\\1\\1')\n    data[col] = data[col].str.replace(r'([a-zA-Z])\\1\\1{2,}\\B',r'\\1\\1\\1')\n    data[col] = data[col].str.replace(r'[ ]{2,}',' ').str.strip()   \n    data[col] = data[col].str.replace(r'[ ]{2,}',' ').str.strip()   \n    data[col] = data[col].apply(lambda x: ' '.join([word for word in x.split() if word not in (stop)]))\n    \n    print('cleaning done!')\n    \n    return data\n\ntrain_df = text_cleaning_2(train_df,'comment_text')","metadata":{"execution":{"iopub.status.busy":"2022-01-04T03:36:21.838797Z","iopub.execute_input":"2022-01-04T03:36:21.839276Z","iopub.status.idle":"2022-01-04T03:36:35.922664Z","shell.execute_reply.started":"2022-01-04T03:36:21.839233Z","shell.execute_reply":"2022-01-04T03:36:35.921913Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(len(train_df))\n\ndef text_cleaning_3(data, col):\n    data[col] = data[col].apply(lambda x: '' if len(x.split(' ')) < config['num_words'] else x)\n    print('cleaning done!')\n    return data\n\ntrain_df = text_cleaning_3(train_df,'comment_text')\ntrain_df = train_df[train_df['comment_text'] != ''].reset_index(drop=True)\nprint(len(train_df))","metadata":{"execution":{"iopub.status.busy":"2022-01-04T03:36:35.923927Z","iopub.execute_input":"2022-01-04T03:36:35.924184Z","iopub.status.idle":"2022-01-04T03:36:35.983839Z","shell.execute_reply.started":"2022-01-04T03:36:35.924155Z","shell.execute_reply":"2022-01-04T03:36:35.983139Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 1.4 Score clipping\nWhen config['clipping_score'] < 10, we clip the score.","metadata":{}},{"cell_type":"code","source":"train_df['score'].value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-01-04T03:36:43.360497Z","iopub.execute_input":"2022-01-04T03:36:43.361384Z","iopub.status.idle":"2022-01-04T03:36:43.372672Z","shell.execute_reply.started":"2022-01-04T03:36:43.361321Z","shell.execute_reply":"2022-01-04T03:36:43.371886Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"clipping_score = config['clipping_score']\n\ntrain_df['score'] = train_df['score'].where(train_df['score'] < clipping_score, clipping_score)\ntrain_df['score'].value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-01-04T03:36:44.275653Z","iopub.execute_input":"2022-01-04T03:36:44.276431Z","iopub.status.idle":"2022-01-04T03:36:44.287677Z","shell.execute_reply.started":"2022-01-04T03:36:44.276381Z","shell.execute_reply":"2022-01-04T03:36:44.286907Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(f'Mean toxicity score: {train_df[\"score\"].mean()}\\n'\n      f'Standard deviation: {train_df[\"score\"].std()}')\n\nbins = math.ceil(train_df['score'].max())\n\nplt.hist(train_df['score'], bins=bins)\nplt.title('Scores Distribution: Adjusted Sum')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-04T03:37:05.061656Z","iopub.execute_input":"2022-01-04T03:37:05.061938Z","iopub.status.idle":"2022-01-04T03:37:05.256576Z","shell.execute_reply.started":"2022-01-04T03:37:05.061907Z","shell.execute_reply":"2022-01-04T03:37:05.255746Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 1.5 Upsampling","metadata":{}},{"cell_type":"code","source":"threshold = config['upsampling_threshold']\n\ntrain_df_over = train_df.query(f'score >= {threshold}')\nn_over = len(train_df_over)\n\ntrain_df_under = train_df.query(f'score < {threshold}')\nn_under = len(train_df_under)\n\nunder_over_ratio_factor = config['under_over_ratio_factor']\nunder_over_ratio = round(n_under / n_over * under_over_ratio_factor)\nprint(under_over_ratio)\ntrain_df_over_repeat = pd.concat([train_df_over] * under_over_ratio)\ntrain_df_upsampling = pd.concat([train_df_under, train_df_over_repeat])\ntrain_df_upsampling = train_df_upsampling.reset_index(drop=True)\ntrain_df = train_df_upsampling\n\nplt.hist(train_df['score'], bins=bins)\nplt.title('Scores Distribution: Adjusted Sum')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-04T03:37:09.154352Z","iopub.execute_input":"2022-01-04T03:37:09.154812Z","iopub.status.idle":"2022-01-04T03:37:09.373864Z","shell.execute_reply.started":"2022-01-04T03:37:09.154771Z","shell.execute_reply":"2022-01-04T03:37:09.373201Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# capping the overflow\nn_6 = len(train_df[train_df['score'] == 6.0])\nn_0 = len(train_df[train_df['score'] == 0.0])\nprint(n_6, n_0)\n\ntrain_df_score_6_undersample = train_df[train_df['score'] == 6.0].sample(n=n_0, random_state=global_seed)\ntrain_df_score_not_6 = train_df.query('score != 6.0')\ntrain_df_capped = pd.concat([train_df_score_6_undersample, train_df_score_not_6]).reset_index(drop=True)\n\nn_6 = len(train_df_capped[train_df_capped['score'] == 6.0])\nn_0 = len(train_df_capped[train_df_capped['score'] == 0.0])\nprint(n_6, n_0)\n#train_df_capped['score'].value_counts()\n\nif config['capped']:\n    train_df = train_df_capped\n    plt.hist(train_df_capped['score'], bins=bins)\n    plt.title('Scores Distribution: Adjusted Sum')\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2022-01-04T03:38:01.407327Z","iopub.execute_input":"2022-01-04T03:38:01.408167Z","iopub.status.idle":"2022-01-04T03:38:01.440406Z","shell.execute_reply.started":"2022-01-04T03:38:01.408124Z","shell.execute_reply":"2022-01-04T03:38:01.439693Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df['score'].value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-01-04T03:38:02.949723Z","iopub.execute_input":"2022-01-04T03:38:02.950364Z","iopub.status.idle":"2022-01-04T03:38:02.958738Z","shell.execute_reply.started":"2022-01-04T03:38:02.950324Z","shell.execute_reply":"2022-01-04T03:38:02.957897Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 1.6 Validation Data Split","metadata":{}},{"cell_type":"code","source":"train_df.describe()","metadata":{"execution":{"iopub.status.busy":"2022-01-04T03:38:17.113758Z","iopub.execute_input":"2022-01-04T03:38:17.114333Z","iopub.status.idle":"2022-01-04T03:38:17.155514Z","shell.execute_reply.started":"2022-01-04T03:38:17.114293Z","shell.execute_reply":"2022-01-04T03:38:17.154786Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"n_folds = config['n_folds']\n\nskf = StratifiedKFold(n_splits=n_folds, shuffle=True, random_state=global_seed)\nfor nfold, (train_index, val_index) in enumerate(skf.split(X=train_df.index,\n                                                           y=train_df.toxic_label)):\n    train_df.loc[val_index, 'fold'] = nfold\n#print(train_df.groupby(['fold', train_df.toxic_label]).size())\n\np_fold = 0\np_train_df = train_df.query(f'fold != {p_fold}').reset_index(drop=True)\np_valid_df = train_df.query(f'fold == {p_fold}').reset_index(drop=True)\n\nprint(len(p_train_df))\nprint(len(p_valid_df))","metadata":{"execution":{"iopub.status.busy":"2022-01-04T03:38:17.868536Z","iopub.execute_input":"2022-01-04T03:38:17.869403Z","iopub.status.idle":"2022-01-04T03:38:17.919389Z","shell.execute_reply.started":"2022-01-04T03:38:17.869359Z","shell.execute_reply":"2022-01-04T03:38:17.918601Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"p_train_df.describe()","metadata":{"execution":{"iopub.status.busy":"2022-01-04T03:38:18.649040Z","iopub.execute_input":"2022-01-04T03:38:18.649687Z","iopub.status.idle":"2022-01-04T03:38:18.694013Z","shell.execute_reply.started":"2022-01-04T03:38:18.649650Z","shell.execute_reply":"2022-01-04T03:38:18.693273Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"p_valid_df.describe()","metadata":{"execution":{"iopub.status.busy":"2022-01-04T03:38:19.451496Z","iopub.execute_input":"2022-01-04T03:38:19.451987Z","iopub.status.idle":"2022-01-04T03:38:19.489381Z","shell.execute_reply.started":"2022-01-04T03:38:19.451949Z","shell.execute_reply":"2022-01-04T03:38:19.488714Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"p_train_df = p_train_df[['comment_text', 'score']].rename(columns={'comment_text': 'text'})\np_valid_df = p_valid_df[['comment_text', 'score']].rename(columns={'comment_text': 'text'})\nprint('done!')","metadata":{"execution":{"iopub.status.busy":"2022-01-04T03:38:20.946925Z","iopub.execute_input":"2022-01-04T03:38:20.947655Z","iopub.status.idle":"2022-01-04T03:38:20.959613Z","shell.execute_reply.started":"2022-01-04T03:38:20.947616Z","shell.execute_reply":"2022-01-04T03:38:20.958228Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 2. DataSet","metadata":{}},{"cell_type":"code","source":"train_ds = datasets.Dataset.from_pandas(p_train_df)\nvalid_ds = datasets.Dataset.from_pandas(p_valid_df)\n\nprint(train_ds)\nprint(valid_ds)","metadata":{"execution":{"iopub.status.busy":"2022-01-04T02:20:35.314619Z","iopub.execute_input":"2022-01-04T02:20:35.314884Z","iopub.status.idle":"2022-01-04T02:20:35.364660Z","shell.execute_reply.started":"2022-01-04T02:20:35.314838Z","shell.execute_reply":"2022-01-04T02:20:35.363988Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"checkpoint = 'roberta-base'\n\n# Downloading tokenizer (Internet required)\n#tokenizer = transformers.AutoTokenizer.from_pretrained(checkpoint)\ntokenizer = transformers.AutoTokenizer.from_pretrained(config['tokenizer_path'])\n\ndef tokenize_function(example):\n    return tokenizer(example[\"text\"], truncation=True, max_length=128)\n\ntokenized_train_ds = train_ds.map(tokenize_function, batched=True)\ntokenized_valid_ds = valid_ds.map(tokenize_function, batched=True)\n\nprint(tokenized_train_ds)\nprint(tokenized_valid_ds)","metadata":{"execution":{"iopub.status.busy":"2022-01-04T02:20:36.392758Z","iopub.execute_input":"2022-01-04T02:20:36.393293Z","iopub.status.idle":"2022-01-04T02:20:43.714919Z","shell.execute_reply.started":"2022-01-04T02:20:36.393257Z","shell.execute_reply":"2022-01-04T02:20:43.714007Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data_collator = transformers.DataCollatorWithPadding(tokenizer=tokenizer)\n\ntf_train_ds = tokenized_train_ds.to_tf_dataset(\n    columns=[\"attention_mask\", \"input_ids\"],\n    label_cols=[\"score\"],\n    shuffle=True,\n    collate_fn=data_collator,\n    batch_size=config['batch_size'],\n)\n\ntf_valid_ds = tokenized_valid_ds.to_tf_dataset(\n    columns=[\"attention_mask\", \"input_ids\"],\n    label_cols=[\"score\"],\n    shuffle=False,\n    collate_fn=data_collator,\n    batch_size=config['batch_size'],\n)\n\nprint(len(tf_train_ds))\nprint(len(tf_valid_ds))","metadata":{"execution":{"iopub.status.busy":"2021-12-30T06:16:21.191141Z","iopub.execute_input":"2021-12-30T06:16:21.191405Z","iopub.status.idle":"2021-12-30T06:16:26.310587Z","shell.execute_reply.started":"2021-12-30T06:16:21.191376Z","shell.execute_reply":"2021-12-30T06:16:26.309864Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 3. Model Training","metadata":{}},{"cell_type":"code","source":"from transformers import TFAutoModel\n\n# Downloading model (Internet required)\n#roberta_model = TFAutoModel.from_pretrained(checkpoint)\nroberta_model = TFAutoModel.from_pretrained(config['model_path'])","metadata":{"execution":{"iopub.status.busy":"2021-12-30T06:16:30.637327Z","iopub.execute_input":"2021-12-30T06:16:30.637593Z","iopub.status.idle":"2021-12-30T06:16:39.304754Z","shell.execute_reply.started":"2021-12-30T06:16:30.637563Z","shell.execute_reply":"2021-12-30T06:16:39.304056Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 3.1 Model","metadata":{}},{"cell_type":"code","source":"class MultiHeadAttentionRegressor(tf.keras.Model):\n    def __init__(self, *args, **kwargs):\n        super().__init__()\n        self.roberta_layer = roberta_model\n\n        self.query_1 = tf.keras.layers.Dense(128, use_bias=False, activation=None)\n        self.query_2 = tf.keras.layers.Dense(128, use_bias=False, activation=None)\n        self.query_3 = tf.keras.layers.Dense(128, use_bias=False, activation=None)\n\n        self.key_1 = tf.keras.layers.Dense(128, use_bias=False, activation=None)\n        self.key_2 = tf.keras.layers.Dense(128, use_bias=False, activation=None)\n        self.key_3 = tf.keras.layers.Dense(128, use_bias=False, activation=None)\n        \n        self.regressor_1 = tf.keras.models.Sequential([\n            tf.keras.layers.Dense(512, activation='selu'),\n            tf.keras.layers.Dropout(0.2),\n        ])\n        \n        self.regressor_2 = tf.keras.models.Sequential([\n            tf.keras.layers.Dense(512, activation='selu'),\n            tf.keras.layers.Dropout(0.2)\n        ])\n        \n        self.regressor_3 = tf.keras.models.Sequential([\n            tf.keras.layers.Dense(512, activation='selu'),\n            tf.keras.layers.Dropout(0.3),\n            tf.keras.layers.Dense(128, activation='selu'),\n            tf.keras.layers.Dropout(0.2),\n            tf.keras.layers.Dense(1, activation=None)\n            ])\n        \n    def call(self, inputs, training=None):\n        roberta_outputs = self.roberta_layer(inputs)\n        \n        pooler_outputs = roberta_outputs['pooler_output'] ## TensorShape([batch_num, 768])\n        output_1 = self.regressor_1(pooler_outputs) ## TensorShape([batch_num, 512])\n        \n        attention_mask = tf.expand_dims(inputs['attention_mask'], -1) ## TensorShape([batch_num, max_len, 1])\n        attention_mask = tf.cast(attention_mask, dtype=tf.float32)\n        last_hidden_states = roberta_outputs['last_hidden_state'] * attention_mask ## TensorShape([batch_num, max_len, 768])\n\n        lhs_1 = last_hidden_states[:, :, :256] ## TensorShape([batch_num(8), max_len, 256])\n        lhs_2 = last_hidden_states[:, :, 256:512]\n        lhs_3 = last_hidden_states[:, :, 512:]\n\n        q_1 = self.query_1(lhs_1) ## TensorShape([8, max_len, 128])\n        k_1 = tf.expand_dims(self.key_1(pooler_outputs), -1) ## TensorShape([8, 128, 1])\n        a_scores_1 = tf.linalg.matmul(q_1, k_1) / tf.math.sqrt(128.) ## TensorShape([8, max_len, 1])\n        a_weights_1 = tf.keras.layers.Softmax(axis=1)(a_scores_1) ## TensorShape([8, max_len, 1])\n        average_hidden_states_1 = tf.math.reduce_sum(lhs_1 * a_weights_1, axis=1) ## TensorShape([8, 256])\n\n        q_2 = self.query_1(lhs_2)\n        k_2 = tf.expand_dims(self.key_2(pooler_outputs), -1)\n        a_scores_2 = tf.linalg.matmul(q_2, k_2) / tf.math.sqrt(128.)\n        a_weights_2 = tf.keras.layers.Softmax(axis=1)(a_scores_2)\n        average_hidden_states_2 = tf.math.reduce_sum(lhs_2 * a_weights_2, axis=1)\n\n        q_3 = self.query_3(lhs_3)\n        k_3 = tf.expand_dims(self.key_3(pooler_outputs), -1)\n        a_scores_3 = tf.linalg.matmul(q_3, k_3) / tf.math.sqrt(128.)\n        a_weights_3 = tf.keras.layers.Softmax(axis=1)(a_scores_3)\n        average_hidden_states_3 = tf.math.reduce_sum(lhs_3 * a_weights_3, axis=1)\n\n        average_hidden_states = tf.concat([average_hidden_states_1,\n                                           average_hidden_states_2,\n                                           average_hidden_states_3], axis=-1) ## TensorShape([8, 768])\n        output_2 = self.regressor_2(average_hidden_states) ## TensorShape([8, 512])\n        \n        output_3 = tf.concat([output_1, output_2], axis=-1) ## TensorShape([8, 1024])\n        outputs = self.regressor_3(output_3)\n        \n        return outputs\n\nmodel = MultiHeadAttentionRegressor()","metadata":{"execution":{"iopub.status.busy":"2021-12-30T06:16:52.03808Z","iopub.execute_input":"2021-12-30T06:16:52.038404Z","iopub.status.idle":"2021-12-30T06:16:52.071799Z","shell.execute_reply.started":"2021-12-30T06:16:52.03836Z","shell.execute_reply":"2021-12-30T06:16:52.07111Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 3.2 Training","metadata":{}},{"cell_type":"code","source":"model.roberta_layer.trainable = False\n\nnum_epochs = 2\nnum_train_steps = len(tf_train_ds) * num_epochs\n\nlr_scheduler = tf.keras.optimizers.schedules.PolynomialDecay(\n    initial_learning_rate=5e-4, end_learning_rate=5e-5, decay_steps=num_train_steps\n)\n\nmodel.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=lr_scheduler),\n              loss=tf.keras.losses.MeanSquaredError()\n             )\n\nfor data, label in tf_train_ds.take(1):\n    example = data\nresult = model(example)\nprint(result)\nmodel.summary()","metadata":{"execution":{"iopub.status.busy":"2021-12-30T06:16:56.822781Z","iopub.execute_input":"2021-12-30T06:16:56.823463Z","iopub.status.idle":"2021-12-30T06:16:57.150757Z","shell.execute_reply.started":"2021-12-30T06:16:56.823425Z","shell.execute_reply":"2021-12-30T06:16:57.149897Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fit_history = model.fit(tf_train_ds,\n                        epochs=num_epochs,\n                        validation_data=tf_valid_ds,\n                        verbose=1)","metadata":{"execution":{"iopub.status.busy":"2021-12-28T03:28:38.448718Z","iopub.execute_input":"2021-12-28T03:28:38.449396Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model.roberta_layer.trainable = True\n\nnum_epochs = 2\nnum_train_steps = len(tf_train_ds) * num_epochs\n\nlr_scheduler = tf.keras.optimizers.schedules.PolynomialDecay(\n    initial_learning_rate=2e-5, end_learning_rate=2e-6, decay_steps=num_train_steps\n)\n\nmodel.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=lr_scheduler),\n              loss=tf.keras.losses.MeanSquaredError()\n             )\n\n#result = model(example)\n#print(result)\nmodel.summary()","metadata":{"execution":{"iopub.status.busy":"2021-12-23T02:24:25.42634Z","iopub.execute_input":"2021-12-23T02:24:25.426545Z","iopub.status.idle":"2021-12-23T02:24:25.464484Z","shell.execute_reply.started":"2021-12-23T02:24:25.42652Z","shell.execute_reply":"2021-12-23T02:24:25.463345Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fit_history = model.fit(tf_train_ds,\n                        epochs=num_epochs,\n                        validation_data=tf_valid_ds,\n                        verbose=1)","metadata":{"execution":{"iopub.status.busy":"2021-12-23T02:24:25.465525Z","iopub.execute_input":"2021-12-23T02:24:25.466226Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 4. Prediction & Submit","metadata":{}},{"cell_type":"code","source":"test_df = pd.read_csv(\"../input/jigsaw-toxic-severity-rating/comments_to_score.csv\")\ntest_df['text'] = test_df['text'].apply(text_cleaning)\ntest_df = text_cleaning_2(test_df,'text')\ntest_df.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_ds = datasets.Dataset.from_pandas(test_df)\n\ntokenized_test_ds = test_ds.map(tokenize_function, batched=True)\ntf_test_ds = tokenized_test_ds.to_tf_dataset(\n    columns=[\"attention_mask\", \"input_ids\"],\n    shuffle=False,\n    collate_fn=data_collator,\n    batch_size=config['batch_size'],\n)\n\nprint(len(tf_test_ds))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"result = model.predict(tf_test_ds)\ntest_df['score'] = result\nsubmission_df = test_df[['comment_id', 'score']]\n\nsubmission_df.to_csv(\"submission.csv\", index=False)\nsubmission_df","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 4.1 validation","metadata":{}},{"cell_type":"code","source":"# New data for validation: text pairs.\ndata_valid = pd.read_csv('../input/jigsaw-toxic-severity-rating/validation_data.csv')\ndata_valid['less_toxic'] = data_valid['less_toxic'].apply(text_cleaning)\ndata_valid['more_toxic'] = data_valid['more_toxic'].apply(text_cleaning)\ndata_valid = text_cleaning_2(data_valid,'less_toxic')\ndata_valid = text_cleaning_2(data_valid,'more_toxic')\ndata_valid.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"more_or_less_toxic_ds = datasets.Dataset.from_pandas(data_valid)\nmore_or_less_toxic_ds","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def less_toxic_tokenize(example):\n    return tokenizer(example['less_toxic'], truncation=True, max_length=128)\n\ndef more_toxic_tokenize(example):\n    return tokenizer(example['more_toxic'], truncation=True, max_length=128)\n\nless_toxic_ds = more_or_less_toxic_ds.map(less_toxic_tokenize, batched=True)\nmore_toxic_ds = more_or_less_toxic_ds.map(more_toxic_tokenize, batched=True)\n\nprint(less_toxic_ds)\nprint(more_toxic_ds)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tf_less_ds = less_toxic_ds.to_tf_dataset(\n    columns=[\"attention_mask\", \"input_ids\"],\n    shuffle=False,\n    collate_fn=data_collator,\n    batch_size=config['batch_size'],\n)\n\ntf_more_ds = more_toxic_ds.to_tf_dataset(\n    columns=[\"attention_mask\", \"input_ids\"],\n    shuffle=False,\n    collate_fn=data_collator,\n    batch_size=config['batch_size'],\n)\n\nprint(len(tf_less_ds))\nprint(len(tf_more_ds))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"less_scores = model.predict(tf_less_ds)\nmore_scores = model.predict(tf_more_ds)\n\ndata_valid['less_score'] = less_scores\ndata_valid['more_score'] = more_scores\n\ndata_valid['correct'] = 1\ndata_valid['correct'] = data_valid['correct'].where(data_valid['less_score'] < data_valid['more_score'], 0)\n\naccuracy = data_valid['correct'].sum() / len(data_valid)\naccuracy","metadata":{"trusted":true},"execution_count":null,"outputs":[]}]}