{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport re\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 5GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As we have seen from EDA - the main problem of th data we have is that classes are unbalanced: toxic comments are only 10% of the whole dataset. \nThis hurts training a lot - let me explain how\n\nIf you have balanced classes then it is highly likely that in every training mini-batch there will be approximately equal number of positive and negative class objects, so the model regularly gets the equal signal what to change about both classes.\n\nBut if proportion of classes is 1 to 9 as we have then the model gets much less signal on positive class than on negative. That can lead to the situation where model will try to regard all samlples as negative or most of them as negative because false estimation of positive class sents too low signal to the loss function because of \nfew samples in mini-batch comparatively to negative class.\n\nthese are the links there this problem is shortly discussed\n\nhttps://machinelearningmastery.com/tactics-to-combat-imbalanced-classes-in-your-machine-learning-dataset/\nhttps://towardsdatascience.com/handling-imbalanced-datasets-in-machine-learning-7a0e84220f28","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"I will summarize what helped me the most: my aim is to make classes balanced for training\n\nFirst approach which is the most intuitive is to change loss function in such a way that the error on prediction toxic examples is higher than on non-toxic class\n\nLets remember how binary crossentropy looks for a single observation : loss = y_i * ln(p_i) + (1 - y_i)ln(1 - p_i), where y_i is a true class: either 1 or 0\np_i is the estimation of probability to be class == 1. We can change this loss: loss = alpha_1 * y_i * ln(p_i) + alpha_2 * (1 - y_i)ln(1 - p_i), \nwhere alphas are multiplieres by which we want to send to model larger signal for false prediction of positive class and lower for the negative class\n\nyou can check this guide to know more about it: https://www.tensorflow.org/tutorials/structured_data/imbalanced_data#calculate_class_weights\n\nWe can manually create such loss function or we can use class_weight option in the model.fit method\n\nIt should look like this:\n\ninitialize class weights: class_weights = [alpha_1, alpha_2]\n\nand then pass it to the model.fit(..., class_weight=class_weights) - which automatically applies class weightening to the loss","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"However, this approach has its drawbacks. In spite of the signal is strong on every positive prediction error, it's not regular\nSo the model makes big updates of the weights if there are some positive examples are in the batch and almost no updates if there are\nno or only a few positive samples in batch. There comes next two intuitive approaches:\n1. Oversampling\n2. Downsampling\n\nOversampling is a copying random n samples of lower class so to make dataset balanced\n\nDownsampling is a subsampling of examples of larger class\n\nDue to these techniques regular signal on both classes while training is provided. However, the following problems araise.\nDownsampling hurts training as it downsizes the amount of data - in our case we need to throw away about 180k samples in small dataset\nand about 1,5M in large dataset to balance the data - that's a lot!\nOversampling makes more data which is good, but at the same time it can lead to the fail in generalization and overfitting as the model sees\nonly a few examples repeated many times. Nevertheless, there are some doubts that those examples are representative of all posible variants. But\nthe more model sees only such examples, the more it gets confident that these are the only possible ones.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"My decision in tackling this problem is to apply something between. Let me show how","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Assume we have train indeces - encoded comment text and train labels - 1 or 0 for toxic and non-toxic\n\nWe first split data in positive and negative classes\n\n    pos = train_ids[np.where(train_labels == 1)[0]]\n    neg = train_ids[np.where(train_labels == 0)[0]]\n\n    pos_labels = train_labels[np.where(train_labels==1)[0]]\n    neg_labels = train_labels[np.where(train_labels==0)[0]]\n\n\n    def make_ds(features, labels):\n\n        ds = tf.data.Dataset.from_tensor_slices((features, labels))#.cache()\n    \n        ds = ds.shuffle(1500000).repeat()\n    \n        return ds\n    \nThen create tf.data.Dataset with positive ang negative data\n    \n    pos_ds = make_ds(pos, pos_labels)\n    neg_ds = make_ds(neg, neg_labels)\n\nAnd then we use method sample_from_datasets which takes from pos_ds and neg_ps those proportions of neg and pos data, which is passed to the weights option\n\n    resampled_ds = tf.data.experimental.sample_from_datasets([pos_ds, neg_ds], weights=[0.5, 0.5])\n    resampled_ds = resampled_ds.batch(BATCH_SIZE).prefetch(AUTO)\n\nAnd finally resampled_ds is passed into model.fit method ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"This method may work as downsampling as well as upsampling depending of batch_size and steps_per_epoch while training\n\nLets consider the example\n\nwe have 20k of toxic samples and 200k of non-toxic samples\n\nbatch_size = 256\n\nif steps_per_epoch == len(train_ids) // batch_size and every batch has 0.5*batch_size=128 toxic and non-toxic samples\n\nThen, all the toxic data will be passed to the model at 157 step of the training, while we defined that there will be 860 steps per epoch - that means that already used toxic samples will be passed to the model repeatedly which is equal to just upsampling.\n\nIf we define steps_per_epoch == len(pos)//batch_size//2 - in order to end epoch when unique toxic samples end - then it is equal to he downsampling\n\nSo my decision is to make steps_per_epoch is somewhere in the middle so that as much as possible non-toxic data is included and toxic comments are not copied 10 times.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Data augmentation","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Although we have found a way how to make training balanced, more unique data is always make things better. But when we don't have more unique data, we can create it by augmenting given samples.\n\nHere is the quick overview of methods i want to propose: https://towardsdatascience.com/these-are-the-easiest-data-augmentation-techniques-in-natural-language-processing-you-can-think-of-88e393fd610\n\nThese are \n1. Synonym Replacement: Randomly choose n words from the sentence that are not stop words. Replace each of these words with one of its synonyms chosen at random.\n2. Random Insertion: Find a random synonym of a random word in the sentence that is not a stop word. Insert that synonym into a random position in the sentence. Do this n times\n3. Random Swap: Randomly choose two words in the sentence and swap their positions. Do this n times\n4. Random Deletion: Randomly remove each word in the sentence with probability p","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"I've taken this functions from their Github: https://github.com/jasonwei20/eda_nlp\nThere is nice job done!","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"len(ds) * 5","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import random\nfrom random import shuffle\nrandom.seed(1)\n\nstop_words = ['i', 'me', 'my', 'myself', 'we', 'our', \n            'ours', 'ourselves', 'you', 'your', 'yours', \n            'yourself', 'yourselves', 'he', 'him', 'his', \n            'himself', 'she', 'her', 'hers', 'herself', \n            'it', 'its', 'itself', 'they', 'them', 'their', \n            'theirs', 'themselves', 'what', 'which', 'who', \n            'whom', 'this', 'that', 'these', 'those', 'am', \n            'is', 'are', 'was', 'were', 'be', 'been', 'being', \n            'have', 'has', 'had', 'having', 'do', 'does', 'did',\n            'doing', 'a', 'an', 'the', 'and', 'but', 'if', 'or',\n            'because', 'as', 'until', 'while', 'of', 'at', \n            'by', 'for', 'with', 'about', 'against', 'between',\n            'into', 'through', 'during', 'before', 'after', \n            'above', 'below', 'to', 'from', 'up', 'down', 'in',\n            'out', 'on', 'off', 'over', 'under', 'again', \n            'further', 'then', 'once', 'here', 'there', 'when', \n            'where', 'why', 'how', 'all', 'any', 'both', 'each', \n            'few', 'more', 'most', 'other', 'some', 'such', 'no', \n            'nor', 'not', 'only', 'own', 'same', 'so', 'than', 'too', \n            'very', 's', 't', 'can', 'will', 'just', 'don', \n            'should', 'now', '']\n\nimport re\ndef get_only_chars(line):\n\n    clean_line = \"\"\n\n    line = line.replace(\"’\", \"\")\n    line = line.replace(\"'\", \"\")\n    line = line.replace(\"-\", \" \") #replace hyphens with spaces\n    line = line.replace(\"\\t\", \" \")\n    line = line.replace(\"\\n\", \" \")\n    line = line.lower()\n\n    for char in line:\n        if char in 'qwertyuiopasdfghjklzxcvbnm ':\n            clean_line += char\n        else:\n            clean_line += ' '\n\n    clean_line = re.sub(' +',' ',clean_line) #delete extra spaces\n    #if clean_line[0] == ' ':\n    #    clean_line = clean_line[1:]\n    return clean_line\nfrom nltk.corpus import wordnet \n\ndef synonym_replacement(words, n):\n    new_words = words.copy()\n    random_word_list = list(set([word for word in words if word not in stop_words]))\n    random.shuffle(random_word_list)\n    num_replaced = 0\n    for random_word in random_word_list:\n        synonyms = get_synonyms(random_word)\n        if len(synonyms) >= 1:\n            synonym = random.choice(list(synonyms))\n            new_words = [synonym if word == random_word else word for word in new_words]\n            #print(\"replaced\", random_word, \"with\", synonym)\n            num_replaced += 1\n        if num_replaced >= n: #only replace up to n words\n            break\n\n    #this is stupid but we need it, trust me\n    sentence = ' '.join(new_words)\n    new_words = sentence.split(' ')\n\n    return new_words\n\ndef get_synonyms(word):\n    synonyms = set()\n    for syn in wordnet.synsets(word): \n        for l in syn.lemmas(): \n            synonym = l.name().replace(\"_\", \" \").replace(\"-\", \" \").lower()\n            synonym = \"\".join([char for char in synonym if char in ' qwertyuiopasdfghjklzxcvbnm'])\n            synonyms.add(synonym) \n    if word in synonyms:\n        synonyms.remove(word)\n    return list(synonyms)\n\ndef random_deletion(words, p):\n\n    #obviously, if there's only one word, don't delete it\n    if len(words) == 1:\n        return words\n\n    #randomly delete words with probability p\n    new_words = []\n    for word in words:\n        r = random.uniform(0, 1)\n        if r > p:\n            new_words.append(word)\n\n    #if you end up deleting all words, just return a random word\n    if len(new_words) == 0:\n        rand_int = random.randint(0, len(words)-1)\n        return [words[rand_int]]\n\n    return new_words\n\ndef random_swap(words, n):\n    new_words = words.copy()\n    for _ in range(n):\n        new_words = swap_word(new_words)\n    return new_words\n\ndef swap_word(new_words):\n    random_idx_1 = random.randint(0, len(new_words)-1)\n    random_idx_2 = random_idx_1\n    counter = 0\n    while random_idx_2 == random_idx_1:\n        random_idx_2 = random.randint(0, len(new_words)-1)\n        counter += 1\n        if counter > 3:\n            return new_words\n    new_words[random_idx_1], new_words[random_idx_2] = new_words[random_idx_2], new_words[random_idx_1] \n    return new_words\n\ndef random_insertion(words, n):\n    new_words = words.copy()\n    for _ in range(n):\n        add_word(new_words)\n    return new_words\n\ndef add_word(new_words):\n    synonyms = []\n    counter = 0\n    while len(synonyms) < 1:\n        random_word = new_words[random.randint(0, len(new_words)-1)]\n        synonyms = get_synonyms(random_word)\n        counter += 1\n        if counter >= 10:\n            return\n    random_synonym = synonyms[0]\n    random_idx = random.randint(0, len(new_words)-1)\n    new_words.insert(random_idx, random_synonym)\n\ndef eda(sentence, alpha_sr=0.1, alpha_ri=0.1, alpha_rs=0.1, p_rd=0.1, num_aug=9):\n\n    sentence = get_only_chars(sentence)\n    words = sentence.split(' ')\n    words = [word for word in words if word is not '']\n    num_words = len(words)\n\n    if num_words == 0:\n        return [\"\"] * num_aug\n    augmented_sentences = []\n    num_new_per_technique = int(num_aug/4)+1\n    n_sr = max(1, int(alpha_sr*num_words))\n    n_ri = max(1, int(alpha_ri*num_words))\n    n_rs = max(1, int(alpha_rs*num_words))\n\n    #sr\n    for _ in range(num_new_per_technique):\n        a_words = synonym_replacement(words, n_sr)\n        augmented_sentences.append(' '.join(a_words))\n\n    #ri\n    for _ in range(num_new_per_technique):\n        a_words = random_insertion(words, n_ri)\n        augmented_sentences.append(' '.join(a_words))\n\n    #rs\n    for _ in range(num_new_per_technique):\n        a_words = random_swap(words, n_rs)\n        augmented_sentences.append(' '.join(a_words))\n\n#rd\n    for _ in range(num_new_per_technique):\n        a_words = random_deletion(words, p_rd)\n        augmented_sentences.append(' '.join(a_words))\n\n    augmented_sentences = [get_only_chars(sentence) for sentence in augmented_sentences]\n    shuffle(augmented_sentences)\n\n#trim so that we have the desired number of augmented sentences\n    if num_aug >= 1:\n        augmented_sentences = np.random.choice(augmented_sentences, num_aug, replace=False)\n    else:\n        keep_prob = num_aug / len(augmented_sentences)\n        augmented_sentences = [s for s in augmented_sentences if random.uniform(0, 1) < keep_prob]\n\n\n    return augmented_sentences","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#lets look what function does\nsentence = \"Hello! This function creates new examples from data\"\naugs = eda(sentence)\nfor element in augs:\n    print(element)\n    print()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Seems like only a slice changes've been made - let's raise the parameters","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"sentence = \" Hello! This function creates new examples from data\"\naugs = eda(sentence, alpha_sr=0.5, alpha_ri=0.1, alpha_rs=0.1, p_rd=0.1)\nfor element in augs:\n    print(element)\n    print()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#try with toxic\nsentence = \"What a fuck are you doing man. It's a fucking bullshit\"\naugs = eda(sentence, alpha_sr=0.3, alpha_ri=0.3, alpha_rs=0.3, p_rd=0.3)\nfor element in augs:\n    print(element)\n    print()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#try with toxic\nsentence = \"What a fuck are you doing man. It's a fucking bullshit\"\naugs = eda(sentence, alpha_sr=0.3, alpha_ri=0.2, alpha_rs=0.2, p_rd=0.2)\nfor element in augs:\n    print(element)\n    print()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"And also I've written my own function to augment the dataset","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def augment_data(toxic_df, alpha_sr=0.3, alpha_ri=0.2, alpha_rs=0.2, p_rd=0.2, num_aug=4):\n    pos = 0\n    sentences = []\n    while pos < len(toxic_df):\n        for sent in toxic_df[\"comment_text\"][pos:pos+10000]:\n            sentences.extend(eda(sent, num_aug=num_aug))\n        pos += 10000\n        print(\"Processed\", pos, \"sentences\")\n    #sentences = np.concatenate(sentences).tolist()\n    labels = len(sentences) * [1]\n    return sentences, labels","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"large_ds = pd.read_csv(\"../input/jigsaw-multilingual-toxic-comment-classification/jigsaw-unintended-bias-train.csv\", usecols=[\"comment_text\",\"toxic\"]).query(\"toxic > 0.5\")\nsmall_ds = pd.read_csv(\"../input/jigsaw-multilingual-toxic-comment-classification/jigsaw-toxic-comment-train.csv\", usecols=[\"comment_text\",\"toxic\"]).query(\"toxic==1\")\n\nds = pd.concat((large_ds,small_ds))\nds[\"length\"] = ds.comment_text.str.split().apply(len)\ndel large_ds\ndel small_ds\nds = ds[ds[\"length\"] > 0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"aug_sents, aug_labels = augment_data(ds)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"aug_df = pd.DataFrame({\"comment_text\":aug_sents, \"toxic\":aug_labels})\naug_df.to_csv(\"aug.csv\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Backtranslating","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Another way to get more toxic data is to make backtranslation via for example google api. This works as result of backtranslation ( from english to russian and back to english) slighty differs from original sentence, while the sense saves.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"You can use following code with your own credentials of google api to do it (you'll need to have an active account)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"    from google.oauth2 import service_account\n    from google.cloud import translate_v2 as translate\n    \n    credentials = service_account.Credentials.from_service_account_file(\"PATH_TO_YOUR_JSON.json\")\n    translate_client = translate.Client(credentials=credentials)\n    text = \"Hello you motherfucking son of a bitch? How are you doing?\"\n    print(text)\n    result = translate_client.translate(\n        text, target_language=\"ru\")[\"translatedText\"]\n    print(result)\n    back_res = translate_client.translate(result, target_language=\"eng\")[\"translatedText\"]\n    print(back_res)\n    \n    def back_translate_sent(text, lang=\"ru\"):\n    res = translate_client.translate(text, target_language=lang)[\"translatedText\"]\n    back_res = translate_client.translate(res, target_language=\"eng\")[\"translatedText\"]\n    return back_res\n\n    def back_translate_n(text, lang_list=[\"ru\", \"tr\", \"it\"], samples_per_sent=3):\n        sentences = []\n        for l in lang_list:\n            sentences.append(back_translate_sent(text, lang=l))\n        return sentences\n    \n    def back_translate_data(ds, samples_per_sent=3):\n        pos = 0\n        sentences = []\n        while pos < len(ds):\n            sentences.extend(ds[\"comment_text\"][pos:pos+10000].apply(back_translate_n).values)\n            pos += 10000\n            print(\"Processed\", pos, \"sentences\")\n        labels = len(sentences) * [1]\n        return sentences, labels\n\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"One more idea in balancing classes and training is to use comments we have but translated to the languages in test data. Models we use can handle multilingual inputs and may be it will enrich the training results\n\nHere is the link to the public dataset with translated comments for this competition: https://www.kaggle.com/miklgr500/jigsaw-train-multilingual-coments-google-api","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}