{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## Training Pipeline by [@shonenkov](https://www.kaggle.com/shonenkov) using multi TPU on PyTorch/XLA\n\nHi everyone!\n\nMy name is Alex Shonenkov, I am researcher, in Love with NLP and DL.\n\nRecently I have published my ideas about this competition:\n\n- [[TPU-Inference] Super Fast XLMRoberta](https://www.kaggle.com/shonenkov/tpu-inference-super-fast-xlmroberta)\n- [NLP Albumentations](https://www.kaggle.com/shonenkov/nlp-albumentations)\n- [Hack with Parallel Corpus](https://www.kaggle.com/shonenkov/hack-with-parallel-corpus)\n- [Class Balance with PyTorch/XLA](https://www.kaggle.com/shonenkov/class-balance-with-pytorch-xla)\n- [open-subtitles-toxic-pseudo-labeling](https://www.kaggle.com/shonenkov/open-subtitles-toxic-pseudo-labeling)\n\nif you didn't see this kernels and datasets, I recommend to read all of them because it may help you for better understand this kernel and achieve success in competition :)","metadata":{}},{"cell_type":"code","source":"# 少這個 import torch_xla 時好像會出錯\n!curl https://raw.githubusercontent.com/pytorch/xla/master/contrib/scripts/env-setup.py -o pytorch-xla-env-setup.py > /dev/null\n!python pytorch-xla-env-setup.py --version 20200420 --apt-packages libomp5 libopenblas-dev > /dev/null\n!pip install transformers==2.5.1 > /dev/null\n!pip install pandarallel > /dev/null\n!pip install catalyst==20.4.2 > /dev/null","metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-07-25T17:31:00.516391Z","iopub.execute_input":"2021-07-25T17:31:00.516901Z","iopub.status.idle":"2021-07-25T17:32:52.553221Z","shell.execute_reply.started":"2021-07-25T17:31:00.51677Z","shell.execute_reply":"2021-07-25T17:32:52.551873Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\n\nimport os\nos.environ['XLA_USE_BF16'] = \"1\"\n\nfrom glob import glob\n\nimport torch\nimport torch.nn as nn\nfrom torch.utils.data import Dataset,DataLoader\nfrom torch.autograd import Variable\nfrom torch.utils.data.sampler import SequentialSampler, RandomSampler\nimport sklearn\n\nimport time\nimport random\nfrom datetime import datetime\nfrom tqdm import tqdm # 好用的進度條工具\ntqdm.pandas()\n\n\nfrom transformers import BertModel, BertTokenizer # Bert 的 library\nfrom transformers import XLMRobertaModel, XLMRobertaTokenizer # XLMRoberta 的 library\nfrom transformers import AdamW, get_linear_schedule_with_warmup, get_constant_schedule # 只有 AdamW有用到，後面兩個式schedule的方式\nfrom catalyst.data.sampler import DistributedSamplerWrapper, BalanceClassSampler # sampler，後面有詳細的介紹\n\nimport gc # 垃圾回收機制(garbage collection)，根據記憶體的分配和釋放情況呼叫\nimport re # regular expression簡稱re(正規表示式)，一些處理字串的方法\n\n\n# !pip install nltk > /dev/null\nimport nltk\nnltk.download('punkt')\n\nfrom nltk import sent_tokenize # nltk:自然語言處理工具箱\n                               # sent_tokenize:幫助我們斷句\n\n# pandarallela目前不支援windows，用類似df.parallel_apply(func)的方式平行化處理\nfrom pandarallel import pandarallel \npandarallel.initialize(nb_workers=4, progress_bar=False) # 宣告有幾個CPU來處理，progress_bar=True會顯示不同CPU的工作進度","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-07-25T17:33:16.188825Z","iopub.execute_input":"2021-07-25T17:33:16.18921Z","iopub.status.idle":"2021-07-25T17:33:24.42997Z","shell.execute_reply.started":"2021-07-25T17:33:16.189171Z","shell.execute_reply":"2021-07-25T17:33:24.428697Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"SEED = 42\n\nMAX_LENGTH = 224 # 設定斷句的最大長度，不同模型會有不同設定\nBACKBONE_PATH = 'xlm-roberta-large'\nROOT_PATH = f'..'\n# ROOT_PATH = f'/content/drive/My Drive/jigsaw2020-kaggle-public-baseline' # for colab\n\n\ndef seed_everything(seed): # 設定隨機的seed\n    random.seed(seed)\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    np.random.seed(seed)\n    torch.manual_seed(seed)\n    torch.cuda.manual_seed(seed)\n    torch.backends.cudnn.deterministic = True\n    torch.backends.cudnn.benchmark = True\n\nseed_everything(SEED)","metadata":{"execution":{"iopub.status.busy":"2021-07-25T17:33:24.432033Z","iopub.execute_input":"2021-07-25T17:33:24.43236Z","iopub.status.idle":"2021-07-25T17:33:24.442452Z","shell.execute_reply.started":"2021-07-25T17:33:24.432325Z","shell.execute_reply":"2021-07-25T17:33:24.441492Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### [NLP Albumentations](https://www.kaggle.com/shonenkov/nlp-albumentations)\n做資料擴增/數據增強(資料處理)，連結裡有示範怎麼增加，就是對一些字詞順序調換之類的處理。\n是現在 NLP 一種很厲害的 Data Augmentation","metadata":{}},{"cell_type":"code","source":"from nltk import sent_tokenize # sent_tokenize:幫助我們斷句\nfrom random import shuffle\nimport random\nimport albumentations # NLP 資料擴增的 library\nfrom albumentations.core.transforms_interface import DualTransform, BasicTransform\n\n# 語言名稱的縮寫，通常 dataset 會標明甚麼語言，這個 competition 應該只有英文的 dataset\nLANGS = { \n    'en': 'english',\n    'it': 'italian', \n    'fr': 'french', \n    'es': 'spanish',\n    'tr': 'turkish', \n    'ru': 'russian',\n    'pt': 'portuguese'\n}\n\ndef get_sentences(text, lang='en'):\n    return sent_tokenize(text, LANGS.get(lang, 'english')) # 斷句，只有英文\n\ndef exclude_duplicate_sentences(text, lang='en'): # 排除重複的句子\n    sentences = []\n    for sentence in get_sentences(text, lang): \n        sentence = sentence.strip() # 移除指定開頭及結尾的字元，斷句完會有空格\n        if sentence not in sentences: # 非重複才加入\n            sentences.append(sentence)\n    return ' '.join(sentences)\n\n# re.sub(替代) 會替換配對的項目，回傳一個經替換的字串，如果沒有則回傳原字串\n# ex: re.sub(r'\\s+', '-', text) 會將 ' ' 替代成 '-'\ndef clean_text(text, lang='en'):\n    text = str(text)\n    text = re.sub(r'[0-9\"]', '', text) # 去除數字\n    text = re.sub(r'#[\\S]+\\b', '', text) # 去除帶有井字號的詞，像是: #kaggle\n    text = re.sub(r'@[\\S]+\\b', '', text) # 去除帶有小老鼠的詞，像是: @kaggle\n    text = re.sub(r'https?\\S+', '', text) # 去除帶有 http 的詞\n    text = re.sub(r'\\s+', ' ', text) # 將多個空格簡化成一個\n    text = exclude_duplicate_sentences(text, lang)\n    return text.strip()\n\n\nclass NLPTransform(BasicTransform):\n    @property\n    def targets(self):\n        return {\"data\": self.apply}\n    \n    def update_params(self, params, **kwargs):  # **kwargs:是可變的keyword arguments列表\n        if hasattr(self, \"interpolation\"): # 檢查有沒有 \"interpolation\" 這個\n            params[\"interpolation\"] = self.interpolation\n        if hasattr(self, \"fill_value\"):\n            params[\"fill_value\"] = self.fill_value\n        return params\n\n    def get_sentences(self, text, lang='en'):\n        return sent_tokenize(text, LANGS.get(lang, 'english'))\n\nclass ShuffleSentencesTransform(NLPTransform): # ShuffleSentences變換\n    \"\"\" Do shuffle by sentence \"\"\"\n    def __init__(self, always_apply=False, p=0.5):\n        super(ShuffleSentencesTransform, self).__init__(always_apply, p)\n\n    def apply(self, data, **params):\n        text, lang = data\n        sentences = self.get_sentences(text, lang)\n        random.shuffle(sentences)\n        return ' '.join(sentences), lang\n\nclass ExcludeDuplicateSentencesTransform(NLPTransform): # 排除重複的\n    \"\"\" Exclude equal sentences \"\"\"\n    def __init__(self, always_apply=False, p=0.5):\n        super(ExcludeDuplicateSentencesTransform, self).__init__(always_apply, p)\n\n    def apply(self, data, **params):\n        text, lang = data\n        sentences = []\n        for sentence in self.get_sentences(text, lang):\n            sentence = sentence.strip()\n            if sentence not in sentences:\n                sentences.append(sentence)\n        return ' '.join(sentences), lang\n\nclass ExcludeNumbersTransform(NLPTransform): \n    \"\"\" exclude any numbers \"\"\"\n    def __init__(self, always_apply=False, p=0.5):\n        super(ExcludeNumbersTransform, self).__init__(always_apply, p)\n\n    def apply(self, data, **params):\n        text, lang = data\n        text = re.sub(r'[0-9]', '', text)\n        text = re.sub(r'\\s+', ' ', text)\n        return text, lang\n\nclass ExcludeHashtagsTransform(NLPTransform): # 去除帶有井字號的詞，像是: #kaggle\n    \"\"\" Exclude any hashtags with # \"\"\"\n    def __init__(self, always_apply=False, p=0.5):\n        super(ExcludeHashtagsTransform, self).__init__(always_apply, p)\n\n    def apply(self, data, **params):\n        text, lang = data\n        text = re.sub(r'#[\\S]+\\b', '', text)\n        text = re.sub(r'\\s+', ' ', text)\n        return text, lang\n\nclass ExcludeUsersMentionedTransform(NLPTransform): # 去除帶有小老鼠的詞，像是: @kaggle\n    \"\"\" Exclude @users \"\"\"\n    def __init__(self, always_apply=False, p=0.5):\n        super(ExcludeUsersMentionedTransform, self).__init__(always_apply, p)\n\n    def apply(self, data, **params):\n        text, lang = data\n        text = re.sub(r'@[\\S]+\\b', '', text)\n        text = re.sub(r'\\s+', ' ', text)\n        return text, lang\n\nclass ExcludeUrlsTransform(NLPTransform): # 去除帶有 http 的詞\n    \"\"\" Exclude urls \"\"\"\n    def __init__(self, always_apply=False, p=0.5):\n        super(ExcludeUrlsTransform, self).__init__(always_apply, p)\n\n    def apply(self, data, **params):\n        text, lang = data\n        text = re.sub(r'https?\\S+', '', text)\n        text = re.sub(r'\\s+', ' ', text)\n        return text, lang","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-07-25T17:33:24.444365Z","iopub.execute_input":"2021-07-25T17:33:24.444766Z","iopub.status.idle":"2021-07-25T17:33:25.341479Z","shell.execute_reply.started":"2021-07-25T17:33:24.444728Z","shell.execute_reply":"2021-07-25T17:33:25.340345Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### [Pseudo-labeling with open-subtitles](https://www.kaggle.com/shonenkov/hack-with-parallel-corpus)\n目的：得到其他語言 toxic 的詞。\n\n做法：先載入英文對其他語言的轉換語料庫，由英文去預測是否為 toxic ，再由語料庫去比對。\n\n\nMore noise with mix of languages can help. I have used [pseudo-labeled open-subtitles dataset](https://www.kaggle.com/shonenkov/open-subtitles-toxic-pseudo-labeling) for this approach. \n\nIt is some analogue for Cutmix in Computer Vision:\n\n<img src='https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2605845%2Ff29492171d83dfa6b6fcae2af414fcf8%2FCutmix_exmaple.png?generation=1579343294489994&alt=media' align=\"left\"> \n\n","metadata":{}},{"cell_type":"code","source":"# 利用 OpenSubtitles Community 所提供的資料來做合成\nclass SynthesicOpenSubtitlesTransform(NLPTransform):\n    def __init__(self, always_apply=False, p=0.5):\n        super(SynthesicOpenSubtitlesTransform, self).__init__(always_apply, p)\n        # 讀取 OpenSubtitles Community 提供的資料庫\n        df = pd.read_csv(f'{ROOT_PATH}/input/open-subtitles-toxic-pseudo-labeling/open-subtitles-synthesic.csv', index_col='id')[['comment_text', 'toxic', 'lang']]\n        # 資料清洗\n        df = df[~df['comment_text'].isna()] # 刪除有缺失項\n        df['comment_text'] = df.parallel_apply(lambda x: clean_text(x['comment_text'], x['lang']), axis=1)\n        df = df.drop_duplicates(subset='comment_text') # 刪除重複項\n        df['toxic'] = df['toxic'].round().astype(np.int)\n\n        self.synthesic_toxic = df[df['toxic'] == 1].comment_text.values\n        self.synthesic_non_toxic = df[df['toxic'] == 0].comment_text.values\n\n        del df\n        gc.collect(); # 記憶體回收\n\n    def generate_synthesic_sample(self, text, toxic): # 隨機的方式選擇增加的字詞\n        texts = [text]\n        if toxic == 0:\n            for i in range(random.randint(1,5)):\n                texts.append(random.choice(self.synthesic_non_toxic))\n        else:\n            for i in range(random.randint(0,2)):\n                texts.append(random.choice(self.synthesic_non_toxic))\n            \n            for i in range(random.randint(1,3)):\n                texts.append(random.choice(self.synthesic_toxic))\n        random.shuffle(texts)\n        return ' '.join(texts)\n\n    def apply(self, data, **params):\n        text, toxic = data\n        text = self.generate_synthesic_sample(text, toxic)\n        return text, toxic","metadata":{"execution":{"iopub.status.busy":"2021-07-25T17:33:25.34314Z","iopub.execute_input":"2021-07-25T17:33:25.343482Z","iopub.status.idle":"2021-07-25T17:33:25.358089Z","shell.execute_reply.started":"2021-07-25T17:33:25.343445Z","shell.execute_reply":"2021-07-25T17:33:25.356788Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_train_transforms(): # 轉成訓練需要的格式，包含前面的所有 transform\n    return albumentations.Compose([\n        ExcludeUsersMentionedTransform(p=0.95),\n        ExcludeUrlsTransform(p=0.95),\n        ExcludeNumbersTransform(p=0.95),\n        ExcludeHashtagsTransform(p=0.95),\n        ExcludeDuplicateSentencesTransform(p=0.95),\n    ], p=1.0)\n\ndef get_synthesic_transforms(): # 合成的 transform\n    return SynthesicOpenSubtitlesTransform(p=0.5)\n\n# 宣告 Transformer\ntrain_transforms = get_train_transforms(); \nsynthesic_transforms = get_synthesic_transforms()\nshuffle_transforms = ShuffleSentencesTransform(always_apply=True)\n\n# 拿 pre-trained 好的 tokenizer(不同的模型需要不同的 tokenizer)\ntokenizer = XLMRobertaTokenizer.from_pretrained(BACKBONE_PATH) ","metadata":{"execution":{"iopub.status.busy":"2021-07-25T17:33:26.312105Z","iopub.execute_input":"2021-07-25T17:33:26.312466Z","iopub.status.idle":"2021-07-25T17:34:21.735941Z","shell.execute_reply.started":"2021-07-25T17:33:26.312435Z","shell.execute_reply":"2021-07-25T17:34:21.734829Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def onehot(size, target): # 建立 one-hot encoding 的 matrix\n    vec = torch.zeros(size, dtype=torch.float32)\n    vec[target] = 1.\n    return vec\n\nclass DatasetRetriever(Dataset):\n\n    def __init__(self, labels_or_ids, comment_texts, langs, use_train_transforms=False, test=False):\n        # 把 dataset 讀取進來的初始化資訊\n        self.test = test # 將訓練(train + validation)和測試(test)分開\n        self.labels_or_ids = labels_or_ids # 是否為惡意評論\n        self.comment_texts = comment_texts\n        self.langs = langs\n        self.use_train_transforms = use_train_transforms\n        \n    def get_tokens(self, text):\n        # 用預訓練 XLMRobertaTokenizer 得到 attention 權重\n        encoded = tokenizer.encode_plus(\n            text,  # 需要轉換的對象\n            add_special_tokens=True,  # 會用指定的 model 轉換，這邊指 XLMRobertaTokenizer，否則用 BERT\n            max_length=MAX_LENGTH, # 224\n            pad_to_max_length=True\n        )\n        return encoded['input_ids'], encoded['attention_mask'] # 回傳處理好的 attention feature\n    \n    def __len__(self): # 回傳評論的長度，dataset有給，不確定為甚麼不用\n        return self.comment_texts.shape[0]\n\n    def __getitem__(self, idx):\n        text = self.comment_texts[idx]\n        lang = self.langs[idx]\n        if self.test is False: # 為 train 的資料需要產生 one-hot encoding array\n            label = self.labels_or_ids[idx]\n            target = onehot(2, label)\n\n        if self.use_train_transforms: # 有做資料擴增\n            text, _ = train_transforms(data=(text, lang))['data']\n            tokens, attention_mask = self.get_tokens(str(text))\n            token_length = sum(attention_mask)\n            # (有可能再增加資料集，同時做transform)\n            if token_length > 0.8*MAX_LENGTH: # 若大於一定長度，則丟入shuffle transformer\n                text, _ = shuffle_transforms(data=(text, lang))['data']\n            elif token_length < 60: # 若太短則合成更多資料(增加資料集)\n                text, _ = synthesic_transforms(data=(text, label))['data']\n            else:\n                tokens, attention_mask = torch.tensor(tokens), torch.tensor(attention_mask)\n                return target, tokens, attention_mask\n\n        tokens, attention_mask = self.get_tokens(str(text))\n        tokens, attention_mask = torch.tensor(tokens), torch.tensor(attention_mask)\n\n        if self.test is False:\n            return target, tokens, attention_mask\n        return self.labels_or_ids[idx], tokens, attention_mask # 若為測試 data 直接回傳\n\n    def get_labels(self): # 回傳是否回惡意評論 & 語言類別\n        return list(np.char.add(self.labels_or_ids.astype(str), self.langs))","metadata":{"execution":{"iopub.status.busy":"2021-07-25T17:34:21.737854Z","iopub.execute_input":"2021-07-25T17:34:21.738257Z","iopub.status.idle":"2021-07-25T17:34:21.754567Z","shell.execute_reply.started":"2021-07-25T17:34:21.738215Z","shell.execute_reply":"2021-07-25T17:34:21.753094Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Here I have used [this kernel](https://www.kaggle.com/shonenkov/prepare-training-data) for merging all train data ","metadata":{}},{"cell_type":"code","source":"%%time\n\ndf_train = pd.read_csv(f'{ROOT_PATH}/input/jigsaw-public-baseline-train-data/train_data.csv')\n\n\ntrain_dataset = DatasetRetriever(\n    labels_or_ids=df_train['toxic'].values,  # 是否為惡意評論\n    comment_texts=df_train['comment_text'].values, # 評論內容\n    langs=df_train['lang'].values, # 語言類別\n    use_train_transforms=True,\n)\n\ndel df_train \ngc.collect(); # 釋放記憶體\n\nfor targets, tokens, attention_masks in train_dataset:\n    break\n    \nprint(targets)\nprint(tokens.shape)\nprint(attention_masks.shape)","metadata":{"execution":{"iopub.status.busy":"2021-07-25T17:34:21.757114Z","iopub.execute_input":"2021-07-25T17:34:21.757503Z","iopub.status.idle":"2021-07-25T17:35:00.6281Z","shell.execute_reply.started":"2021-07-25T17:34:21.757468Z","shell.execute_reply":"2021-07-25T17:35:00.626958Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Class Balance\n\nAfter some experiments I have decided that [class balance](https://www.kaggle.com/shonenkov/class-balance-with-pytorch-xla) in this competition is very important. Also I noticed impact if use balancing dataset by languages.\n\nHere you can see unique values for get_labels method:\n\n維持資料的平均程度","metadata":{}},{"cell_type":"code","source":"np.unique(train_dataset.get_labels())","metadata":{"execution":{"iopub.status.busy":"2021-07-25T17:35:00.629726Z","iopub.execute_input":"2021-07-25T17:35:00.630033Z","iopub.status.idle":"2021-07-25T17:35:10.895958Z","shell.execute_reply.started":"2021-07-25T17:35:00.630004Z","shell.execute_reply":"2021-07-25T17:35:10.89496Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"### 讀取驗證的 dataset，index_col 可以將 column 設定成 index\ndf_val = pd.read_csv(f'{ROOT_PATH}/input/jigsaw-multilingual-toxic-comment-classification/validation.csv', index_col='id')\n\nvalidation_tune_dataset = DatasetRetriever(\n    labels_or_ids=df_val['toxic'].values, \n    comment_texts=df_val['comment_text'].values, \n    langs=df_val['lang'].values,\n    use_train_transforms=True, # 有座資料擴增\n)\n\ndf_val['comment_text'] = df_val.parallel_apply(lambda x: clean_text(x['comment_text'], x['lang']), axis=1)\n\nvalidation_dataset = DatasetRetriever(\n    labels_or_ids=df_val['toxic'].values,  # 是否為惡意評論\n    comment_texts=df_val['comment_text'].values, # 評論內容\n    langs=df_val['lang'].values, # 語言類別\n    use_train_transforms=False, \n)\n\ndel df_val\ngc.collect();\n\nfor targets, tokens, attention_masks in validation_dataset:\n    break\n\nprint(targets)\nprint(tokens.shape)\nprint(attention_masks.shape)","metadata":{"execution":{"iopub.status.busy":"2021-07-25T17:35:10.899508Z","iopub.execute_input":"2021-07-25T17:35:10.899874Z","iopub.status.idle":"2021-07-25T17:35:12.885616Z","shell.execute_reply.started":"2021-07-25T17:35:10.899837Z","shell.execute_reply":"2021-07-25T17:35:12.884328Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_test = pd.read_csv(f'{ROOT_PATH}/input/jigsaw-multilingual-toxic-comment-classification/test.csv', index_col='id')\n\n# parallel_apply 可以平行化處理，平行化處理\"資料清洗\"(因為訓練的資料有做)\ndf_test['comment_text'] = df_test.parallel_apply(lambda x: clean_text(x['content'], x['lang']), axis=1)\n\ntest_dataset = DatasetRetriever(\n    labels_or_ids=df_test.index.values, \n    comment_texts=df_test['comment_text'].values, \n    langs=df_test['lang'].values,\n    use_train_transforms=False,\n    test=True # 標示為測試資料\n)\n\ndel df_test\ngc.collect(); # 釋放記憶體\n\nfor ids, tokens, attention_masks in test_dataset:\n    break\n\nprint(ids)\nprint(tokens.shape)\nprint(attention_masks.shape)","metadata":{"execution":{"iopub.status.busy":"2021-07-25T17:35:12.887106Z","iopub.execute_input":"2021-07-25T17:35:12.887418Z","iopub.status.idle":"2021-07-25T17:35:25.220777Z","shell.execute_reply.started":"2021-07-25T17:35:12.887383Z","shell.execute_reply":"2021-07-25T17:35:25.219558Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 算分數\n可以參考 https://gist.github.com/nlpjoe/50c16a210a5898d2589ac341274db4bb#file-rocaucmeter-md\n\n基本上是在講 F1-score 的概念","metadata":{}},{"cell_type":"code","source":"class RocAucMeter(object):\n    def __init__(self):\n        self.reset()\n\n    def reset(self):\n        self.y_true = np.array([0,1])\n        self.y_pred = np.array([0.5,0.5])\n        self.score = 0\n\n    def update(self, y_true, y_pred):\n        y_true = y_true.cpu().numpy().argmax(axis=1)\n        y_pred = nn.functional.softmax(y_pred, dim=1).data.cpu().numpy()[:,1]\n        self.y_true = np.hstack((self.y_true, y_true))\n        self.y_pred = np.hstack((self.y_pred, y_pred))\n        self.score = sklearn.metrics.roc_auc_score(self.y_true, self.y_pred, labels=np.array([0, 1]))\n    \n    @property\n    def avg(self):\n        return self.score\n\nclass AverageMeter(object):\n    \"\"\"Computes and stores the average and current value\"\"\"\n    def __init__(self):\n        self.reset()\n\n    def reset(self):\n        self.val = 0\n        self.avg = 0\n        self.sum = 0\n        self.count = 0\n\n    def update(self, val, n=1):\n        self.val = val\n        self.sum += val * n\n        self.count += n\n        self.avg = self.sum / self.count","metadata":{"execution":{"iopub.status.busy":"2021-07-25T17:35:25.222468Z","iopub.execute_input":"2021-07-25T17:35:25.223088Z","iopub.status.idle":"2021-07-25T17:35:25.234926Z","shell.execute_reply.started":"2021-07-25T17:35:25.223041Z","shell.execute_reply":"2021-07-25T17:35:25.234249Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Label Smoothing is all you need\n[inception v3] 一種正歸化、控制loss，主要防止 overfitting\n\n想知道詳細運作原理可以參考https://zhuanlan.zhihu.com/p/377084763","metadata":{}},{"cell_type":"code","source":"class LabelSmoothing(nn.Module): # 繼承 nn.Module\n    def __init__(self, smoothing = 0.1):\n        super(LabelSmoothing, self).__init__()\n        self.confidence = 1.0 - smoothing\n        self.smoothing = smoothing\n    # training 的時候 loss 控制\n    def forward(self, x, target):\n        if self.training:\n            x = x.float()\n            target = target.float()\n            logprobs = torch.nn.functional.log_softmax(x, dim = -1)\n            nll_loss = -logprobs * target\n            nll_loss = nll_loss.sum(-1)\n            smooth_loss = -logprobs.mean(dim=-1)\n            loss = self.confidence * nll_loss + self.smoothing * smooth_loss\n            return loss.mean()\n        else:\n            return torch.nn.functional.cross_entropy(x, target)","metadata":{"execution":{"iopub.status.busy":"2021-07-25T17:35:25.237105Z","iopub.execute_input":"2021-07-25T17:35:25.237691Z","iopub.status.idle":"2021-07-25T17:35:25.248207Z","shell.execute_reply.started":"2021-07-25T17:35:25.237647Z","shell.execute_reply":"2021-07-25T17:35:25.247311Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Custom TPU Fitter\n\n<img src='https://image.made-in-china.com/202f0j10dPkYMNLhhabE/Children-Bicycle-Baby-Kids-BMX-Bike.jpg' width=250 align=\"left\"> \n\nP.S. Lets go to do contributing [Catalyst](https://github.com/catalyst-team/catalyst) with TPU backend :)\n\n<img src='https://raw.githubusercontent.com/catalyst-team/catalyst-pics/master/pics/catalyst_logo.png' width=100 align=\"center\">\n","metadata":{}},{"cell_type":"code","source":"# 少這個 import torch_xla 時好像會出錯\n!curl https://raw.githubusercontent.com/pytorch/xla/master/contrib/scripts/env-setup.py -o pytorch-xla-env-setup.py > /dev/null\n!python pytorch-xla-env-setup.py --version 20200420 --apt-packages libomp5 libopenblas-dev > /dev/null\n!pip install transformers==2.5.1 > /dev/null\n!pip install pandarallel > /dev/null\n!pip install catalyst==20.4.2 > /dev/null","metadata":{"execution":{"iopub.status.busy":"2021-07-25T17:40:52.444748Z","iopub.execute_input":"2021-07-25T17:40:52.445198Z","iopub.status.idle":"2021-07-25T17:42:07.322868Z","shell.execute_reply.started":"2021-07-25T17:40:52.445159Z","shell.execute_reply":"2021-07-25T17:42:07.321691Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import warnings\n\nwarnings.filterwarnings(\"ignore\") # 處理警告的方式—忽略警告\n\n# XLA 設備的 pytorch library \n# https://pytorch.apachecn.org/docs/1.4/69.html\nimport torch_xla \nimport torch_xla.core.xla_model as xm\nimport torch_xla.distributed.parallel_loader as pl # PyTorch DataLoader\nimport torch_xla.distributed.xla_multiprocessing as xmp\n\n'''\nhttps://catalyst-team.github.io/catalyst/api/data.html#catalyst.data.sampler.BalanceClassSampler\nDistributedSamplerWrapper: 用於分佈式訓練\nBalanceClassSampler: 在 unbalanced 的情況產生安全的 sample\n'''\nfrom catalyst.data.sampler import DistributedSamplerWrapper, BalanceClassSampler\n\n# 在講如何放進 TPU 訓練\nclass TPUFitter:\n    def __init__(self, model, device, config): # 傳入的參數為 (訓練的網路、指定的TPU、訓練參數)\n        if not os.path.exists('node_submissions'):\n            os.makedirs('node_submissions')\n\n        self.config = config\n        self.epoch = 0\n        self.log_path = 'log.txt'\n\n        self.model = model\n        self.device = device\n\n        param_optimizer = list(self.model.named_parameters())\n        no_decay = ['bias', 'LayerNorm.bias', 'LayerNorm.weight'] # weightdecay 的變化方式\n        optimizer_grouped_parameters = [\n            {'params': [p for n, p in param_optimizer if not any(nd in n for nd in no_decay)], 'weight_decay': 0.001},\n            {'params': [p for n, p in param_optimizer if any(nd in n for nd in no_decay)], 'weight_decay': 0.0}\n        ]\n\n        self.optimizer = AdamW(optimizer_grouped_parameters, lr=config.lr*xm.xrt_world_size()) # optimizer 選 AdamW\n        self.scheduler = config.SchedulerClass(self.optimizer, **config.scheduler_params) # loss 變化的 scheduler\n\n        self.criterion = config.criterion\n        xm.master_print(f'Fitter prepared. Device is {self.device}')\n\n    def fit(self, train_loader, validation_loader):\n        for e in range(self.config.n_epochs):\n            if self.config.verbose:\n                lr = self.optimizer.param_groups[0]['lr'] # learning rate\n                timestamp = datetime.utcnow().isoformat() # 顯示時間\n                self.log(f'\\n{timestamp}\\nLR: {lr}')\n\n            t = time.time()\n            para_loader = pl.ParallelLoader(train_loader, [self.device]) # train_loader 的宣告\n            # \n            losses, final_scores = self.train_one_epoch(para_loader.per_device_loader(self.device)) # 訓練(得到 losses)\n            \n            self.log(f'[RESULT]: Train. Epoch: {self.epoch}, loss: {losses.avg:.5f}, final_score: {final_scores.avg:.5f}, time: {(time.time() - t):.5f}')\n\n            t = time.time()\n            para_loader = pl.ParallelLoader(validation_loader, [self.device]) # validation_loader 的宣告\n            losses, final_scores = self.validation(para_loader.per_device_loader(self.device)) # 驗證(得到 losses)\n\n            self.log(f'[RESULT]: Validation. Epoch: {self.epoch}, loss: {losses.avg:.5f}, final_score: {final_scores.avg:.5f}, time: {(time.time() - t):.5f}')\n\n            if self.config.validation_scheduler:\n                self.scheduler.step(metrics=final_scores.avg)\n\n            self.epoch += 1\n            \n    # 經資料擴增的 inference 方式\n    def run_tuning_and_inference(self, test_loader, validation_tune_loader):\n        for e in range(2):\n            self.optimizer.param_groups[0]['lr'] = self.config.lr*xm.xrt_world_size() / (e + 1)\n            para_loader = pl.ParallelLoader(validation_tune_loader, [self.device])\n            losses, final_scores = self.train_one_epoch(para_loader.per_device_loader(self.device))\n            para_loader = pl.ParallelLoader(test_loader, [self.device])\n            self.run_inference(para_loader.per_device_loader(self.device))\n            \n    # 跟(下方) train_one_epoch 類似，就是寫怎麼做驗證\n    def validation(self, val_loader): \n        self.model.eval()\n        losses = AverageMeter()\n        final_scores = RocAucMeter()\n\n        t = time.time()\n        for step, (targets, inputs, attention_masks) in enumerate(val_loader):\n            if self.config.verbose:\n                if step % self.config.verbose_step == 0:\n                    xm.master_print(\n                        f'Valid Step {step}, loss: ' + \\\n                        f'{losses.avg:.5f}, final_score: {final_scores.avg:.5f}, ' + \\\n                        f'time: {(time.time() - t):.5f}'\n                    )\n            with torch.no_grad():\n                inputs = inputs.to(self.device, dtype=torch.long) \n                attention_masks = attention_masks.to(self.device, dtype=torch.long) \n                targets = targets.to(self.device, dtype=torch.float) \n\n                outputs = self.model(inputs, attention_masks)\n                loss = self.criterion(outputs, targets)\n                \n                batch_size = inputs.size(0)\n\n                final_scores.update(targets, outputs)\n                losses.update(loss.detach().item(), batch_size)\n                \n        return losses, final_scores\n    \n    def train_one_epoch(self, train_loader): # 訓練一個 epoch\n        self.model.train()\n        # 計算 loss\n        losses = AverageMeter() \n        final_scores = RocAucMeter() \n        t = time.time()\n        for step, (targets, inputs, attention_masks) in enumerate(train_loader): # 分批執行\n            if self.config.verbose:\n                if step % self.config.verbose_step == 0:\n                    self.log( # 顯示訓練的過程\n                        f'Train Step {step}, loss: ' + \\\n                        f'{losses.avg:.5f}, final_score: {final_scores.avg:.5f}, ' + \\\n                        f'time: {(time.time() - t):.5f}'\n                    )\n            # (網路)訓練過程開始\n            inputs = inputs.to(self.device, dtype=torch.long) # 放入 device\n            attention_masks = attention_masks.to(self.device, dtype=torch.long) # 放入 device\n            targets = targets.to(self.device, dtype=torch.float) # 放入 device\n\n            self.optimizer.zero_grad() # 把梯度(gradient)歸 0\n\n            outputs = self.model(inputs, attention_masks) # 放入網路得到單次 epoch 結果\n            loss = self.criterion(outputs, targets) # 計算 loss\n\n            batch_size = inputs.size(0)\n            \n            final_scores.update(targets, outputs)\n            \n            losses.update(loss.detach().item(), batch_size) # 更新losses\n            \n            # 計算 backward\n            loss.backward() \n            xm.optimizer_step(self.optimizer)\n\n            if self.config.step_scheduler:\n                self.scheduler.step()\n            # (網路)訓練過程結束\n        self.model.eval()\n        self.save('last-checkpoint.bin') # 儲存網路\n        return losses, final_scores\n\n    def run_inference(self, test_loader): # 輸出預測結果(id和toxic)\n        self.model.eval()\n        result = {'id': [], 'toxic': []}\n        t = time.time()\n        for step, (ids, inputs, attention_masks) in enumerate(test_loader):\n            if self.config.verbose:\n                if step % self.config.verbose_step == 0:\n                    xm.master_print(f'Prediction Step {step}, time: {(time.time() - t):.5f}')\n\n            with torch.no_grad():\n                inputs = inputs.to(self.device, dtype=torch.long) \n                attention_masks = attention_masks.to(self.device, dtype=torch.long) # 得到 attention feature\n                outputs = self.model(inputs, attention_masks) # 放入模型預測\n                toxics = nn.functional.softmax(outputs, dim=1).data.cpu().numpy()[:,1] # 整理 output，只要 toxics 項\n\n            result['id'].extend(ids.cpu().numpy())\n            result['toxic'].extend(toxics)\n\n        result = pd.DataFrame(result)\n        node_count = len(glob('node_submissions/*.csv'))\n        result.to_csv(f'node_submissions/submission_{node_count}_{datetime.utcnow().microsecond}_{random.random()}.csv', index=False)\n\n    def save(self, path):        \n        xm.save(self.model.state_dict(), path) # 模型保存\n\n\n    def log(self, message):\n        if self.config.verbose:\n            xm.master_print(message)\n        with open(self.log_path, 'a+') as logger:\n            xm.master_print(f'{message}', logger)","metadata":{"execution":{"iopub.status.busy":"2021-07-25T17:42:07.325565Z","iopub.execute_input":"2021-07-25T17:42:07.326083Z","iopub.status.idle":"2021-07-25T17:42:32.266377Z","shell.execute_reply.started":"2021-07-25T17:42:07.326021Z","shell.execute_reply":"2021-07-25T17:42:32.263995Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Model","metadata":{}},{"cell_type":"code","source":"from transformers import XLMRobertaModel\n# 搭建訓練的網路\nclass ToxicSimpleNNModel(nn.Module): \n    # 宣告有甚麼層\n    def __init__(self):\n        super(ToxicSimpleNNModel, self).__init__()\n        self.backbone = XLMRobertaModel.from_pretrained(BACKBONE_PATH) # 主要的backbone\n        self.dropout = nn.Dropout(0.3)\n        self.linear = nn.Linear(\n            in_features=self.backbone.pooler.dense.out_features*2,\n            out_features=2, # 記得要和答案的大小一樣\n        )\n    # 完整的網路描述\n    def forward(self, input_ids, attention_masks):\n        bs, seq_length = input_ids.shape\n        seq_x, _ = self.backbone(input_ids=input_ids, attention_mask=attention_masks)\n        apool = torch.mean(seq_x, 1) # average pool\n        mpool, _ = torch.max(seq_x, 1) # max pool\n        x = torch.cat((apool, mpool), 1) # 將 A_pool, Max_pool 的結果拼(concatnate)在一起\n        x = self.dropout(x)\n        return self.linear(x) # 最後轉 linear 才能語答案(one-hot encoding)比較","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"net = ToxicSimpleNNModel() # 宣告網路的物件","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Custom Config","metadata":{}},{"cell_type":"code","source":"class TrainGlobalConfig: # 訓練的參數設定\n    num_workers = 0 \n    batch_size = 16 \n    n_epochs = 3\n    lr = 0.5 * 1e-5\n\n    # -------------------\n    verbose = True\n    verbose_step = 50 # 每幾步顯示訓練過程\n    # -------------------\n\n    # --------------------\n    step_scheduler = False  # 在 optimizer.step 之後做 scheduler.step\n    validation_scheduler = True  # 在驗證階段做 scheduler.step \n    SchedulerClass = torch.optim.lr_scheduler.ReduceLROnPlateau\n    scheduler_params = dict(\n        mode='max',\n        factor=0.7,\n        patience=0,\n        verbose=False, \n        threshold=0.0001,\n        threshold_mode='abs',\n        cooldown=0, \n        min_lr=1e-8,\n        eps=1e-08\n    )\n    # --------------------\n\n    # -------------------\n    criterion = LabelSmoothing()\n    # -------------------","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Main method\n\n就是先宣告 sampler，再宣告 loader\n\n分 train, validation(original和tuned), test\n\n最後在宣告有關 TPU 的設定 + 放入模型","metadata":{}},{"cell_type":"code","source":"def _mp_fn(rank, flags):\n    device = xm.xla_device()\n    net.to(device) # 模型初始化\n    \n    '''\n    可參考 https://catalyst-team.github.io/catalyst/api/data.html#catalyst.data.sampler.BalanceClassSampler\n    DistributedSamplerWrapper: 用於分佈式訓練\n    BalanceClassSampler: 在 unbalanced 的情況產生安全的 sample\n                         mode 須為 downsampling 或 upsampling\n    '''\n    train_sampler = DistributedSamplerWrapper( # 宣告分布式訓練的採樣器(sampler)\n        sampler=BalanceClassSampler(labels=train_dataset.get_labels(), mode=\"downsampling\"),\n        num_replicas=xm.xrt_world_size(), # 參與分佈式訓練的 process 數\n        rank=xm.get_ordinal(), # 當前 process 在 num_replicas 中的排名\n        shuffle=True \n    )\n    # 資料載入，可參考 https://zhuanlan.zhihu.com/p/117270644\n    train_loader = torch.utils.data.DataLoader( \n        train_dataset,\n        batch_size=TrainGlobalConfig.batch_size,\n        sampler=train_sampler, # 呼叫宣告好的 sampler\n        pin_memory=False, # 選擇是否要將load進來的數據複製到pin_memory區中\n        drop_last=True, # 選擇是否要丟棄最後一個不完整的batch\n        num_workers=TrainGlobalConfig.num_workers,\n    )\n    # validation\n    validation_sampler = torch.utils.data.distributed.DistributedSampler(\n        validation_dataset,\n        num_replicas=xm.xrt_world_size(),\n        rank=xm.get_ordinal(),\n        shuffle=False\n    )\n    validation_loader = torch.utils.data.DataLoader(\n        validation_dataset,\n        batch_size=TrainGlobalConfig.batch_size,\n        sampler=validation_sampler,\n        pin_memory=False,\n        drop_last=False,\n        num_workers=TrainGlobalConfig.num_workers\n    )\n    # validation - 有做資料擴增\n    validation_tune_sampler = torch.utils.data.distributed.DistributedSampler(\n        validation_tune_dataset,\n        num_replicas=xm.xrt_world_size(),\n        rank=xm.get_ordinal(),\n        shuffle=True\n    )\n    validation_tune_loader = torch.utils.data.DataLoader(\n        validation_tune_dataset,\n        batch_size=TrainGlobalConfig.batch_size,\n        sampler=validation_tune_sampler,\n        pin_memory=False,\n        drop_last=False,\n        num_workers=TrainGlobalConfig.num_workers\n    )\n    # test\n    test_sampler = torch.utils.data.distributed.DistributedSampler(\n        test_dataset,\n        num_replicas=xm.xrt_world_size(),\n        rank=xm.get_ordinal(),\n        shuffle=False\n    )\n    test_loader = torch.utils.data.DataLoader(\n        test_dataset,\n        batch_size=TrainGlobalConfig.batch_size,\n        sampler=test_sampler,\n        pin_memory=False,\n        drop_last=False,\n        num_workers=TrainGlobalConfig.num_workers\n    )\n\n    if rank == 0:\n        time.sleep(1)\n    \n    # 訓練\n    fitter = TPUFitter(model=net, device=device, config=TrainGlobalConfig) # TPU 宣告\n    fitter.fit(train_loader, validation_loader) \n    fitter.run_tuning_and_inference(test_loader, validation_tune_loader) # training","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Colab Notebook\n\nI hope Kaggle Team will increase RAM memory for tpu notebook as soon as possible. But now I recommend you use colab pro with HIGH RAM mode :)\n\n[Here](https://drive.google.com/drive/folders/1hbcSRfvtTTlERs7remsRST2amIWAFVry?usp=sharing) I have created public read-only google drive with colab notebook! You can save copy and start training right now!\n\nAlso you can run this code here with nprocs=1, if you need. It works! But it is very slow (~1.5 P100).","metadata":{}},{"cell_type":"code","source":"# FLAGS={}\n# 設定在 XLA 設備上要執行的工作\n# xmp.spawn(_mp_fn, args=(FLAGS,), nprocs=8, start_method='fork') \n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# 將所有csv (.csv') 依 id 集合變成一個 csv\n# submission = pd.concat([pd.read_csv(path) for path in glob('node_submissions/*.csv')]).groupby('id').mean()\n# submission['toxic'].hist(bins=100)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's imagine that this logs have got using Kaggle:","metadata":{}},{"cell_type":"code","source":"file = open('../input/jigsaw-public-baseline-results/log.txt', 'r')\nfor line in file.readlines():\n    print(line[:-1])\nfile.close()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Submission\n\n","metadata":{}},{"cell_type":"code","source":"# 輸出成比賽需要的格式\nsubmission = pd.read_csv('../input/jigsaw-public-baseline-results/submission.csv', index_col='id')\nsubmission.hist(bins=100)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission.to_csv('submission.csv')","metadata":{"trusted":true},"execution_count":null,"outputs":[]}]}