{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## Training Pipeline by [@shonenkov](https://www.kaggle.com/shonenkov) using multi TPU on PyTorch/XLA\n\nHi everyone!\n\nMy name is Alex Shonenkov, I am researcher, in Love with NLP and DL.\n\nRecently I have published my ideas about this competition:\n\n- [[TPU-Inference] Super Fast XLMRoberta](https://www.kaggle.com/shonenkov/tpu-inference-super-fast-xlmroberta)\n- [NLP Albumentations](https://www.kaggle.com/shonenkov/nlp-albumentations)\n- [Hack with Parallel Corpus](https://www.kaggle.com/shonenkov/hack-with-parallel-corpus)\n- [Class Balance with PyTorch/XLA](https://www.kaggle.com/shonenkov/class-balance-with-pytorch-xla)\n- [open-subtitles-toxic-pseudo-labeling](https://www.kaggle.com/shonenkov/open-subtitles-toxic-pseudo-labeling)\n\nif you didn't see this kernels and datasets, I recommend to read all of them because it may help you for better understand this kernel and achieve success in competition :)","metadata":{}},{"cell_type":"markdown","source":"## MAIN IDEA\n\nI spent a lot of time for create working kernel on the kaggle, I have tried to optimize it for 16GB RAM. But I was not able to do it for distributed MULTI TPU here, because of my datasets is too big for this aims.\n\nHere I would like to demonstrate my training pipeline without running and also I would like to provide you, my firends, prepared Colab notebook with kaggle structure!\n\nSo lets start!","metadata":{}},{"cell_type":"code","source":"!curl https://raw.githubusercontent.com/pytorch/xla/master/contrib/scripts/env-setup.py -o pytorch-xla-env-setup.py > /dev/null\n!python pytorch-xla-env-setup.py --version 20200420 --apt-packages libomp5 libopenblas-dev > /dev/null\n!pip install transformers==2.5.1 > /dev/null\n!pip install pandarallel > /dev/null\n!pip install catalyst==20.4.2 > /dev/null","metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-07-28T06:44:56.245891Z","iopub.execute_input":"2021-07-28T06:44:56.246434Z","iopub.status.idle":"2021-07-28T06:46:14.270181Z","shell.execute_reply.started":"2021-07-28T06:44:56.246332Z","shell.execute_reply":"2021-07-28T06:46:14.269051Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 修改區域\n此份程式使用 pytorch 進行訓練，且善用 XLA 可以進一步使用 Kaggle 的 TPU 進行訓練，可以進一步進行加速<BR>\n此部分可以由添加的環境變數 `os.environ['XLA_USE_BF16'] = \"1\"` 了解<br>\n- bfloat16　參考資料　https://cloud.google.com/tpu/docs/bfloat16?hl=zh-tw\n　簡單來說，就是一個可以搭配 TPU 運算使用的浮點數格式\n    \n原本底下的 Code 應該由上方完成，但其安裝網址及版本有誤，或甚至過期導致無法正常安裝<br>\n經由底下的修改正常化後續使用到相關 XLA 的 Code，避免無法呼叫的錯誤。\n    \n- XLA (加速線性代數) 參考資料 https://www.tensorflow.org/xla?hl=zh-tw\n- Pytorch 之 XLA 官方 Code https://github.com/pytorch/xla","metadata":{}},{"cell_type":"code","source":"# 下載 https://raw.githubusercontent.com/pytorch/xla/master/contrib/scripts/env-setup.py ，命名為 pytorch-xla-env-setup.py\n!curl https://raw.githubusercontent.com/pytorch/xla/master/contrib/scripts/env-setup.py -o pytorch-xla-env-setup.py\n# 安裝 配有 XLA 之 Pytorch，版本為 nightly ，可理解為最新但非穩定版本\n!python pytorch-xla-env-setup.py --version \"nightly\"","metadata":{"execution":{"iopub.status.busy":"2021-07-28T06:46:14.272189Z","iopub.execute_input":"2021-07-28T06:46:14.272528Z","iopub.status.idle":"2021-07-28T06:47:06.643135Z","shell.execute_reply.started":"2021-07-28T06:46:14.272486Z","shell.execute_reply":"2021-07-28T06:47:06.641805Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\n\nimport os\nos.environ['XLA_USE_BF16'] = \"1\"\n\nfrom glob import glob\n\nimport torch\nimport torch.nn as nn\nfrom torch.utils.data import Dataset,DataLoader\nfrom torch.autograd import Variable\nfrom torch.utils.data.sampler import SequentialSampler, RandomSampler\nimport sklearn\n\nimport time\nimport random\nfrom datetime import datetime\nfrom tqdm import tqdm\ntqdm.pandas()\n\nfrom transformers import BertModel, BertTokenizer\nfrom transformers import XLMRobertaModel, XLMRobertaTokenizer\nfrom transformers import AdamW, get_linear_schedule_with_warmup, get_constant_schedule\nfrom catalyst.data.sampler import DistributedSamplerWrapper, BalanceClassSampler\n\nimport gc\nimport re\n\n# !pip install nltk > /dev/null\nimport nltk\nnltk.download('punkt')\n\nfrom nltk import sent_tokenize\n\nfrom pandarallel import pandarallel\n\npandarallel.initialize(nb_workers=4, progress_bar=False)","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-07-28T06:47:06.647794Z","iopub.execute_input":"2021-07-28T06:47:06.648154Z","iopub.status.idle":"2021-07-28T06:47:10.883641Z","shell.execute_reply.started":"2021-07-28T06:47:06.648118Z","shell.execute_reply":"2021-07-28T06:47:10.882278Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"SEED = 42\n\nMAX_LENGTH = 224\nBACKBONE_PATH = 'xlm-roberta-large'\nROOT_PATH = f'..'\n# ROOT_PATH = f'/content/drive/My Drive/jigsaw2020-kaggle-public-baseline' # for colab\n\n\ndef seed_everything(seed):\n    random.seed(seed)\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    np.random.seed(seed)\n    torch.manual_seed(seed)\n    torch.cuda.manual_seed(seed)\n    torch.backends.cudnn.deterministic = True\n    torch.backends.cudnn.benchmark = True\n\nseed_everything(SEED)","metadata":{"execution":{"iopub.status.busy":"2021-07-28T06:47:10.885424Z","iopub.execute_input":"2021-07-28T06:47:10.885784Z","iopub.status.idle":"2021-07-28T06:47:10.899149Z","shell.execute_reply.started":"2021-07-28T06:47:10.885747Z","shell.execute_reply":"2021-07-28T06:47:10.897604Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### [NLP Albumentations](https://www.kaggle.com/shonenkov/nlp-albumentations)","metadata":{}},{"cell_type":"code","source":"from nltk import sent_tokenize\nfrom random import shuffle\nimport random\nimport albumentations\nfrom albumentations.core.transforms_interface import DualTransform, BasicTransform\n\n\nLANGS = {\n    'en': 'english',\n    'it': 'italian', \n    'fr': 'french', \n    'es': 'spanish',\n    'tr': 'turkish', \n    'ru': 'russian',\n    'pt': 'portuguese'\n}\n\ndef get_sentences(text, lang='en'):\n    return sent_tokenize(text, LANGS.get(lang, 'english'))\n\ndef exclude_duplicate_sentences(text, lang='en'):\n    sentences = []\n    for sentence in get_sentences(text, lang):\n        sentence = sentence.strip()\n        if sentence not in sentences:\n            sentences.append(sentence)\n    return ' '.join(sentences)\n\ndef clean_text(text, lang='en'):\n    text = str(text)\n    text = re.sub(r'[0-9\"]', '', text)\n    text = re.sub(r'#[\\S]+\\b', '', text)\n    text = re.sub(r'@[\\S]+\\b', '', text)\n    text = re.sub(r'https?\\S+', '', text)\n    text = re.sub(r'\\s+', ' ', text)\n    text = exclude_duplicate_sentences(text, lang)\n    return text.strip()\n\n\nclass NLPTransform(BasicTransform):\n    \"\"\" Transform for nlp task.\"\"\"\n\n    @property\n    def targets(self):\n        return {\"data\": self.apply}\n    \n    def update_params(self, params, **kwargs):\n        if hasattr(self, \"interpolation\"):\n            params[\"interpolation\"] = self.interpolation\n        if hasattr(self, \"fill_value\"):\n            params[\"fill_value\"] = self.fill_value\n        return params\n\n    def get_sentences(self, text, lang='en'):\n        return sent_tokenize(text, LANGS.get(lang, 'english'))\n\nclass ShuffleSentencesTransform(NLPTransform):\n    \"\"\" Do shuffle by sentence \"\"\"\n    def __init__(self, always_apply=False, p=0.5):\n        super(ShuffleSentencesTransform, self).__init__(always_apply, p)\n\n    def apply(self, data, **params):\n        text, lang = data\n        sentences = self.get_sentences(text, lang)\n        random.shuffle(sentences)\n        return ' '.join(sentences), lang\n\nclass ExcludeDuplicateSentencesTransform(NLPTransform):\n    \"\"\" Exclude equal sentences \"\"\"\n    def __init__(self, always_apply=False, p=0.5):\n        super(ExcludeDuplicateSentencesTransform, self).__init__(always_apply, p)\n\n    def apply(self, data, **params):\n        text, lang = data\n        sentences = []\n        for sentence in self.get_sentences(text, lang):\n            sentence = sentence.strip()\n            if sentence not in sentences:\n                sentences.append(sentence)\n        return ' '.join(sentences), lang\n\nclass ExcludeNumbersTransform(NLPTransform):\n    \"\"\" exclude any numbers \"\"\"\n    def __init__(self, always_apply=False, p=0.5):\n        super(ExcludeNumbersTransform, self).__init__(always_apply, p)\n\n    def apply(self, data, **params):\n        text, lang = data\n        text = re.sub(r'[0-9]', '', text)\n        text = re.sub(r'\\s+', ' ', text)\n        return text, lang\n\nclass ExcludeHashtagsTransform(NLPTransform):\n    \"\"\" Exclude any hashtags with # \"\"\"\n    def __init__(self, always_apply=False, p=0.5):\n        super(ExcludeHashtagsTransform, self).__init__(always_apply, p)\n\n    def apply(self, data, **params):\n        text, lang = data\n        text = re.sub(r'#[\\S]+\\b', '', text)\n        text = re.sub(r'\\s+', ' ', text)\n        return text, lang\n\nclass ExcludeUsersMentionedTransform(NLPTransform):\n    \"\"\" Exclude @users \"\"\"\n    def __init__(self, always_apply=False, p=0.5):\n        super(ExcludeUsersMentionedTransform, self).__init__(always_apply, p)\n\n    def apply(self, data, **params):\n        text, lang = data\n        text = re.sub(r'@[\\S]+\\b', '', text)\n        text = re.sub(r'\\s+', ' ', text)\n        return text, lang\n\nclass ExcludeUrlsTransform(NLPTransform):\n    \"\"\" Exclude urls \"\"\"\n    def __init__(self, always_apply=False, p=0.5):\n        super(ExcludeUrlsTransform, self).__init__(always_apply, p)\n\n    def apply(self, data, **params):\n        text, lang = data\n        text = re.sub(r'https?\\S+', '', text)\n        text = re.sub(r'\\s+', ' ', text)\n        return text, lang","metadata":{"execution":{"iopub.status.busy":"2021-07-28T06:47:10.900925Z","iopub.execute_input":"2021-07-28T06:47:10.901356Z","iopub.status.idle":"2021-07-28T06:47:11.344723Z","shell.execute_reply.started":"2021-07-28T06:47:10.901308Z","shell.execute_reply":"2021-07-28T06:47:11.343495Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### [Pseudo-labeling with open-subtitles](https://www.kaggle.com/shonenkov/hack-with-parallel-corpus)\n\nMore noise with mix of languages can help. I have used [pseudo-labeled open-subtitles dataset](https://www.kaggle.com/shonenkov/open-subtitles-toxic-pseudo-labeling) for this approach. \n\nIt is some analogue for Cutmix in Computer Vision:\n\n<img src='https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2605845%2Ff29492171d83dfa6b6fcae2af414fcf8%2FCutmix_exmaple.png?generation=1579343294489994&alt=media' align=\"left\"> ","metadata":{}},{"cell_type":"code","source":"# 利用 OpenSubtitles的資料做合成，主要合成多種語言的 noise 讓模型更強\nclass SynthesicOpenSubtitlesTransform(NLPTransform):\n    def __init__(self, always_apply=False, p=0.5):\n        # 此行意指一建 class 便讀入相關變數進行資料初始化\n        super(SynthesicOpenSubtitlesTransform, self).__init__(always_apply, p)\n        # 讀入 csv\n        df = pd.read_csv(f'{ROOT_PATH}/input/open-subtitles-toxic-pseudo-labeling/open-subtitles-synthesic.csv', index_col='id')[['comment_text', 'toxic', 'lang']]\n        # 排除 nan ('comment_text' 為空者)，~df['comment_text'].isna() 代表 df['comment_text'].isna() 的反選\n        df = df[~df['comment_text'].isna()]\n        # parallel_apply 可利用多 CPU 對於 Dataframe 進行處理，此行意指針對 df 的 comment_text 的每一欄進行 clean_text 處理\n        # clean_text : 去除特殊符號及數字等影響判斷的字元\n        df['comment_text'] = df.parallel_apply(lambda x: clean_text(x['comment_text'], x['lang']), axis=1)\n        # 去除資料重複者\n        df = df.drop_duplicates(subset='comment_text')\n        # 原本 toxic 中資料可能有浮點數，若有浮點數則四捨五入且設資料型態為整數 int\n        df['toxic'] = df['toxic'].round().astype(np.int)\n        # 取出合成資料，toxic 為 1 者即 toxic，為 0 者則為非 toxic，藉此強化資料區別性\n        self.synthesic_toxic = df[df['toxic'] == 1].comment_text.values\n        self.synthesic_non_toxic = df[df['toxic'] == 0].comment_text.values\n\n        # 此兩行為整理資料，清除不需要的資料及排除掉可能占用 CPU 的垃圾資料\n        del df\n        gc.collect();\n\n    # 產生合成資料\n    def generate_synthesic_sample(self, text, toxic):\n        texts = [text]\n        # 若 toxic 為 0 者，在 non toxic 的詞彙隨機選取\n        if toxic == 0:\n            for i in range(random.randint(1,5)):\n                texts.append(random.choice(self.synthesic_non_toxic))\n        # 若 toxic 為 1 者，在其中加入一些 non toxic 的詞彙，並與 toxic 詞彙合成\n        else:\n            for i in range(random.randint(0,2)):\n                texts.append(random.choice(self.synthesic_non_toxic))\n            \n            for i in range(random.randint(1,3)):\n                texts.append(random.choice(self.synthesic_toxic))\n        # 打亂順序\n        random.shuffle(texts)\n        # 回傳詞彙中加入空格的句子\n        return ' '.join(texts)\n\n    # 定義進行資料合成的步驟，也就是呼叫產生合成資料的 Function，寫這個可以讓 dataframe 使用\n    def apply(self, data, **params):\n        text, toxic = data\n        text = self.generate_synthesic_sample(text, toxic)\n        return text, toxic","metadata":{"execution":{"iopub.status.busy":"2021-07-28T06:47:11.346322Z","iopub.execute_input":"2021-07-28T06:47:11.346669Z","iopub.status.idle":"2021-07-28T06:47:11.36127Z","shell.execute_reply.started":"2021-07-28T06:47:11.346635Z","shell.execute_reply":"2021-07-28T06:47:11.359989Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_train_transforms():\n    return albumentations.Compose([\n        ExcludeUsersMentionedTransform(p=0.95),\n        ExcludeUrlsTransform(p=0.95),\n        ExcludeNumbersTransform(p=0.95),\n        ExcludeHashtagsTransform(p=0.95),\n        ExcludeDuplicateSentencesTransform(p=0.95),\n    ], p=1.0)\n\ndef get_synthesic_transforms():\n    return SynthesicOpenSubtitlesTransform(p=0.5)\n\n\ntrain_transforms = get_train_transforms();\nsynthesic_transforms = get_synthesic_transforms()\ntokenizer = XLMRobertaTokenizer.from_pretrained(BACKBONE_PATH)\nshuffle_transforms = ShuffleSentencesTransform(always_apply=True)","metadata":{"execution":{"iopub.status.busy":"2021-07-28T06:47:11.362876Z","iopub.execute_input":"2021-07-28T06:47:11.363274Z","iopub.status.idle":"2021-07-28T06:48:07.957186Z","shell.execute_reply.started":"2021-07-28T06:47:11.363234Z","shell.execute_reply":"2021-07-28T06:48:07.956116Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def onehot(size, target):\n    vec = torch.zeros(size, dtype=torch.float32)\n    vec[target] = 1.\n    return vec\n\nclass DatasetRetriever(Dataset):\n\n    def __init__(self, labels_or_ids, comment_texts, langs, use_train_transforms=False, test=False):\n        self.test = test\n        self.labels_or_ids = labels_or_ids\n        self.comment_texts = comment_texts\n        self.langs = langs\n        self.use_train_transforms = use_train_transforms\n        \n    def get_tokens(self, text):\n        encoded = tokenizer.encode_plus(\n            text, \n            add_special_tokens=True, \n            max_length=MAX_LENGTH, \n            pad_to_max_length=True\n        )\n        return encoded['input_ids'], encoded['attention_mask']\n\n    def __len__(self):\n        return self.comment_texts.shape[0]\n\n    def __getitem__(self, idx):\n        text = self.comment_texts[idx]\n        lang = self.langs[idx]\n        if self.test is False:\n            label = self.labels_or_ids[idx]\n            target = onehot(2, label)\n\n        if self.use_train_transforms:\n            text, _ = train_transforms(data=(text, lang))['data']\n            tokens, attention_mask = self.get_tokens(str(text))\n            token_length = sum(attention_mask)\n            if token_length > 0.8*MAX_LENGTH:\n                text, _ = shuffle_transforms(data=(text, lang))['data']\n            elif token_length < 60:\n                text, _ = synthesic_transforms(data=(text, label))['data']\n            else:\n                tokens, attention_mask = torch.tensor(tokens), torch.tensor(attention_mask)\n                return target, tokens, attention_mask\n\n        tokens, attention_mask = self.get_tokens(str(text))\n        tokens, attention_mask = torch.tensor(tokens), torch.tensor(attention_mask)\n\n        if self.test is False:\n            return target, tokens, attention_mask\n        return self.labels_or_ids[idx], tokens, attention_mask\n\n    def get_labels(self):\n        return list(np.char.add(self.labels_or_ids.astype(str), self.langs))","metadata":{"execution":{"iopub.status.busy":"2021-07-28T06:48:07.96139Z","iopub.execute_input":"2021-07-28T06:48:07.96217Z","iopub.status.idle":"2021-07-28T06:48:07.979384Z","shell.execute_reply.started":"2021-07-28T06:48:07.962107Z","shell.execute_reply":"2021-07-28T06:48:07.978025Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Here I have used [this kernel](https://www.kaggle.com/shonenkov/prepare-training-data) for merging all train data ","metadata":{}},{"cell_type":"code","source":"%%time\n\ndf_train = pd.read_csv(f'{ROOT_PATH}/input/jigsaw-public-baseline-train-data/train_data.csv')\n\n\ntrain_dataset = DatasetRetriever(\n    labels_or_ids=df_train['toxic'].values, \n    comment_texts=df_train['comment_text'].values, \n    langs=df_train['lang'].values,\n    use_train_transforms=True,\n)\n\ndel df_train\ngc.collect();\n\nfor targets, tokens, attention_masks in train_dataset:\n    break\n    \nprint(targets)\nprint(tokens.shape)\nprint(attention_masks.shape)","metadata":{"execution":{"iopub.status.busy":"2021-07-28T06:48:07.981377Z","iopub.execute_input":"2021-07-28T06:48:07.9817Z","iopub.status.idle":"2021-07-28T06:48:30.041374Z","shell.execute_reply.started":"2021-07-28T06:48:07.981668Z","shell.execute_reply":"2021-07-28T06:48:30.040433Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Class Balance\n\nAfter some experiments I have decided that [class balance](https://www.kaggle.com/shonenkov/class-balance-with-pytorch-xla) in this competition is very important. Also I noticed impact if use balancing dataset by languages.\n\nHere you can see unique values for get_labels method:","metadata":{}},{"cell_type":"code","source":"np.unique(train_dataset.get_labels())","metadata":{"execution":{"iopub.status.busy":"2021-07-28T06:48:30.0427Z","iopub.execute_input":"2021-07-28T06:48:30.043208Z","iopub.status.idle":"2021-07-28T06:48:39.896402Z","shell.execute_reply.started":"2021-07-28T06:48:30.04317Z","shell.execute_reply":"2021-07-28T06:48:39.895511Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_val = pd.read_csv(f'{ROOT_PATH}/input/jigsaw-multilingual-toxic-comment-classification/validation.csv', index_col='id')\n\nvalidation_tune_dataset = DatasetRetriever(\n    labels_or_ids=df_val['toxic'].values, \n    comment_texts=df_val['comment_text'].values, \n    langs=df_val['lang'].values,\n    use_train_transforms=True,\n)\n\ndf_val['comment_text'] = df_val.parallel_apply(lambda x: clean_text(x['comment_text'], x['lang']), axis=1)\n\nvalidation_dataset = DatasetRetriever(\n    labels_or_ids=df_val['toxic'].values, \n    comment_texts=df_val['comment_text'].values, \n    langs=df_val['lang'].values,\n    use_train_transforms=False,\n)\n\ndel df_val\ngc.collect();\n\nfor targets, tokens, attention_masks in validation_dataset:\n    break\n\nprint(targets)\nprint(tokens.shape)\nprint(attention_masks.shape)","metadata":{"execution":{"iopub.status.busy":"2021-07-28T06:48:39.897667Z","iopub.execute_input":"2021-07-28T06:48:39.898164Z","iopub.status.idle":"2021-07-28T06:48:41.84699Z","shell.execute_reply.started":"2021-07-28T06:48:39.89812Z","shell.execute_reply":"2021-07-28T06:48:41.845654Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_test = pd.read_csv(f'{ROOT_PATH}/input/jigsaw-multilingual-toxic-comment-classification/test.csv', index_col='id')\ndf_test['comment_text'] = df_test.parallel_apply(lambda x: clean_text(x['content'], x['lang']), axis=1)\n\ntest_dataset = DatasetRetriever(\n    labels_or_ids=df_test.index.values, \n    comment_texts=df_test['comment_text'].values, \n    langs=df_test['lang'].values,\n    use_train_transforms=False,\n    test=True\n)\n\ndel df_test\ngc.collect();\n\nfor ids, tokens, attention_masks in test_dataset:\n    break\n\nprint(ids)\nprint(tokens.shape)\nprint(attention_masks.shape)","metadata":{"execution":{"iopub.status.busy":"2021-07-28T06:48:41.848658Z","iopub.execute_input":"2021-07-28T06:48:41.84906Z","iopub.status.idle":"2021-07-28T06:48:53.653229Z","shell.execute_reply.started":"2021-07-28T06:48:41.84902Z","shell.execute_reply":"2021-07-28T06:48:53.651858Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class RocAucMeter(object):\n    def __init__(self):\n        self.reset()\n\n    def reset(self):\n        self.y_true = np.array([0,1])\n        self.y_pred = np.array([0.5,0.5])\n        self.score = 0\n\n    def update(self, y_true, y_pred):\n        y_true = y_true.cpu().numpy().argmax(axis=1)\n        y_pred = nn.functional.softmax(y_pred, dim=1).data.cpu().numpy()[:,1]\n        self.y_true = np.hstack((self.y_true, y_true))\n        self.y_pred = np.hstack((self.y_pred, y_pred))\n        self.score = sklearn.metrics.roc_auc_score(self.y_true, self.y_pred, labels=np.array([0, 1]))\n    \n    @property\n    def avg(self):\n        return self.score\n\nclass AverageMeter(object):\n    \"\"\"Computes and stores the average and current value\"\"\"\n    def __init__(self):\n        self.reset()\n\n    def reset(self):\n        self.val = 0\n        self.avg = 0\n        self.sum = 0\n        self.count = 0\n\n    def update(self, val, n=1):\n        self.val = val\n        self.sum += val * n\n        self.count += n\n        self.avg = self.sum / self.count","metadata":{"execution":{"iopub.status.busy":"2021-07-28T06:48:53.65515Z","iopub.execute_input":"2021-07-28T06:48:53.655646Z","iopub.status.idle":"2021-07-28T06:48:53.668719Z","shell.execute_reply.started":"2021-07-28T06:48:53.655596Z","shell.execute_reply":"2021-07-28T06:48:53.667513Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Label Smoothing is all you need\nNow we can use translating and augmenting data for training with this Loss: ","metadata":{}},{"cell_type":"code","source":"class LabelSmoothing(nn.Module):\n    def __init__(self, smoothing = 0.1):\n        super(LabelSmoothing, self).__init__()\n        self.confidence = 1.0 - smoothing\n        self.smoothing = smoothing\n\n    def forward(self, x, target):\n        if self.training:\n            x = x.float()\n            target = target.float()\n            logprobs = torch.nn.functional.log_softmax(x, dim = -1)\n            nll_loss = -logprobs * target\n            nll_loss = nll_loss.sum(-1)\n            smooth_loss = -logprobs.mean(dim=-1)\n            loss = self.confidence * nll_loss + self.smoothing * smooth_loss\n            return loss.mean()\n        else:\n            return torch.nn.functional.cross_entropy(x, target)","metadata":{"execution":{"iopub.status.busy":"2021-07-28T06:48:53.670478Z","iopub.execute_input":"2021-07-28T06:48:53.671047Z","iopub.status.idle":"2021-07-28T06:48:53.696104Z","shell.execute_reply.started":"2021-07-28T06:48:53.670984Z","shell.execute_reply":"2021-07-28T06:48:53.695197Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Custom TPU Fitter\n\n<img src='https://image.made-in-china.com/202f0j10dPkYMNLhhabE/Children-Bicycle-Baby-Kids-BMX-Bike.jpg' width=250 align=\"left\"> \n\nP.S. Lets go to do contributing [Catalyst](https://github.com/catalyst-team/catalyst) with TPU backend :)\n\n<img src='https://raw.githubusercontent.com/catalyst-team/catalyst-pics/master/pics/catalyst_logo.png' width=100 align=\"center\">\n","metadata":{}},{"cell_type":"markdown","source":"# 可注意區域\n下方 code 即可看到有引入相關 `torch_xla`的部分，若沒有正常安裝則無法正確執行底下的 code 。","metadata":{}},{"cell_type":"code","source":"import warnings\n\nwarnings.filterwarnings(\"ignore\")\n\nimport torch_xla\nimport torch_xla.core.xla_model as xm\nimport torch_xla.distributed.parallel_loader as pl\nimport torch_xla.distributed.xla_multiprocessing as xmp\n\nfrom catalyst.data.sampler import DistributedSamplerWrapper, BalanceClassSampler\n\n# TPU 訓練使用，會使用到 torch_xla\nclass TPUFitter:\n    # 初始化\n    def __init__(self, model, device, config):\n        # 建立 node_submissions 資料夾\n        if not os.path.exists('node_submissions'):\n            os.makedirs('node_submissions')\n        # config 為設定資料， epoch 初始化為 0 ，log存放位置為 log.txt\n        self.config = config\n        self.epoch = 0\n        self.log_path = 'log.txt'\n        # 引入 model 及確定要使用的 device\n        self.model = model\n        self.device = device\n        # 引入 optimizer 所用到的 parameters\n        param_optimizer = list(self.model.named_parameters())\n        # 此會用於框選出不需要 decay 的 parameter\n        no_decay = ['bias', 'LayerNorm.bias', 'LayerNorm.weight']\n        # 將需要 decay 及不需要 decay 的 parameter 分組，需要 decay 者 weight_decay 為 0.001\n        optimizer_grouped_parameters = [\n            {'params': [p for n, p in param_optimizer if not any(nd in n for nd in no_decay)], 'weight_decay': 0.001},\n            {'params': [p for n, p in param_optimizer if any(nd in n for nd in no_decay)], 'weight_decay': 0.0}\n        ]\n        # Optimizer 設定為 AdamW\n        self.optimizer = AdamW(optimizer_grouped_parameters, lr=config.lr*xm.xrt_world_size())\n        # 讀入設定資料的排程工具\n        self.scheduler = config.SchedulerClass(self.optimizer, **config.scheduler_params)\n        # 讀入設定資料中的 criterion\n        self.criterion = config.criterion\n        xm.master_print(f'Fitter prepared. Device is {self.device}')\n\n    def fit(self, train_loader, validation_loader):\n        for e in range(self.config.n_epochs):\n            if self.config.verbose:\n                lr = self.optimizer.param_groups[0]['lr']\n                timestamp = datetime.utcnow().isoformat()\n                self.log(f'\\n{timestamp}\\nLR: {lr}')\n\n            t = time.time()\n            para_loader = pl.ParallelLoader(train_loader, [self.device])\n            losses, final_scores = self.train_one_epoch(para_loader.per_device_loader(self.device))\n            \n            self.log(f'[RESULT]: Train. Epoch: {self.epoch}, loss: {losses.avg:.5f}, final_score: {final_scores.avg:.5f}, time: {(time.time() - t):.5f}')\n\n            t = time.time()\n            para_loader = pl.ParallelLoader(validation_loader, [self.device])\n            losses, final_scores = self.validation(para_loader.per_device_loader(self.device))\n\n            self.log(f'[RESULT]: Validation. Epoch: {self.epoch}, loss: {losses.avg:.5f}, final_score: {final_scores.avg:.5f}, time: {(time.time() - t):.5f}')\n\n            if self.config.validation_scheduler:\n                self.scheduler.step(metrics=final_scores.avg)\n\n            self.epoch += 1\n    \n    # tuning 及推論用\n    def run_tuning_and_inference(self, test_loader, validation_tune_loader):\n        for e in range(2):\n            # 學習率下降及調整\n            self.optimizer.param_groups[0]['lr'] = self.config.lr*xm.xrt_world_size() / (e + 1)\n            # 平行化讀入 validation 資料\n            para_loader = pl.ParallelLoader(validation_tune_loader, [self.device])\n            # 訓練一個 Epoch，產出對應 loss 及分數\n            losses, final_scores = self.train_one_epoch(para_loader.per_device_loader(self.device))\n            # 平行化讀入 test 資料\n            para_loader = pl.ParallelLoader(test_loader, [self.device])\n            # 進行推論\n            self.run_inference(para_loader.per_device_loader(self.device))\n    # Validation\n    def validation(self, val_loader):\n        # 對 model 進行評估\n        self.model.eval()\n        # loss 使用 AverageMeter\n        losses = AverageMeter()\n        # 分數評估使用 RocAucMeter\n        final_scores = RocAucMeter()\n        # 計時用\n        t = time.time()\n        # 定義每一個 step 會做的事\n        for step, (targets, inputs, attention_masks) in enumerate(val_loader):\n            # 若開啟 verbose 模式，則print出目前的 step數, loss 及 score\n            if self.config.verbose:\n                if step % self.config.verbose_step == 0:\n                    xm.master_print(\n                        f'Valid Step {step}, loss: ' + \\\n                        f'{losses.avg:.5f}, final_score: {final_scores.avg:.5f}, ' + \\\n                        f'time: {(time.time() - t):.5f}'\n                    )\n            # 訓練設定，validation 不進行梯度計算\n            with torch.no_grad():\n                # 將 inputs, attention_masks 及 targets 以其對應的適合格式放到 device 上\n                inputs = inputs.to(self.device, dtype=torch.long) \n                attention_masks = attention_masks.to(self.device, dtype=torch.long) \n                targets = targets.to(self.device, dtype=torch.float) \n                \n                # 輸出結果\n                outputs = self.model(inputs, attention_masks)\n                loss = self.criterion(outputs, targets)\n                \n                batch_size = inputs.size(0)\n                # 根據輸出結果更新資料\n                final_scores.update(targets, outputs)\n                losses.update(loss.detach().item(), batch_size)\n                \n        return losses, final_scores\n    # 訓練用 function\n    def train_one_epoch(self, train_loader):\n        # 進行訓練\n        self.model.train()\n        # loss 使用 AverageMeter\n        losses = AverageMeter()\n        # 分數評估使用 RocAucMeter\n        final_scores = RocAucMeter()\n        # 計時用\n        t = time.time()\n        # 定義每一個 step 會做的事\n        for step, (targets, inputs, attention_masks) in enumerate(train_loader):\n            # 若開啟 verbose 模式，則print出目前的 step數, loss 及 score\n            if self.config.verbose:\n                if step % self.config.verbose_step == 0:\n                    self.log(\n                        f'Train Step {step}, loss: ' + \\\n                        f'{losses.avg:.5f}, final_score: {final_scores.avg:.5f}, ' + \\\n                        f'time: {(time.time() - t):.5f}'\n                    )\n            # 將 inputs, attention_masks 及 targets 以其對應的適合格式放到 device 上\n            inputs = inputs.to(self.device, dtype=torch.long)\n            attention_masks = attention_masks.to(self.device, dtype=torch.long)\n            targets = targets.to(self.device, dtype=torch.float)\n            # 梯度初始化\n            self.optimizer.zero_grad()\n            # 輸出結果\n            outputs = self.model(inputs, attention_masks)\n            loss = self.criterion(outputs, targets)\n\n            batch_size = inputs.size(0)\n            # 更新資料\n            final_scores.update(targets, outputs)\n            \n            losses.update(loss.detach().item(), batch_size)\n            # 反向傳播\n            loss.backward()\n            # 梯度下降\n            xm.optimizer_step(self.optimizer)\n\n            if self.config.step_scheduler:\n                self.scheduler.step()\n        \n        self.model.eval()\n        self.save('last-checkpoint.bin')\n        return losses, final_scores\n    \n    # inference 使用\n    def run_inference(self, test_loader):\n        # 進行 evaluate\n        self.model.eval()\n        # 初始化結果存放空間\n        result = {'id': [], 'toxic': []}\n        # 計時用\n        t = time.time()\n        for step, (ids, inputs, attention_masks) in enumerate(test_loader):\n            # 若開啟 verbose 模式，則print出目前的 step數 及 時間\n            if self.config.verbose:\n                if step % self.config.verbose_step == 0:\n                    xm.master_print(f'Prediction Step {step}, time: {(time.time() - t):.5f}')\n            # 不進行梯度計算\n            with torch.no_grad():\n                # 將 inputs, attention_masks 及 targets 以其對應的適合格式放到 device 上\n                inputs = inputs.to(self.device, dtype=torch.long) \n                attention_masks = attention_masks.to(self.device, dtype=torch.long)\n                outputs = self.model(inputs, attention_masks)\n                # 根據結果進行分類\n                toxics = nn.functional.softmax(outputs, dim=1).data.cpu().numpy()[:,1]\n            \n            # 將結果放入已初始化的空間\n            result['id'].extend(ids.cpu().numpy())\n            result['toxic'].extend(toxics)\n        # 將結果存成 DataFrame 並且輸出成 csv 供 submit 使用\n        result = pd.DataFrame(result)\n        node_count = len(glob('node_submissions/*.csv'))\n        result.to_csv(f'node_submissions/submission_{node_count}_{datetime.utcnow().microsecond}_{random.random()}.csv', index=False)\n    \n    # 儲存模型用\n    def save(self, path):        \n        xm.save(self.model.state_dict(), path)\n    # 儲存 log 用\n    def log(self, message):\n        if self.config.verbose:\n            xm.master_print(message)\n        with open(self.log_path, 'a+') as logger:\n            xm.master_print(f'{message}', logger)","metadata":{"execution":{"iopub.status.busy":"2021-07-28T06:48:53.697744Z","iopub.execute_input":"2021-07-28T06:48:53.698398Z","iopub.status.idle":"2021-07-28T06:48:54.36924Z","shell.execute_reply.started":"2021-07-28T06:48:53.698347Z","shell.execute_reply":"2021-07-28T06:48:54.367776Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Model","metadata":{}},{"cell_type":"code","source":"from transformers import XLMRobertaModel\n\nclass ToxicSimpleNNModel(nn.Module):\n\n    def __init__(self):\n        super(ToxicSimpleNNModel, self).__init__()\n        self.backbone = XLMRobertaModel.from_pretrained(BACKBONE_PATH)\n        self.dropout = nn.Dropout(0.3)\n        self.linear = nn.Linear(\n            in_features=self.backbone.pooler.dense.out_features*2,\n            out_features=2,\n        )\n\n    def forward(self, input_ids, attention_masks):\n        bs, seq_length = input_ids.shape\n        seq_x, _ = self.backbone(input_ids=input_ids, attention_mask=attention_masks)\n        apool = torch.mean(seq_x, 1)\n        mpool, _ = torch.max(seq_x, 1)\n        x = torch.cat((apool, mpool), 1)\n        x = self.dropout(x)\n        return self.linear(x)","metadata":{"execution":{"iopub.status.busy":"2021-07-28T06:48:54.371247Z","iopub.execute_input":"2021-07-28T06:48:54.371622Z","iopub.status.idle":"2021-07-28T06:48:54.38139Z","shell.execute_reply.started":"2021-07-28T06:48:54.371582Z","shell.execute_reply":"2021-07-28T06:48:54.380215Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"net = ToxicSimpleNNModel()","metadata":{"execution":{"iopub.status.busy":"2021-07-28T06:48:54.38287Z","iopub.execute_input":"2021-07-28T06:48:54.383265Z","iopub.status.idle":"2021-07-28T06:50:12.424149Z","shell.execute_reply.started":"2021-07-28T06:48:54.383229Z","shell.execute_reply":"2021-07-28T06:50:12.42311Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Custom Config","metadata":{}},{"cell_type":"code","source":"class TrainGlobalConfig:\n    num_workers = 0 \n    batch_size = 16 \n    n_epochs = 3\n    lr = 0.5 * 1e-5\n\n    # -------------------\n    verbose = True\n    verbose_step = 50\n    # -------------------\n\n    # --------------------\n    step_scheduler = False  # do scheduler.step after optimizer.step\n    validation_scheduler = True  # do scheduler.step after validation stage loss\n    SchedulerClass = torch.optim.lr_scheduler.ReduceLROnPlateau\n    scheduler_params = dict(\n        mode='max',\n        factor=0.7,\n        patience=0,\n        verbose=False, \n        threshold=0.0001,\n        threshold_mode='abs',\n        cooldown=0, \n        min_lr=1e-8,\n        eps=1e-08\n    )\n    # --------------------\n\n    # -------------------\n    criterion = LabelSmoothing()\n    # -------------------","metadata":{"execution":{"iopub.status.busy":"2021-07-28T06:50:12.4264Z","iopub.execute_input":"2021-07-28T06:50:12.426974Z","iopub.status.idle":"2021-07-28T06:50:12.436839Z","shell.execute_reply.started":"2021-07-28T06:50:12.426911Z","shell.execute_reply":"2021-07-28T06:50:12.435348Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Main method","metadata":{}},{"cell_type":"code","source":"def _mp_fn(rank, flags):\n    device = xm.xla_device()\n    net.to(device)\n\n    train_sampler = DistributedSamplerWrapper(\n        sampler=BalanceClassSampler(labels=train_dataset.get_labels(), mode=\"downsampling\"),\n        num_replicas=xm.xrt_world_size(),\n        rank=xm.get_ordinal(),\n        shuffle=True\n    )\n    train_loader = torch.utils.data.DataLoader(\n        train_dataset,\n        batch_size=TrainGlobalConfig.batch_size,\n        sampler=train_sampler,\n        pin_memory=False,\n        drop_last=True,\n        num_workers=TrainGlobalConfig.num_workers,\n    )\n    validation_sampler = torch.utils.data.distributed.DistributedSampler(\n        validation_dataset,\n        num_replicas=xm.xrt_world_size(),\n        rank=xm.get_ordinal(),\n        shuffle=False\n    )\n    validation_loader = torch.utils.data.DataLoader(\n        validation_dataset,\n        batch_size=TrainGlobalConfig.batch_size,\n        sampler=validation_sampler,\n        pin_memory=False,\n        drop_last=False,\n        num_workers=TrainGlobalConfig.num_workers\n    )\n    validation_tune_sampler = torch.utils.data.distributed.DistributedSampler(\n        validation_tune_dataset,\n        num_replicas=xm.xrt_world_size(),\n        rank=xm.get_ordinal(),\n        shuffle=True\n    )\n    validation_tune_loader = torch.utils.data.DataLoader(\n        validation_tune_dataset,\n        batch_size=TrainGlobalConfig.batch_size,\n        sampler=validation_tune_sampler,\n        pin_memory=False,\n        drop_last=False,\n        num_workers=TrainGlobalConfig.num_workers\n    )\n    test_sampler = torch.utils.data.distributed.DistributedSampler(\n        test_dataset,\n        num_replicas=xm.xrt_world_size(),\n        rank=xm.get_ordinal(),\n        shuffle=False\n    )\n    test_loader = torch.utils.data.DataLoader(\n        test_dataset,\n        batch_size=TrainGlobalConfig.batch_size,\n        sampler=test_sampler,\n        pin_memory=False,\n        drop_last=False,\n        num_workers=TrainGlobalConfig.num_workers\n    )\n\n    if rank == 0:\n        time.sleep(1)\n    \n    fitter = TPUFitter(model=net, device=device, config=TrainGlobalConfig)\n    fitter.fit(train_loader, validation_loader)\n    fitter.run_tuning_and_inference(test_loader, validation_tune_loader)","metadata":{"execution":{"iopub.status.busy":"2021-07-28T06:50:12.438613Z","iopub.execute_input":"2021-07-28T06:50:12.439023Z","iopub.status.idle":"2021-07-28T06:50:12.45969Z","shell.execute_reply.started":"2021-07-28T06:50:12.438956Z","shell.execute_reply":"2021-07-28T06:50:12.458754Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Colab Notebook\n\nI hope Kaggle Team will increase RAM memory for tpu notebook as soon as possible. But now I recommend you use colab pro with HIGH RAM mode :)\n\n[Here](https://drive.google.com/drive/folders/1hbcSRfvtTTlERs7remsRST2amIWAFVry?usp=sharing) I have created public read-only google drive with colab notebook! You can save copy and start training right now!\n\nAlso you can run this code here with nprocs=1, if you need. It works! But it is very slow (~1.5 P100).","metadata":{}},{"cell_type":"code","source":"# FLAGS={}\n# xmp.spawn(_mp_fn, args=(FLAGS,), nprocs=8, start_method='fork')","metadata":{"execution":{"iopub.status.busy":"2021-07-28T06:50:12.461359Z","iopub.execute_input":"2021-07-28T06:50:12.461712Z","iopub.status.idle":"2021-07-28T06:50:12.478754Z","shell.execute_reply.started":"2021-07-28T06:50:12.461676Z","shell.execute_reply":"2021-07-28T06:50:12.477491Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# submission = pd.concat([pd.read_csv(path) for path in glob('node_submissions/*.csv')]).groupby('id').mean()\n# submission['toxic'].hist(bins=100)","metadata":{"execution":{"iopub.status.busy":"2021-07-28T06:50:12.480683Z","iopub.execute_input":"2021-07-28T06:50:12.481141Z","iopub.status.idle":"2021-07-28T06:50:12.491893Z","shell.execute_reply.started":"2021-07-28T06:50:12.481101Z","shell.execute_reply":"2021-07-28T06:50:12.490143Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's imagine that this logs have got using Kaggle:","metadata":{}},{"cell_type":"code","source":"file = open('../input/jigsaw-public-baseline-results/log.txt', 'r')\nfor line in file.readlines():\n    print(line[:-1])\nfile.close()","metadata":{"execution":{"iopub.status.busy":"2021-07-28T06:50:12.494073Z","iopub.execute_input":"2021-07-28T06:50:12.494541Z","iopub.status.idle":"2021-07-28T06:50:12.554727Z","shell.execute_reply.started":"2021-07-28T06:50:12.494492Z","shell.execute_reply":"2021-07-28T06:50:12.548082Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This model should be trained ~10 epoch, I have run only 3 epoch for this kernel.","metadata":{}},{"cell_type":"markdown","source":"## Submission\n\nIf you want to get high score ~0.945-0.946 such as [[TPU-Inference] Super Fast XLMRoberta](https://www.kaggle.com/shonenkov/tpu-inference-super-fast-xlmroberta) you should do blend such as [here](https://www.kaggle.com/hamditarek/ensemble), but I would like to make submission with only this kernel","metadata":{}},{"cell_type":"code","source":"submission = pd.read_csv('../input/jigsaw-public-baseline-results/submission.csv', index_col='id')\nsubmission.hist(bins=100)","metadata":{"execution":{"iopub.status.busy":"2021-07-28T06:50:12.561104Z","iopub.execute_input":"2021-07-28T06:50:12.561502Z","iopub.status.idle":"2021-07-28T06:50:13.106567Z","shell.execute_reply.started":"2021-07-28T06:50:12.561466Z","shell.execute_reply":"2021-07-28T06:50:13.105714Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission.to_csv('submission.csv')","metadata":{"execution":{"iopub.status.busy":"2021-07-28T06:50:13.107692Z","iopub.execute_input":"2021-07-28T06:50:13.108023Z","iopub.status.idle":"2021-07-28T06:50:13.362435Z","shell.execute_reply.started":"2021-07-28T06:50:13.10799Z","shell.execute_reply":"2021-07-28T06:50:13.361214Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Thank you for reading my kernel!\n\n[Here](https://drive.google.com/drive/folders/1hbcSRfvtTTlERs7remsRST2amIWAFVry?usp=sharing) I have created public read-only google drive with colab notebook! You can save copy and start training right now!\n\n\nIf you like this format of notebooks I would like continue to make kernels with realizations of my ideas.","metadata":{}}]}