{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Token Classification Approach\n\nI looked at some of the most popular notebooks, and it looks like they are all approaching the task as text classification and using the following format as input: \n\n```discourse_type + “ “ + discourse_text + sep + essay_text```\n\nIt looks like this approach is working well, but it means that each discourse_text requires one forward pass through the model. I thought of a different way that only needs one forward pass *per essay*. Thus the number of forward passes, without considering batching, drops from 36k to 4k. \n\nIn essence, I treat it as a token classification problem instead of text classification. I insert in CLS tokens before each discourse text within the essay text and then I use the output from the classification head only at those CLS tokens to calculate class scores. \n\nTo prepare the input, I add CLS tokens to the essay at the beginning of each discourse text. I also decided to add END tokens so that the model understands where the discourse text ends. I decided to make one CLS/END token for each discourse type.\n\n### Example essay text\n> Driverless cars are exaclty what you would expect them to be. Cars that will drive without a person actually behind the wheel controlling the actions of the vehicle. The idea of driverless cars going in to developement shows the amount of technological increase that the wolrd has made. The leader of this idea of driverless cars are the automobiles they call Google cars. \n>\n> With all these things that could possibly go wrong with these driverless cars there is no way that the developement of them should continue any further. In today's society if something bad COULD happen or something COULD go wrong, it WILL happen, and it WILL go wrong. There are just way too many safety hazards that come along with these driverless cars. Becuase of all of these problems that arise with the cars it is just a gargantuan risk to implement these cars into our lifestyles. \n\n\n### Discourse texts\n> LEAD (ADEQUATE): Driverless cars are exaclty what you would expect them to be. Cars that will drive without a person actually behind the wheel controlling the actions of the vehicle. The idea of driverless cars going in to developement shows the amount of technological increase that the wolrd has made. The leader of this idea of driverless cars are the automobiles they call Google cars.\n>\n> CONCLUDING STATEMENT (INEFFECTIVE): With all these things that could possibly go wrong with these driverless cars there is no way that the developement of them should continue any further. In today's society if something bad COULD happen or something COULD go wrong, it WILL happen, and it WILL go wrong. There are just way too many safety hazards that come along with these driverless cars. Becuase of all of these problems that arise with the cars it is just a gargantuan risk to implement these cars into our lifestyles.  \n\n\n### Essay text with CLS/END tokens inserted\n\n> \\<s\\>[CLS_LEAD] Driverless cars are exaclty what you would expect them to be. Cars that will drive without a person actually behind the wheel controlling the actions of the vehicle. The idea of driverless cars going in to developement shows the amount of technological increase that the wolrd has made. The leader of this idea of driverless cars are the automobiles they call Google cars. The arduous task of creating safe driverless cars has not been fully mastered yet. [END_LEAD]   \n>\n> [CLS_CONCLUDING STATEMENT] With all these things that could possibly go wrong with these driverless cars there is no way that the developement of them should continue any further. In today's society if something bad COULD happen or something COULD go wrong, it WILL happen, and it WILL go wrong. There are just way too many safety hazards that come along with these driverless cars. Becuase of all of these problems that arise with the cars it is just a gargantuan risk to implement these cars into our lifestyles. [END_CONCLUDING STATEMENT]\\</s\\>\n\n\nThe next step involves putting -100 labels for each token except for the CLS tokens so the loss is calculated only for CLS tokens. This is all illustrated in the image below:\n\n![token classification diagram](https://raw.githubusercontent.com/nbroad1881/kaggle-images/main/feedback-prize-effectiveness/token-classification-diagram.png)\n\n\n# Notes\n- Adding CLS and END tokens means a search must be done to find the start and end indexes of the discourse_text in the essay.  Searching for the discourse_text sometimes results in no matches. I noticed this mainly happens due to different whitespace at the end. When stripping the whitespace from discourse_text and searching, there is only one instance of the discourse_text not being in the essay. In this case, the discourse_text contains different words compared to the associated essay, as noticed in [this post here](https://www.kaggle.com/competitions/feedback-prize-effectiveness/discussion/331544).\n    - Sometimes the discourse_text is in multiple locations in the essay. To deal with this, I assume that the order of rows in train.csv is from the beginning of the essay to the end. If I search one at a time, starting at the beginning, I’ll take the first occurrence of the span. I can keep track of the end index of the last discourse_text that I searched for, so the next span must start after the previous one. \n    - I added a check that confirms that only 1 discourse_text was not matched correctly.\n- The CV does not use grouped or stratified splits. \n- I used longformer base in my notebook and it takes about 10 minutes per epoch using 80% of the data for training.\n\nI don’t know if this can beat the standard approach, but maybe it will be good enough and fast enough to be competitive for the efficiency prize. Or maybe it will just add some diversity to your ensemble.","metadata":{}},{"cell_type":"markdown","source":"# Config","metadata":{}},{"cell_type":"code","source":"cfg = {\n    \"num_proc\": 2,\n    # data\n    \"k_folds\": 5,\n    \"max_length\": 2048,\n    \"padding\": False,\n    \"stride\": 0,\n    \"data_dir\": \"../input/feedback-prize-effectiveness\",\n    \"load_from_disk\": None, # if you already tokenized, you can load it through this\n    \"pad_multiple\": 512,\n    # model\n    \"model_name_or_path\": \"allenai/longformer-base-4096\",\n    \"dropout\": 0.1,\n    # to put in TrainingArguments\n    \"trainingargs\": {\n        \"output_dir\": \"output\",\n        \"do_train\": True,\n        \"do_eval\": True,\n        \"per_device_train_batch_size\": 2,\n        \"per_device_eval_batch_size\": 4,\n        \"learning_rate\": 9e-6,\n        \"weight_decay\": 0.01,\n        \"num_train_epochs\": 3,\n        \"warmup_ratio\": 0.1,\n        \"optim\": 'adamw_torch',\n        \"logging_steps\": 50,\n        \"save_strategy\": \"epoch\",\n        \"evaluation_strategy\": \"epoch\",\n        \"report_to\": \"none\",\n        \"group_by_length\": True,\n        \"save_total_limit\": 1,\n        \"metric_for_best_model\": \"loss\",\n        \"greater_is_better\": False,\n        \"seed\": 18,\n        # you should probably set \"fp16\" to True, but it doesn't really matter on Kaggle\n    }\n}","metadata":{"execution":{"iopub.status.busy":"2022-07-19T15:41:05.921291Z","iopub.execute_input":"2022-07-19T15:41:05.921689Z","iopub.status.idle":"2022-07-19T15:41:05.932658Z","shell.execute_reply.started":"2022-07-19T15:41:05.921657Z","shell.execute_reply":"2022-07-19T15:41:05.931096Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Functions for reading text files and correcting encoding errors\n\n\nCorrection code from: https://www.kaggle.com/competitions/feedback-prize-2021/discussion/313330","metadata":{}},{"cell_type":"code","source":"import re\nimport pickle\nimport codecs\nimport warnings\nimport logging\nfrom functools import partial\nfrom pathlib import Path\nfrom itertools import chain\nfrom text_unidecode import unidecode\nfrom typing import Any, Optional, Tuple\n\nimport pandas as pd\nfrom sklearn.model_selection import KFold\nfrom transformers import AutoTokenizer, set_seed\n\nfrom datasets import Dataset, load_from_disk\n\n# https://www.kaggle.com/competitions/feedback-prize-2021/discussion/313330\ndef replace_encoding_with_utf8(error: UnicodeError) -> Tuple[bytes, int]:\n    return error.object[error.start : error.end].encode(\"utf-8\"), error.end\n\n\ndef replace_decoding_with_cp1252(error: UnicodeError) -> Tuple[str, int]:\n    return error.object[error.start : error.end].decode(\"cp1252\"), error.end\n\n\n# Register the encoding and decoding error handlers for `utf-8` and `cp1252`.\ncodecs.register_error(\"replace_encoding_with_utf8\", replace_encoding_with_utf8)\ncodecs.register_error(\"replace_decoding_with_cp1252\", replace_decoding_with_cp1252)\n\n\ndef resolve_encodings_and_normalize(text: str) -> str:\n    \"\"\"Resolve the encoding problems and normalize the abnormal characters.\"\"\"\n    text = (\n        text.encode(\"raw_unicode_escape\")\n        .decode(\"utf-8\", errors=\"replace_decoding_with_cp1252\")\n        .encode(\"cp1252\", errors=\"replace_encoding_with_utf8\")\n        .decode(\"utf-8\", errors=\"replace_decoding_with_cp1252\")\n    )\n    text = unidecode(text)\n    return text\n\n\ndef read_text_files(example, data_dir):\n\n    id_ = example[\"essay_id\"]\n\n    with open(data_dir / \"train\" / f\"{id_}.txt\", \"r\") as fp:\n        example[\"text\"] = resolve_encodings_and_normalize(fp.read())\n\n    return example\n\nset_seed(cfg[\"trainingargs\"][\"seed\"])\n\n# change logging to not be bombarded by messages\n# if you are debugging, the messages will likely be helpful\nwarnings.simplefilter('ignore')\nlogging.disable(logging.WARNING)","metadata":{"execution":{"iopub.status.busy":"2022-07-19T15:41:05.936173Z","iopub.execute_input":"2022-07-19T15:41:05.937123Z","iopub.status.idle":"2022-07-19T15:41:15.842864Z","shell.execute_reply.started":"2022-07-19T15:41:05.937076Z","shell.execute_reply":"2022-07-19T15:41:15.841609Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Adding essay texts to dataset and adding special tokens to tokenizer","metadata":{}},{"cell_type":"code","source":"data_dir = Path(cfg[\"data_dir\"])\n\nif cfg[\"load_from_disk\"]:\n    if not cfg[\"load_from_disk\"].endswith(\".dataset\"):\n        cfg[\"load_from_disk\"] += \".dataset\"\n    ds = load_from_disk(cfg['load_from_disk'])\n    \n    pkl_file = f\"{cfg['load_from_disk'][:-len('.dataset')]}_pkl\"\n    with open(pkl_file, \"rb\") as fp:\n        grouped = pickle.load(fp)\n    \n    print(\"Loading from saved files\")\nelse:\n    train_df = pd.read_csv(data_dir / \"train.csv\")\n\n    text_ds = Dataset.from_dict({\"essay_id\": train_df.essay_id.unique()})\n\n    text_ds = text_ds.map(\n        partial(read_text_files, data_dir=data_dir),\n        num_proc=cfg[\"num_proc\"],\n        batched=False,\n        desc=\"Loading text files\",\n    )\n\n    text_df = text_ds.to_pandas()\n\n    train_df[\"discourse_text\"] = [\n        resolve_encodings_and_normalize(x) for x in train_df[\"discourse_text\"]\n    ]\n\n    train_df = train_df.merge(text_df, on=\"essay_id\", how=\"left\")\n\ndisc_types = [\n    \"Claim\",\n    \"Concluding Statement\",\n    \"Counterclaim\",\n    \"Evidence\",\n    \"Lead\",\n    \"Position\",\n    \"Rebuttal\",\n]\ncls_tokens_map = {label: f\"[CLS_{label.upper()}]\" for label in disc_types}\nend_tokens_map = {label: f\"[END_{label.upper()}]\" for label in disc_types}\n\nlabel2id = {\n    \"Adequate\": 0,\n    \"Effective\": 1,\n    \"Ineffective\": 2,\n}\n\ntokenizer = AutoTokenizer.from_pretrained(cfg[\"model_name_or_path\"])\ntokenizer.add_special_tokens(\n    {\"additional_special_tokens\": list(cls_tokens_map.values())+list(end_tokens_map.values())}\n)\ncls_id_map = {\n    label: tokenizer.encode(tkn)[1]\n    for label, tkn in cls_tokens_map.items()\n}","metadata":{"execution":{"iopub.status.busy":"2022-07-19T15:41:15.846057Z","iopub.execute_input":"2022-07-19T15:41:15.846456Z","iopub.status.idle":"2022-07-19T15:41:39.825847Z","shell.execute_reply.started":"2022-07-19T15:41:15.846407Z","shell.execute_reply":"2022-07-19T15:41:39.824497Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Functions for locating discourse_texts in essay_text and tokenizing","metadata":{}},{"cell_type":"code","source":"def find_positions(example):\n\n    text = example[\"text\"][0]\n    \n    # keeps track of what has already\n    # been located\n    min_idx = 0\n    \n    # stores start and end indexes of discourse_texts\n    idxs = []\n    \n    for dt in example[\"discourse_text\"]:\n        # calling strip is essential\n        matches = list(re.finditer(re.escape(dt.strip()), text))\n        \n        # If there are multiple matches, take the first one\n        # that is past the previous discourse texts.\n        if len(matches) > 1:\n            for m in matches:\n                if m.start() >= min_idx:\n                    break\n        # If no matches are found\n        elif len(matches) == 0:\n            idxs.append([-1]) # will filter out later\n            continue  \n        # If one match is found\n        else:\n            m = matches[0]\n            \n        idxs.append([m.start(), m.end()])\n\n        min_idx = m.start()\n\n    return idxs\n\ndef tokenize(example):\n    example[\"idxs\"] = find_positions(example)\n\n    text = example[\"text\"][0]\n    chunks = []\n    labels = []\n    prev = 0\n\n    zipped = zip(\n        example[\"idxs\"],\n        example[\"discourse_type\"],\n        example[\"discourse_effectiveness\"],\n    )\n    for idxs, disc_type, disc_effect in zipped:\n        # when the discourse_text wasn't found\n        if idxs == [-1]:\n            continue\n\n        s, e = idxs\n\n        # if the start of the current discourse_text is not \n        # at the end of the previous one.\n        # (text in between discourse_texts)\n        if s != prev:\n            chunks.append(text[prev:s])\n            prev = s\n\n        # if the start of the current discourse_text is \n        # the same as the end of the previous discourse_text\n        if s == prev:\n            chunks.append(cls_tokens_map[disc_type])\n            chunks.append(text[s:e])\n            chunks.append(end_tokens_map[disc_type])\n        \n        prev = e\n\n        labels.append(label2id[disc_effect])\n\n    tokenized = tokenizer(\n        \" \".join(chunks),\n        padding=False,\n        truncation=True,\n        max_length=cfg[\"max_length\"],\n        add_special_tokens=True,\n    )\n    \n    # at this point, labels is not the same shape as input_ids.\n    # The following loop will add -100 so that the loss function\n    # ignores all tokens except CLS tokens\n\n    # idx for labels list\n    idx = 0\n    final_labels = []\n    for id_ in tokenized[\"input_ids\"]:\n        # if this id belongs to a CLS token\n        if id_ in cls_id_map.values():\n            final_labels.append(labels[idx])\n            idx += 1\n        else:\n            # -100 will be ignored by loss function\n            final_labels.append(-100)\n    \n    tokenized[\"labels\"] = final_labels\n\n    return tokenized","metadata":{"execution":{"iopub.status.busy":"2022-07-19T15:41:39.830938Z","iopub.execute_input":"2022-07-19T15:41:39.831227Z","iopub.status.idle":"2022-07-19T15:41:39.851513Z","shell.execute_reply.started":"2022-07-19T15:41:39.831192Z","shell.execute_reply":"2022-07-19T15:41:39.850110Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# I frequently restart my notebook, so to reduce time\n# you can set this to just load the tokenized dataset from disk.\n# It gets loaded in the 3rd code cell, but a check is done here\n# to skip tokenizing\nif cfg[\"load_from_disk\"] is None:\n\n    # make lists of discourse_text, discourse_effectiveness\n    # for each essay\n    grouped = train_df.groupby([\"essay_id\"]).agg(list)\n\n    ds = Dataset.from_pandas(grouped)\n\n    ds = ds.map(\n        tokenize,\n        batched=False,\n        num_proc=cfg[\"num_proc\"],\n        desc=\"Tokenizing\",\n    )\n\n    save_dir = f\"{cfg['trainingargs']['output_dir']}\"\n    ds.save_to_disk(f\"{save_dir}.dataset\")\n    with open(f\"{save_dir}_pkl\", \"wb\") as fp:\n        pickle.dump(grouped, fp)\n    print(\"Saving dataset to disk:\", cfg['trainingargs']['output_dir'])\n    \n\n\n# basic kfold \ndef get_folds(df, k_folds=5):\n\n    kf = KFold(n_splits=k_folds)\n    return [\n        val_idx\n        for _, val_idx in kf.split(df)\n    ]\n\nfold_idxs = get_folds(ds[\"labels\"], cfg[\"k_folds\"])","metadata":{"execution":{"iopub.status.busy":"2022-07-19T15:41:39.854034Z","iopub.execute_input":"2022-07-19T15:41:39.854408Z","iopub.status.idle":"2022-07-19T15:42:32.997826Z","shell.execute_reply.started":"2022-07-19T15:41:39.854359Z","shell.execute_reply":"2022-07-19T15:42:32.996473Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Check data to make sure all discourse texts are represented as CLS tokens and labels\n\n1 fails, but that seems like an error by whomever made the feedback prize dataset. The discourse_text ends with the word florida whereas the essay text ends in LOCATION_NAME. I didn't do anything to fix this, but with more sophisticated matching, you should be able to catch instances like this. Hopefully the hidden set doesn't have any, but it wouldn't be too hard to probe for this.","metadata":{}},{"cell_type":"code","source":"bad_matches = []\ncls_ids = set(list(cls_id_map.values()))\nfor id_, l, ids, dt in zip(ds[\"essay_id\"], ds[\"labels\"], ds[\"input_ids\"], grouped.discourse_text):\n    \n    # count number of labels (ignoring -100)\n    num_cls_label = sum([x!=-100 for x in l])\n    # count number of cls ids\n    num_cls_id = sum([x in cls_ids for x in ids])\n    # true number of discourse_texts\n    num_dt = len(dt)\n    \n    if num_cls_label != num_dt or num_cls_id != num_dt:\n        bad_matches.append((id_, l, ids, dt))\n        \nprint(\"Num bad matches\", len(bad_matches))\ntemp = train_df[train_df[\"essay_id\"]==bad_matches[0][0]]\ntemp_txt = temp.text.values[0]\nprint(temp_txt)\nprint(\"*\"*100)\nprint([x for x in temp.discourse_text if x.strip() not in temp_txt])","metadata":{"execution":{"iopub.status.busy":"2022-07-19T15:42:32.999738Z","iopub.execute_input":"2022-07-19T15:42:33.000380Z","iopub.status.idle":"2022-07-19T15:42:35.943968Z","shell.execute_reply.started":"2022-07-19T15:42:33.000318Z","shell.execute_reply":"2022-07-19T15:42:35.942519Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Compare discourse texts, essay with CLS/END tokens injected, and bare essay text\n\nThis is one final visual check to make sure it looks alright.","metadata":{}},{"cell_type":"code","source":"for t in ds[0][\"discourse_text\"]:\n    print(t, \"\\n\")\nprint(\"*\"*100)\nprint(tokenizer.decode(ds[0][\"input_ids\"]))\nprint(\"*\"*100)\nprint(ds[0][\"text\"][0])","metadata":{"execution":{"iopub.status.busy":"2022-07-19T15:42:35.945642Z","iopub.execute_input":"2022-07-19T15:42:35.945972Z","iopub.status.idle":"2022-07-19T15:42:37.468824Z","shell.execute_reply.started":"2022-07-19T15:42:35.945927Z","shell.execute_reply":"2022-07-19T15:42:37.467477Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Training","metadata":{}},{"cell_type":"code","source":"import gc\nimport torch\nfrom transformers import Trainer, TrainingArguments, AutoConfig, AutoModelForTokenClassification, DataCollatorForTokenClassification\n\nargs = TrainingArguments(**cfg[\"trainingargs\"])\n\n# If using longformer, you will want to pad to a multiple of 512\n# For most others, you'll want to pad to a multiple of 8\ncollator = DataCollatorForTokenClassification(\n    tokenizer=tokenizer, pad_to_multiple_of=cfg[\"pad_multiple\"], padding=True\n)\n\noutput = args.output_dir\nfor fold in range(cfg[\"k_folds\"]):\n    \n    args.output_dir = f\"{output}-fold{fold}\"\n    \n    model_config = AutoConfig.from_pretrained(\n            cfg[\"model_name_or_path\"],\n        )\n    model_config.update(\n        {\n            \"num_labels\": 3,\n            \"cls_tokens\": list(cls_id_map.values()),\n            \"label2id\": label2id,\n            \"id2label\": {v:k for k, v in label2id.items()},\n        }\n    )\n    \n    model = AutoModelForTokenClassification.from_pretrained(cfg[\"model_name_or_path\"], config=model_config)\n    \n    # Because tokens were added, it is important to resize the embeddings\n    model.resize_token_embeddings(len(tokenizer)) \n\n    # split dataset to train and eval\n    keep_cols = {\"input_ids\", \"attention_mask\", \"labels\"}\n    train_idxs =  list(chain(*[i for f, i in enumerate(fold_idxs) if f != fold]))\n    train_dataset = ds.select(train_idxs).remove_columns([c for c in ds.column_names if c not in keep_cols])\n    eval_dataset = ds.select(fold_idxs[fold]).remove_columns([c for c in ds.column_names if c not in keep_cols])\n\n\n    trainer = Trainer(\n        model=model,\n        args=args,\n        train_dataset=train_dataset,\n        eval_dataset=eval_dataset,\n        tokenizer=tokenizer,\n        data_collator=collator,\n    )\n    \n    trainer.train()\n    \n    del model\n    gc.collect()\n    torch.cuda.empty_cache()","metadata":{"execution":{"iopub.status.busy":"2022-07-19T15:42:37.470631Z","iopub.execute_input":"2022-07-19T15:42:37.471754Z","iopub.status.idle":"2022-07-19T18:27:59.239144Z","shell.execute_reply.started":"2022-07-19T15:42:37.471704Z","shell.execute_reply":"2022-07-19T18:27:59.230450Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Get best scores\n\nI'm sure with more tuning (and a larger model), the score can improve even more!","metadata":{}},{"cell_type":"code","source":"import json\nbest_metrics = []\n\nfor fold in range(cfg[\"k_folds\"]):\n    folder = Path(f\"{output}-fold{fold}\")\n    checkpoint = sorted(list(folder.glob(\"checkpoint*\")))[-1]\n    with open(checkpoint/\"trainer_state.json\", \"r\") as fp:\n        data = json.load(fp)\n        best_metrics.append(data[\"best_metric\"])\n    \nprint(best_metrics)\naverage = sum(best_metrics)/len(best_metrics)\naverage","metadata":{"execution":{"iopub.status.busy":"2022-07-19T18:27:59.241316Z","iopub.execute_input":"2022-07-19T18:27:59.241632Z","iopub.status.idle":"2022-07-19T18:27:59.294806Z","shell.execute_reply.started":"2022-07-19T18:27:59.241585Z","shell.execute_reply":"2022-07-19T18:27:59.293469Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}