{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Competition/Problem Description\nWhen you visit a doctor, how they interpret your symptoms can determine whether your diagnosis is accurate. By the time they’re licensed, physicians have had a lot of practice writing patient notes that document the history of the patient’s complaint, physical exam findings, possible diagnoses, and follow-up care. Learning and assessing the skill of writing patient notes requires feedback from other doctors, a time-intensive process that could be improved with the addition of machine learning.\n\nUntil recently, the Step 2 Clinical Skills examination was one component of the United States Medical Licensing Examination® (USMLE®). The exam required test-takers to interact with Standardized Patients (people trained to portray specific clinical cases) and write a patient note. Trained physician raters later scored patient notes with rubrics that outlined each case’s important concepts (referred to as features). The more such features found in a patient note, the higher the score (among other factors that contribute to the final score for the exam).\n\nHowever, having physicians score patient note exams requires significant time, along with human and financial resources. Approaches using natural language processing have been created to address this problem, but patient notes can still be challenging to score computationally because features may be expressed in many ways. For example, the feature \"loss of interest in activities\" can be expressed as \"no longer plays tennis.\" Other challenges include the need to map concepts by combining multiple text segments, or cases of ambiguous negation such as “no cold intolerance, hair loss, palpitations, or tremor” corresponding to the key essential “lack of other thyroid symptoms.”\n\nIn this competition, you’ll identify specific clinical concepts in patient notes. Specifically, you'll develop an automated method to map clinical concepts from an exam rubric (e.g., “diminished appetite”) to various ways in which these concepts are expressed in clinical patient notes written by medical students (e.g., “eating less,” “clothes fit looser”). Great solutions will be both accurate and reliable.\n\nIf successful, you'll help tackle the biggest practical barriers in patient note scoring, making the approach more transparent, interpretable, and easing the development and administration of such assessments. As a result, medical practitioners will be able to explore the full potential of patient notes to reveal information relevant to clinical skills assessment.\n\nThis competition is sponsored by the National Board of Medical Examiners® (NBME®). Through research and innovation, NBME supports medical school and residency program educators in addressing issues around the evolution of teaching, learning, technology, and the need for meaningful feedback. NBME offers high-quality assessments and educational services for students, professionals, educators, regulators, and institutions dedicated to the evolving needs of medical education and health care. To serve these communities, NBME collaborates with a diverse and comprehensive array of practicing health professionals, medical educators, state medical board members, test developers, academic researchers, scoring experts and public representatives.\n\nNBME gratefully acknowledges the valuable input of Dr Le An Ha from the University of Wolverhampton’s Research Group in Computational Linguistics.","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-07-04T13:45:28.751283Z","iopub.execute_input":"2022-07-04T13:45:28.751714Z","iopub.status.idle":"2022-07-04T13:45:28.781800Z","shell.execute_reply.started":"2022-07-04T13:45:28.751629Z","shell.execute_reply":"2022-07-04T13:45:28.780636Z"}}},{"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# huggingface fun\nfrom datasets import Dataset\nfrom datasets import load_dataset\nfrom transformers import AutoTokenizer\n\nimport torch","metadata":{"execution":{"iopub.status.busy":"2022-07-10T17:45:38.626871Z","iopub.execute_input":"2022-07-10T17:45:38.627226Z","iopub.status.idle":"2022-07-10T17:45:38.638186Z","shell.execute_reply.started":"2022-07-10T17:45:38.627195Z","shell.execute_reply":"2022-07-10T17:45:38.637194Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"base_dir = \"/kaggle/input/nbme-score-clinical-patient-notes\"\n\ntrain = pd.read_csv(f\"{base_dir}/train.csv\")\npatient_notes = pd.read_csv(f\"{base_dir}/patient_notes.csv\")\nfeatures = pd.read_csv(f\"{base_dir}/features.csv\")\ntest = pd.read_csv(f\"{base_dir}/test.csv\")\nsample_submission = pd.read_csv(f\"{base_dir}/sample_submission.csv\")","metadata":{"execution":{"iopub.status.busy":"2022-07-10T16:23:19.774874Z","iopub.execute_input":"2022-07-10T16:23:19.778170Z","iopub.status.idle":"2022-07-10T16:23:20.789686Z","shell.execute_reply.started":"2022-07-10T16:23:19.778130Z","shell.execute_reply":"2022-07-10T16:23:20.788654Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Split the train datatset into an 80/20 split train-validation","metadata":{}},{"cell_type":"code","source":"validation_dataset = pd.DataFrame()\nfor case_num in train.case_num.unique():\n    validation_dataset = pd.concat([validation_dataset, train[train.case_num == case_num].sample(frac=0.2, random_state=24)])\n    \ntrain_dataset = train.drop(validation_dataset.index)\nprint(train_dataset.shape)\nprint(validation_dataset.shape)","metadata":{"execution":{"iopub.status.busy":"2022-07-10T16:23:23.138199Z","iopub.execute_input":"2022-07-10T16:23:23.138650Z","iopub.status.idle":"2022-07-10T16:23:23.196041Z","shell.execute_reply.started":"2022-07-10T16:23:23.138610Z","shell.execute_reply":"2022-07-10T16:23:23.195120Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Its common to have 2 sub-ner-tags per ner-tag, indicated by a prefix. \n# Typically `B-` is used to indicate the beginning of an ner tag and `I-` is used to indicate a token is included in an ner-tag\n# I'll start with just the `I-` indicator to try first our target tokens\n# For now, im just going to group the feature texts into a single list, something could be said for splitting on case_num\n# '0' Used to indicate a token doesnt have an interesting ner-tag\nner_tag_list = ['0']\nfor i, r in features.feature_text.iteritems():\n    ner_tag_list.append(\"I-\" + r)\n    \nner_tag_list[:7]","metadata":{"execution":{"iopub.status.busy":"2022-07-10T16:23:24.678678Z","iopub.execute_input":"2022-07-10T16:23:24.679090Z","iopub.status.idle":"2022-07-10T16:23:24.699380Z","shell.execute_reply.started":"2022-07-10T16:23:24.679050Z","shell.execute_reply":"2022-07-10T16:23:24.698309Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# train_dataset[train_dataset.pn_num == 61497]\ntrain_dataset.sample(8)","metadata":{"execution":{"iopub.status.busy":"2022-07-10T16:23:27.345980Z","iopub.execute_input":"2022-07-10T16:23:27.346327Z","iopub.status.idle":"2022-07-10T16:23:27.362979Z","shell.execute_reply.started":"2022-07-10T16:23:27.346298Z","shell.execute_reply":"2022-07-10T16:23:27.361597Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def tag_and_tokenize(dataset):\n    tokenized_dict = {\n        \"tokens\": [],\n        \"ner_tags\": []\n    }\n\n    size = dataset.pn_num.unique().shape[0]\n    pos = 0\n    for pn_num in dataset.pn_num.unique():\n        print(f\"Tokenized: {pn_num} ({pos+1}/{size})\", end=\"\\r\")\n        pn_history = patient_notes[patient_notes.pn_num == pn_num].squeeze().pn_history\n\n        # Naive split for now\n        pn_tokens = pn_history.split()\n        ner_tag_indices = [0 for _ in pn_tokens]\n\n        for i, r in dataset[dataset.pn_num == pn_num].iterrows():\n            location_list = eval(r.location)\n            feature = features[features.feature_num == r.feature_num].squeeze().feature_text\n            ner_tag_index = ner_tag_list.index(\"I-\"+feature)\n\n            # No feature/annotations skip\n            if len(location_list) == 0:\n                continue\n\n            for location in location_list:\n                for sub_loc in location.split(\";\"):\n                    char_token_start = eval(sub_loc.split()[0])\n                    char_token_end = eval(sub_loc.split()[1])\n\n                    # Used to find the token in the word list via space offset value\n                    space_offset = 0 if pn_history[char_token_start-1] == \" \" or char_token_start == 0 else 1\n\n                    token_start_idx = len(pn_history[0:char_token_start].split()) - space_offset\n                    token_end_idx = token_start_idx + len(pn_history[char_token_start:char_token_end].split())\n\n                    for i in range(token_start_idx, token_end_idx):\n                        ner_tag_indices[i] = ner_tag_index\n        #             print(f\"{sub_loc} || {pn_history[char_token_start:char_token_end]} || {feature} || {pn_tokens[token_start_idx:token_end_idx]}\")\n\n        tokenized_dict[\"tokens\"].append(pn_tokens)\n        tokenized_dict[\"ner_tags\"].append(ner_tag_indices)\n        pos += 1\n    \n    return tokenized_dict\n\ndef generate_datasets():\n    tokenized_train_dict = tag_and_tokenize(train_dataset)\n    tokenized_val_dict = tag_and_tokenize(validation_dataset)\n    \n    return (Dataset.from_pandas(pd.DataFrame(tokenized_train_dict)), Dataset.from_pandas(pd.DataFrame(tokenized_val_dict)))\n\ntrain_HF_dataset, validation_HF_dataset = generate_datasets()","metadata":{"execution":{"iopub.status.busy":"2022-07-10T16:23:32.273069Z","iopub.execute_input":"2022-07-10T16:23:32.273422Z","iopub.status.idle":"2022-07-10T16:23:43.266787Z","shell.execute_reply.started":"2022-07-10T16:23:32.273381Z","shell.execute_reply":"2022-07-10T16:23:43.265809Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_HF_dataset","metadata":{"execution":{"iopub.status.busy":"2022-07-10T16:23:50.088083Z","iopub.execute_input":"2022-07-10T16:23:50.088506Z","iopub.status.idle":"2022-07-10T16:23:50.095024Z","shell.execute_reply.started":"2022-07-10T16:23:50.088469Z","shell.execute_reply":"2022-07-10T16:23:50.094089Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Source: https://huggingface.co/emilyalsentzer/Bio_ClinicalBERT\nmodel_checkpoint = \"emilyalsentzer/Bio_ClinicalBERT\"\ntokenizer = AutoTokenizer.from_pretrained(model_checkpoint)","metadata":{"execution":{"iopub.status.busy":"2022-07-10T16:23:53.064592Z","iopub.execute_input":"2022-07-10T16:23:53.065179Z","iopub.status.idle":"2022-07-10T16:23:57.490104Z","shell.execute_reply.started":"2022-07-10T16:23:53.065132Z","shell.execute_reply":"2022-07-10T16:23:57.489196Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Source: https://huggingface.co/docs/transformers/main/en/tasks/token_classification\ndef tokenize_and_align_labels(examples):\n    tokenized_inputs = tokenizer(examples[\"tokens\"], truncation=True, is_split_into_words=True)\n\n    labels = []\n    for i, label in enumerate(examples[f\"ner_tags\"]):\n        word_ids = tokenized_inputs.word_ids(batch_index=i)  # Map tokens to their respective word.\n        previous_word_idx = None\n        label_ids = []\n        for word_idx in word_ids:  # Set the special tokens to -100.\n            if word_idx is None:\n                label_ids.append(-100)\n            elif word_idx != previous_word_idx:  # Only label the first token of a given word.\n                label_ids.append(label[word_idx])\n            else:\n                label_ids.append(-100)\n            previous_word_idx = word_idx\n        labels.append(label_ids)\n\n    tokenized_inputs[\"labels\"] = labels\n    return tokenized_inputs","metadata":{"execution":{"iopub.status.busy":"2022-07-10T18:11:47.644113Z","iopub.execute_input":"2022-07-10T18:11:47.644460Z","iopub.status.idle":"2022-07-10T18:11:47.652637Z","shell.execute_reply.started":"2022-07-10T18:11:47.644430Z","shell.execute_reply":"2022-07-10T18:11:47.651466Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tokenized_train_dataset = train_HF_dataset.map(tokenize_and_align_labels, batched=True)\ntokenized_validation_dataset = validation_HF_dataset.map(tokenize_and_align_labels, batched=True)","metadata":{"execution":{"iopub.status.busy":"2022-07-10T18:11:51.619134Z","iopub.execute_input":"2022-07-10T18:11:51.619594Z","iopub.status.idle":"2022-07-10T18:11:54.809572Z","shell.execute_reply.started":"2022-07-10T18:11:51.619555Z","shell.execute_reply":"2022-07-10T18:11:54.808538Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from transformers import DataCollatorForTokenClassification\n\ndata_collator = DataCollatorForTokenClassification(tokenizer=tokenizer)","metadata":{"execution":{"iopub.status.busy":"2022-07-10T18:11:59.293971Z","iopub.execute_input":"2022-07-10T18:11:59.294324Z","iopub.status.idle":"2022-07-10T18:11:59.299389Z","shell.execute_reply.started":"2022-07-10T18:11:59.294294Z","shell.execute_reply":"2022-07-10T18:11:59.298488Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from transformers import AutoModelForTokenClassification, TrainingArguments, Trainer\n\nmodel = AutoModelForTokenClassification.from_pretrained(model_checkpoint, num_labels=len(ner_tag_list))","metadata":{"execution":{"iopub.status.busy":"2022-07-10T18:12:02.362524Z","iopub.execute_input":"2022-07-10T18:12:02.363203Z","iopub.status.idle":"2022-07-10T18:12:04.535770Z","shell.execute_reply.started":"2022-07-10T18:12:02.363163Z","shell.execute_reply":"2022-07-10T18:12:04.534819Z"},"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"os.environ[\"WANDB_DISABLED\"] = \"true\"","metadata":{"execution":{"iopub.status.busy":"2022-07-10T17:47:49.438003Z","iopub.execute_input":"2022-07-10T17:47:49.438353Z","iopub.status.idle":"2022-07-10T17:47:49.443073Z","shell.execute_reply.started":"2022-07-10T17:47:49.438323Z","shell.execute_reply":"2022-07-10T17:47:49.442060Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"training_args = TrainingArguments(\n    output_dir=\"./results\",\n    evaluation_strategy=\"epoch\",\n    learning_rate=2e-5,\n    per_device_train_batch_size=16,\n    per_device_eval_batch_size=16,\n    num_train_epochs=3,\n    weight_decay=0.01,\n)\n\ntrainer = Trainer(\n    model=model,\n    args=training_args,\n    train_dataset=tokenized_train_dataset,\n    eval_dataset=tokenized_validation_dataset,\n    tokenizer=tokenizer,\n    data_collator=data_collator,\n)\n\ntrainer.train()","metadata":{"execution":{"iopub.status.busy":"2022-07-10T18:12:17.638125Z","iopub.execute_input":"2022-07-10T18:12:17.638485Z","iopub.status.idle":"2022-07-10T18:14:01.613621Z","shell.execute_reply.started":"2022-07-10T18:12:17.638454Z","shell.execute_reply":"2022-07-10T18:14:01.612559Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"trainer.evaluate()","metadata":{"execution":{"iopub.status.busy":"2022-07-10T18:15:17.564444Z","iopub.execute_input":"2022-07-10T18:15:17.565395Z","iopub.status.idle":"2022-07-10T18:15:25.223292Z","shell.execute_reply.started":"2022-07-10T18:15:17.565356Z","shell.execute_reply":"2022-07-10T18:15:25.222366Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model_filename=\"clinical_ner_04.model\"\ntrainer.save_model(model_filename)","metadata":{"execution":{"iopub.status.busy":"2022-07-10T18:15:49.994244Z","iopub.execute_input":"2022-07-10T18:15:49.994898Z","iopub.status.idle":"2022-07-10T18:15:50.655852Z","shell.execute_reply.started":"2022-07-10T18:15:49.994858Z","shell.execute_reply":"2022-07-10T18:15:50.654869Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from transformers import AutoTokenizer, AutoModelForTokenClassification","metadata":{"execution":{"iopub.status.busy":"2022-07-10T18:15:55.724082Z","iopub.execute_input":"2022-07-10T18:15:55.724449Z","iopub.status.idle":"2022-07-10T18:15:55.731800Z","shell.execute_reply.started":"2022-07-10T18:15:55.724400Z","shell.execute_reply":"2022-07-10T18:15:55.730799Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tokenizer = AutoTokenizer.from_pretrained(model_filename)\nner_model = AutoModelForTokenClassification.from_pretrained(model_filename, num_labels=len(ner_tag_list))","metadata":{"execution":{"iopub.status.busy":"2022-07-10T18:16:05.295367Z","iopub.execute_input":"2022-07-10T18:16:05.295978Z","iopub.status.idle":"2022-07-10T18:16:06.342332Z","shell.execute_reply.started":"2022-07-10T18:16:05.295931Z","shell.execute_reply":"2022-07-10T18:16:06.341328Z"},"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ex_note = patient_notes.sample().squeeze().pn_history\nex_note_tokens = ex_note.split()\ntokens = tokenizer(ex_note_tokens, truncation=True, is_split_into_words=True)\ntorch.tensor(tokens['input_ids']).unsqueeze(0).size()","metadata":{"execution":{"iopub.status.busy":"2022-07-10T18:19:39.061052Z","iopub.execute_input":"2022-07-10T18:19:39.061452Z","iopub.status.idle":"2022-07-10T18:19:39.075221Z","shell.execute_reply.started":"2022-07-10T18:19:39.061397Z","shell.execute_reply":"2022-07-10T18:19:39.074227Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"predictions = ner_model.forward(input_ids=torch.tensor(tokens['input_ids']).unsqueeze(0), attention_mask=torch.tensor(tokens['attention_mask']).unsqueeze(0))\npredictions = torch.argmax(predictions.logits.squeeze(), axis=1)\npredictions = [ner_tag_list[i] for i in predictions]\nprint(predictions)","metadata":{"execution":{"iopub.status.busy":"2022-07-10T18:19:42.535171Z","iopub.execute_input":"2022-07-10T18:19:42.535544Z","iopub.status.idle":"2022-07-10T18:19:42.917402Z","shell.execute_reply.started":"2022-07-10T18:19:42.535513Z","shell.execute_reply":"2022-07-10T18:19:42.916326Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"len(predictions)","metadata":{"execution":{"iopub.status.busy":"2022-07-10T18:19:47.362247Z","iopub.execute_input":"2022-07-10T18:19:47.362609Z","iopub.status.idle":"2022-07-10T18:19:47.369177Z","shell.execute_reply.started":"2022-07-10T18:19:47.362578Z","shell.execute_reply":"2022-07-10T18:19:47.368122Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"words = tokenizer.batch_decode(tokens['input_ids'])\nprint([word.replace(\"##\", \"\") for word in words])","metadata":{"execution":{"iopub.status.busy":"2022-07-10T18:19:49.935243Z","iopub.execute_input":"2022-07-10T18:19:49.935905Z","iopub.status.idle":"2022-07-10T18:19:49.945837Z","shell.execute_reply.started":"2022-07-10T18:19:49.935860Z","shell.execute_reply":"2022-07-10T18:19:49.943756Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(ex_note)","metadata":{"execution":{"iopub.status.busy":"2022-07-10T18:19:53.347474Z","iopub.execute_input":"2022-07-10T18:19:53.348135Z","iopub.status.idle":"2022-07-10T18:19:53.353121Z","shell.execute_reply.started":"2022-07-10T18:19:53.348099Z","shell.execute_reply":"2022-07-10T18:19:53.352141Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"predictions = ner_model.forward(input_ids=torch.tensor(tokens['input_ids']).unsqueeze(0), attention_mask=torch.tensor(tokens['attention_mask']).unsqueeze(0))\npredictions = torch.argmax(predictions.logits.squeeze(), axis=1)\npredictions = [ner_tag_list[i] for i in predictions]\nprint(predictions)\n\n# Source: https://huggingface.co/docs/transformers/main/en/tasks/token_classification\ndef align_predictions_with_input(predictions, tokenized_input):\n    labels = []\n    for i, label in enumerate(examples[f\"ner_tags\"]):\n        word_ids = tokenized_inputs.word_ids(batch_index=i)  # Map tokens to their respective word.\n        previous_word_idx = None\n        label_ids = []\n        for word_idx in word_ids:  # Set the special tokens to -100.\n            if word_idx is None:\n                label_ids.append(-100)\n            elif word_idx != previous_word_idx:  # Only label the first token of a given word.\n                label_ids.append(label[word_idx])\n            else:\n                label_ids.append(-100)\n            previous_word_idx = word_idx\n        labels.append(label_ids)\n\n    tokenized_inputs[\"labels\"] = labels\n    return tokenized_inputs","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ex_note","metadata":{"execution":{"iopub.status.busy":"2022-07-10T18:07:02.781233Z","iopub.execute_input":"2022-07-10T18:07:02.781588Z","iopub.status.idle":"2022-07-10T18:07:02.790473Z","shell.execute_reply.started":"2022-07-10T18:07:02.781559Z","shell.execute_reply":"2022-07-10T18:07:02.787037Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"words = [word.replace(\"##\", \"\") for word in tokenizer.batch_decode(tokens['input_ids'])]\nrebuilt_note = \"\"\nprev_idx = None\nfor i, idx in enumerate(tokens.word_ids()):\n    # ignore special tokens\n    if idx == None:\n        continue\n    \n    # append space before if new id\n    if prev_idx is not None and prev_idx != idx:\n        rebuilt_note += \" \"\n    \n    rebuilt_note += words[i]\n    \n    prev_idx = idx\n    \nprint(rebuilt_note, \"\\n\")\nprint(ex_note.lower().replace(\"\\r\\n\", \" \"))","metadata":{"execution":{"iopub.status.busy":"2022-07-10T18:19:58.890597Z","iopub.execute_input":"2022-07-10T18:19:58.891044Z","iopub.status.idle":"2022-07-10T18:19:58.902812Z","shell.execute_reply.started":"2022-07-10T18:19:58.891003Z","shell.execute_reply":"2022-07-10T18:19:58.901796Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"rebuilt_note == ex_note.lower().replace(\"\\r\\n\", \" \")","metadata":{"execution":{"iopub.status.busy":"2022-07-10T18:18:00.212263Z","iopub.execute_input":"2022-07-10T18:18:00.212627Z","iopub.status.idle":"2022-07-10T18:18:00.218853Z","shell.execute_reply.started":"2022-07-10T18:18:00.212596Z","shell.execute_reply":"2022-07-10T18:18:00.217897Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"len(tokenizer.batch_decode(tokens['input_ids']))","metadata":{"execution":{"iopub.status.busy":"2022-07-10T18:33:45.448522Z","iopub.execute_input":"2022-07-10T18:33:45.448861Z","iopub.status.idle":"2022-07-10T18:33:45.460430Z","shell.execute_reply.started":"2022-07-10T18:33:45.448832Z","shell.execute_reply":"2022-07-10T18:33:45.459384Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tokens2 = tokenizer(ex_note_tokens, truncation=True, is_split_into_words=True, max_length=128)","metadata":{"execution":{"iopub.status.busy":"2022-07-10T18:34:05.777963Z","iopub.execute_input":"2022-07-10T18:34:05.778835Z","iopub.status.idle":"2022-07-10T18:34:05.786289Z","shell.execute_reply.started":"2022-07-10T18:34:05.778785Z","shell.execute_reply":"2022-07-10T18:34:05.785254Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"len(tokenizer.batch_decode(tokens2['input_ids']))","metadata":{"execution":{"iopub.status.busy":"2022-07-10T18:34:36.244688Z","iopub.execute_input":"2022-07-10T18:34:36.245134Z","iopub.status.idle":"2022-07-10T18:34:36.261630Z","shell.execute_reply.started":"2022-07-10T18:34:36.245095Z","shell.execute_reply":"2022-07-10T18:34:36.260580Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}