{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# ⭐️⭐️Feedback Prize - Predicting Effective Arguments⭐️⭐️\n\n## 📢***First step of this competition : https://www.kaggle.com/code/venkatkumar001/nlpstarter-3-simple-ml-baseline-algo-with-kfold***\n\n## 📢***Second step of this competition - Some try of basic transformer : https://www.kaggle.com/code/venkatkumar001/nlpstarter4-distilbert-bert-base-cased***\n\n\n\n
\n
\n\n\n\n## ***Full credit-This guy introduce new one : https://www.kaggle.com/code/maroberti/fastai-with-transformers-bert-roberta***\n\n# ⭐️⭐️Let's try it Guys!⭐️⭐️\n\n\n\n","metadata":{}},{"cell_type":"markdown","source":"# **1. Import Necessary library**","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nfrom pathlib import Path\nimport os\nimport random\n\nimport torch\nimport torch.optim as optim","metadata":{"execution":{"iopub.status.busy":"2022-07-23T08:26:46.610804Z","iopub.execute_input":"2022-07-23T08:26:46.611501Z","iopub.status.idle":"2022-07-23T08:26:46.622267Z","shell.execute_reply.started":"2022-07-23T08:26:46.611408Z","shell.execute_reply":"2022-07-23T08:26:46.621212Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **Particular this version used - good of this code (Old version of fastai and transformers)**","metadata":{}},{"cell_type":"code","source":"!pip install fastai==1.0.58","metadata":{"execution":{"iopub.status.busy":"2022-07-23T08:26:47.159014Z","iopub.execute_input":"2022-07-23T08:26:47.159574Z","iopub.status.idle":"2022-07-23T08:26:56.918290Z","shell.execute_reply.started":"2022-07-23T08:26:47.159536Z","shell.execute_reply":"2022-07-23T08:26:56.917017Z"},"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!pip install transformers==2.5.1","metadata":{"execution":{"iopub.status.busy":"2022-07-23T08:26:56.922892Z","iopub.execute_input":"2022-07-23T08:26:56.923669Z","iopub.status.idle":"2022-07-23T08:27:06.088166Z","shell.execute_reply.started":"2022-07-23T08:26:56.923625Z","shell.execute_reply":"2022-07-23T08:27:06.087002Z"},"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Fast.ai\nfrom fastai import *\nfrom fastai.text import *\nfrom fastai.callbacks import *\n\n# transformers\nfrom transformers import PreTrainedModel, PreTrainedTokenizer, PretrainedConfig\n\nfrom transformers import BertForSequenceClassification, BertTokenizer, BertConfig\nfrom transformers import RobertaForSequenceClassification, RobertaTokenizer, RobertaConfig\nfrom transformers import XLNetForSequenceClassification, XLNetTokenizer, XLNetConfig\nfrom transformers import XLMForSequenceClassification, XLMTokenizer, XLMConfig\nfrom transformers import DistilBertForSequenceClassification, DistilBertTokenizer, DistilBertConfig\n\nimport fastai\nimport transformers\n\nprint('Fastai_Version: ',fastai.__version__)\nprint('Transformers_Version: ', transformers.__version__)\nprint('Pytorch: ', torch.__version__)","metadata":{"execution":{"iopub.status.busy":"2022-07-23T08:27:10.251753Z","iopub.execute_input":"2022-07-23T08:27:10.252131Z","iopub.status.idle":"2022-07-23T08:27:11.249148Z","shell.execute_reply.started":"2022-07-23T08:27:10.252091Z","shell.execute_reply":"2022-07-23T08:27:11.248132Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for dirname, _,filenames in os.walk('/kaggle/input'):\n for filename in filenames:\n print(os.path.join(dirname,filename))","metadata":{"execution":{"iopub.status.busy":"2022-07-23T08:27:11.260770Z","iopub.execute_input":"2022-07-23T08:27:11.261364Z","iopub.status.idle":"2022-07-23T08:27:12.302039Z","shell.execute_reply.started":"2022-07-23T08:27:11.261322Z","shell.execute_reply":"2022-07-23T08:27:12.301044Z"},"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **2. Load Data**","metadata":{}},{"cell_type":"code","source":"train = pd.read_csv('../input/feedback-prize-effectiveness/train.csv')\ntest = pd.read_csv('../input/feedback-prize-effectiveness/test.csv')\nsample = pd.read_csv('../input/feedback-prize-effectiveness/sample_submission.csv')\nprint(f'Train_Shape: {train.shape},Test_Shape: {test.shape},Sample_Shape: {sample.shape}')\ndisplay(train.sample(2))\ndisplay(test.sample(2))\ndisplay(sample.sample(2))","metadata":{"execution":{"iopub.status.busy":"2022-07-23T08:27:12.303056Z","iopub.execute_input":"2022-07-23T08:27:12.303426Z","iopub.status.idle":"2022-07-23T08:27:12.485667Z","shell.execute_reply.started":"2022-07-23T08:27:12.303387Z","shell.execute_reply":"2022-07-23T08:27:12.484576Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **3. Fast.ai and transformer: All about transfer learning**\n\nInearly 2018, Jeremy Howard(co-founder of fast.ai) and Sebastian Ruder introduced the Universal Language Model Fine-tuning for Text Classification (ULMFiT) method. ULMFiT was the first Transfer Learning method applied to NLP. As a result, besides significantly outperforming many state-of-the-art tasks, it allowed, with only 100 labeled examples, to match performances equivalent to models trained on 100× more data. (thats why fast.ai is different)\n\n
\n\n## **So that i am trying fastai+transformer!**","metadata":{}},{"cell_type":"markdown","source":"## **Main transformers classes**\nIn transformers, each model architecture is associated with 3 main types of classes:\n\n1. A model class to load/store a particular pre-train model.\n2. A tokenizer class to pre-process the data and make it compatible with a particular model.\n3. A configuration class to load/store the configuration of a particular model.","metadata":{}},{"cell_type":"markdown","source":"## **ModelClass**","metadata":{}},{"cell_type":"code","source":"MODEL_CLASSES = {\n 'bert': (BertForSequenceClassification, BertTokenizer, BertConfig),\n 'xlnet': (XLNetForSequenceClassification, XLNetTokenizer, XLNetConfig),\n 'xlm': (XLMForSequenceClassification, XLMTokenizer, XLMConfig),\n 'roberta': (RobertaForSequenceClassification, RobertaTokenizer, RobertaConfig),\n 'distilbert': (DistilBertForSequenceClassification, DistilBertTokenizer, DistilBertConfig)\n}","metadata":{"execution":{"iopub.status.busy":"2022-07-23T08:27:12.487595Z","iopub.execute_input":"2022-07-23T08:27:12.488043Z","iopub.status.idle":"2022-07-23T08:27:12.494458Z","shell.execute_reply.started":"2022-07-23T08:27:12.487999Z","shell.execute_reply":"2022-07-23T08:27:12.492932Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **Configuration**","metadata":{}},{"cell_type":"code","source":"# Parameters\nseed = 42\nuse_fp16 = False\nbs = 16\n\nmodel_type = 'roberta'\npretrained_model_name = 'roberta-base'\n\n# model_type = 'bert'\n# pretrained_model_name='bert-base-uncased'\n\n# model_type = 'distilbert'\n# pretrained_model_name = 'distilbert-base-uncased'\n\n#model_type = 'xlm'\n#pretrained_model_name = 'xlm-clm-enfr-1024'\n\n","metadata":{"execution":{"iopub.status.busy":"2022-07-23T08:27:12.497183Z","iopub.execute_input":"2022-07-23T08:27:12.497928Z","iopub.status.idle":"2022-07-23T08:27:12.504968Z","shell.execute_reply.started":"2022-07-23T08:27:12.497887Z","shell.execute_reply":"2022-07-23T08:27:12.503215Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model_class,tokenizer_class, config_class = MODEL_CLASSES[model_type]","metadata":{"execution":{"iopub.status.busy":"2022-07-23T08:27:12.505983Z","iopub.execute_input":"2022-07-23T08:27:12.507819Z","iopub.status.idle":"2022-07-23T08:27:12.512915Z","shell.execute_reply.started":"2022-07-23T08:27:12.507690Z","shell.execute_reply":"2022-07-23T08:27:12.511850Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **All key category objects of robert transformer**","metadata":{}},{"cell_type":"code","source":"model_class.pretrained_model_archive_map.keys()\n","metadata":{"execution":{"iopub.status.busy":"2022-07-23T08:27:12.516984Z","iopub.execute_input":"2022-07-23T08:27:12.517689Z","iopub.status.idle":"2022-07-23T08:27:12.525206Z","shell.execute_reply.started":"2022-07-23T08:27:12.517656Z","shell.execute_reply":"2022-07-23T08:27:12.523820Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Seed**","metadata":{}},{"cell_type":"code","source":"def seed_all(seed_value):\n random.seed(seed_value) # Python\n np.random.seed(seed_value) # cpu vars\n torch.manual_seed(seed_value) # cpu vars\n \n if torch.cuda.is_available(): \n torch.cuda.manual_seed(seed_value)\n torch.cuda.manual_seed_all(seed_value) # gpu vars\n torch.backends.cudnn.deterministic = True #needed\n torch.backends.cudnn.benchmark = False","metadata":{"execution":{"iopub.status.busy":"2022-07-23T08:32:41.297298Z","iopub.execute_input":"2022-07-23T08:32:41.297688Z","iopub.status.idle":"2022-07-23T08:32:41.304097Z","shell.execute_reply.started":"2022-07-23T08:32:41.297655Z","shell.execute_reply":"2022-07-23T08:32:41.303083Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"seed_all(seed)","metadata":{"execution":{"iopub.status.busy":"2022-07-23T08:32:42.395958Z","iopub.execute_input":"2022-07-23T08:32:42.396650Z","iopub.status.idle":"2022-07-23T08:32:42.404005Z","shell.execute_reply.started":"2022-07-23T08:32:42.396602Z","shell.execute_reply":"2022-07-23T08:32:42.403020Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **4. Data processing - Transformer process**\n\n1. The TokenizeProcessor object takes as tokenizer argument a Tokenizer object.\n2. The Tokenizer object takes as tok_func argument a BaseTokenizer object.\n3. The BaseTokenizer object implement the function tokenizer(t:str) → List[str] that take a text t and returns the list of its tokens.\n","metadata":{}},{"cell_type":"code","source":"class TransformersBaseTokenizer(BaseTokenizer):\n \"\"\"Wrapper around PreTrainedTokenizer to be compatible with fast.ai\"\"\"\n def __init__(self, pretrained_tokenizer: PreTrainedTokenizer, model_type = 'bert', **kwargs):\n self._pretrained_tokenizer = pretrained_tokenizer\n self.max_seq_len = pretrained_tokenizer.max_len\n self.model_type = model_type\n\n def __call__(self, *args, **kwargs): \n return self\n\n def tokenizer(self, t:str) -> List[str]:\n \"\"\"Limits the maximum sequence length and add the spesial tokens\"\"\"\n CLS = self._pretrained_tokenizer.cls_token\n SEP = self._pretrained_tokenizer.sep_token\n if self.model_type in ['roberta']:\n tokens = self._pretrained_tokenizer.tokenize(t, add_prefix_space=True)[:self.max_seq_len - 2]\n tokens = [CLS] + tokens + [SEP]\n else:\n tokens = self._pretrained_tokenizer.tokenize(t)[:self.max_seq_len - 2]\n if self.model_type in ['xlnet']:\n tokens = tokens + [SEP] + [CLS]\n else:\n tokens = [CLS] + tokens + [SEP]\n return tokens","metadata":{"execution":{"iopub.status.busy":"2022-07-23T08:37:36.205825Z","iopub.execute_input":"2022-07-23T08:37:36.206539Z","iopub.status.idle":"2022-07-23T08:37:36.216762Z","shell.execute_reply.started":"2022-07-23T08:37:36.206491Z","shell.execute_reply":"2022-07-23T08:37:36.215775Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"transformer_tokenizer = tokenizer_class.from_pretrained(pretrained_model_name)\ntransformer_base_tokenizer = TransformersBaseTokenizer(pretrained_tokenizer = transformer_tokenizer, model_type = model_type)\nfastai_tokenizer = Tokenizer(tok_func = transformer_base_tokenizer, pre_rules=[], post_rules=[])","metadata":{"execution":{"iopub.status.busy":"2022-07-23T08:40:58.777762Z","iopub.execute_input":"2022-07-23T08:40:58.778477Z","iopub.status.idle":"2022-07-23T08:40:59.596801Z","shell.execute_reply.started":"2022-07-23T08:40:58.778440Z","shell.execute_reply":"2022-07-23T08:40:59.595833Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **In hugging face transformer approach - Text of transformer some structure and proceed**\n\n1. bert: [CLS] + tokens + [SEP] + padding\n\n2. roberta: [CLS] + prefix_space + tokens + [SEP] + padding\n\n3. distilbert: [CLS] + tokens + [SEP] + padding\n\n4. xlm: [CLS] + tokens + [SEP] + padding\n\n5. xlnet: padding + tokens + [SEP] + [CLS]\n","metadata":{}},{"cell_type":"code","source":"class TransformersVocab(Vocab):\n def __init__(self, tokenizer: PreTrainedTokenizer):\n super(TransformersVocab, self).__init__(itos = [])\n self.tokenizer = tokenizer\n \n def numericalize(self, t:Collection[str]) -> List[int]:\n \"Convert a list of tokens `t` to their ids.\"\n return self.tokenizer.convert_tokens_to_ids(t)\n #return self.tokenizer.encode(t)\n\n def textify(self, nums:Collection[int], sep=' ') -> List[str]:\n \"Convert a list of `nums` to their tokens.\"\n nums = np.array(nums).tolist()\n return sep.join(self.tokenizer.convert_ids_to_tokens(nums)) if sep is not None else self.tokenizer.convert_ids_to_tokens(nums)\n \n def __getstate__(self):\n return {'itos':self.itos, 'tokenizer':self.tokenizer}\n\n def __setstate__(self, state:dict):\n self.itos = state['itos']\n self.tokenizer = state['tokenizer']\n self.stoi = collections.defaultdict(int,{v:k for k,v in enumerate(self.itos)})\n","metadata":{"execution":{"iopub.status.busy":"2022-07-23T08:41:58.494780Z","iopub.execute_input":"2022-07-23T08:41:58.495160Z","iopub.status.idle":"2022-07-23T08:41:58.507079Z","shell.execute_reply.started":"2022-07-23T08:41:58.495128Z","shell.execute_reply":"2022-07-23T08:41:58.505977Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"transformer_vocab = TransformersVocab(tokenizer = transformer_tokenizer)\nnumericalize_processor = NumericalizeProcessor(vocab=transformer_vocab)\n\ntokenize_processor = TokenizeProcessor(tokenizer=fastai_tokenizer, include_bos=False, include_eos=False)\n\ntransformer_processor = [tokenize_processor, numericalize_processor]","metadata":{"execution":{"iopub.status.busy":"2022-07-23T08:42:12.453160Z","iopub.execute_input":"2022-07-23T08:42:12.453542Z","iopub.status.idle":"2022-07-23T08:42:12.458521Z","shell.execute_reply.started":"2022-07-23T08:42:12.453486Z","shell.execute_reply":"2022-07-23T08:42:12.457263Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pad_first = bool(model_type in ['xlnet'])\npad_idx = transformer_tokenizer.pad_token_id","metadata":{"execution":{"iopub.status.busy":"2022-07-23T08:42:43.980786Z","iopub.execute_input":"2022-07-23T08:42:43.981382Z","iopub.status.idle":"2022-07-23T08:42:43.985969Z","shell.execute_reply.started":"2022-07-23T08:42:43.981347Z","shell.execute_reply":"2022-07-23T08:42:43.984989Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tokens = transformer_tokenizer.tokenize('Now big fan of fastai!')\nprint(tokens)\nids = transformer_tokenizer.convert_tokens_to_ids(tokens)\nprint(ids)\ntransformer_tokenizer.convert_ids_to_tokens(ids)","metadata":{"execution":{"iopub.status.busy":"2022-07-23T08:43:19.729683Z","iopub.execute_input":"2022-07-23T08:43:19.730573Z","iopub.status.idle":"2022-07-23T08:43:19.740719Z","shell.execute_reply.started":"2022-07-23T08:43:19.730501Z","shell.execute_reply":"2022-07-23T08:43:19.739577Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Mapping my target value category to binary**","metadata":{}},{"cell_type":"code","source":"effectiveness_map = {'Ineffective' : 0, 'Adequate':1,'Effective':2}\ntrain['target'] = train['discourse_effectiveness'].map(effectiveness_map)\ntrain.sample(2)","metadata":{"execution":{"iopub.status.busy":"2022-07-23T08:45:42.295736Z","iopub.execute_input":"2022-07-23T08:45:42.296691Z","iopub.status.idle":"2022-07-23T08:45:42.321436Z","shell.execute_reply.started":"2022-07-23T08:45:42.296644Z","shell.execute_reply":"2022-07-23T08:45:42.320434Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **Fast.ai - Databunch api all in one place like compile the data's**","metadata":{}},{"cell_type":"code","source":"databunch = (TextList.from_df(train, cols='discourse_text', processor=transformer_processor)\n .split_by_rand_pct(0.1,seed=seed)\n .label_from_df(cols= 'target')\n .add_test(test)\n .databunch(bs=bs, pad_first=pad_first, pad_idx=pad_idx))\n","metadata":{"execution":{"iopub.status.busy":"2022-07-23T08:47:12.342667Z","iopub.execute_input":"2022-07-23T08:47:12.343366Z","iopub.status.idle":"2022-07-23T08:47:29.041376Z","shell.execute_reply.started":"2022-07-23T08:47:12.343328Z","shell.execute_reply":"2022-07-23T08:47:29.040071Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **Check tokenizer,batch,normalize values**","metadata":{}},{"cell_type":"code","source":"print('[CLS] token :', transformer_tokenizer.cls_token)\nprint('[SEP] token :', transformer_tokenizer.sep_token)\nprint('[PAD] token :', transformer_tokenizer.pad_token)\ndatabunch.show_batch()","metadata":{"execution":{"iopub.status.busy":"2022-07-23T08:47:41.490098Z","iopub.execute_input":"2022-07-23T08:47:41.490482Z","iopub.status.idle":"2022-07-23T08:47:51.224446Z","shell.execute_reply.started":"2022-07-23T08:47:41.490447Z","shell.execute_reply":"2022-07-23T08:47:51.223476Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('[CLS] id :', transformer_tokenizer.cls_token_id)\nprint('[SEP] id :', transformer_tokenizer.sep_token_id)\nprint('[PAD] id :', pad_idx)\ntest_one_batch = databunch.one_batch()[0]\nprint('Batch shape : ',test_one_batch.shape)\nprint(test_one_batch)","metadata":{"execution":{"iopub.status.busy":"2022-07-23T08:48:19.817844Z","iopub.execute_input":"2022-07-23T08:48:19.818207Z","iopub.status.idle":"2022-07-23T08:48:22.090863Z","shell.execute_reply.started":"2022-07-23T08:48:19.818176Z","shell.execute_reply":"2022-07-23T08:48:22.087749Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **Transformer custom model- Robert-base**","metadata":{}},{"cell_type":"code","source":"# defining our model architecture \nclass CustomTransformerModel(nn.Module):\n def __init__(self, transformer_model: PreTrainedModel):\n super(CustomTransformerModel,self).__init__()\n self.transformer = transformer_model\n \n def forward(self, input_ids, attention_mask=None):\n \n # attention_mask\n # Mask to avoid performing attention on padding token indices.\n # Mask values selected in ``[0, 1]``:\n # ``1`` for tokens that are NOT MASKED, ``0`` for MASKED tokens.\n attention_mask = (input_ids!=pad_idx).type(input_ids.type()) \n \n logits = self.transformer(input_ids,\n attention_mask = attention_mask)[0] \n return logits\n","metadata":{"execution":{"iopub.status.busy":"2022-07-23T08:48:44.420490Z","iopub.execute_input":"2022-07-23T08:48:44.421467Z","iopub.status.idle":"2022-07-23T08:48:44.429015Z","shell.execute_reply.started":"2022-07-23T08:48:44.421429Z","shell.execute_reply":"2022-07-23T08:48:44.427742Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **Configure the models - input data's**","metadata":{}},{"cell_type":"code","source":"config = config_class.from_pretrained(pretrained_model_name)\nconfig.num_labels = 3\nconfig.use_bfloat16 = use_fp16\nprint(config)","metadata":{"execution":{"iopub.status.busy":"2022-07-23T08:49:06.101630Z","iopub.execute_input":"2022-07-23T08:49:06.102410Z","iopub.status.idle":"2022-07-23T08:49:06.379282Z","shell.execute_reply.started":"2022-07-23T08:49:06.102371Z","shell.execute_reply":"2022-07-23T08:49:06.378312Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"transformer_model = model_class.from_pretrained(pretrained_model_name, config = config)\n# transformer_model = model_class.from_pretrained(pretrained_model_name, num_labels = 5)\n\ncustom_transformer_model = CustomTransformerModel(transformer_model = transformer_model)","metadata":{"execution":{"iopub.status.busy":"2022-07-23T08:49:29.746480Z","iopub.execute_input":"2022-07-23T08:49:29.747210Z","iopub.status.idle":"2022-07-23T08:49:45.250321Z","shell.execute_reply.started":"2022-07-23T08:49:29.747171Z","shell.execute_reply":"2022-07-23T08:49:45.249329Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **Learner - custom optimizer / custom metrics**\n\nAnd from fastai.callbacks import * ---> this one only run this above version of fast.ai ok!","metadata":{}},{"cell_type":"code","source":"from fastai.callbacks import *\nfrom transformers import AdamW\nfrom functools import partial\n\nCustomAdamW = partial(AdamW, correct_bias=False)\n\nlearner = Learner(databunch, \n custom_transformer_model, \n opt_func = CustomAdamW, \n metrics=[accuracy, error_rate])\n\n# Show graph of learner stats and metrics after each epoch.\nlearner.callbacks.append(ShowGraph(learner))\n\n# Put learn in FP16 precision mode. --> Seems to not working\nif use_fp16: learner = learner.to_fp16()","metadata":{"execution":{"iopub.status.busy":"2022-07-23T08:58:58.092349Z","iopub.execute_input":"2022-07-23T08:58:58.093304Z","iopub.status.idle":"2022-07-23T08:58:58.117267Z","shell.execute_reply.started":"2022-07-23T08:58:58.093263Z","shell.execute_reply":"2022-07-23T08:58:58.116411Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(learner.model)","metadata":{"execution":{"iopub.status.busy":"2022-07-23T09:00:41.455573Z","iopub.execute_input":"2022-07-23T09:00:41.456250Z","iopub.status.idle":"2022-07-23T09:00:41.463020Z","shell.execute_reply.started":"2022-07-23T09:00:41.456200Z","shell.execute_reply":"2022-07-23T09:00:41.462017Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **Optional one - how to implement the manual format - this guy shows great easily understand**","metadata":{}},{"cell_type":"code","source":"# For DistilBERT\n# list_layers = [learner.model.transformer.distilbert.embeddings,\n# learner.model.transformer.distilbert.transformer.layer[0],\n# learner.model.transformer.distilbert.transformer.layer[1],\n# learner.model.transformer.distilbert.transformer.layer[2],\n# learner.model.transformer.distilbert.transformer.layer[3],\n# learner.model.transformer.distilbert.transformer.layer[4],\n# learner.model.transformer.distilbert.transformer.layer[5],\n# learner.model.transformer.pre_classifier]\n\n# # For xlnet-base-cased\n# list_layers = [learner.model.transformer.transformer.word_embedding,\n# learner.model.transformer.transformer.layer[0],\n# learner.model.transformer.transformer.layer[1],\n# learner.model.transformer.transformer.layer[2],\n# learner.model.transformer.transformer.layer[3],\n# learner.model.transformer.transformer.layer[4],\n# learner.model.transformer.transformer.layer[5],\n# learner.model.transformer.transformer.layer[6],\n# learner.model.transformer.transformer.layer[7],\n# learner.model.transformer.transformer.layer[8],\n# learner.model.transformer.transformer.layer[9],\n# learner.model.transformer.transformer.layer[10],\n# learner.model.transformer.transformer.layer[11],\n# learner.model.transformer.sequence_summary]\n\n# For roberta-base\nlist_layers = [learner.model.transformer.roberta.embeddings,\n learner.model.transformer.roberta.encoder.layer[0],\n learner.model.transformer.roberta.encoder.layer[1],\n learner.model.transformer.roberta.encoder.layer[2],\n learner.model.transformer.roberta.encoder.layer[3],\n learner.model.transformer.roberta.encoder.layer[4],\n learner.model.transformer.roberta.encoder.layer[5],\n learner.model.transformer.roberta.encoder.layer[6],\n learner.model.transformer.roberta.encoder.layer[7],\n learner.model.transformer.roberta.encoder.layer[8],\n learner.model.transformer.roberta.encoder.layer[9],\n learner.model.transformer.roberta.encoder.layer[10],\n learner.model.transformer.roberta.encoder.layer[11],\n learner.model.transformer.roberta.pooler]","metadata":{"execution":{"iopub.status.busy":"2022-07-23T09:01:59.998443Z","iopub.execute_input":"2022-07-23T09:01:59.998824Z","iopub.status.idle":"2022-07-23T09:02:00.008645Z","shell.execute_reply.started":"2022-07-23T09:01:59.998791Z","shell.execute_reply":"2022-07-23T09:02:00.007614Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"learner.split(list_layers)\nnum_groups = len(learner.layer_groups)\nprint('Learner split in',num_groups,'groups')\nprint(learner.layer_groups)","metadata":{"execution":{"iopub.status.busy":"2022-07-23T09:02:27.987951Z","iopub.execute_input":"2022-07-23T09:02:27.988399Z","iopub.status.idle":"2022-07-23T09:02:28.056377Z","shell.execute_reply.started":"2022-07-23T09:02:27.988357Z","shell.execute_reply":"2022-07-23T09:02:28.055457Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **5.Training process**","metadata":{}},{"cell_type":"code","source":"learner.save('untrain')","metadata":{"execution":{"iopub.status.busy":"2022-07-23T09:03:17.668418Z","iopub.execute_input":"2022-07-23T09:03:17.669097Z","iopub.status.idle":"2022-07-23T09:03:18.430193Z","shell.execute_reply.started":"2022-07-23T09:03:17.669062Z","shell.execute_reply":"2022-07-23T09:03:18.429209Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"seed_all(seed)\nlearner.load('untrain');","metadata":{"execution":{"iopub.status.busy":"2022-07-23T09:03:28.627898Z","iopub.execute_input":"2022-07-23T09:03:28.628258Z","iopub.status.idle":"2022-07-23T09:03:29.214087Z","shell.execute_reply.started":"2022-07-23T09:03:28.628230Z","shell.execute_reply":"2022-07-23T09:03:29.213151Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### **group of classification process-1**","metadata":{}},{"cell_type":"code","source":"learner.freeze_to(-1)","metadata":{"execution":{"iopub.status.busy":"2022-07-23T09:03:40.350480Z","iopub.execute_input":"2022-07-23T09:03:40.351397Z","iopub.status.idle":"2022-07-23T09:03:40.362197Z","shell.execute_reply.started":"2022-07-23T09:03:40.351357Z","shell.execute_reply":"2022-07-23T09:03:40.361261Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"learner.summary()","metadata":{"execution":{"iopub.status.busy":"2022-07-23T09:03:46.547747Z","iopub.execute_input":"2022-07-23T09:03:46.548356Z","iopub.status.idle":"2022-07-23T09:03:49.517481Z","shell.execute_reply.started":"2022-07-23T09:03:46.548318Z","shell.execute_reply":"2022-07-23T09:03:49.516440Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **Find best learning rate**","metadata":{}},{"cell_type":"code","source":"learner.lr_find()","metadata":{"execution":{"iopub.status.busy":"2022-07-23T09:04:38.672744Z","iopub.execute_input":"2022-07-23T09:04:38.673173Z","iopub.status.idle":"2022-07-23T09:04:45.046113Z","shell.execute_reply.started":"2022-07-23T09:04:38.673136Z","shell.execute_reply":"2022-07-23T09:04:45.044027Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"learner.recorder.plot(skip_end=10,suggestion=True)","metadata":{"execution":{"iopub.status.busy":"2022-07-23T09:05:12.079300Z","iopub.execute_input":"2022-07-23T09:05:12.079684Z","iopub.status.idle":"2022-07-23T09:05:12.489617Z","shell.execute_reply.started":"2022-07-23T09:05:12.079648Z","shell.execute_reply":"2022-07-23T09:05:12.488693Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **Just one cycle-try yourwise**","metadata":{}},{"cell_type":"code","source":"learner.fit_one_cycle(4,max_lr=1e-03,moms=(0.8,0.7))","metadata":{"execution":{"iopub.status.busy":"2022-07-23T09:05:54.998674Z","iopub.execute_input":"2022-07-23T09:05:54.999059Z","iopub.status.idle":"2022-07-23T09:07:40.130818Z","shell.execute_reply.started":"2022-07-23T09:05:54.999025Z","shell.execute_reply":"2022-07-23T09:07:40.129520Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"learner.save('first_cycle')","metadata":{"execution":{"iopub.status.busy":"2022-07-23T09:07:40.133059Z","iopub.execute_input":"2022-07-23T09:07:40.133663Z","iopub.status.idle":"2022-07-23T09:07:41.012101Z","shell.execute_reply.started":"2022-07-23T09:07:40.133620Z","shell.execute_reply":"2022-07-23T09:07:41.011074Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"seed_all(seed)\nlearner.load('first_cycle');","metadata":{"execution":{"iopub.status.busy":"2022-07-23T09:07:41.013768Z","iopub.execute_input":"2022-07-23T09:07:41.014155Z","iopub.status.idle":"2022-07-23T09:07:41.617120Z","shell.execute_reply.started":"2022-07-23T09:07:41.014117Z","shell.execute_reply":"2022-07-23T09:07:41.616141Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"learner.freeze_to(-2)\nlr = 1e-3","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"learner.fit_one_cycle(6, max_lr=slice(lr*0.95**num_groups, lr), moms=(0.8, 0.9))","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"learner.save('second_cycle')","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"seed_all(seed)\nlearner.load('second_cycle');","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"learner.freeze_to(-3)","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"learner.fit_one_cycle(2, max_lr=slice(lr*0.95**num_groups, lr), moms=(0.8, 0.9))","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"learner.save('third_cycle')","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"seed_all(seed)\nlearner.load('third_cycle');","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"learner.unfreeze()","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"learner.fit_one_cycle(6, max_lr=slice(lr*0.95**num_groups, lr), moms=(0.8, 0.9))","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **Predict**","metadata":{}},{"cell_type":"code","source":"learner.predict('Vk is beast mode of NLP using fastai')","metadata":{"execution":{"iopub.status.busy":"2022-07-23T09:07:46.847962Z","iopub.execute_input":"2022-07-23T09:07:46.848465Z","iopub.status.idle":"2022-07-23T09:07:46.882700Z","shell.execute_reply.started":"2022-07-23T09:07:46.848427Z","shell.execute_reply":"2022-07-23T09:07:46.881617Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **Save the model - Pickle file----->Its used inference approach**","metadata":{}},{"cell_type":"code","source":"learner.export(file = 'transformer.pkl');","metadata":{"execution":{"iopub.status.busy":"2022-07-23T09:08:10.112651Z","iopub.execute_input":"2022-07-23T09:08:10.113042Z","iopub.status.idle":"2022-07-23T09:08:11.530226Z","shell.execute_reply.started":"2022-07-23T09:08:10.113010Z","shell.execute_reply":"2022-07-23T09:08:11.529224Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Load the pickle file and predict**","metadata":{}},{"cell_type":"code","source":"path = '/kaggle/working'\nexport_learner = load_learner(path, file = 'transformer.pkl')","metadata":{"execution":{"iopub.status.busy":"2022-07-23T09:08:32.758686Z","iopub.execute_input":"2022-07-23T09:08:32.759634Z","iopub.status.idle":"2022-07-23T09:08:33.269404Z","shell.execute_reply.started":"2022-07-23T09:08:32.759596Z","shell.execute_reply":"2022-07-23T09:08:33.268457Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"learner.predict('Vk is beast mode of NLP using fastai')","metadata":{"execution":{"iopub.status.busy":"2022-07-23T09:08:49.709863Z","iopub.execute_input":"2022-07-23T09:08:49.710802Z","iopub.status.idle":"2022-07-23T09:08:49.732406Z","shell.execute_reply.started":"2022-07-23T09:08:49.710766Z","shell.execute_reply":"2022-07-23T09:08:49.731386Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Predict output of test data**","metadata":{}},{"cell_type":"code","source":"def get_preds_as_nparray(ds_type) -> np.ndarray:\n \"\"\"\n the get_preds method does not yield the elements in order by default\n we borrow the code from the RNNLearner to resort the elements into their correct order\n \"\"\"\n preds = learner.get_preds(ds_type)[0].detach().cpu().numpy()\n sampler = [i for i in databunch.dl(ds_type).sampler]\n reverse_sampler = np.argsort(sampler)\n return preds[reverse_sampler, :]\n\ntest_preds = get_preds_as_nparray(DatasetType.Test)","metadata":{"execution":{"iopub.status.busy":"2022-07-23T09:09:15.843434Z","iopub.execute_input":"2022-07-23T09:09:15.844375Z","iopub.status.idle":"2022-07-23T09:09:16.093829Z","shell.execute_reply.started":"2022-07-23T09:09:15.844337Z","shell.execute_reply":"2022-07-23T09:09:16.092399Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **6. Generate Submission file**","metadata":{}},{"cell_type":"code","source":"sample.loc[:,\"Ineffective\"] = test_preds[:,0]\nsample.loc[:,\"Adequate\"] = test_preds[:,1]\nsample.loc[:,\"Effective\"] = test_preds[:,2]\nsample.to_csv('submission.csv',index=False)","metadata":{"execution":{"iopub.status.busy":"2022-07-23T09:10:38.042697Z","iopub.execute_input":"2022-07-23T09:10:38.043287Z","iopub.status.idle":"2022-07-23T09:10:38.063379Z","shell.execute_reply.started":"2022-07-23T09:10:38.043249Z","shell.execute_reply":"2022-07-23T09:10:38.062439Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample","metadata":{"execution":{"iopub.status.busy":"2022-07-23T09:10:46.324416Z","iopub.execute_input":"2022-07-23T09:10:46.325101Z","iopub.status.idle":"2022-07-23T09:10:46.337647Z","shell.execute_reply.started":"2022-07-23T09:10:46.325065Z","shell.execute_reply":"2022-07-23T09:10:46.336571Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 📢 **Predict your self -- Inference approach**","metadata":{}},{"cell_type":"markdown","source":"## **⭐️⭐️Thanks for visiting guys!⭐️⭐️**\n\n## **Full Credits:**\n\n1. https://www.kaggle.com/code/maroberti/fastai-with-transformers-bert-roberta\n2. https://www.kaggle.com/code/venkatkumar001/nlpstarter-3-simple-ml-baseline-algo-with-kfold\n3. https://www.kaggle.com/code/venkatkumar001/nlpstarter4-distilbert-bert-base-cased\n4. https://towardsdatascience.com/fastai-with-transformers-bert-roberta-xlnet-xlm-distilbert-4f41ee18ecb2\n\n## **If you know more about Huggingface transformer:**\n1. https://www.kaggle.com/code/venkatkumar001/nlp-starter2-hf-pretrain-finetune\n2. https://www.kaggle.com/code/venkatkumar001/transformeranatomy-encoder","metadata":{}}]}