{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## This notebook is a fork of brilliant [codebert-pairwise notebook](https://www.kaggle.com/code/thedevastator/codebert-pairwise )\n### with added graphcodebert model\n\n_____\n**Credits:**<br>\nThis notebook demonstrates a simple ensemble method for ranking problems. It is **based on these incredible notebooks:**\n- **[Stronger baseline with code cells](https://www.kaggle.com/code/suicaokhoailang/stronger-baseline-with-code-cells)** by [suicaokhoailang](https://www.kaggle.com/suicaokhoailang)\n- **[AI4Code Pairwise BertSmall inference](https://www.kaggle.com/code/yuanzhezhou/ai4code-pairwise-bertsmall-inference)** by [yuanzhezhou](https://www.kaggle.com/yuanzhezhou)<br>\n- **[ai4code-pytorch-graphcodebert](https://www.kaggle.com/code/kojimar/ai4code-pytorch-graphcodebert/notebook)** by [kojimar](https://www.kaggle.com/kojimar)<br>\n\nAll credits for the models themselves (both training and prediction) belogs to the original authors! I simply cloned their code and retrained my own version.\n_____\n\n# Ensembling Rank Based Submissions\n\nWe are a month away from the finish line and yet there is no high scoring ensemble in sight!\n\nLet's fix this. \n\nBut how do you actually combine rank based predictions? Let alone predictions that come from completly different approaches (direct rank prediction / pairwise).<br>\nActually, this is pretty simple: **Average the indices of the elements.**<br>\nThis way, we can sort the final prediction by the ensembled indices and it will simply represent an aggragated representation of the element's location.<br>","metadata":{}},{"cell_type":"code","source":"def read_notebook(path):\n    import pandas as pd\n    return (\n        pd.read_json(\n            path,\n            dtype={'cell_type': 'category', 'source': 'str'})\n        .assign(id=path.stem)\n        .rename_axis('cell_id')\n    )","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def clean_code(cell): return str(cell).replace(\"\\\\n\", \"\\n\")\n\ndef sample_cells(cells, n):\n    import numpy as np\n    cells = [clean_code(cell) for cell in cells]\n    if n >= len(cells): return [cell[:200] for cell in cells]\n    else:\n        results = []\n        step = len(cells) / n\n        idx = 0\n        while int(np.round(idx)) < len(cells):\n            results.append(cells[int(np.round(idx))])\n            idx += step        \n        if cells[-1] not in results: results[-1] = cells[-1]\n        return results\n\ndef get_features(df):\n    from tqdm import tqdm\n    features = dict()\n    df = df.sort_values(\"rank\").reset_index(drop=True)\n    for idx, sub_df in tqdm(df.groupby(\"id\")):\n        features[idx] = dict()\n        total_md = sub_df[sub_df.cell_type == \"markdown\"].shape[0]\n        code_sub_df = sub_df[sub_df.cell_type == \"code\"]\n        total_code = code_sub_df.shape[0]\n        codes = sample_cells(code_sub_df.source.values, 20)\n        features[idx][\"total_code\"] = total_code\n        features[idx][\"total_md\"] = total_md\n        features[idx][\"codes\"] = codes\n    return features","metadata":{"papermill":{"duration":0.023767,"end_time":"2022-05-23T03:29:09.929422","exception":false,"start_time":"2022-05-23T03:29:09.905655","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-07-27T06:05:56.691552Z","iopub.execute_input":"2022-07-27T06:05:56.691847Z","iopub.status.idle":"2022-07-27T06:05:56.708031Z","shell.execute_reply.started":"2022-07-27T06:05:56.691809Z","shell.execute_reply":"2022-07-27T06:05:56.707128Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def read_data(data): return tuple(d.cuda() for d in data[:-1]), data[-1].cuda()\n\ndef validate(model, val_loader):    \n    import sys\n    import torch    \n    import numpy as np\n    from tqdm import tqdm    \n    model.eval()    \n    tbar = tqdm(val_loader, file=sys.stdout)    \n    preds = []\n    labels = []\n    with torch.no_grad():\n        for idx, data in enumerate(tbar):\n            inputs, target = read_data(data)\n            pred = model(*inputs)\n            preds.append(pred.detach().cpu().numpy().ravel())\n            labels.append(target.detach().cpu().numpy().ravel())    \n    return np.concatenate(labels), np.concatenate(preds)\n\ndef predict_caller(args): return predict(args[0], args[1],args[2])\n    \ndef predict(model_path, ckpt_path,notebook_version):\n    \n    import gc\n    import json\n    import sys, os\n    import numpy as np\n    import pandas as pd\n    from tqdm import tqdm\n    from pathlib import Path\n    from scipy import sparse\n\n    data_dir = Path('../input/AI4Code')\n    paths_test = list((data_dir / 'test').glob('*.json'))\n    notebooks_test = [\n        read_notebook(path) for path in tqdm(paths_test, desc='Test NBs')\n    ]\n    test_df = (\n        pd.concat(notebooks_test)\n        .set_index('id', append=True)\n        .swaplevel()\n        .sort_index(level='id', sort_remaining=False)\n    ).reset_index()\n    test_df[\"rank\"] = test_df.groupby([\"id\", \"cell_type\"]).cumcount()\n    test_df[\"pred\"] = test_df.groupby([\"id\", \"cell_type\"])[\"rank\"].rank(pct=True)\n\n    import torch\n    import torch.nn as nn\n    import torch.nn.functional as F\n    from torch.utils.data import DataLoader, Dataset\n    from transformers import AutoModel, AutoTokenizer\n\n    class MarkdownModel(nn.Module):\n        def __init__(self, model_path):\n            super(MarkdownModel, self).__init__()\n            self.model = AutoModel.from_pretrained(model_path)\n            self.top = nn.Linear(769, 1)\n\n        def forward(self, ids, mask, fts):\n            x = self.model(ids, mask)[0]\n            x = self.top(torch.cat((x[:, 0, :], fts),1))\n            return x\n\n\n    class MarkdownDataset(Dataset):\n\n        def __init__(self, df, model_name_or_path, total_max_len, md_max_len, fts):\n            super().__init__()\n            self.df = df.reset_index(drop=True)\n            self.md_max_len = md_max_len\n            self.total_max_len = total_max_len  # maxlen allowed by model config\n            self.tokenizer = AutoTokenizer.from_pretrained(model_name_or_path)\n            self.fts = fts\n\n        def __getitem__(self, index):\n            row = self.df.iloc[index]\n\n            inputs = self.tokenizer.encode_plus(\n                row.source,\n                None,\n                add_special_tokens=True,\n                max_length=self.md_max_len,\n                padding=\"max_length\",\n                return_token_type_ids=True,\n                truncation=True\n            )\n            code_inputs = self.tokenizer.batch_encode_plus(\n                [str(x) for x in self.fts[row.id][\"codes\"]],\n                add_special_tokens=True,\n                max_length=23,\n                padding=\"max_length\",\n                truncation=True\n            )\n            n_md = self.fts[row.id][\"total_md\"]\n            n_code = self.fts[row.id][\"total_code\"]\n            if n_md + n_code == 0:\n                fts = torch.FloatTensor([0])\n            else:\n                fts = torch.FloatTensor([n_md / (n_md + n_code)])\n\n            ids = inputs['input_ids']\n            for x in code_inputs['input_ids']:\n                ids.extend(x[:-1])\n            ids = ids[:self.total_max_len]\n            if len(ids) != self.total_max_len:\n                ids = ids + [self.tokenizer.pad_token_id, ] * (self.total_max_len - len(ids))\n            ids = torch.LongTensor(ids)\n\n            mask = inputs['attention_mask']\n            for x in code_inputs['attention_mask']:\n                mask.extend(x[:-1])\n            mask = mask[:self.total_max_len]\n            if len(mask) != self.total_max_len:\n                mask = mask + [self.tokenizer.pad_token_id, ] * (self.total_max_len - len(mask))\n            mask = torch.LongTensor(mask)\n\n            assert len(ids) == self.total_max_len\n\n            return ids, mask, fts, torch.FloatTensor([row.pct_rank])\n\n        def __len__(self):\n            return self.df.shape[0]\n    \n    model = MarkdownModel(model_path)\n    model = model.cuda()\n    model.eval()\n    model.load_state_dict(torch.load(ckpt_path))\n    BS = 32\n    NW = 8\n    MAX_LEN = 64\n    test_df[\"pct_rank\"] = 0\n    test_fts = get_features(test_df)\n    test_ds = MarkdownDataset(test_df[test_df[\"cell_type\"] == \"markdown\"].reset_index(drop=True), md_max_len=64,total_max_len=512, model_name_or_path=model_path, fts=test_fts)\n    test_loader = DataLoader(test_ds, batch_size=BS, shuffle=False, num_workers=NW,\n                              pin_memory=False, drop_last=False)\n    _, y_test = validate(model, test_loader)\n    model.to(torch.device('cpu'))\n    torch.cuda.empty_cache()    \n    del model, test_loader, test_ds\n    gc.collect()      \n    \n    test_df.loc[test_df[\"cell_type\"] == \"markdown\", \"pred\"] = y_test\n    sub_df = test_df.sort_values(\"pred\").groupby(\"id\")[\"cell_id\"].apply(lambda x: \" \".join(x)).reset_index()\n    sub_df.rename(columns={\"cell_id\": \"cell_order\"}, inplace=True)\n    sub_df.head()\n    sub_df.to_csv(f\"submission_{notebook_version}.csv\", index=False)\n\n    del test_df, paths_test, notebooks_test, test_fts, model_path, ckpt_path, sub_df\n    del json, np, pd, tqdm, Path, sparse, torch, sys, os, nn, F, AutoModel, AutoTokenizer\n    gc.collect()\n    ","metadata":{"papermill":{"duration":0.027037,"end_time":"2022-05-23T03:29:16.100816","exception":false,"start_time":"2022-05-23T03:29:16.073779","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-07-27T06:05:56.733575Z","iopub.execute_input":"2022-07-27T06:05:56.733956Z","iopub.status.idle":"2022-07-27T06:05:56.778422Z","shell.execute_reply.started":"2022-07-27T06:05:56.733926Z","shell.execute_reply":"2022-07-27T06:05:56.777409Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import gc\nckpt_path = \"../input/uclanlp-plbart/uclanlp_plbart.bin\"\nmodel_path = \"../input/uclanlp-plbart/\"\n\nfrom tqdm.contrib.concurrent import process_map\nprocess_map(predict_caller, [(model_path, ckpt_path,\"plbart\")])[0]\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2022-07-27T06:05:56.782381Z","iopub.execute_input":"2022-07-27T06:05:56.783172Z","iopub.status.idle":"2022-07-27T06:06:27.571656Z","shell.execute_reply.started":"2022-07-27T06:05:56.783132Z","shell.execute_reply":"2022-07-27T06:06:27.570644Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import gc\nckpt_path = \"../input/codebert-base-aicode/codebert-base.bin\"\nmodel_path = \"../input/codebert-base/codebert-base/\"\n\nfrom tqdm.contrib.concurrent import process_map\nprocess_map(predict_caller, [(model_path, ckpt_path,1)])[0]\ngc.collect()","metadata":{"papermill":{"duration":20.374059,"end_time":"2022-05-23T03:29:36.523092","exception":false,"start_time":"2022-05-23T03:29:16.149033","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-07-27T06:06:27.573212Z","iopub.execute_input":"2022-07-27T06:06:27.573589Z","iopub.status.idle":"2022-07-27T06:06:46.801068Z","shell.execute_reply.started":"2022-07-27T06:06:27.57355Z","shell.execute_reply":"2022-07-27T06:06:46.800262Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import json\nimport numpy as np\nimport pandas as pd\nfrom tqdm import tqdm\nfrom pathlib import Path\nfrom scipy import sparse\n\npd.options.display.width = 180\npd.options.display.max_colwidth = 120\n\nBERT_PATH = \"../input/huggingface-bert-variants/distilbert-base-uncased/distilbert-base-uncased\"\n\ndata_dir = Path('../input/AI4Code')\nNUM_TRAIN = 200\n\ndef read_notebook(path):\n    return (\n        pd.read_json(\n            path,\n            dtype={'cell_type': 'category', 'source': 'str'})\n        .assign(id=path.stem)\n        .rename_axis('cell_id')\n    )\n\npaths_train = list((data_dir / 'train').glob('*.json'))[:NUM_TRAIN]\nnotebooks_train = [\n    read_notebook(path) for path in tqdm(paths_train, desc='Train NBs')\n]\ndf = (\n    pd.concat(notebooks_train)\n    .set_index('id', append=True)\n    .swaplevel()\n    .sort_index(level='id', sort_remaining=False)\n)\n\ndf","metadata":{"papermill":{"duration":0.017111,"end_time":"2022-05-23T03:29:36.734096","exception":false,"start_time":"2022-05-23T03:29:36.716985","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-07-27T06:06:46.803257Z","iopub.execute_input":"2022-07-27T06:06:46.80371Z","iopub.status.idle":"2022-07-27T06:06:51.264235Z","shell.execute_reply.started":"2022-07-27T06:06:46.803671Z","shell.execute_reply":"2022-07-27T06:06:51.263271Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Get an example notebook\nnb_id = df.index.unique('id')[6]\nprint('Notebook:', nb_id)\n\nprint(\"The disordered notebook:\")\nnb = df.loc[nb_id, :]\ndisplay(nb)\nprint()","metadata":{"execution":{"iopub.status.busy":"2022-07-27T06:06:51.265982Z","iopub.execute_input":"2022-07-27T06:06:51.266277Z","iopub.status.idle":"2022-07-27T06:06:51.285152Z","shell.execute_reply.started":"2022-07-27T06:06:51.266236Z","shell.execute_reply":"2022-07-27T06:06:51.284393Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\ndf_orders = pd.read_csv(\n    data_dir / 'train_orders.csv',\n    index_col='id',\n    squeeze=True,\n).str.split()  # Split the string representation of cell_ids into a list\n\ndf_orders","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-07-27T06:06:51.286307Z","iopub.execute_input":"2022-07-27T06:06:51.287024Z","iopub.status.idle":"2022-07-27T06:06:53.729344Z","shell.execute_reply.started":"2022-07-27T06:06:51.286985Z","shell.execute_reply":"2022-07-27T06:06:53.72845Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"len(df_orders.loc[\"002ba502bdac45\"])","metadata":{"execution":{"iopub.status.busy":"2022-07-27T06:06:53.730838Z","iopub.execute_input":"2022-07-27T06:06:53.731218Z","iopub.status.idle":"2022-07-27T06:06:53.740095Z","shell.execute_reply.started":"2022-07-27T06:06:53.731167Z","shell.execute_reply":"2022-07-27T06:06:53.739323Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cell_order = df_orders.loc[nb_id]\n\nprint(\"The ordered notebook:\")\nnb.loc[cell_order, :]","metadata":{"execution":{"iopub.status.busy":"2022-07-27T06:06:53.741311Z","iopub.execute_input":"2022-07-27T06:06:53.741837Z","iopub.status.idle":"2022-07-27T06:06:53.760243Z","shell.execute_reply.started":"2022-07-27T06:06:53.741795Z","shell.execute_reply":"2022-07-27T06:06:53.759486Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_ranks(base, derived):\n    return [base.index(d) for d in derived]\n\ncell_ranks = get_ranks(cell_order, list(nb.index))\nnb.insert(0, 'rank', cell_ranks)\n\nnb","metadata":{"execution":{"iopub.status.busy":"2022-07-27T06:06:53.763271Z","iopub.execute_input":"2022-07-27T06:06:53.764334Z","iopub.status.idle":"2022-07-27T06:06:53.780502Z","shell.execute_reply.started":"2022-07-27T06:06:53.764293Z","shell.execute_reply":"2022-07-27T06:06:53.779678Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_orders_ = df_orders.to_frame().join(\n    df.reset_index('cell_id').groupby('id')['cell_id'].apply(list),\n    how='right',\n)\n\nranks = {}\nfor id_, cell_order, cell_id in df_orders_.itertuples():\n    ranks[id_] = {'cell_id': cell_id, 'rank': get_ranks(cell_order, cell_id)}\n\ndf_ranks = (\n    pd.DataFrame\n    .from_dict(ranks, orient='index')\n    .rename_axis('id')\n    .apply(pd.Series.explode)\n    .set_index('cell_id', append=True)\n)\n\ndf_ranks","metadata":{"execution":{"iopub.status.busy":"2022-07-27T06:06:53.781468Z","iopub.execute_input":"2022-07-27T06:06:53.781694Z","iopub.status.idle":"2022-07-27T06:06:53.880878Z","shell.execute_reply.started":"2022-07-27T06:06:53.781668Z","shell.execute_reply":"2022-07-27T06:06:53.879888Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_ancestors = pd.read_csv(data_dir / 'train_ancestors.csv', index_col='id')\ndf_ancestors","metadata":{"execution":{"iopub.status.busy":"2022-07-27T06:06:53.882327Z","iopub.execute_input":"2022-07-27T06:06:53.882685Z","iopub.status.idle":"2022-07-27T06:06:54.108232Z","shell.execute_reply.started":"2022-07-27T06:06:53.882645Z","shell.execute_reply":"2022-07-27T06:06:54.107386Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df = df.reset_index().merge(df_ranks, on=[\"id\", \"cell_id\"]).merge(df_ancestors, on=[\"id\"])\ndf","metadata":{"execution":{"iopub.status.busy":"2022-07-27T06:06:54.109535Z","iopub.execute_input":"2022-07-27T06:06:54.109899Z","iopub.status.idle":"2022-07-27T06:06:54.18704Z","shell.execute_reply.started":"2022-07-27T06:06:54.109858Z","shell.execute_reply":"2022-07-27T06:06:54.186231Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df[\"pct_rank\"] = df[\"rank\"] / df.groupby(\"id\")[\"cell_id\"].transform(\"count\")\ndf[\"pct_rank\"].hist(bins=10)","metadata":{"execution":{"iopub.status.busy":"2022-07-27T06:06:54.188277Z","iopub.execute_input":"2022-07-27T06:06:54.188951Z","iopub.status.idle":"2022-07-27T06:06:54.442137Z","shell.execute_reply.started":"2022-07-27T06:06:54.188904Z","shell.execute_reply":"2022-07-27T06:06:54.441364Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dict_cellid_source = dict(zip(df['cell_id'].values, df['source'].values))\nimport os\nimport re\nimport numpy as np\nimport pandas as pd\nfrom tqdm import tqdm\nfrom pathlib import Path\nimport matplotlib.pyplot as plt\nfrom sklearn.metrics.pairwise import cosine_similarity\nfrom nltk.stem import WordNetLemmatizer\nimport nltk; nltk.download('wordnet')\n\nstemmer = WordNetLemmatizer()\n\ndef preprocess_text(document):\n        # Remove all the special characters\n        document = re.sub(r'\\W', ' ', str(document))\n\n        # remove all single characters\n        document = re.sub(r'\\s+[a-zA-Z]\\s+', ' ', document)\n\n        # Remove single characters from the start\n        document = re.sub(r'\\^[a-zA-Z]\\s+', ' ', document)\n\n        # Substituting multiple spaces with single space\n        document = re.sub(r'\\s+', ' ', document, flags=re.I)\n\n        # Removing prefixed 'b'\n        document = re.sub(r'^b\\s+', '', document)\n\n        # Converting to Lowercase\n        document = document.lower()\n        #return document\n\n        # Lemmatization\n        tokens = document.split()\n        tokens = [stemmer.lemmatize(word) for word in tokens]\n        tokens = [word for word in tokens if len(word) > 3]\n\n        preprocessed_text = ' '.join(tokens)\n        return preprocessed_text\n\n    \ndef preprocess_df(df):\n    \"\"\"\n    This function is for processing sorce of notebook\n    returns preprocessed dataframe\n    \"\"\"\n    return [preprocess_text(message) for message in df.source]\n\ndf.source = df.source.apply(preprocess_text)","metadata":{"execution":{"iopub.status.busy":"2022-07-27T06:06:54.443692Z","iopub.execute_input":"2022-07-27T06:06:54.443961Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.model_selection import GroupShuffleSplit\n\nNVALID = 0.1  # size of validation set\n\nsplitter = GroupShuffleSplit(n_splits=1, test_size=NVALID, random_state=0)\n\ntrain_ind, val_ind = next(splitter.split(df, groups=df[\"ancestor_id\"]))\n\ntrain_df = df.loc[train_ind].reset_index(drop=True)\nval_df = df.loc[val_ind].reset_index(drop=True)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from tqdm.notebook import tqdm\n\ndef generate_triplet(df, mode='train'):\n  triplets = []\n  ids = df.id.unique()\n  random_drop = np.random.random(size=10000)>0.9\n  count = 0\n\n  for id, df_tmp in tqdm(df.groupby('id')):\n    df_tmp_markdown = df_tmp[df_tmp['cell_type']=='markdown']\n\n    df_tmp_code = df_tmp[df_tmp['cell_type']=='code']\n    df_tmp_code_rank = df_tmp_code['rank'].values\n    df_tmp_code_cell_id = df_tmp_code['cell_id'].values\n\n    for cell_id, rank in df_tmp_markdown[['cell_id', 'rank']].values:\n      labels = np.array([(r==(rank+1)) for r in df_tmp_code_rank]).astype('int')\n\n      for cid, label in zip(df_tmp_code_cell_id, labels):\n        count += 1\n        if label==1:\n          triplets.append( [cell_id, cid, label] )\n          # triplets.append( [cid, cell_id, label] )\n        elif mode == 'test':\n          triplets.append( [cell_id, cid, label] )\n          # triplets.append( [cid, cell_id, label] )\n        elif random_drop[count%10000]:\n          triplets.append( [cell_id, cid, label] )\n          # triplets.append( [cid, cell_id, label] )\n    \n  return triplets\n\ntriplets = generate_triplet(train_df)\nval_triplets = generate_triplet(val_df, mode = 'test')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"val_df.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from bisect import bisect\n\n\ndef count_inversions(a):\n    inversions = 0\n    sorted_so_far = []\n    for i, u in enumerate(a):\n        j = bisect(sorted_so_far, u)\n        inversions += i - j\n        sorted_so_far.insert(j, u)\n    return inversions\n\n\ndef kendall_tau(ground_truth, predictions):\n    total_inversions = 0\n    total_2max = 0  # twice the maximum possible inversions across all instances\n    for gt, pred in zip(ground_truth, predictions):\n        ranks = [gt.index(x) for x in pred]  # rank predicted order in terms of ground truth\n        total_inversions += count_inversions(ranks)\n        n = len(gt)\n        total_2max += n * (n - 1)\n    return 1 - 4 * total_inversions / total_2max","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from transformers import RobertaTokenizer, RobertaConfig, RobertaModel\nimport torch.nn.functional as F\nimport torch.nn as nn\nimport torch\nfrom transformers import AutoModelWithLMHead, AutoTokenizer, AutoModel\n\nMAX_LEN = 128\n\n    \nclass MarkdownModel(nn.Module):\n    def __init__(self):\n        super(MarkdownModel, self).__init__()\n        self.distill_bert = AutoModel.from_pretrained(\"../input/mymodelpairbertsmallpretrained/models/checkpoint-18000\")\n        self.top = nn.Linear(512, 1)\n\n        self.dropout = nn.Dropout(0.2)\n        \n    def forward(self, ids, mask):\n        x = self.distill_bert(ids, mask)[0]\n        x = self.dropout(x)\n        x = self.top(x[:, 0, :])\n        x = torch.sigmoid(x) \n        return x","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from torch.utils.data import DataLoader, Dataset\n\n\n\nclass MarkdownDataset(Dataset):\n    \n    def __init__(self, df, max_len, mode='train'):\n        super().__init__()\n        self.df = df\n        self.max_len = max_len\n        self.tokenizer = AutoTokenizer.from_pretrained(\"../input/mymodelpairbertsmallpretrained/my_own_tokenizer\", do_lower_case=True)\n        self.mode=mode\n\n    def __getitem__(self, index):\n        row = self.df[index]\n\n        label = row[-1]\n\n        txt = dict_cellid_source[row[0]] + '[SEP]' + dict_cellid_source[row[1]]\n\n        inputs = self.tokenizer.encode_plus(\n            txt,\n            None,\n            add_special_tokens=True,\n            max_length=self.max_len,\n            padding=\"max_length\",\n            return_token_type_ids=True,\n            truncation=True\n        )\n        ids = torch.LongTensor(inputs['input_ids'])\n        mask = torch.LongTensor(inputs['attention_mask'])\n\n        return ids, mask, torch.FloatTensor([label])\n\n    def __len__(self):\n        return len(self.df)\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def adjust_lr(optimizer, epoch):\n    if epoch < 1:\n        lr = 5e-5\n    elif epoch < 2:\n        lr = 1e-3\n    elif epoch < 5:\n        lr = 1e-4\n    else:\n        lr = 1e-5\n\n    for p in optimizer.param_groups:\n        p['lr'] = lr\n    return lr\n    \ndef get_optimizer(net):\n    optimizer = torch.optim.Adam(filter(lambda p: p.requires_grad, net.parameters()), lr=3e-4, betas=(0.9, 0.999), eps=1e-08)\n    return optimizer\n\nBS = 128\nNW = 8","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def read_data(data):\n    return tuple(d.cuda() for d in data[:-1]), data[-1].cuda()\n\ndef validate(model, val_loader, mode='train'):\n    model.eval()\n    \n    tbar = tqdm(val_loader, file=sys.stdout)\n    \n    preds = np.zeros(len(val_loader.dataset), dtype='float32')\n    labels = []\n    count = 0\n\n    with torch.no_grad():\n        for idx, data in enumerate(tbar):\n            inputs, target = read_data(data)\n\n            pred = model(inputs[0], inputs[1]).detach().cpu().numpy().ravel()\n\n            preds[count:count+len(pred)] = pred\n            count += len(pred)\n            \n            if mode=='test':\n              labels.append(target.detach().cpu().numpy().ravel())\n    if mode=='test':\n      return preds\n    else:\n      return np.concatenate(labels), np.concatenate(preds)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"paths_test = list((data_dir / 'test').glob('*.json'))\nnotebooks_test = [\n    read_notebook(path) for path in tqdm(paths_test, desc='Test NBs')\n]\ntest_df = (\n    pd.concat(notebooks_test)\n    .set_index('id', append=True)\n    .swaplevel()\n    .sort_index(level='id', sort_remaining=False)\n).reset_index()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_df.source = test_df.source.apply(preprocess_text)\ndict_cellid_source = dict(zip(test_df['cell_id'].values, test_df['source'].values))\ntest_df[\"rank\"] = test_df.groupby([\"id\", \"cell_type\"]).cumcount()\ntest_df[\"pred\"] = test_df.groupby([\"id\", \"cell_type\"])[\"rank\"].rank(pct=False)\ntest_triplets = generate_triplet(test_df, mode = 'test')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_df[\"pct_rank\"] = 0\ntest_ds = MarkdownDataset(test_triplets, max_len=MAX_LEN)\ntest_loader = DataLoader(test_ds, batch_size=BS * 4, shuffle=False, num_workers=NW, pin_memory=False, drop_last=False)\n\nimport gc \ngc.collect()\nlen(test_ds), test_ds[0]","metadata":{"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import sys \n\nmodel = MarkdownModel()\nmodel = model.cuda()\nmodel.load_state_dict(torch.load('../input/mymodelbertsmallpretrained120000/my_own_model.bin'))\ny_test = validate(model, test_loader, mode='test')","metadata":{"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"preds_copy = y_test\npred_vals = []\ncount = 0\nfor id, df_tmp in tqdm(test_df.groupby('id')):\n  df_tmp_mark = df_tmp[df_tmp['cell_type']=='markdown']\n  df_tmp_code = df_tmp[df_tmp['cell_type']!='markdown']\n  df_tmp_code_rank = df_tmp_code['rank'].rank().values\n  N_code = len(df_tmp_code_rank)\n  N_mark = len(df_tmp_mark)\n\n  preds_tmp = preds_copy[count:count+N_mark * N_code]\n\n  count += N_mark * N_code\n\n  for i in range(N_mark):\n    pred = preds_tmp[i*N_code:i*N_code+N_code] \n\n    softmax = np.exp((pred-np.mean(pred)) *20)/np.sum(np.exp((pred-np.mean(pred)) *20)) \n\n    rank = np.sum(softmax * df_tmp_code_rank)\n    pred_vals.append(rank)\n\ndel model\ndel test_triplets[:]\ndel dict_cellid_source\ngc.collect()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_df.loc[test_df[\"cell_type\"] == \"markdown\", \"pred\"] = pred_vals\nsub_df = test_df.sort_values(\"pred\").groupby(\"id\")[\"cell_id\"].apply(lambda x: \" \".join(x)).reset_index()\nsub_df.rename(columns={\"cell_id\": \"cell_order\"}, inplace=True)\nsub_df.to_csv(\"submission_2.csv\", index=False)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import json\nfrom pathlib import Path\nimport random\nimport os\nimport sys\n\nimport numpy as np\nimport pandas as pd\nfrom scipy import sparse\nfrom tqdm import tqdm\n\nfrom transformers import AutoModel, AutoTokenizer\nfrom torch.utils.data import DataLoader, Dataset\nimport torch.nn.functional as F\nimport torch.nn as nn\nimport torch\n\ntotal_max_len = 512\ndata_dir = Path('../input/AI4Code')\nmodel_path = \"../input/aicode-graphcodebert\"\nckpt_path = \"../input/aicode-graphcodebert/graphcodebert.bin\"\ntokenizer_path = \"../input/aicode-graphcodebert\"","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def seed_everything(seed=42):\n    random.seed(seed)\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    np.random.seed(seed)\n    torch.manual_seed(seed)\n    torch.cuda.manual_seed(seed)\n    torch.backends.cudnn.deterministic = True\n    \nseed_everything(42)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def read_notebook(path):\n    return (\n        pd.read_json(\n            path,\n            dtype={'cell_type': 'category', 'source': 'str'})\n        .assign(id=path.stem)\n        .rename_axis('cell_id')\n    )\n\npaths_test = list((data_dir / 'test').glob('*.json'))\nnotebooks_test = [\n    read_notebook(path) for path in tqdm(paths_test, desc='Test NBs')\n]\ntest_df = (\n    pd.concat(notebooks_test)\n    .set_index('id', append=True)\n    .swaplevel()\n    .sort_index(level='id', sort_remaining=False)\n).reset_index()\ntest_df[\"rank\"] = test_df.groupby([\"id\", \"cell_type\"]).cumcount()\ntest_df[\"pred\"] = test_df.groupby([\"id\", \"cell_type\"])[\"rank\"].rank(pct=True)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Additional code cells\ndef clean_code(cell):\n    return str(cell).replace(\"\\\\n\", \"\\n\")\n\n\ndef sample_cells(cells, n):\n    cells = [clean_code(cell) for cell in cells]\n    if n >= len(cells):\n        return [cell[:200] for cell in cells]\n    else:\n        results = []\n        step = len(cells) / n\n        idx = 0\n        while int(np.round(idx)) < len(cells):\n            results.append(cells[int(np.round(idx))])\n            idx += step\n        assert cells[0] in results\n        if cells[-1] not in results:\n            results[-1] = cells[-1]\n        return results\n\n\ndef get_features(df):\n    features = dict()\n    df = df.sort_values(\"rank\").reset_index(drop=True)\n    for idx, sub_df in tqdm(df.groupby(\"id\")):\n        features[idx] = dict()\n        total_md = sub_df[sub_df.cell_type == \"markdown\"].shape[0]\n        code_sub_df = sub_df[sub_df.cell_type == \"code\"]\n        total_code = code_sub_df.shape[0]\n        codes = sample_cells(code_sub_df.source.values, 20)\n        features[idx][\"total_code\"] = total_code\n        features[idx][\"total_md\"] = total_md\n        features[idx][\"codes\"] = codes\n    return features","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_fts = get_features(test_df)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class MarkdownModel(nn.Module):\n    def __init__(self, model_path):\n        super(MarkdownModel, self).__init__()\n        self.model = AutoModel.from_pretrained(model_path)\n        self.top = nn.Linear(769, 1)\n        \n    def forward(self, ids, mask, fts):\n        x = self.model(ids, mask)[0]\n        x = self.top(torch.cat((x[:, 0, :], fts),1))\n        return x\n\nclass MarkdownDataset(Dataset):\n\n    def __init__(self, df, tokenizer_path, total_max_len, md_max_len, fts):\n        super().__init__()\n        self.df = df.reset_index(drop=True)\n        self.md_max_len = md_max_len\n        self.total_max_len = total_max_len  # maxlen allowed by model config\n        self.tokenizer = AutoTokenizer.from_pretrained(tokenizer_path)\n        self.fts = fts\n\n    def __getitem__(self, index):\n        row = self.df.iloc[index]\n\n        inputs = self.tokenizer.encode_plus(\n            row.source,\n            None,\n            add_special_tokens=True,\n            max_length=self.md_max_len,\n            padding=\"max_length\",\n            return_token_type_ids=True,\n            truncation=True\n        )\n        code_inputs = self.tokenizer.batch_encode_plus(\n            [str(x) for x in self.fts[row.id][\"codes\"]],\n            add_special_tokens=True,\n            max_length=23,\n            padding=\"max_length\",\n            truncation=True\n        )\n        n_md = self.fts[row.id][\"total_md\"]\n        n_code = self.fts[row.id][\"total_code\"]\n        if n_md + n_code == 0:\n            fts = torch.FloatTensor([0])\n        else:\n            fts = torch.FloatTensor([n_md / (n_md + n_code)])\n\n        ids = inputs['input_ids']\n        for x in code_inputs['input_ids']:\n            ids.extend(x[:-1])\n        ids = ids[:self.total_max_len]\n        if len(ids) != self.total_max_len:\n            ids = ids + [self.tokenizer.pad_token_id, ] * (self.total_max_len - len(ids))\n        ids = torch.LongTensor(ids)\n\n        mask = inputs['attention_mask']\n        for x in code_inputs['attention_mask']:\n            mask.extend(x[:-1])\n        mask = mask[:self.total_max_len]\n        if len(mask) != self.total_max_len:\n            mask = mask + [self.tokenizer.pad_token_id, ] * (self.total_max_len - len(mask))\n        mask = torch.LongTensor(mask)\n\n        assert len(ids) == self.total_max_len\n\n        return ids, mask, fts, torch.FloatTensor([row.pct_rank])\n\n    def __len__(self):\n        return self.df.shape[0]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def read_data(data):\n    return tuple(d.cuda() for d in data[:-1]), data[-1].cuda()\n\n\ndef validate(model, val_loader):\n    model.eval()\n    \n    tbar = tqdm(val_loader, file=sys.stdout)\n    \n    preds = []\n    labels = []\n\n    with torch.no_grad():\n        for idx, data in enumerate(tbar):\n            inputs, target = read_data(data)\n\n            pred = model(*inputs)\n\n            preds.append(pred.detach().cpu().numpy().ravel())\n            labels.append(target.detach().cpu().numpy().ravel())\n    \n    return np.concatenate(labels), np.concatenate(preds)\n\ndef predict(model_path, ckpt_path, tokenizer_path):\n    model = MarkdownModel(model_path)\n    model = model.to('cuda:0')\n    model.eval()\n    model.load_state_dict(torch.load(ckpt_path, map_location='cuda:0'))\n    BS = 32\n    NW = 8\n    MAX_LEN = 64\n    test_df[\"pct_rank\"] = 0\n    test_ds = MarkdownDataset(test_df[test_df[\"cell_type\"] == \"markdown\"].reset_index(drop=True), \n                              md_max_len=64,\n                              tokenizer_path = tokenizer_path,\n                              total_max_len = total_max_len,\n                              fts=test_fts)\n    test_loader = DataLoader(test_ds, batch_size=BS, \n                             shuffle=False,\n                             num_workers=NW,\n                             pin_memory=False,\n                             drop_last=False)\n    _, y_test = validate(model, test_loader)\n    return y_test","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_test = predict(model_path, ckpt_path, tokenizer_path)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_df.loc[test_df[\"cell_type\"] == \"markdown\", \"pred\"] = y_test","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sub_df = test_df.sort_values(\"pred\").groupby(\"id\")[\"cell_id\"].apply(lambda x: \" \".join(x)).reset_index()\nsub_df.rename(columns={\"cell_id\": \"cell_order\"}, inplace=True)\nsub_df.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sub_df.to_csv(\"submission_3.csv\", index=False)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Rank Ensemble\nAnd now for the moment we have all been waiting for: **Ensemling rank based submissions.**\n\nBut how are we going to do this?\n\n- Let's say that we have two different submissions: \"submission_1.csv\", \"submission_2.csv\". Each containing a list of sorted strings per row.\n- We would like to create a new submission such that each row contains a sorted list that is an aggregation of the sorted list in the same row of both submissions.\n- To do this, we simply ensemble the indices. The index is nothing but a rank of a particular string. From the highest likelyhood of the string being in it's expected package to the lowest.\n- Then sort the strings by their ensembled index.","metadata":{}},{"cell_type":"code","source":"df_1 = pd.read_csv('submission_2.csv')\ndf_2 = pd.read_csv('submission_1.csv')\ndf_3 = pd.read_csv('submission_3.csv')\ndf_4 = pd.read_csv('submission_plbart.csv')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"new_samples = []\nfor sample_idx in range(len(df_1)):\n    sample_1 = {k: v for v, k in enumerate(df_1.iloc[sample_idx]['cell_order'].split(' '))}\n    sample_2 = {k: v for v, k in enumerate(df_2.iloc[sample_idx]['cell_order'].split(' '))}\n    sample_3 = {k: v for v, k in enumerate(df_3.iloc[sample_idx]['cell_order'].split(' '))}\n    sample_4 = {k: v for v, k in enumerate(df_4.iloc[sample_idx]['cell_order'].split(' '))}\n    for key in sample_1: sample_1[key] = ( (sample_1[key] * 0.120) + (sample_2[key] * 0.270) + (sample_3[key] * 0.370)+ (sample_4[key] * 0.240))\n    new_samples.append(' '.join([i[0] for i in list(sorted(sample_1.items(), key=lambda x:x[1]))]))\ndf_1['cell_order'] = new_samples","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_1.to_csv('submission.csv', index = False)\ndf_1","metadata":{"trusted":true},"execution_count":null,"outputs":[]}]}