{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"\n\n

- Data Understanding & Language Analysis -

\n\n> ๐Ÿ“œ **Goal**: Predict the correct ordering of the **cells** within a Jupyter Notebook.\n\n**What is a ๐Ÿฆ  cell** - notice here that a **cell** can be both:\n* a `coding` cell - where you write code\n* a `markdown` cell - where you can write text, add images etc.\n\n**โ— Important** - within the `.json` filesthe `code` cells are in their *correct* order - only the `markdown` cells have been *shuffled*.\n\nOh boy, this competition sounds like FUN. I owe MOST of what I've learned as Data Scientist to notebooks. I never was an \"only code\" person, so discovering that you can explain, comment, add images and schemas to your code in order to make it more *readable* was a life changer for me.\n\nAlso, notebooks can have a \"learning-teaching\" experience that a raw `.py` file just ... doesn't cut it for me.\n\nSo let's get started!\n\n### โฌ‡ Libraries","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"}},{"cell_type":"code","source":"!pip install spacy-language-detection","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-07-12T10:23:29.413717Z","iopub.execute_input":"2022-07-12T10:23:29.414256Z","iopub.status.idle":"2022-07-12T10:23:58.154736Z","shell.execute_reply.started":"2022-07-12T10:23:29.414134Z","shell.execute_reply":"2022-07-12T10:23:58.153453Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Libraries\nimport os\nimport gc\nimport wandb\nfrom time import time\nimport random\nimport math\nimport glob\nimport json\nfrom bisect import bisect\nfrom scipy.sparse import vstack\nfrom tqdm import tqdm\nimport warnings\nimport pandas as pd\nimport numpy as np\nimport seaborn as sns\nimport matplotlib as mpl\nfrom matplotlib import cm\nimport matplotlib.patches as patches\nimport matplotlib.pyplot as plt\nimport matplotlib.image as mpimg\nfrom matplotlib.offsetbox import AnnotationBbox, OffsetImage\nfrom matplotlib.colors import ListedColormap, LinearSegmentedColormap\nfrom matplotlib.patches import Rectangle\nfrom IPython.display import display_html\nplt.rcParams.update({'font.size': 16})\n\n# Spacy Language Detector\nimport spacy\nfrom spacy.language import Language\nfrom spacy_language_detection import LanguageDetector\n\n# Environment check\nwarnings.filterwarnings(\"ignore\")\nos.environ[\"WANDB_SILENT\"] = \"true\"\nCONFIG = {'competition': 'AI4Code', '_wandb_kernel': 'aot'}\n\n# Custom colors\nclass clr:\n S = '\\033[1m' + '\\033[93m'\n E = '\\033[0m'\n \nmy_colors = [\"#CDFC74\", \"#F3EA56\", \"#EBB43D\", \n \"#DF7D27\", \"#D14417\", \"#B80A0A\", \"#9C0042\"]\nmy_pastels = [\"#A5EC9B\", \"#B4E185\", \"#C3D973\", \n \"#CDCD61\", \"#CCB049\", \"#CB812D\", \"#B93221\"]\nmy_darks = [\"#FCF238\", \"#F19321\", \"#E54F14\", \n \"#C22318\", \"#B01028\", \"#9D0642\", \"#85006C\"]\n\ngradient1 = [\"#a5ec9b\", \"#abe890\", \"#b0e485\", \"#b7e07b\", \"#bddb71\", \n \"#c3d667\", \"#cad15e\", \"#d1cc55\", \"#d7c64e\", \"#dec147\", \"#e5ba41\", \"#ebb43d\"]\nCMAP1 = ListedColormap(my_colors)\nCMAP2 = ListedColormap(my_colors[-1])\n\nprint(clr.S+\"Notebook Color Schemes:\"+clr.E)\nsns.palplot(sns.color_palette(my_pastels))\nsns.palplot(sns.color_palette(my_colors))\nsns.palplot(sns.color_palette(my_darks))\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-12T10:23:58.158408Z","iopub.execute_input":"2022-07-12T10:23:58.159239Z","iopub.status.idle":"2022-07-12T10:24:09.193474Z","shell.execute_reply.started":"2022-07-12T10:23:58.159194Z","shell.execute_reply":"2022-07-12T10:24:09.192786Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### ๐Ÿ W&B Fork & Run\n\nIn order to run this notebook you will need to input your own **secret API key** within the `! wandb login $secret_value_0` line. \n\n๐Ÿ**How do you get your own API key?**\n\nSuper simple! Go to **https://wandb.ai/site** -> Login -> Click on your profile in the top right corner -> Settings -> Scroll down to API keys -> copy your very own key (for more info check [this amazing notebook for ML Experiment Tracking on Kaggle](https://www.kaggle.com/ayuraj/experiment-tracking-with-weights-and-biases)).\n\n
","metadata":{}},{"cell_type":"code","source":"# ๐Ÿ Secrets\nfrom kaggle_secrets import UserSecretsClient\nuser_secrets = UserSecretsClient()\nsecret_value_0 = user_secrets.get_secret(\"wandb\")\n\n! wandb login $secret_value_0","metadata":{"execution":{"iopub.status.busy":"2022-07-12T10:24:09.195044Z","iopub.execute_input":"2022-07-12T10:24:09.197370Z","iopub.status.idle":"2022-07-12T10:24:12.044748Z","shell.execute_reply.started":"2022-07-12T10:24:09.197264Z","shell.execute_reply":"2022-07-12T10:24:12.042447Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### โฌ‡ Helper Functions","metadata":{}},{"cell_type":"code","source":"def count_inversions(a):\n '''src: https://www.kaggle.com/code/ryanholbrook/getting-started-with-ai4code'''\n inversions = 0\n sorted_so_far = []\n for i, u in enumerate(a):\n j = bisect(sorted_so_far, u)\n inversions += i - j\n sorted_so_far.insert(j, u)\n return inversions\n\n\ndef kendall_tau(ground_truth, predictions):\n '''src: https://www.kaggle.com/code/ryanholbrook/getting-started-with-ai4code'''\n total_inversions = 0\n total_2max = 0 # twice the maximum possible inversions across all instances\n for gt, pred in zip(ground_truth, predictions):\n ranks = [gt.index(x) for x in pred] # rank predicted order in terms of ground truth\n total_inversions += count_inversions(ranks)\n n = len(gt)\n total_2max += n * (n - 1)\n return 1 - 4 * total_inversions / total_2max\n\n\ndef show_values_on_bars(axs, h_v=\"v\", space=0.4):\n '''Plots the value at the end of the a seaborn barplot.\n axs: the ax of the plot\n h_v: weather or not the barplot is vertical/ horizontal'''\n \n def _show_on_single_plot(ax):\n if h_v == \"v\":\n for p in ax.patches:\n _x = p.get_x() + p.get_width() / 2\n _y = p.get_y() + p.get_height()\n value = int(p.get_height())\n ax.text(_x, _y, format(value, ','), ha=\"center\") \n elif h_v == \"h\":\n for p in ax.patches:\n _x = p.get_x() + p.get_width() + float(space)\n _y = p.get_y() + p.get_height()\n value = int(p.get_width())\n ax.text(_x, _y, format(value, ','), ha=\"left\")\n\n if isinstance(axs, np.ndarray):\n for idx, ax in np.ndenumerate(axs):\n _show_on_single_plot(ax)\n else:\n _show_on_single_plot(axs)\n \n \n# === ๐Ÿ W&B ===\ndef save_dataset_artifact(run_name, artifact_name, path):\n '''Saves dataset to W&B Artifactory.\n run_name: name of the experiment\n artifact_name: under what name should the dataset be stored\n path: path to the dataset'''\n \n run = wandb.init(project='AI4Code', \n name=run_name, \n config=CONFIG)\n artifact = wandb.Artifact(name=artifact_name, \n type='dataset')\n artifact.add_file(path)\n\n wandb.log_artifact(artifact)\n wandb.finish()\n print(\"Artifact has been saved successfully.\")\n \n \ndef create_wandb_plot(x_data=None, y_data=None, x_name=None, y_name=None, title=None, log=None, plot=\"line\"):\n '''Create and save lineplot/barplot in W&B Environment.\n x_data & y_data: Pandas Series containing x & y data\n x_name & y_name: strings containing axis names\n title: title of the graph\n log: string containing name of log'''\n \n data = [[label, val] for (label, val) in zip(x_data, y_data)]\n table = wandb.Table(data=data, columns = [x_name, y_name])\n \n if plot == \"line\":\n wandb.log({log : wandb.plot.line(table, x_name, y_name, title=title)})\n elif plot == \"bar\":\n wandb.log({log : wandb.plot.bar(table, x_name, y_name, title=title)})\n elif plot == \"scatter\":\n wandb.log({log : wandb.plot.scatter(table, x_name, y_name, title=title)})\n \n \ndef create_wandb_hist(x_data=None, x_name=None, title=None, log=None):\n '''Create and save histogram in W&B Environment.\n x_data: Pandas Series containing x values\n x_name: strings containing axis name\n title: title of the graph\n log: string containing name of log'''\n \n data = [[x] for x in x_data]\n table = wandb.Table(data=data, columns=[x_name])\n wandb.log({log : wandb.plot.histogram(table, x_name, title=title)})\n \n \n# ๐Ÿ Log Cover Photo\nrun = wandb.init(project='AI4Code', name='CoverPhoto', config=CONFIG)\ncover = plt.imread(\"../input/ai4code-processed-data/AI4Code Cover.png\")\nwandb.log({\"example\": wandb.Image(cover)})\nwandb.finish()","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true},"execution":{"iopub.status.busy":"2022-07-12T10:24:12.048526Z","iopub.execute_input":"2022-07-12T10:24:12.048908Z","iopub.status.idle":"2022-07-12T10:24:32.238935Z","shell.execute_reply.started":"2022-07-12T10:24:12.048872Z","shell.execute_reply":"2022-07-12T10:24:32.237953Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 1. The Data\n\n## 1.1 The .csv files\n* `train_orders.csv` - TRAIN DATA\n * `id` - unique ID of the notebook\n * `cell_order` - gives the correct order of *each cell* within this notebook\n\n\n* `train_ancestors`\n * `id` - unique ID of the notebook (same as for `train_orders.csv`)\n * `ancestor_id` - notebook that has a common origin or *ancestor* (good as grouping factor when constructing validation splits)\n * `parent_id` - the *original* notebook, which may be present in the train data or not","metadata":{}},{"cell_type":"code","source":"# ๐Ÿ W&B Experiment\nrun = wandb.init(project='AI4Code', name='metadata-explore', config=CONFIG)\n\n# Read in original data\norders = pd.read_csv(\"../input/AI4Code/train_orders.csv\")\nancestors = pd.read_csv(\"../input/AI4Code/train_ancestors.csv\")","metadata":{"execution":{"iopub.status.busy":"2022-07-12T10:24:32.240652Z","iopub.execute_input":"2022-07-12T10:24:32.242066Z","iopub.status.idle":"2022-07-12T10:24:41.135438Z","shell.execute_reply.started":"2022-07-12T10:24:32.242022Z","shell.execute_reply":"2022-07-12T10:24:41.134043Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"wandb.log({\"unique_notebooks\" : orders.id.nunique()})\n\nprint(clr.S+\"~~~~ TRAIN ~~~~\"+clr.E)\nprint(clr.S+\"Orders:\"+clr.E)\nprint(f\"Shape: {orders.shape} with {orders.id.nunique()} unique IDs.\", \"\\n\")\norders.head()","metadata":{"execution":{"iopub.status.busy":"2022-07-12T10:24:41.139058Z","iopub.execute_input":"2022-07-12T10:24:41.139676Z","iopub.status.idle":"2022-07-12T10:24:43.025767Z","shell.execute_reply.started":"2022-07-12T10:24:41.139625Z","shell.execute_reply":"2022-07-12T10:24:43.024586Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"wandb.log({\"unique_ancestors\" : ancestors.ancestor_id.nunique()})\n\nprint(clr.S+\"Ancestors:\"+clr.E)\nprint(f\"Shape: {ancestors.shape} also with {ancestors.id.nunique()} unique IDs.\")\nprint(f\"There are also {ancestors.ancestor_id.nunique()} unique ancestor_id and {ancestors.parent_id.nunique()} unique parent_id.\")\nancestors.head()","metadata":{"execution":{"iopub.status.busy":"2022-07-12T10:24:43.027428Z","iopub.execute_input":"2022-07-12T10:24:43.027832Z","iopub.status.idle":"2022-07-12T10:24:44.891650Z","shell.execute_reply.started":"2022-07-12T10:24:43.027795Z","shell.execute_reply":"2022-07-12T10:24:44.890434Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### ๐Ÿ•ธ Network Analysis on Ancestors","metadata":{}},{"cell_type":"code","source":"# Get frequency per ancestor_id\ndata = ancestors.groupby(\"ancestor_id\")[\"id\"].count().reset_index().\\\n sort_values(\"id\", ascending=False).reset_index(drop=True)\ndata.columns = [\"ancestor_id\", \"count\"]\n\n\n# Basic metrics\ntotal_singles = data[data[\"count\"]==1].shape[0]\ntotal_double_plus = data[data[\"count\"]>1].shape[0]\n\nprint(clr.S+\"Total number of ids with only 1 ancestor:\"+clr.E, total_singles, \"\\n\"+ \"\\t\"*4+\n clr.S+\"percent:\"+clr.E, round(total_singles/len(data), 3), \"\\n\")\nprint(clr.S+\"Total number of ids with 2+ ancestors:\"+clr.E, total_double_plus, \"\\n\"+ \"\\t\"*4+\n clr.S+\"percent:\"+clr.E, round(total_double_plus/len(data), 3))","metadata":{"execution":{"iopub.status.busy":"2022-07-12T10:24:44.893518Z","iopub.execute_input":"2022-07-12T10:24:44.894003Z","iopub.status.idle":"2022-07-12T10:24:46.851699Z","shell.execute_reply.started":"2022-07-12T10:24:44.893961Z","shell.execute_reply":"2022-07-12T10:24:46.850517Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(24, 12))\nfig.suptitle('Analysis on the 6,843 ids that have more than 1 ancestor', \n weight=\"bold\", size=25)\n\n# Violinplot\nsns.violinplot(data=data[data[\"count\"]>1], y=\"count\", ax=ax1, color=my_colors[3])\nax1.set_title(\"Frequency of ancestors for an id\", weight=\"bold\", size=19)\nax1.set_ylabel(\"No. of Ancestors\", size = 18, weight=\"bold\")\nax1.axhline(y=10, linestyle = '--', color=my_pastels[0], lw=4)\nax1.text(x=-0.43, y=8, s=\"most values are < 10\", color=my_darks[3], size=15, weight=\"bold\")\nax1.arrow(x=0.05, y=12, dx=0, dy=45, color=my_pastels[0], lw=4, \n head_width=0.01, head_length=1, linestyle = '--')\nax1.text(x=0.055, y=59, s=\"There are 168 outliers \\n with values > 10\",\n color=my_darks[3], size=15, weight=\"bold\")\n\n# Barplot\nsns.barplot(data=data[data[\"count\"]>1].head(12), x=\"count\", y=\"ancestor_id\", ax=ax2, \n palette=gradient1)\nshow_values_on_bars(axs=ax2, h_v=\"h\", space=0.4)\nax2.set_title(\"Top 12 Ancestor IDs with most connections\", weight=\"bold\", size=19)\nax2.set_ylabel(\"Ancestor ID\", size = 18, weight=\"bold\")\nax2.set_xlabel(\"\")\nax2.set_xticks([])\n\n# Arrow\nstyle = \"Simple, tail_width=5, head_width=16, head_length=23\"\nkw = dict(arrowstyle=style, color=my_darks[3])\narrow = patches.FancyArrowPatch((40, 11.1), (64, 0.7),\n connectionstyle=\"arc3,rad=-.10\", **kw)\nplt.gca().add_patch(arrow)\n\nsns.despine(right=True, top=True);","metadata":{"execution":{"iopub.status.busy":"2022-07-12T10:24:46.854700Z","iopub.execute_input":"2022-07-12T10:24:46.855145Z","iopub.status.idle":"2022-07-12T10:24:48.834257Z","shell.execute_reply.started":"2022-07-12T10:24:46.855112Z","shell.execute_reply":"2022-07-12T10:24:48.833344Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# ๐Ÿ Log plots\nwandb_data = data[data[\"count\"]>1].head(12)\ncreate_wandb_plot(x_data=wandb_data[\"ancestor_id\"],\n y_data=wandb_data[\"count\"],\n x_name=\"Ancestor ID\", \n y_name=\"Frequency\", \n title=\"Top 12 Ancestor IDs with most connections\",\n log=\"top_12\", plot=\"bar\")\n\nwandb_data = data[data[\"count\"]>1]\ncreate_wandb_hist(x_data=wandb_data[\"count\"],\n x_name=\"No. of Ancestors\", \n title=\"Frequency of ancestors for an id\",\n log=\"ancestor_freq\")","metadata":{"execution":{"iopub.status.busy":"2022-07-12T10:24:48.837402Z","iopub.execute_input":"2022-07-12T10:24:48.837742Z","iopub.status.idle":"2022-07-12T10:24:51.623560Z","shell.execute_reply.started":"2022-07-12T10:24:48.837714Z","shell.execute_reply":"2022-07-12T10:24:51.622636Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"random.seed(25)\n\n# Add a fictive y\n# this \"y\" doesn't mean anything, it's just for\n# showcasing purposes\ndata[\"y\"] = [random.randint(0, 100) for i in range(len(data))]\nperc = round(data[data[\"count\"]<=10].shape[0]/len(data), 3)*100\n\nplt.figure(figsize=(24, 10))\nsns.scatterplot(data=data, x=\"count\", y=\"y\", size=\"count\", alpha=0.65, sizes=(100, 7000),\n hue=\"count\", palette=CMAP1)\n\nplt.title(\"Distribution of Ancestor IDs per frequency\", weight=\"bold\", size=25)\nplt.xlabel(\"No. of Ancestors\", size = 18, weight=\"bold\")\nplt.ylabel(\"\")\nplt.yticks([])\n\nplt.axvline(x=10, linestyle = '--', color=\"#757F62\", lw=4)\nplt.text(x=11, y=85, s=f\"{perc}% of the data is here\", color=\"#757F62\", size=17, weight=\"bold\")\nplt.arrow(x=25, y=83, dx=-17, dy=0, color=\"#757F62\", lw=4, \n head_width=1.2, head_length=0.5)\n\nplt.text(x=54, y=50, s=f\"Biggest outlier\", color=my_colors[-1], size=17, weight=\"bold\")\nplt.arrow(x=54, y=48, dx=8, dy=0, color=my_colors[-1], lw=4, \n head_width=1.2, head_length=0.5)\n\nplt.legend('',frameon=False)\n\nsns.despine(right=True, top=True, left=True);","metadata":{"execution":{"iopub.status.busy":"2022-07-12T10:24:51.625393Z","iopub.execute_input":"2022-07-12T10:24:51.625779Z","iopub.status.idle":"2022-07-12T10:24:57.196173Z","shell.execute_reply.started":"2022-07-12T10:24:51.625750Z","shell.execute_reply":"2022-07-12T10:24:57.195019Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# ๐Ÿ Finish this experiment\nwandb.finish()","metadata":{"execution":{"iopub.status.busy":"2022-07-12T10:24:57.197897Z","iopub.execute_input":"2022-07-12T10:24:57.198385Z","iopub.status.idle":"2022-07-12T10:25:27.076797Z","shell.execute_reply.started":"2022-07-12T10:24:57.198339Z","shell.execute_reply":"2022-07-12T10:25:27.075895Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 1.2 The .json files\n\nโ— **Important**: the `.json` files have a `dict` structure and contain first the `code cells` (that are in correct order) and then the `markdown cells` that have been shuffled.\n\n> The `.json` files have the following structure (e.g.):\n```\n{\n'cell_type': \n\n {'1862f0a6': 'code',\n '2a9e43d6': 'code',\n '038b763d': 'code',\n ...\n '21616367': 'markdown',\n 'fcb6792d': 'markdown',\n '63c26fa2': 'markdown',\n ...\n },\n 'source': \n {'1862f0a6': '# This Python 3 environment comes with many helpful analytics libraries ....',\n '2a9e43d6': 'import numpy as np\\nimport pandas as pd\\nimport random\\n\\nfrom sklearn.model_selection ...',\n '038b763d': \"import warnings\\nwarnings.filterwarnings('ignore')\",\n '2eefe0ef': \"matplotlib.rcParams.update({'font.size': 14})\",\n ...\n 'aaad8355': '*ะขะธะฟ ะดะฐะฝะฝั‹ั… ะพะฑัƒั‡ะฐัŽั‰ะตะณะพ ัะตั‚ะฐ*',\n '503926eb': 'ะ˜ะฝะธั†ะธะฐะปะธะทะฐั†ะธั ะบะปะฐััะฐ Data',\n '3e5f860d': 'ะŸั€ะธะทะฝะฐะบะธ Rooms, KitchenSquare, HouseFloor ะธะผะตัŽั‚ ะฒ ะฝะตะบะพั‚ะพั€ั‹ั… ะฝะฐะฑะปัŽะดะตะฝะธัั… ะฝัƒะปะตะฒั‹ะต ะทะฝะฐั‡ะตะฝะธั'\n }\n}\n```\n\n
","metadata":{}},{"cell_type":"code","source":"def get_json_data(ID):\n '''\n Returns a df containing the .json information.\n ID: name of file\n return :: a df comtaining cols \"cell_id\", \"cell_type\", \"source\"\n '''\n\n # Read in the .json file\n file = json.load(open(f\"../input/AI4Code/train/{ID}.json\"))\n\n # Create an empty dataframe of size n\n # where n = numver of cell ids in the notebook\n n = len(file[\"cell_type\"].keys())\n df = pd.DataFrame(index=range(n),columns=[\"cell_id\", \"cell_type\", \"source\"])\n\n # Get all sources in order\n all_sources = list(file[\"source\"].values())\n\n # Add cell id and type to dataframe\n for k, (cell_id, cell_type) in enumerate(file[\"cell_type\"].items()):\n df.loc[k, \"cell_id\"] = cell_id\n df.loc[k, \"cell_type\"] = cell_type\n # as cell_id is in the same order for both \"cell_type\" and \"source\"\n df.loc[k, \"source\"] = all_sources[k]\n \n return df","metadata":{"execution":{"iopub.status.busy":"2022-07-12T10:25:27.078198Z","iopub.execute_input":"2022-07-12T10:25:27.079049Z","iopub.status.idle":"2022-07-12T10:25:27.085922Z","shell.execute_reply.started":"2022-07-12T10:25:27.079010Z","shell.execute_reply":"2022-07-12T10:25:27.084963Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Show an example\nID = \"00001756c60be8\"\n\nexample_df = get_json_data(ID)\nexample_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-07-12T10:25:27.087301Z","iopub.execute_input":"2022-07-12T10:25:27.087667Z","iopub.status.idle":"2022-07-12T10:25:27.139424Z","shell.execute_reply.started":"2022-07-12T10:25:27.087638Z","shell.execute_reply":"2022-07-12T10:25:27.138412Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 2. Language Detection\n\n> ๐Ÿ“œ **Note**: The first example we get is a notebook that has all the `markdown` cells in what seems to be Russian! This means that throughout the notebooks there are **other languages that we could encounter besides English**.\n\nLet's see what is the proportion on languages.\n\n## 2.1 Set up the Language Detector\n\nI will be using the `spacy` library for this part.","metadata":{}},{"cell_type":"code","source":"# Language Detector Function\ndef get_lang_detector(nlp, name):\n return LanguageDetector(seed=42)\n\n# Load spacy model\nnlp_model = spacy.load(\"en_core_web_sm\")\n\n# Create instance for language detection\nLanguage.factory(\"language_detector\", func=get_lang_detector)\nnlp_model.add_pipe('language_detector', last=True)","metadata":{"execution":{"iopub.status.busy":"2022-07-12T10:25:27.141362Z","iopub.execute_input":"2022-07-12T10:25:27.142112Z","iopub.status.idle":"2022-07-12T10:25:28.474796Z","shell.execute_reply.started":"2022-07-12T10:25:27.142068Z","shell.execute_reply":"2022-07-12T10:25:28.473652Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2.2 Function to extract the language\n\nThe function `get_document_language()` simply accesses the information from a `.json` file and returns a dictionary of the form `{\"language\":\"en\", \"score\":0.97655}`, where:\n* `language` - specifies the preponderent language of the notebook\n* `score` - specifies the probability\n\n*TODO: Add multiple language detection e.g.: 70% english + 25% french + 5% spanish*","metadata":{}},{"cell_type":"code","source":"def get_document_language(ID):\n '''\n Returns the language of the document.\n ID: name of file\n return :: dictionary containing the language and score (probability)\n '''\n # Retrieve .json df\n df = get_json_data(ID)\n\n # Get a string of all doc text\n # Keep only first 200 chars to not overload memory\n all_doc_text = \" \".join(df[df[\"cell_type\"]==\"markdown\"][\"source\"].tolist())[:200]\n\n # Get document language\n doc = nlp_model(all_doc_text)\n language = doc._.language\n \n return language","metadata":{"execution":{"iopub.status.busy":"2022-07-12T10:25:28.476025Z","iopub.execute_input":"2022-07-12T10:25:28.476354Z","iopub.status.idle":"2022-07-12T10:25:28.481997Z","shell.execute_reply.started":"2022-07-12T10:25:28.476310Z","shell.execute_reply":"2022-07-12T10:25:28.480801Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# An example\nID = \"00001756c60be8\"\nlanguage = get_document_language(ID)\n\nprint(clr.S+f\"--- Notebook: {ID} ---\"+clr.E)\nprint(clr.S+\"The language of this document is:\"+clr.E, language[\"language\"])\nprint(clr.S+\"With a probability of:\"+clr.E, language[\"score\"])","metadata":{"execution":{"iopub.status.busy":"2022-07-12T10:25:28.483734Z","iopub.execute_input":"2022-07-12T10:25:28.484382Z","iopub.status.idle":"2022-07-12T10:25:28.542482Z","shell.execute_reply.started":"2022-07-12T10:25:28.484314Z","shell.execute_reply":"2022-07-12T10:25:28.541764Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2.3 Retrieve language for all 140k files\n\n> ๐Ÿ“œ **Note**: Because the cell below takes ~ 1hr and 15 mins to run, I have commented it and saved the result to a separate [dataset](https://www.kaggle.com/datasets/andradaolteanu/ai4code-processed-data) and into my [W&B Dashboard](https://wandb.ai/andrada/AI4Code?workspace=user-andrada) for easy access.","metadata":{}},{"cell_type":"code","source":"# # === Uncomment this cell to run it ===\n\n# # Retrieve all languages for all notebooks\n# all_languages = []\n\n# # This takes ~ 1hr 15 mins\n# for k, ID in tqdm(enumerate(orders[\"id\"])):\n# all_languages.append(get_document_language(ID))\n \n# # Convert to dataframe\n# all_lang_df = pd.DataFrame(all_languages)\n# all_lang_df[\"id\"] = orders[\"id\"]\n\n# # Save file\n# # .parquet is smaller than .csv\n# all_lang_df.to_parquet(\"all_languages.parquet\", index=False)","metadata":{"execution":{"iopub.status.busy":"2022-07-12T10:25:28.543709Z","iopub.execute_input":"2022-07-12T10:25:28.544148Z","iopub.status.idle":"2022-07-12T10:25:28.548004Z","shell.execute_reply.started":"2022-07-12T10:25:28.544119Z","shell.execute_reply":"2022-07-12T10:25:28.547381Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Read in the languages\nall_lang_df = pd.read_parquet(\"../input/ai4code-processed-data/all_languages.parquet\")\n\n# ๐Ÿ Save artifact to W&B\nsave_dataset_artifact(run_name=\"languages-data\", \n artifact_name=\"language\", \n path=\"../input/ai4code-processed-data/all_languages.parquet\")","metadata":{"execution":{"iopub.status.busy":"2022-07-12T10:25:28.549005Z","iopub.execute_input":"2022-07-12T10:25:28.549686Z","iopub.status.idle":"2022-07-12T10:25:45.629397Z","shell.execute_reply.started":"2022-07-12T10:25:28.549652Z","shell.execute_reply":"2022-07-12T10:25:45.628095Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2.4 Add full-name mapping\n\nAs we have seen, the `spacy` library mapps using the convention ISO Code 2 - where the mapping is made using **2 letters** instead of the full name of the language.\n\nHence I have imported [this dataset from wikipedia](https://www.kaggle.com/datasets/andradaolteanu/iso-language-codes) to map the codes to the full language.","metadata":{}},{"cell_type":"code","source":"# Import external mapping of the languages\niso_codes = pd.read_csv(\"../input/iso-language-codes/ISO_languages_codes.csv\")\n\n# Add full name\nall_lang_df = all_lang_df.merge(right=iso_codes, \n left_on=\"language\", right_on=\"2_letter_code\", \n how=\"left\").iloc[:, :4]\n\nall_lang_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-07-12T10:25:45.631423Z","iopub.execute_input":"2022-07-12T10:25:45.632346Z","iopub.status.idle":"2022-07-12T10:25:45.715331Z","shell.execute_reply.started":"2022-07-12T10:25:45.632274Z","shell.execute_reply":"2022-07-12T10:25:45.714061Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Save file\nall_lang_df.to_parquet(\"all_languages_mapped.parquet\", index=False)\n\n# ๐Ÿ Save artifact to W&B\nsave_dataset_artifact(run_name=\"languages-data-mapped\", \n artifact_name=\"language_mapped\", \n path=\"../input/ai4code-processed-data/all_languages_mapped.parquet\")","metadata":{"execution":{"iopub.status.busy":"2022-07-12T10:25:45.716785Z","iopub.execute_input":"2022-07-12T10:25:45.717260Z","iopub.status.idle":"2022-07-12T10:26:05.343973Z","shell.execute_reply.started":"2022-07-12T10:25:45.717227Z","shell.execute_reply":"2022-07-12T10:26:05.343207Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 3. Language Analysis on the Notebooks\n\n**๐Ÿ“œ Main takeaways:**\n* *90% of the notebooks are in English* and only 10% of the rest are written in other languages\n* from the 10%, the most encountered languages are:\n * Portuguese - by far the most frequent\n * Russian\n * Turkish\n * Japanese\n * Italian\n * Korean\n * Spanish\n\n**โ— What should we do with this information?**\n* *should we delete* from training the notebooks that are non-english or should we try to *incorporate* them?\n* is this *90-10 proportion the same* for the `test` data too?","metadata":{}},{"cell_type":"code","source":"# ๐Ÿ W&B Experiment\nrun = wandb.init(project='AI4Code', name='language-explore', config=CONFIG)","metadata":{"execution":{"iopub.status.busy":"2022-07-12T10:26:05.345314Z","iopub.execute_input":"2022-07-12T10:26:05.345696Z","iopub.status.idle":"2022-07-12T10:26:12.448115Z","shell.execute_reply.started":"2022-07-12T10:26:05.345663Z","shell.execute_reply":"2022-07-12T10:26:12.446788Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"all_lang_df[\"is_en\"] = all_lang_df[\"language\"].apply(lambda x: \"English\" if x ==\"en\" else \"Other\")\n\n# ๐Ÿ Log into W&B\nwandb.log({\"distinct_languages\" : all_lang_df[\"iso_language_name\"].nunique(),\n \"perc_en_notebooks\" : all_lang_df[\"iso_language_name\"].value_counts()[0]/len(all_lang_df[\"iso_language_name\"])})\n\nprint(clr.S+\"Total number of unique languages present within the notebooks:\"+clr.E,\n all_lang_df[\"iso_language_name\"].nunique())\nprint(clr.S+\"Percentage of notebooks in English:\"+clr.E,\n all_lang_df[\"iso_language_name\"].value_counts()[0]/len(all_lang_df[\"iso_language_name\"]), \"\\n\")\nprint(clr.S+\"Other languages:\"+clr.E,\n all_lang_df[\"iso_language_name\"].value_counts().index[1:].tolist())","metadata":{"execution":{"iopub.status.busy":"2022-07-12T10:26:12.450143Z","iopub.execute_input":"2022-07-12T10:26:12.450844Z","iopub.status.idle":"2022-07-12T10:26:14.260501Z","shell.execute_reply.started":"2022-07-12T10:26:12.450804Z","shell.execute_reply":"2022-07-12T10:26:14.259487Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Barchart data\ndata = all_lang_df[\"iso_language_name\"].value_counts().reset_index()\ndata.columns = [\"language\", \"count\"]\ndata = data[data[\"language\"]!=\"English\"]\n\n# Piechart data\nlabels = all_lang_df[\"is_en\"].value_counts().index.tolist()\nsizes = all_lang_df[\"is_en\"].value_counts().values.tolist()\nexplode = (0, 0.2)\n\n# Plots\nfig, (ax1, ax2) = plt.subplots(1, 2, figsize=(24, 18))\nfig.suptitle('Language Analysis',weight=\"bold\", size=25)\n\n# Pie\nax1.pie(sizes, explode=explode, labels=labels, autopct='%1.1f%%',\n shadow=True, startangle=90, colors=[my_colors[2], my_colors[4]],\n labeldistance=1.06)\nax1.set_title(\"Percentage of English Notebooks vs Other Languages\", weight=\"bold\", size=19)\nax1.axis('equal')\n\nsns.barplot(data=data, x=\"count\", y=\"language\", ax=ax2, \n palette=\"autumn\")\nshow_values_on_bars(axs=ax2, h_v=\"h\", space=0.4)\nax2.set_title(\"Frequency of Other Languages\", weight=\"bold\", size=19)\nax2.set_ylabel(\"Language\", size = 18, weight=\"bold\")\nax2.set_xlabel(\"\")\nax2.set_xticks([])\n\nplt.subplots_adjust(left=None, bottom=None, right=None, top=None, wspace=0.8, hspace=None)\nsns.despine(right=True, top=True, bottom=True);","metadata":{"execution":{"iopub.status.busy":"2022-07-12T10:26:14.262008Z","iopub.execute_input":"2022-07-12T10:26:14.262531Z","iopub.status.idle":"2022-07-12T10:26:16.824732Z","shell.execute_reply.started":"2022-07-12T10:26:14.262488Z","shell.execute_reply":"2022-07-12T10:26:16.823718Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# ๐Ÿ Finish this experiment\nwandb.finish()","metadata":{"execution":{"iopub.status.busy":"2022-07-12T10:26:16.826197Z","iopub.execute_input":"2022-07-12T10:26:16.826976Z","iopub.status.idle":"2022-07-12T10:26:33.466086Z","shell.execute_reply.started":"2022-07-12T10:26:16.826936Z","shell.execute_reply":"2022-07-12T10:26:33.464824Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n\n

- Baseline Model & Hyperparameter Tuning -

\n\n### โฌ‡ Libraries","metadata":{}},{"cell_type":"code","source":"from sklearn.model_selection import GroupShuffleSplit\nfrom sklearn.feature_extraction.text import TfidfVectorizer\nfrom xgboost import XGBRanker\nfrom scipy import sparse","metadata":{"execution":{"iopub.status.busy":"2022-07-12T10:26:33.467804Z","iopub.execute_input":"2022-07-12T10:26:33.468219Z","iopub.status.idle":"2022-07-12T10:26:33.681269Z","shell.execute_reply.started":"2022-07-12T10:26:33.468189Z","shell.execute_reply":"2022-07-12T10:26:33.679939Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 1. Create Rank & Analyse\n\nWe first need to create out `target` feature, which will be the `rank`, or the **order** in witch the cells are organised.","metadata":{}},{"cell_type":"code","source":"# ~~~ For now I will REMOVE all notebooks that are not in english ~~~\n\n# Filter out other languages\nonly_english = all_lang_df[all_lang_df[\"iso_language_name\"]==\"English\"].reset_index(drop=True)\n\n# Merge\ndata = pd.merge(left=orders, right=only_english, on=\"id\").iloc[:, :2]\n\ndel iso_codes, all_lang_df\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2022-07-12T10:26:33.683026Z","iopub.execute_input":"2022-07-12T10:26:33.683539Z","iopub.status.idle":"2022-07-12T10:26:34.362415Z","shell.execute_reply.started":"2022-07-12T10:26:33.683491Z","shell.execute_reply":"2022-07-12T10:26:34.361124Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# ~~~ Create Rank (or Order) for each cell ~~~\n\n# Explode cell_order into multiple rows\ndata[\"cell_order\"] = data[\"cell_order\"].apply(lambda x: x.split())\ndata = data.explode(\"cell_order\").reset_index(drop=True)\n\n# Create rank\ndata['rank'] = 1\ndata['rank'] = data.groupby(['id'])['rank'].cumsum()\n\ndata.head()","metadata":{"execution":{"iopub.status.busy":"2022-07-12T10:26:34.368299Z","iopub.execute_input":"2022-07-12T10:26:34.368689Z","iopub.status.idle":"2022-07-12T10:26:37.398722Z","shell.execute_reply.started":"2022-07-12T10:26:34.368660Z","shell.execute_reply":"2022-07-12T10:26:37.397440Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### How many cells are in a notebook?\n\nWith this new `rank` feature we can now look at a distribution for **the number of cells within notebooks**.\n\n> ๐Ÿ“œ **Note:** I would keep in mind here that the *number of cells* are *NOT necessarily correlated* with how much `code` there is in a notebook. A notebook *can have 2-3 cells with many lines of code or text* in it.","metadata":{}},{"cell_type":"code","source":"no_cells = data.groupby(\"id\")[\"rank\"].max().values\n\n# Plot\nprint(clr.S+\"=== Metrics ===\"+clr.E)\nprint(clr.S+\"Min no. of cells:\"+clr.E, no_cells.min(), \"\\n\" +\n clr.S+\"Mean no. of cells:\"+clr.E, no_cells.mean(), \"\\n\" +\n clr.S+\"Max no. of cells:\"+clr.E, no_cells.max())\n\nplt.figure(figsize=(24, 10))\nsns.distplot(no_cells, rug=True,\n rug_kws={\"color\": my_pastels[0]},\n kde_kws={\"color\": my_darks[-1], \"lw\": 5, \"alpha\": 0.7},\n hist_kws={\"histtype\": \"step\", \"linewidth\": 3, \"alpha\": 1, \"color\": my_pastels[0]})\n\nplt.title(\"Distribution of the Number of Cells in a Notebook\", weight=\"bold\", size=25)\nplt.xlabel(\"No. of Cells\", size = 18, weight=\"bold\")\nplt.ylabel(\"Frequency\")\n\nplt.axvline(x=50, linestyle = '--', color=my_darks[-1], lw=2)\nplt.text(x=60, y=0.015, s=f\"Most notebooks have ~ 50 total cells\", \n color=my_darks[-1], size=17, weight=\"bold\")\n\nsns.despine(right=True, top=True, left=True);","metadata":{"execution":{"iopub.status.busy":"2022-07-12T10:26:37.400002Z","iopub.execute_input":"2022-07-12T10:26:37.400621Z","iopub.status.idle":"2022-07-12T10:26:40.313344Z","shell.execute_reply.started":"2022-07-12T10:26:37.400587Z","shell.execute_reply":"2022-07-12T10:26:40.312374Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del no_cells\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2022-07-12T10:26:40.314563Z","iopub.execute_input":"2022-07-12T10:26:40.314864Z","iopub.status.idle":"2022-07-12T10:26:40.696142Z","shell.execute_reply.started":"2022-07-12T10:26:40.314838Z","shell.execute_reply":"2022-07-12T10:26:40.694576Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 2. Create training dataset\n\nThe second step consists of **retrieving the `source` information** (meaning the `code` and `markdown` info) from the `.json` files.\n\n## 2.1 Retrieving the `source` for each cell\n\n*โ— The cell bellow takes a while, so I have commented it and saved the `train` file to [my dataset](https://www.kaggle.com/datasets/andradaolteanu/ai4code-processed-data).*","metadata":{}},{"cell_type":"code","source":"# # === Uncomment this cell to run it ===\n\n# # Get all data from the .json files\n# all_id_data = []\n\n# for ID in tqdm(data[\"id\"].unique()):\n# id_data = get_json_data(ID)\n# id_data[\"id\"] = [ID] * len(id_data)\n# all_id_data.append(id_data)\n \n# # Concatenate all dataframes together\n# train = pd.DataFrame(columns=[\"cell_id\", \"cell_type\", \"source\", \"id\"])\n# train = pd.concat(all_id_data)\n\n# # Merge Rank info\n# train = pd.merge(left=train, right=data, \n# left_on=[\"id\", \"cell_id\"], right_on=[\"id\", \"cell_order\"])\n# train.drop(columns=\"cell_order\", inplace=True)\n\n# # Compute the percentage rank\n# # Divide each rank to the total number of cells within a notebook\n# train[\"percent_rank\"] = train[\"rank\"] / train.groupby(\"id\")[\"cell_id\"].transform(\"count\")\n\n# # Add ancestor data\n# train = pd.merge(left=train, right=ancestors, on=[\"id\"])\n\n# train.to_parquet(\"train.parquet\", index=False)","metadata":{"execution":{"iopub.status.busy":"2022-07-12T10:26:40.697655Z","iopub.execute_input":"2022-07-12T10:26:40.697967Z","iopub.status.idle":"2022-07-12T10:26:40.709334Z","shell.execute_reply.started":"2022-07-12T10:26:40.697940Z","shell.execute_reply":"2022-07-12T10:26:40.708269Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Load in the saved file\ntrain = pd.read_parquet(\"../input/ai4code-processed-data/train.parquet\")\n\n# ๐Ÿ Save artifact to W&B\nsave_dataset_artifact(run_name=\"train_data\", \n artifact_name=\"train_data\", \n path=\"../input/ai4code-processed-data/train.parquet\")","metadata":{"execution":{"iopub.status.busy":"2022-07-12T10:26:40.712639Z","iopub.execute_input":"2022-07-12T10:26:40.713607Z","iopub.status.idle":"2022-07-12T10:27:27.307194Z","shell.execute_reply.started":"2022-07-12T10:26:40.713565Z","shell.execute_reply":"2022-07-12T10:27:27.305748Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### - Cells Analysis -\n\n๐Ÿ“œ **Notes**:\n* On average in the notebooks there are twice more `code` cells than `markdown` cells.\n* Usually the distribution between `code` and `markdown` matches - let's see the correlation.","metadata":{}},{"cell_type":"code","source":"# Data\ncell_analysis = train.groupby([\"id\", \"cell_type\"])[\"cell_id\"].count().reset_index()\ncell_avg = cell_analysis.groupby(\"cell_type\")[\"cell_id\"].mean().reset_index()\n\nfig, (ax1, ax2) = plt.subplots(1, 2, figsize=(24, 12))\nfig.suptitle('Cell Type Analysis', \n weight=\"bold\", size=25)\n\n# Barplot\nsns.barplot(data=cell_avg, x=\"cell_type\", y=\"cell_id\", ax=ax1,\n palette=[my_colors[4], my_darks[-2]])\nshow_values_on_bars(axs=ax1, h_v=\"v\", space=0.4)\nax1.set_title(\"Average count per Cell Type\", weight=\"bold\", size=19)\nax1.set_ylabel(\"Cell Type\", size = 18, weight=\"bold\")\nax1.set_xlabel(\"\")\nax1.arrow(x=1.5, y=22, dx=-1, dy=0, color=my_colors[4], lw=3, \n head_width=0.2, head_length=0.05, linestyle = '-')\nax1.text(x=0.43, y=23, s=\"Twice more code cells than markdown cells\",\n color=my_colors[4], size=15, weight=\"bold\")\n\n# Hist\nsns.histplot(data=cell_analysis, x=\"cell_id\", hue=\"cell_type\", ax=ax2,\n palette=[my_colors[4], my_darks[-2]])\nax2.set_title(\"Cell Type distribution per Notebook\", weight=\"bold\", size=19)\nax2.set_ylabel(\"Frequency\", size = 18, weight=\"bold\")\n\nsns.despine(right=True, top=True);","metadata":{"execution":{"iopub.status.busy":"2022-07-12T10:27:27.309145Z","iopub.execute_input":"2022-07-12T10:27:27.309522Z","iopub.status.idle":"2022-07-12T10:27:34.386743Z","shell.execute_reply.started":"2022-07-12T10:27:27.309486Z","shell.execute_reply":"2022-07-12T10:27:34.385206Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"๐Ÿ“œ **Notes**:\n* The coorelation shows that in the notebooks with small number of `markdown cells` there is also a small number of `code cells`\n* The relationship works vice versa too - as the correlation is **direct and positive** between these 2 cell types.","metadata":{}},{"cell_type":"code","source":"# Data\nscatter_data = pd.pivot(data=cell_analysis, index=\"id\", columns=\"cell_type\", values=\"cell_id\")\nscatter_data[\"size\"] = 30\n\nplt.figure(figsize=(24, 10))\nsns.scatterplot(data=scatter_data, x=\"code\", y=\"markdown\", size=\"size\",\n hue=\"size\", alpha=0.65, palette=CMAP2, sizes=(300, 6000))\n\nplt.title(\"Correlation between Markdown & Code\", weight=\"bold\", size=25)\nplt.xlabel(\"Code\", size = 18, weight=\"bold\")\nplt.ylabel(\"Markdown\", size = 18, weight=\"bold\")\n\nplt.legend('')\n\nsns.despine(right=True, top=True, left=True);","metadata":{"execution":{"iopub.status.busy":"2022-07-12T10:27:34.388641Z","iopub.execute_input":"2022-07-12T10:27:34.389793Z","iopub.status.idle":"2022-07-12T10:27:39.305983Z","shell.execute_reply.started":"2022-07-12T10:27:34.389738Z","shell.execute_reply":"2022-07-12T10:27:39.304720Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del cell_analysis, cell_avg, scatter_data\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2022-07-12T10:27:39.307265Z","iopub.execute_input":"2022-07-12T10:27:39.307615Z","iopub.status.idle":"2022-07-12T10:27:39.623194Z","shell.execute_reply.started":"2022-07-12T10:27:39.307587Z","shell.execute_reply":"2022-07-12T10:27:39.622175Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2.2 Sample Down the data\n\n> ๐Ÿ“œ **Note**: as the amount of data to work with is huge, **I will be sampling down** considerably in this notebook, in order to make a runable pipeline and perform the hyperparameter tuning.","metadata":{}},{"cell_type":"code","source":"# ~~~ Choose a % of the data ~~~\nPERC_DATA = 0.3\n# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~","metadata":{"execution":{"iopub.status.busy":"2022-07-12T10:27:39.624805Z","iopub.execute_input":"2022-07-12T10:27:39.625169Z","iopub.status.idle":"2022-07-12T10:27:39.634924Z","shell.execute_reply.started":"2022-07-12T10:27:39.625127Z","shell.execute_reply":"2022-07-12T10:27:39.634208Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"random.seed(24)\n\n# Get all unique ids\nunique_ids = train[\"id\"].unique().tolist()\nprint(clr.S+\"Total unique ids:\"+clr.E, len(unique_ids))\n\n# Sample down\nunique_ids = random.sample(unique_ids, k=int(len(unique_ids)*PERC_DATA))\nprint(clr.S+\"Sampled unique ids:\"+clr.E, len(unique_ids))\n\ntrain = train[train[\"id\"].isin(unique_ids)].reset_index(drop=True)\nprint(clr.S+\"Sampled train Shape:\"+clr.E, train.shape)","metadata":{"execution":{"iopub.status.busy":"2022-07-12T10:27:39.636278Z","iopub.execute_input":"2022-07-12T10:27:39.636772Z","iopub.status.idle":"2022-07-12T10:27:41.526374Z","shell.execute_reply.started":"2022-07-12T10:27:39.636741Z","shell.execute_reply":"2022-07-12T10:27:41.525402Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del unique_ids\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2022-07-12T10:27:41.527739Z","iopub.execute_input":"2022-07-12T10:27:41.528808Z","iopub.status.idle":"2022-07-12T10:27:41.839150Z","shell.execute_reply.started":"2022-07-12T10:27:41.528760Z","shell.execute_reply":"2022-07-12T10:27:41.838468Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 3. Data Split\n\n**๐Ÿ™ Code below is from [Getting Started with AI4Code](https://www.kaggle.com/code/ryanholbrook/getting-started-with-ai4code)**.","metadata":{}},{"cell_type":"code","source":"# Size of validation set\nNVALID = 0.1\n\n# Create selection object\nsplitter = GroupShuffleSplit(n_splits=1, test_size=NVALID, random_state=0)\n\n\n# Set column \"id\" as index\ntrain = train.set_index(\"id\")\n\n# Split, keeping notebooks with a common origin (ancestor_id) together\nids = train.index.unique('id')\nancestor_data = ancestors.set_index('id').loc[ids, 'ancestor_id']\nids_train, ids_valid = next(splitter.split(ids, groups=ancestor_data))\nids_train, ids_valid = ids[ids_train], ids[ids_valid]\n\n# Create train and validation sets\ndf_train = train.loc[ids_train, :]\ndf_valid = train.loc[ids_valid, :]\n\nprint(clr.S+\"Original Shape:\"+clr.E, train.shape)\nprint(clr.S+\"Train Shape:\"+clr.E, df_train.shape)\nprint(clr.S+\"Valid Shape:\"+clr.E, df_valid.shape)","metadata":{"execution":{"iopub.status.busy":"2022-07-12T10:27:41.840198Z","iopub.execute_input":"2022-07-12T10:27:41.841069Z","iopub.status.idle":"2022-07-12T10:27:43.422268Z","shell.execute_reply.started":"2022-07-12T10:27:41.841036Z","shell.execute_reply":"2022-07-12T10:27:43.421099Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del train, ids, ancestor_data, ids_train, ids_valid\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2022-07-12T10:27:43.423694Z","iopub.execute_input":"2022-07-12T10:27:43.424174Z","iopub.status.idle":"2022-07-12T10:27:43.815448Z","shell.execute_reply.started":"2022-07-12T10:27:43.424129Z","shell.execute_reply":"2022-07-12T10:27:43.814275Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 4. Feature Engineering\n\nโ—พ **min_df**: when building the vocabulary ignore terms that have a document frequency strictly lower than the given threshold.","metadata":{}},{"cell_type":"code","source":"# Glimpse of how the train dataset looks now\ndf_train.head(3)","metadata":{"execution":{"iopub.status.busy":"2022-07-12T10:27:43.817012Z","iopub.execute_input":"2022-07-12T10:27:43.817491Z","iopub.status.idle":"2022-07-12T10:27:43.834212Z","shell.execute_reply.started":"2022-07-12T10:27:43.817448Z","shell.execute_reply":"2022-07-12T10:27:43.833451Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Remove tokens that have a frequency lower than 15%\ntfidf = TfidfVectorizer(min_df=0.15)\n\n# Create the features from the text\n# within each cell\nX_train = tfidf.fit_transform(df_train['source'].astype(str))\n\n# Create the target variable\n# which is the rank (order) of the cells\ny_train = df_train[\"rank\"].to_numpy()\n\n# Number of cells in each notebook\ngroups = df_train.groupby(\"id\")[\"rank\"].max().values","metadata":{"execution":{"iopub.status.busy":"2022-07-12T10:27:43.835770Z","iopub.execute_input":"2022-07-12T10:27:43.836479Z","iopub.status.idle":"2022-07-12T10:28:31.221160Z","shell.execute_reply.started":"2022-07-12T10:27:43.836442Z","shell.execute_reply":"2022-07-12T10:28:31.219881Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# The same process for data validation\nX_valid = tfidf.transform(df_valid['source'].astype(str))\ny_valid = df_valid[\"rank\"].to_numpy()","metadata":{"execution":{"iopub.status.busy":"2022-07-12T10:28:31.222797Z","iopub.execute_input":"2022-07-12T10:28:31.223261Z","iopub.status.idle":"2022-07-12T10:28:35.742546Z","shell.execute_reply.started":"2022-07-12T10:28:31.223216Z","shell.execute_reply":"2022-07-12T10:28:35.741596Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### โ— Creating data leakage on purpose\n\nA question I've asked myself is: **how do you train the `rank` target while telling the model that it has to rank ONLY for `markdown` cells**?\n\nAs we know, within the `.json` files the code cells are in the correct order and only the `markdown` cells are shuffled. So, we don't really have to do anything for the `cell` codes, but only for markdown.\n\n๐Ÿ“œ In the [Getting Started with AI4Code](https://www.kaggle.com/code/ryanholbrook/getting-started-with-ai4code) notebook, the authors do something that I found SUPER cool. They *leak* the information for the code cells, meaning that they create a **new feature column** that:\n* has the *correct* rank for all `code` cells\n* has a dummy value `0` for all the `markdown` cells\n\nGenius, never seen smth like this before. Let us do the same:","metadata":{}},{"cell_type":"code","source":"X_train = sparse.hstack((X_train, \n np.where(df_train['cell_type']=='code', \n df_train['rank'], 0).reshape(-1, 1)))","metadata":{"execution":{"iopub.status.busy":"2022-07-12T10:28:35.743948Z","iopub.execute_input":"2022-07-12T10:28:35.744268Z","iopub.status.idle":"2022-07-12T10:28:35.950206Z","shell.execute_reply.started":"2022-07-12T10:28:35.744239Z","shell.execute_reply":"2022-07-12T10:28:35.949289Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_valid = sparse.hstack((X_valid, \n np.where(df_valid['cell_type']=='code', \n df_valid['rank'], 0).reshape(-1, 1)))","metadata":{"execution":{"iopub.status.busy":"2022-07-12T10:28:35.951462Z","iopub.execute_input":"2022-07-12T10:28:35.951993Z","iopub.status.idle":"2022-07-12T10:28:35.980283Z","shell.execute_reply.started":"2022-07-12T10:28:35.951950Z","shell.execute_reply":"2022-07-12T10:28:35.979520Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 5. XGBRanker\n\nAs the model we'll be using the `XGBRanker()` from `xgboost`.\n\n> ๐Ÿ“œ **Note**: This algorithm **doesn't try to predict the `rank` per se**, but rather **what is the order importance** of each `cell` within the notebook. \n\nHence, as the end instead of this:\n\n`actual_rank: [1, 2, 3, 4, 5, 6, 7]` & `predicted_rank: [1, 2, 3, 4, 5, 6, 7]`\n\nwe'll have this:\n\n`actual_rank: [1, 2, 3, 4, 5, 6, 7]` & `predicted_rank: [-0.7, 0, 0.33, 0.4, 0.45, 0.8, 0.88]`\n\n## 5.1 Training Function\n\nLet us first create the training function. I'll also initiate a new `wandb` experiment and I will log into the dashboard the `kendall_tau` correlation as the final metric.","metadata":{}},{"cell_type":"code","source":"def train_XGBRanker():\n \n config_defaults = {\"booster\":'gbtree',\n \"objective\":'rank:pairwise',\n \"random_state\":24, \n \"learning_rate\":0.1,\n \"n_estimators\":110}\n \n # ๐Ÿ W&B Experiment\n config_defaults.update(CONFIG)\n run = wandb.init(project='AI4Code', name='xgbRanker', config=config_defaults)\n config = wandb.config\n \n # Initiate the model\n model = XGBRanker(booster=config.booster,\n objective=config.objective,\n random_state=config.random_state, \n learning_rate=config.learning_rate,\n n_estimators=config.n_estimators)\n\n # Train the model\n model.fit(X_train, y_train, group=groups, verbose=True)\n\n # Create df containing the cell_id and the prediction\n predict = pd.DataFrame({\"cell_id\" : df_valid[\"cell_id\"],\n \"pred\" : model.predict(X_valid)}, index = df_valid.index)\n\n # Sort (using the predicted rank) and then group\n predict = predict.sort_values(by = ['id', 'pred'], ascending = [False, True])\\\n .groupby('id')['cell_id'].apply(list)\n\n # Create the same but for actual data\n actual = df_valid.sort_values(by = ['id', 'rank'], ascending = [False, True])\\\n .groupby('id')['cell_id'].apply(list)\n\n # Kendall Metric\n metric = kendall_tau(actual, predict)\n print(clr.S+\"Kendall Tau\"+clr.E, metric)\n wandb.log({\"kendall_tau\": np.float(metric)})","metadata":{"execution":{"iopub.status.busy":"2022-07-12T10:28:35.983920Z","iopub.execute_input":"2022-07-12T10:28:35.984862Z","iopub.status.idle":"2022-07-12T10:28:35.994582Z","shell.execute_reply.started":"2022-07-12T10:28:35.984823Z","shell.execute_reply":"2022-07-12T10:28:35.993391Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 5.2 First Run - Baseline\n\n> ๐Ÿ“œ We get a score of **0.5849**.","metadata":{}},{"cell_type":"code","source":"train_XGBRanker()","metadata":{"execution":{"iopub.status.busy":"2022-07-12T10:28:35.996282Z","iopub.execute_input":"2022-07-12T10:28:35.997976Z","iopub.status.idle":"2022-07-12T10:30:45.461024Z","shell.execute_reply.started":"2022-07-12T10:28:35.997925Z","shell.execute_reply":"2022-07-12T10:30:45.459916Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 5.3 Hyperparameter Tuning [๐Ÿ W&B Sweeps]\n\nFor the hyperparameter tuning part I will be using the W&B integrated [Sweeps for XGBoost](https://docs.wandb.ai/guides/integrations/xgboost) method to log all my experiments.\n\n*๐Ÿ™ The tutorial I am following is [Using_W&B_Sweeps_with_XGBoost](https://colab.research.google.com/github/wandb/examples/blob/master/colabs/boosting/Using_W%26B_Sweeps_with_XGBoost.ipynb#scrollTo=VCRlDRL6_5aA).*\n\n
\n\n> โ— **Note:** for the good functioning of Sweeps it is very important that the training function aka `train_XGBRanker()` does NOT have any **arguments** passed. Hence, the format of the `wandb.agent()` should always be `wandb.agent(sweep_id, train_XGBRanker, count=20)` and **NOT** `wandb.agent(sweep_id, train_XGBRanker(data, model, config), count=20)`.","metadata":{}},{"cell_type":"code","source":"# Sweep Config\nsweep_config = {\n \"method\": \"random\", # grid for all\n \"metric\": {\n \"name\": \"kendall_tau\",\n \"goal\": \"maximize\" \n },\n \"parameters\": {\n \"learning_rate\": {\n \"values\": [0.03, 0.04, 0.05, 0.06, 0.07, 0.08, 0.09]\n },\n \"n_estimators\": {\n \"values\": [130, 140, 150, 160, 170, 180, 190, 200]\n },\n \"random_state\": {\n \"values\": [21, 22, 23, 24, 25, 26, 27, 28]\n }\n }\n}\n\n# Sweep ID\nsweep_id = wandb.sweep(sweep_config, project=\"AI4Code\")","metadata":{"execution":{"iopub.status.busy":"2022-07-12T10:30:45.463267Z","iopub.execute_input":"2022-07-12T10:30:45.464778Z","iopub.status.idle":"2022-07-12T10:30:48.231787Z","shell.execute_reply.started":"2022-07-12T10:30:45.464728Z","shell.execute_reply":"2022-07-12T10:30:48.230639Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# ๐Ÿ RUN SWEEPS\nstart = time()\n\n# count = the number of trials/experiments to run\nwandb.agent(sweep_id, train_XGBRanker, count=20)\nprint(\"Sweeping took:\", round((time()-start)/60, 1), \"mins\")","metadata":{"execution":{"iopub.status.busy":"2022-07-12T10:30:48.233788Z","iopub.execute_input":"2022-07-12T10:30:48.234226Z","iopub.status.idle":"2022-07-12T11:24:07.629575Z","shell.execute_reply.started":"2022-07-12T10:30:48.234185Z","shell.execute_reply":"2022-07-12T11:24:07.628067Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The [Sweeps Dashboard](https://wandb.ai/andrada/AI4Code/sweeps/w957m2lh?workspace=user-andrada) shows the following:\n* All the runs on time vs the performance of the Kendall metric + an importance panel containing the most important features during training.\n
\n\n* A visualization with every experiment and its performance.\n
\n\n
\n

๐Ÿ“œ Best Score So Far: Kendall Tau 0.55 | learning_rate: 0.09 | n_estimators: 130 | learning_rate: 24 | DATA_PERC: 0.04

\n
\n\n
\n

๐Ÿ“œ UPDATE Best Score So Far: Kendall Tau 0.5938 | learning_rate: 0.08 | n_estimators: 190 | learning_rate: 24 | DATA_PERC: 0.1

\n
","metadata":{}},{"cell_type":"code","source":"# ๐Ÿ Finish the Experiment\nwandb.finish()","metadata":{"execution":{"iopub.status.busy":"2022-07-12T11:24:07.631834Z","iopub.execute_input":"2022-07-12T11:24:07.632242Z","iopub.status.idle":"2022-07-12T11:24:09.533467Z","shell.execute_reply.started":"2022-07-12T11:24:07.632201Z","shell.execute_reply":"2022-07-12T11:24:09.532427Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 6. Prediction\n\nThe final step is to save our best model and create the **Submission Pipeline**.\n\n> ๐Ÿ“œ **Note**: Because we are training the final model with ALL the data, I will also merge the `train` and `valid` datasets together, as well as recompute the `groups` instance.","metadata":{}},{"cell_type":"code","source":"# Create the group\n# As now we'll train with ALL the data (train + valid)\nfinal_groups = pd.concat((df_train, df_valid)).groupby(\"id\")[\"rank\"].max().values\n\nbest_configs = {\"booster\":'gbtree',\n \"objective\":'rank:pairwise',\n \"random_state\":22, \n \"learning_rate\":0.09,\n \"n_estimators\":200}\n \n# Initiate the model\nfinal_model = XGBRanker(booster=best_configs[\"booster\"],\n objective=best_configs[\"objective\"],\n random_state=best_configs[\"random_state\"], \n learning_rate=best_configs[\"learning_rate\"],\n n_estimators=best_configs[\"n_estimators\"])\n\n# Train the final model\nfinal_model.fit(vstack((X_train, X_valid)), np.concatenate((y_train, y_valid)),\n group=final_groups, verbose=True)\n\n# Save it\nfinal_model.save_model(\"XGBRanker_best.json\")","metadata":{"execution":{"iopub.status.busy":"2022-07-12T11:24:09.535553Z","iopub.execute_input":"2022-07-12T11:24:09.535946Z","iopub.status.idle":"2022-07-12T11:28:32.122212Z","shell.execute_reply.started":"2022-07-12T11:24:09.535912Z","shell.execute_reply":"2022-07-12T11:28:32.121147Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Load Final Model","metadata":{}},{"cell_type":"code","source":"final_model = XGBRanker()\nfinal_model.load_model(\"../input/ai4code-processed-data/XGBRanker_best.json\")","metadata":{"execution":{"iopub.status.busy":"2022-07-12T11:28:32.123891Z","iopub.execute_input":"2022-07-12T11:28:32.124218Z","iopub.status.idle":"2022-07-12T11:28:34.054895Z","shell.execute_reply.started":"2022-07-12T11:28:32.124188Z","shell.execute_reply":"2022-07-12T11:28:34.053640Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#
","metadata":{"execution":{"iopub.status.busy":"2022-07-12T11:28:34.056366Z","iopub.execute_input":"2022-07-12T11:28:34.056715Z","iopub.status.idle":"2022-07-12T11:28:35.862702Z","shell.execute_reply.started":"2022-07-12T11:28:34.056685Z","shell.execute_reply":"2022-07-12T11:28:35.861636Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n### ๐Ÿ W&B Dashboard\n\n> My [W&B Dashboard](https://wandb.ai/andrada/AI4Code?workspace=user-andrada).\n\n
\n\n
\n\n### My Specs\n\n* ๐Ÿ–ฅ Z8 G4 Workstation\n* ๐Ÿ’พ 2 CPUs & 96GB Memory\n* ๐ŸŽฎ 2x NVIDIA A6000\n* ๐Ÿ’ป Zbook Studio G7 on the go","metadata":{}}]}