Google Slow VS Fast AI Runtime
","metadata":{}},{"cell_type":"markdown","source":"## Competition Understanding\n\nAt first glance this competition has a lot of moving parts, making it hard to understand how to approach. In this notebook I'm going through some of my thought processes to understand the competition and it's data.\n\nInitial thoughts:\n- **What are we predicting?** - Predict the runtime length of ML graphs and configurations.\n- **What does the data look like?** - Graph configurations in npz format.\n - Two \"collection types\": `tile` and `layout` collections\n- **What does the target look like?** - \"Finally, for the layout collections, your job is to predict the order of the indices from best-to-worse configurations (i.e., ones leading to the smallest `d[\"config_runtime\"]`)\"\n- **How are we evaluated?** two evaluation metrics, described below.","metadata":{}},{"cell_type":"markdown","source":"## Imports","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nfrom glob import glob\nfrom pathlib import Path\nfrom tqdm import tqdm\nimport seaborn as sns\nimport os\n","metadata":{"execution":{"iopub.status.busy":"2023-08-31T14:40:18.745608Z","iopub.execute_input":"2023-08-31T14:40:18.746022Z","iopub.status.idle":"2023-08-31T14:40:18.752779Z","shell.execute_reply.started":"2023-08-31T14:40:18.745989Z","shell.execute_reply":"2023-08-31T14:40:18.751509Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Data Structure\n- `npz_all` contains the data.\n - Two collections types : `layout` and `tile` folders.\n - `layout` has `nlp` and `xla`\n - `tile` has `xla`\n -Each collection type has default/random\n -The data is split `train/test/valid`","metadata":{}},{"cell_type":"code","source":"!tree -I *.npz /kaggle/input/predict-ai-model-runtime/","metadata":{"execution":{"iopub.status.busy":"2023-08-31T14:15:42.099432Z","iopub.execute_input":"2023-08-31T14:15:42.099943Z","iopub.status.idle":"2023-08-31T14:15:44.119309Z","shell.execute_reply.started":"2023-08-31T14:15:42.099910Z","shell.execute_reply":"2023-08-31T14:15:44.117803Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Evaluation Metric\n\n1. For collection `tile:xla`\n - We use the (1-slowdown) incurred of the top-K predictions to reflect how much slower the top-K configurations predicted by the model is from the actual fastest configuration\n2. For collection `layout:*`\n - We use the Kendal Tau Correlation (a ranking metric: how well does your model-predicted ranking, correspond to the real ranking of runtimes).","metadata":{}},{"cell_type":"markdown","source":"## EDA","metadata":{}},{"cell_type":"markdown","source":"From the repo we get that:\n\n> Suppose a `.npz` file stores a graph (representing a kernel) with `n` nodes and `m` edges. In addition, suppose we compile the graph with `c` different configurations, and run each on a TPU. Crucially, the configuration is at the graph-level. Then, the `.npz` file stores the following dictionary (can be loaded with `d = dict(np.load(\"npz/tile/xla/train/