{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Predict AI Model Runtime (Tile Only)","metadata":{}},{"cell_type":"markdown","source":"[Google - Fast or Slow? Predict AI Model Runtime](https://www.kaggle.com/competitions/predict-ai-model-runtime) - \nA Kaggle research prediction competition hosted by Google\n\nTrain a model that predicts runtime rankings for different tile configurations when executing machine learning models on TPUs. Since a machine learning model can be represented as a graph, with nodes being operations and edges being tensors, a GNN model can be trained to predict the execution times of different configurations and rank them accordingly. A detailed description of the problem is described in the paper [here](https://arxiv.org/abs/2308.13490).","metadata":{}},{"cell_type":"markdown","source":"## Prerequisites & Setup","metadata":{}},{"cell_type":"markdown","source":" - [PyTorch Geometric](https://pytorch-geometric.readthedocs.io/en/stable/) (aka PyG) is used for the majority a graph-based neural network tasks.\n - [pytorch_scatter](https://github.com/rusty1s/pytorch_scatter) is required for the `global_max_pool` function used in the [model](#The-Model) to function efficiently. Note: This will not compile on Kaggle, so it has been removed.\n - Some warnings are intentionally suppressed, as they are harmless. This may change in the future.","metadata":{}},{"cell_type":"code","source":"import warnings\nwarnings.filterwarnings(\"ignore\", category=UserWarning)\nwarnings.filterwarnings(\"ignore\", category=DeprecationWarning)","metadata":{"execution":{"iopub.status.busy":"2023-10-28T09:03:00.980519Z","iopub.execute_input":"2023-10-28T09:03:00.981471Z","iopub.status.idle":"2023-10-28T09:03:00.994343Z","shell.execute_reply.started":"2023-10-28T09:03:00.981426Z","shell.execute_reply":"2023-10-28T09:03:00.993072Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import os\niskaggle = os.environ.get('KAGGLE_KERNEL_RUN_TYPE', '')\nif iskaggle:\n    !pip -q install torch_geometric","metadata":{"execution":{"iopub.status.busy":"2023-10-28T09:03:00.996442Z","iopub.execute_input":"2023-10-28T09:03:00.996808Z","iopub.status.idle":"2023-10-28T09:03:15.024290Z","shell.execute_reply.started":"2023-10-28T09:03:00.996780Z","shell.execute_reply":"2023-10-28T09:03:15.023052Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import multiprocessing\nimport torch\nimport numpy as np\nimport pandas as pd\nimport torch.nn as nn\nimport torch.nn.init as init\nfrom IPython.display import display, HTML\nfrom pathlib import Path\nfrom tqdm.auto import tqdm\nfrom torch.nn.utils import clip_grad_norm_\nfrom torch.utils.data import Dataset\nfrom torch.optim.lr_scheduler import OneCycleLR\nfrom torch_geometric.data import Data\nfrom torch_geometric.loader import DataLoader\nfrom torch_geometric.nn.models import MLP, GraphSAGE\nfrom torch_geometric.nn.pool import global_mean_pool, global_max_pool\nfrom sklearn.preprocessing import MinMaxScaler\nfrom tqdm.contrib.concurrent import process_map","metadata":{"execution":{"iopub.status.busy":"2023-10-28T09:03:15.025931Z","iopub.execute_input":"2023-10-28T09:03:15.026263Z","iopub.status.idle":"2023-10-28T09:03:21.571942Z","shell.execute_reply.started":"2023-10-28T09:03:15.026234Z","shell.execute_reply":"2023-10-28T09:03:21.571105Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"device = 'cuda' if torch.cuda.is_available() else 'cpu'","metadata":{"execution":{"iopub.status.busy":"2023-10-28T09:03:21.574889Z","iopub.execute_input":"2023-10-28T09:03:21.575741Z","iopub.status.idle":"2023-10-28T09:03:21.602794Z","shell.execute_reply.started":"2023-10-28T09:03:21.575704Z","shell.execute_reply":"2023-10-28T09:03:21.601804Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Import Data","metadata":{}},{"cell_type":"code","source":"base_path = Path('/kaggle/input')\npath = Path(base_path/'predict-ai-model-runtime/npz_all/npz/tile/xla')","metadata":{"execution":{"iopub.status.busy":"2023-10-28T09:03:21.604253Z","iopub.execute_input":"2023-10-28T09:03:21.604499Z","iopub.status.idle":"2023-10-28T09:03:21.622163Z","shell.execute_reply.started":"2023-10-28T09:03:21.604477Z","shell.execute_reply":"2023-10-28T09:03:21.621299Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Fetch each model file as needed and concatenate them into a dictionary of DataFrames according to each split. This is done via `multiprocessing` using `process_map` from `tdqm`. It offers a noticable speed increase due to the I/O limits introduced by `np.load`.","metadata":{}},{"cell_type":"code","source":"def process_split(file):\n    d = dict(np.load(file))\n    d['file'] = file.stem\n    return d\n\ndef load_df(path):\n    splits = ['train', 'valid', 'test']\n    cpu_count = multiprocessing.cpu_count()\n    df_dict = {}\n    for i, split in enumerate(splits, start=1):\n        split_path = path/split\n        files = list(split_path.glob('*.npz'))\n        df_list = []\n        pbar = process_map(process_split, files, unit='file',\n                           desc=f'({i}/{len(splits)}) {split}, Processing',\n                           max_workers=cpu_count, chunksize=1, leave=False)\n        for result in pbar:\n            df_list.append(result)\n        df_dict[split] = pd.DataFrame.from_dict(df_list)\n    return df_dict","metadata":{"execution":{"iopub.status.busy":"2023-10-28T09:03:21.623227Z","iopub.execute_input":"2023-10-28T09:03:21.623496Z","iopub.status.idle":"2023-10-28T09:03:21.633292Z","shell.execute_reply.started":"2023-10-28T09:03:21.623472Z","shell.execute_reply":"2023-10-28T09:03:21.632400Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Define a custom PyTorch `Dataset` for shaping the data from the aformentioned DataFrame dictionary.\n- A PyG `Data` object is returned, as it allows for the simplified management of the graph data.\n- `torch.from_numpy` is used where possible, as it directly maps memory from a numpy array to a PyTorch tensor without copying.\n- `edge_index` has it's axes swapped, as it is stored as `(dest, source)`, and PyG expects edges as `(source, dest)`.\n- `edge_index` is transposed, as PyG expects a shape of `(2, num_edges)` instead of `(num_edges, 2)`.\n- `torch.contiguous` is called on `edge_index`. PyG recommends this to fully utilize PyTorch [sparse tensors](https://pytorch.org/docs/stable/sparse.html#sparse-coo-docs).\n- `config_feat` is multiplied by `100.0`. This helps with training as it results in the magnitude of the graph-level configuration features matching the magnitude of the node-level features.\n- `MinMaxScaler` is used on `target`, as we only care about the ranking order. Also, we do not want to target potentially large integers, especially when our loss function, `MSELoss`, requires tensors with a data type of `float32` to function.","metadata":{}},{"cell_type":"code","source":"class TileDataset(Dataset):\n    def __init__(self, df):\n        self.df = df\n\n    def __len__(self):\n        return len(self.df)\n\n    def __getitem__(self, index):\n        row = self.df.iloc[index]\n        node_feat = torch.from_numpy(row['node_feat'])\n        node_opcode = torch.from_numpy(row['node_opcode'].astype(np.int32))\n        edge_index = torch.from_numpy(row['edge_index'][:, [1, 0]].T).contiguous()\n        config_feat = torch.from_numpy(row['config_feat'] * 100.0)\n        config_feat_length = torch.tensor(config_feat.shape[0])\n        config_runtime = torch.from_numpy(row['config_runtime'])\n        target = (config_runtime / row['config_runtime_normalizers'])\n        target = MinMaxScaler().fit_transform(target.reshape(-1, 1)).flatten()\n        target = torch.from_numpy(target.astype(np.float32))\n        return Data(node_feat=node_feat, node_opcode=node_opcode, edge_index=edge_index,\n                    config_feat=config_feat, target=target, config_runtime=config_runtime,\n                   config_feat_length=config_feat_length).to(device)","metadata":{"execution":{"iopub.status.busy":"2023-10-28T09:03:21.634328Z","iopub.execute_input":"2023-10-28T09:03:21.634624Z","iopub.status.idle":"2023-10-28T09:03:21.644703Z","shell.execute_reply.started":"2023-10-28T09:03:21.634600Z","shell.execute_reply":"2023-10-28T09:03:21.643807Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## The Model","metadata":{}},{"cell_type":"markdown","source":"This model closely resembles the 'late-join' tile model described in the [paper](https://arxiv.org/abs/2308.13490). It is a three layer GraphSAGE model followed by an MLP classifier:\n\n1. All trainable parameters use Kaiming initialization. This is because ReLUs are used throughout. This should help prevent exploding / vanishing gradients, especially at the start of training.\n2. Concatenate the node features with opcode embeddings for each node. An opcode embedding vector of size of `128` was chosen.\n3. Feed the concatenated tensor along with the edge index into a PyG `GraphSAGE` model with the following characteristics:\n   - Three layers\n   - ReLU activation functions (except for the final layer)\n   - No normalization\n   - Mean aggregation\n   - Dropout applied after each aggregation\n4. 'Concat pool' the GNN's output. This process is similar to `AdaptiveConcatPool2d` as used by [fastai](https://github.com/fastai/fastai). This utilizes PyG's `global_mean_pool` and `global_max_pool` functions to aggregate the output of the GNN, resulting in two seperate graph embeddings. These embeddings are then concatenated. This enhances the information content for the MLP to utilize in the next step.\n5. The concatenated graph embedding is then fed into an MLP, which gradually reduces the input into a scalar ranking value for each graph configuration. It has the following features:\n   - Four layers\n   - ReLU activation functions (except the final layer)\n   - No normalization\n   - Dropout applied after each layer\n6. The ranking is then z-normalized, this helps match the magnitude of predictions with the magnitude of the min-maxed target, allowing for sane values to be fed into the loss function at all times and reducing the risk of exploding / vanishing gradients.","metadata":{}},{"cell_type":"code","source":"class TileModel(nn.Module):\n    def __init__(self):\n        super().__init__()\n        emb_dim = 128\n        self.embedding = nn.Embedding(120, emb_dim)\n        self.gnn = GraphSAGE(in_channels=emb_dim + 140, hidden_channels=128,\n                             num_layers=3, out_channels=64, dropout=p)\n        self.postnet = MLP(channel_list=[128 + 24, 64, 32, 16, 1], dropout=p, norm=None)\n        self.initialize_parameters()\n\n    def initialize_parameters(self):\n        for m in self.modules():\n            if isinstance(m, nn.Linear):\n                init.kaiming_normal_(m.weight)\n                if m.bias is not None:\n                    init.zeros_(m.bias)\n\n    def forward(self, data):\n        x = torch.cat([data.node_feat, self.embedding(data.node_opcode)], dim=-1)\n        x = self.gnn(x, data.edge_index)\n        x = torch.cat([global_mean_pool(x, data.batch), global_max_pool(x, data.batch)], dim=-1)\n        x = torch.cat([data.config_feat, x.repeat_interleave(data.config_feat_length, dim=0)], dim=-1)\n        x = self.postnet(x)\n        x = torch.flatten(x)\n        x = (x - x.mean()) / (x.std() + 1e-8)\n        return x","metadata":{"editable":true,"slideshow":{"slide_type":""},"tags":[],"execution":{"iopub.status.busy":"2023-10-28T09:03:21.645830Z","iopub.execute_input":"2023-10-28T09:03:21.646087Z","iopub.status.idle":"2023-10-28T09:03:21.659429Z","shell.execute_reply.started":"2023-10-28T09:03:21.646064Z","shell.execute_reply":"2023-10-28T09:03:21.658580Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Training","metadata":{}},{"cell_type":"markdown","source":"### Hyperparameters","metadata":{}},{"cell_type":"markdown","source":"These closely match those used in the [paper](https://arxiv.org/abs/2308.13490). Some notable differences are outlined below in [Setting the Scheduler](#Setting-the-Scheduler)","metadata":{}},{"cell_type":"code","source":"num_epochs = 6\nlr = 1e-4\nwd = 1e-4\np = 0.1\nmax_norm = 1e-2\nmom = 0.99\npct_start = 0.7\nmodel = TileModel().to(device)\ncriterion = nn.MSELoss()\noptimizer = torch.optim.Adam(model.parameters(), lr=lr, weight_decay=wd)","metadata":{"execution":{"iopub.status.busy":"2023-10-28T09:03:21.660587Z","iopub.execute_input":"2023-10-28T09:03:21.660939Z","iopub.status.idle":"2023-10-28T09:03:24.966370Z","shell.execute_reply.started":"2023-10-28T09:03:21.660915Z","shell.execute_reply":"2023-10-28T09:03:24.965341Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Loading the Data","metadata":{}},{"cell_type":"code","source":"tile_df = load_df(path)","metadata":{"execution":{"iopub.status.busy":"2023-10-28T09:03:24.969704Z","iopub.execute_input":"2023-10-28T09:03:24.970102Z","iopub.status.idle":"2023-10-28T09:03:44.211103Z","shell.execute_reply.started":"2023-10-28T09:03:24.970076Z","shell.execute_reply":"2023-10-28T09:03:44.209781Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_ds = TileDataset(tile_df['train'])\nvalid_ds = TileDataset(tile_df['valid'])\ntest_ds = TileDataset(tile_df['test'])","metadata":{"execution":{"iopub.status.busy":"2023-10-28T09:03:44.212838Z","iopub.execute_input":"2023-10-28T09:03:44.213739Z","iopub.status.idle":"2023-10-28T09:03:44.218556Z","shell.execute_reply.started":"2023-10-28T09:03:44.213698Z","shell.execute_reply":"2023-10-28T09:03:44.217585Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- `shuffle` is enabled on the training `DataLoader`. This prevents the model from memorizing the order of configurations that are fed to it, resulting in a generalized solution.\n- No batching is used, as it added a high amount of complexity to the model as tensors needed to be stacked and padded otherwise. Experimentation revealed that batching resulted in unstable training for the current model.\n- Batching did result a large speed increase, due to better GPU and memory utilization, but the accuracy trade-off was not sufficient in this case.\n- Memory cannot be pinned as the `Data` objects in our `Dataset`s contain [sparse tensors](https://pytorch.org/docs/stable/sparse.html#sparse-coo-docs).","metadata":{}},{"cell_type":"code","source":"train_dl = DataLoader(train_ds, shuffle=True)\nvalid_dl = DataLoader(valid_ds)\ntest_dl = DataLoader(test_ds)","metadata":{"execution":{"iopub.status.busy":"2023-10-28T09:03:44.219697Z","iopub.execute_input":"2023-10-28T09:03:44.219958Z","iopub.status.idle":"2023-10-28T09:03:44.230771Z","shell.execute_reply.started":"2023-10-28T09:03:44.219935Z","shell.execute_reply":"2023-10-28T09:03:44.229703Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Setting the Scheduler","metadata":{}},{"cell_type":"markdown","source":"- A high value of [`max_momentum`](#Hyperparameters) is used. In theory, this should help with training because are only using a single graph configuration at each step without any batching. This results in a noisy set of gradients, as the loss can vary greatly from step to step. Using a high value of momentum smooths this noise inverse to the learning rate (with PyTorch's `OneCycleLR` scheduler), causing the model to converge on a solution in a more consistent manner than it would have otherwise.\n- The startup curve for the [One-Cycle Scheduler](https://arxiv.org/abs/1708.07120) (`OneCycleLR`) is set to [`0.7`](#Hyperparameters), which is very high. In theory, this helps as the learning rate should be low and the momentum high as we begin training. This is because it would be easy to overfit with this particular ranking task at the beginning of training. For example, if a series of similar configurations were coincidentally fed into the model at the beginning (which will often happen), the model will overfit to said configurations and have a hard time recovering. If, instead, a low inital learning rate is used, a 'coarse' representation of rankings would be seen by the model at the beginning of training. Only after multiple epochs, when the data has been 'seen' multiple times, should the learning rate be increased, this way, the model is more likely to converge on a better solution.","metadata":{}},{"cell_type":"code","source":"scheduler = OneCycleLR(optimizer, max_lr=lr, epochs=num_epochs, steps_per_epoch=len(train_dl),\n                       max_momentum=mom, pct_start=pct_start)","metadata":{"execution":{"iopub.status.busy":"2023-10-28T09:03:44.231862Z","iopub.execute_input":"2023-10-28T09:03:44.232115Z","iopub.status.idle":"2023-10-28T09:03:44.242693Z","shell.execute_reply.started":"2023-10-28T09:03:44.232093Z","shell.execute_reply":"2023-10-28T09:03:44.241806Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Evaluation Metric","metadata":{}},{"cell_type":"markdown","source":"Each configuration is given a score using the `score_preds` function used inside the [`eval`](#The-Training-Loop) function according to the following evaluation metric:\n\n$$1 - \\left(\\frac{\\text{The best runtime of the top-k predictions}}{\\text{The best runtime of all configurations}} - 1\\right) = 2 - \\frac{\\min_{i \\in K} y_i}{\\min_{i \\in A} y_i}$$\n\nThis formula measures the amount of slowdown incurred for the top-k (`5` in this case) chosen predictions in the range of `0.0-1.0`. At the end of each epoch, the average slowdown is returned so the model's performance can be evaluated on the validation set. Values closer to `1.0` are good, and values closer to `0.0` are bad. If the value falls outside of this range (`>0.0 & <1.0`), something has likely gone wrong during training.","metadata":{}},{"cell_type":"code","source":"def score_preds(preds, df):\n    score = 0.0\n    df_length = len(df)\n    for index in range(df_length):\n        config_runtime = df.iloc[index]['config_runtime']\n        best_pred = min(config_runtime[preds[index]])\n        best_targ = min(config_runtime)\n        score += 2.0 - best_pred / best_targ\n    score /= df_length\n    return score","metadata":{"execution":{"iopub.status.busy":"2023-10-28T09:03:44.243720Z","iopub.execute_input":"2023-10-28T09:03:44.243965Z","iopub.status.idle":"2023-10-28T09:03:44.252830Z","shell.execute_reply.started":"2023-10-28T09:03:44.243943Z","shell.execute_reply":"2023-10-28T09:03:44.252027Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### The Training Loop","metadata":{}},{"cell_type":"code","source":"def train():\n    model.train()\n    pbar = tqdm(train_dl, unit='model', leave=False)\n    loss_total = 0.0\n    n = 0\n    optimizer.zero_grad()\n    for data in pbar:\n        assert len(data) == 1, 'Batching is unsupported.'\n        optimizer.zero_grad()\n        out = model(data)\n        loss = criterion(out, data.target)\n        loss.backward()\n        clip_grad_norm_(model.parameters(), max_norm)\n        optimizer.step()\n        scheduler.step()\n        current_lr = optimizer.param_groups[0]['lr']\n        loss_total += loss.item()\n        n += 1\n        pbar.set_description(f'Training: lr: {current_lr:.2e} loss: {(loss_total / n):.4f}')\n    return loss_total / n","metadata":{"editable":true,"slideshow":{"slide_type":""},"tags":[],"execution":{"iopub.status.busy":"2023-10-28T09:03:44.253936Z","iopub.execute_input":"2023-10-28T09:03:44.254249Z","iopub.status.idle":"2023-10-28T09:03:44.265991Z","shell.execute_reply.started":"2023-10-28T09:03:44.254219Z","shell.execute_reply":"2023-10-28T09:03:44.265207Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def eval():\n    preds = []\n    model.eval()\n    loss_total = 0.0\n    pbar = tqdm(valid_dl, desc='Evaluating', unit='model', leave=False)\n    for data in pbar:\n        out = model(data)\n        loss = criterion(out, data.target)\n        loss_total += loss.item()\n        out = list(out.split(data.config_feat_length.tolist()))\n        top_five = [torch.argsort(t.to('cpu').detach())[:5] for t in out]\n        preds.extend(top_five)\n    loss_total /= len(valid_dl)\n    score = score_preds(preds, tile_df['valid'])\n    return loss_total, score","metadata":{"editable":true,"slideshow":{"slide_type":""},"tags":[],"execution":{"iopub.status.busy":"2023-10-28T09:03:44.267098Z","iopub.execute_input":"2023-10-28T09:03:44.267723Z","iopub.status.idle":"2023-10-28T09:03:44.276786Z","shell.execute_reply.started":"2023-10-28T09:03:44.267689Z","shell.execute_reply":"2023-10-28T09:03:44.275912Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"epoch = 1\nfor epoch in range(num_epochs):\n    train_loss = train()\n    valid_loss, score = eval()\n    display(HTML(f'<div style=\"font-size: 12px;\"><b>Epoch \\\n                {epoch + 1}/{num_epochs}</b>, Train Loss: {train_loss:.4f} \\\n                Valid Loss: {valid_loss:.4f}, Score: {score:.4f}</div>'))\n    epoch += 1","metadata":{"execution":{"iopub.status.busy":"2023-10-28T09:03:44.277911Z","iopub.execute_input":"2023-10-28T09:03:44.278190Z","iopub.status.idle":"2023-10-28T09:09:28.578614Z","shell.execute_reply.started":"2023-10-28T09:03:44.278167Z","shell.execute_reply":"2023-10-28T09:09:28.577668Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The model typically scores `0.90` and above, meaning that it's predicted configurations for each model in the validation set result in an approximate slowdown of 10% compared to the optimal configurations.\n\nAssuming the test set contains models different to the validation set, it will hopefully perform about the same on the test set.","metadata":{}},{"cell_type":"markdown","source":"## Inference","metadata":{}},{"cell_type":"code","source":"preds = []\nmodel.eval()\npbar = tqdm(test_dl, desc='Inference', leave=False)\nfor data in pbar:\n    out = model(data)\n    out = list(out.split(data.config_feat_length.tolist()))\n    top_five = [torch.argsort(t.to('cpu').detach())[:5].tolist() for t in out]\n    preds.extend(top_five)","metadata":{"execution":{"iopub.status.busy":"2023-10-28T09:09:28.580025Z","iopub.execute_input":"2023-10-28T09:09:28.580314Z","iopub.status.idle":"2023-10-28T09:09:33.232801Z","shell.execute_reply.started":"2023-10-28T09:09:28.580290Z","shell.execute_reply":"2023-10-28T09:09:33.231882Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for index, pred in enumerate(preds[:10]):\n    file = tile_df['test'].iloc[index]['file']\n    print(f'{file}: {pred}')","metadata":{"scrolled":true,"execution":{"iopub.status.busy":"2023-10-28T09:09:33.234122Z","iopub.execute_input":"2023-10-28T09:09:33.234408Z","iopub.status.idle":"2023-10-28T09:09:33.240677Z","shell.execute_reply.started":"2023-10-28T09:09:33.234383Z","shell.execute_reply":"2023-10-28T09:09:33.239705Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Submitting the Results","metadata":{}},{"cell_type":"code","source":"submission = pd.read_csv(base_path/'predict-ai-model-runtime/sample_submission.csv')\nfor i, file in enumerate(tile_df['test']['file'].values):\n    id = f'tile:xla:{file}'\n    submission.loc[submission['ID'] == id, 'TopConfigs'] = ';'.join(map(str, preds[i]))\nsubmission.to_csv('./submission.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2023-10-28T09:09:33.241889Z","iopub.execute_input":"2023-10-28T09:09:33.242216Z","iopub.status.idle":"2023-10-28T09:09:33.673053Z","shell.execute_reply.started":"2023-10-28T09:09:33.242185Z","shell.execute_reply":"2023-10-28T09:09:33.672075Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission","metadata":{"execution":{"iopub.status.busy":"2023-10-28T09:09:33.674252Z","iopub.execute_input":"2023-10-28T09:09:33.674529Z","iopub.status.idle":"2023-10-28T09:09:33.689009Z","shell.execute_reply.started":"2023-10-28T09:09:33.674504Z","shell.execute_reply":"2023-10-28T09:09:33.687984Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The above results look sane to me, and match the format of the sample submission. I will use the [Kaggle API](https://www.kaggle.com/docs/api) to directly submit the predictions for the [competition](https://www.kaggle.com/competitions/predict-ai-model-runtime).","metadata":{}},{"cell_type":"code","source":"submit = False\nif submit:\n    from kaggle import api\n    comp = 'predict-ai-model-runtime'\n    api.competition_submit_cli('./submission.csv', 'Tile GraphSAGE v2', comp)","metadata":{"execution":{"iopub.status.busy":"2023-10-28T09:09:33.690150Z","iopub.execute_input":"2023-10-28T09:09:33.690836Z","iopub.status.idle":"2023-10-28T09:09:33.695312Z","shell.execute_reply.started":"2023-10-28T09:09:33.690792Z","shell.execute_reply":"2023-10-28T09:09:33.694314Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Improvements / Reflections","metadata":{}},{"cell_type":"markdown","source":"- Batching seemed to cause the model to perform worse with the current implementation. I only have my theories, I think it perhaps has trouble finding a 'better' solution as it seems to converge very early on, even with a low learning rate.\n- Any normalization while batching also caused the model to perform worse. It would converge on a single solution for each configuration. So I removed all batch normalization, and then I removed all batching. I have tried several experiments, but I cannot figure out what's going wrong.\n- Having the option for 'early-joining' the graph configuration features can improve performance according to the [paper](https://arxiv.org/abs/2308.13490). This would require quite a few changes, and a different notebook may be a better option for pursuing this implementation.\n- Using a ranking loss ([ListMLE](http://icml2008.cs.helsinki.fi/papers/167.pdf)) may also improve performance of the model, however it will also increase complexity, as I found most common ranking loss functions were difficult to implement with the current model as they were not differentiable in a straight-forward manner, with PyTorch's [Autograd](https://pytorch.org/tutorials/beginner/introyt/autogradyt_tutorial.html) being very disagreeable at every step of the process.\n- Performance could be increased very slightly in the `forward` function of the model, as well as in the `score_preds` function, as some residual batching code is still left over (`repeat_interleave`, some extra tensor dimensions).\n- Ensembling could be used to increase the accuracy of predictions very easily. Simply switch out `GraphSAGE` for `GAT` and `GCN` in the [model](#The-Model) and aggregate the results.","metadata":{}}]}